IT/Software career thread: Invert binary trees for dollars.

  • Guest, it's time once again for the hotly contested and exciting FoH Asshat Tournament!



    Go here and fill out your bracket!
    Who's been the biggest Asshat in the last year? Once again, only you can decide!

Neranja

<Bronze Donator>
2,605
4,143
Everything has tradeoffs. The cost of doing a truly standalone system are way too much comparing it to doing it ourselves or paying licenses for our stuff.
If you pay 100 to 200k for AWS a year you are on the smaller side and can most likely survive if Jeff wants a new yacht.

The department I work for buys at least a million dollars worth of server hardware every year, and it's one of the smaller ones in the company. Also, they have lots of red tape if data has to leave the company in any form, so cloud providers have to be certified by their IT security department. Last time I was involved in someone trying this "cloud" thing they went "LOL NOPE!", and that was it.
 

Lendarios

Trump's Staff
<Gold Donor>
19,360
-17,424
If you pay 100 to 200k for AWS a year you are on the smaller side and can most likely survive if Jeff wants a new yacht.

The department I work for buys at least a million dollars worth of server hardware every year, and it's one of the smaller ones in the company. Also, they have lots of red tape if data has to leave the company in any form, so cloud providers have to be certified by their IT security department. Last time I was involved in someone trying this "cloud" thing they went "LOL NOPE!", and that was it.
I think you guys are throwing money away by not using the cloud. But I know there is other things to consider when making a cloud move. Personnel being the most important one. The cloud is always cheaper than doing everything yourself for half the performance.

Serverless is a way of life.
 

TJT

Mr. Poopybutthole
<Gold Donor>
40,701
102,086
The place I work at spends $300k a month on cloud server services in addition to having our own data centers.
 

Neranja

<Bronze Donator>
2,605
4,143
The cloud is always cheaper than doing everything yourself for half the performance.
Actually, we have evaluated this on multiple occasions, and I have told this personally to Google engineers in those conference calls: For our types of jobs cloud servers are worthless unless they have predictable performance (no overbooking), low latency interconnects (InfiniBand, IBoE/RoCE, Myrinet, doesn't really matter what as long as it works with an MPI), and scratch disk space with high IOPS.

Performance on their latest processors was around 25% compared to around two year old hardware. I think the cop-out after the results was that we "they couldn't optimize the software for the GCE, so that more cores would offset the perfromance loss due to the increase in latency between nodes."

Ironically, this is licensed off the shelf standard software, for which the software vendor wants to see more money the more cores you use. Also, it is so fucking expensive that the costs for it in a month are higher than what you pay for your whole AWS bill in a year. So of course you buy optimized hardware for it.

Compare the prices on Amazon between the instance types you normally use, and the instances that have the required features above. The key word on Amazon is EFA.
 
  • 1Like
Reactions: 1 user

Lendarios

Trump's Staff
<Gold Donor>
19,360
-17,424
Neranja Neranja
We don't use instances.

That is the thing. All our apps and software are designed to be serverless from the ground up. The only thing we have with an Kubernetes is a graph Federation that everyone uses to communicate with each other. And we did it on Kubernetes so we don't have to wait the extra seconds from a cold lambda startup.

The only one that breaks that is me, because I have to process almost 200GB of data a month to feed our processes from zips, but those can take whatever time. So its not a super critical machine, so I have an EC2 machine for this.

BTW taking quickbooks for example and putting on EC2, is not using the cloud. Using the cloud is ditching QuickBooks and instead develop a cloudbase serverless solution
 

Neranja

<Bronze Donator>
2,605
4,143
We don't use instances.
See, that's the thing: We have real engineering, with real jobs, with government regulations and approvals on the line, and things to archive for decades .. for legal reasons.

Regular jobs around here take around 512 to 1024 cores, and have to be distributed onto multiple machines because the job won't fit in memory otherwise. Even with 512 GB per machine it's a tight fit, as the engineers are always wanting bigger, better, faster, more. Failed jobs because of "out of memory" are a daily occurrence here. If you do the math a job around here takes around 6 to 8 TB of RAM. Result files can also range into the terabytes.

We don't do serverless, because all those machines have to work in lockstep, as each machine only has a part of the model geometry. And when one machine is slightly slower than the others it affects the whole job, as the simulation only progresses to the next iteration when all machines are done with the current one. We also don't this "virtualization" thing, because even 3% performance loss is around 10 minutes of lost time on a 6 hour job. Those are 10 minutes that the machines and licenses can't run another job, and we sometimes have jobs that can run for weeks, sometimes more than a month.
 
  • 1Like
Reactions: 1 user

Lendarios

Trump's Staff
<Gold Donor>
19,360
-17,424
See, that's the thing: We have real engineering, with real jobs, with government regulations and approvals on the line, and things to archive for decades .. for legal reasons.

Regular jobs around here take around 512 to 1024 cores, and have to be distributed onto multiple machines because the job won't fit in memory otherwise. Even with 512 GB per machine it's a tight fit, as the engineers are always wanting bigger, better, faster, more. Failed jobs because of "out of memory" are a daily occurrence here. If you do the math a job around here takes around 6 to 8 TB of RAM. Result files can also range into the terabytes.

We don't do serverless, because all those machines have to work in lockstep, as each machine only has a part of the model geometry. And when one machine is slightly slower than the others it affects the whole job, as the simulation only progresses to the next iteration when all machines are done with the current one. We also don't this "virtualization" thing, because even 3% performance loss is around 10 minutes of lost time on a 6 hour job. Those are 10 minutes that the machines and licenses can't run another job, and we sometimes have jobs that can run for weeks, sometimes more than a month.

If you are having memory problems is a sign that your software frankly sucks.

How big is your development team? Do you do any developing in house of the tools you use?
 
  • 1Picard
Reactions: 1 user

Neranja

<Bronze Donator>
2,605
4,143
If you are having memory problems is a sign that your software frankly sucks.

How big is your development team? Do you do any developing in house of the tools you use?
Did you even read my posts? This is industry standard, off the shelf software. And no, the software doesn't suck, some of it was even engineered by NASA back in the 60s, when computers didn't even have gigabytes of memory. It's just that the models being simulated are getting bigger and bigger, and there is not much you can do about it.

Also, there is a certain pressure to use industry standard tools, because first of all you need to verify the results, and second because development is quite expensive. There are some efforts like OpenFOAM, but for most of the industry you need complete software stacks that work together from design to manufacturing, and nowadays recycling.

Remember, this is real world engineering, not some "AI" data science business bullshit. If those fuck up some people lose money and a company may go out of business. If engineering fucks up people die.
 

agripa

Molten Core Raider
585
495
Regarding the cloud/local. We are very happy with our AWS cloud solution. We use extensively the AWS suite of products, and probably we pay around 100-200k a year for it.

But it is a lot cheaper than in housing all of that, plus we really cant decouple from the SNS/SQS/S3 as we use those as part of our application design.
If you don't have good change control and finops mgmt costs can sneak up. We had a dev tinkering around in powerbi incur 30k in KMS and S3 t2 charges before we noticed over a couple day period. We had another account paying 10k a month in unused EBS volumes for like a year fun times.
 
  • 1Like
Reactions: 1 user

TomServo

<Bronze Donator>
6,305
8,095
If you don't have good change control and finops mgmt costs can sneak up. We had a dev tinkering around in powerbi incur 30k in KMS and S3 t2 charges before we noticed over a couple day period. We had another account paying 10k a month in unused EBS volumes for like a year fun times.
Kms decrypt calls on huge fucking data set?
 

Phazael

Confirmed Beta Shitlord, Fat Bastard
<Aristocrat╭ರ_•́>
14,021
29,921
One of the softphone clients? lol.
Yeah. Like the entire business was on Avaya, with half of it being physical desk phones, but the rest soft phones. Shit would fucking conk out all the god damn time and getting anything configured on the back end in a place like DHS (like say if you had to rescope some VLANs) was like pulling fucking teeth.
 

Mist

Eeyore Enthusiast
<Gold Donor>
30,275
22,008
Yeah. Like the entire business was on Avaya, with half of it being physical desk phones, but the rest soft phones. Shit would fucking conk out all the god damn time and getting anything configured on the back end in a place like DHS (like say if you had to rescope some VLANs) was like pulling fucking teeth.
Which of the horrible softphones was it? One-X Communicator? One-X Agent?

Avaya Workplace and Avaya Agent for Desktop have actually gotten pretty acceptable (after 2+ years of customers beating up Avaya about how fucking bad they were during COVID.)

To be fair, no one makes a really good softphone client. Even Microsoft with all their money, charging customers for Teams E5 licenses, the Teams softphone is not great.
 

Phazael

Confirmed Beta Shitlord, Fat Bastard
<Aristocrat╭ರ_•́>
14,021
29,921
One-X, the bane of my fucking existence. Things would fuck up pulling IPs all the god damn time. And god help you if you ever had to set up a call que for soft phone users with that piece of shit.

But yeah Teams Softphone stinks and is a bandwidth pig, though their conference room integration is somewhat decent. Zoom is ok, but its backend is shit and missing a lot of options/features. Zoom Rooms can die in a fucking fire though.
 

Mist

Eeyore Enthusiast
<Gold Donor>
30,275
22,008
Things would fuck up pulling IPs all the god damn time.
Not exactly sure what you're trying to say here, but the entire first year of COVID I logged countless billable hours helping clients reconfigure their VPNs for softphones. People would do stuff like try to use dynamic NAT and that's just a big fucking nope.
 

TJT

Mr. Poopybutthole
<Gold Donor>
40,701
102,086
One-X, the bane of my fucking existence. Things would fuck up pulling IPs all the god damn time. And god help you if you ever had to set up a call que for soft phone users with that piece of shit.

But yeah Teams Softphone stinks and is a bandwidth pig, though their conference room integration is somewhat decent. Zoom is ok, but its backend is shit and missing a lot of options/features. Zoom Rooms can die in a fucking fire though.
We use Dialup for our soft phones. Never hear any complaints from anyone at our company.
 

Mist

Eeyore Enthusiast
<Gold Donor>
30,275
22,008
Dialpad and Ringcentral are the best everyday cloudphone vendors for most normal companies, but if you have a complex call center operation with lots of IVRs and various integrations, you need to go with a product that you and third party vendors can develop your own solutions on top of, and that pretty much gives you a choice of Avaya, Cisco, and a handful of others.
 
  • 1Like
Reactions: 1 user