IT/Software career thread: Invert binary trees for dollars.

Phelps McManus

Phelps McManus

<Silver Donator>
311
321
TJT said:
First, how many Airflow instances do you have? If you have thousands of DAGs spread across many instances Astro becomes more useful.
Click to expand...

We only run around 90 DAGs, so it fits on a single instance (one web server, one scheduler - still on 2.x). I have 3 environments though, each completely isolated: prod, dev, and local. We use Azure DevOps with CI/CD pipelines so you work locally, debug and iterate as needed, then PR to dev which updates that server in less than 30s. Make sure your changes doesn’t break other DAGs (dev runs mirror of every process) and then PR to main to roll it into production. This setup has worked really well for us.

I may have reinvented the wheel, but I built my own connection/variable system that syncs each environment with Azure KeyVault. That includes rapidly standing up new local dev instances using Docker. We have over a dozen external systems we connect to and this makes rotating certificates a breeze.

Still, having third party support, presumably 24/7, sounds like a dream.
 
TJT

TJT

Mr. Poopybutthole
<Gold Donor>
47,932
132,014
Phelps McManus said:
We only run around 90 DAGs, so it fits on a single instance (one web server, one scheduler - still on 2.x). I have 3 environments though, each completely isolated: prod, dev, and local. We use Azure DevOps with CI/CD pipelines so you work locally, debug and iterate as needed, then PR to dev which updates that server in less than 30s. Make sure your changes doesn’t break other DAGs (dev runs mirror of every process) and then PR to main to roll it into production. This setup has worked really well for us.

I may have reinvented the wheel, but I built my own connection/variable system that syncs each environment with Azure KeyVault. That includes rapidly standing up new local dev instances using Docker. We have over a dozen external systems we connect to and this makes rotating certificates a breeze.

Still, having third party support, presumably 24/7, sounds like a dream.
Click to expand...
Yeah a small solution like that it doesn't get you a ton of benefits but it is still superior to a setup like that. Our data platform and the use of it is growing a lot too because we got giga funding from the ME last month.

We have like 3500 DAGs total.
 
ShakyJake

ShakyJake

<Donor>
8,848
21,644
Aldarion said:
Anyone who enjoyed learning new things can relate to this.

A large part of the joy in every job I've ever enjoyed was learning something new and then becoming good at it.

Thats gone in many industries. Now you just have your assistant do the job, and skip the whole "learning and getting good at it" part. If you enjoyed that, AI has taken it away for many.

The interesting part to me is learning that was never universal - apparently a lot of people never enjoyed that part at all, and it certainly wasnt their main motivation.
Click to expand...
You still have to get good at using the thing. At work, the split is pretty obvious. The people who’ve actually sat down and learned how to use AI get very different results from the people who still treat it like a chatbot. Same tool, but the output is night and day.

The learning hasn’t disappeared. A lot of it has just moved. Much of the old grind can be skipped now, and plenty of people never liked that part in the first place, so they’re fine with it.

Personally, I feel like AI gives me a huge advantage over a developer who isn’t using it. I’m less limited by what I already know or which tools I happen to be familiar with. I spend less time tweaking individual lines of code or digging through docs for some esoteric library that has little to do with the actual problem I’m trying to solve. That leaves me more room to take ideas further and put my judgment and creativity to work. That’s been a pretty exciting change for me.
 
Noodleface

Noodleface

A Mod Real Quick
40,285
20,244
Control said:
Hmm, I don't necessarily think any of those are in conflict.
It will put a crazy amount of people out of work. It will keep people from learning things they needed to know before. And that probably isn't the greatest path. And it's absolutely amazing at the things its good at and still fucking terrible at the things it isn't, while arguing very convincingly and confidently that it is. However, the bucket of stuff it isn't good at is getting smaller crazy fast.
(I'm undecided about whether or not the datacenter peepee touching is appropriate...)

Pretty sure I've done more "coding" since Opus 5.5 landed than I could do in a lifetime. Fucking thing still sucks at a lot of art things though. It's made massive progress compared to a few months ago, but visuals, especially 3d are still a giant bottleneck for game-y things.
"Claude, make WoW"
"No problem, back in an hour!"
"Um, characters are supposed to wear gear, and it's supposed to move with them you know?"
"It does!"
"Um, look at this screenshot..."
"Go fuck yourself meatbag!"
(possibly paraphrased)
Click to expand...
We are vibe coding my game. It did it entirely in 3d in 5.5 and we asked it to play test it for us. We actually watched it play the game and it made observations on what needed fixing. It's a crazy frontier.
 
You must log in or register to reply here.