IT/Software career thread: Invert binary trees for dollars.

Fucker said:

Microsoft 365 Principal SRE here. I can provide some context. At approximately 21:40 UTC, an engineer deployed a routine configuration change to Azure Front Door that inadvertently redirected our authentication traffic to a SharePoint site last updated in 2017. We immediately initiated a rollback, but the rollback pipeline requires MFA approval through Microsoft Authenticator, which is currently waiting for a Conditional Access policy evaluation from the same infrastructure we're trying to restore. We have a break-glass admin account, but unfortunately its password is stored in a OneNote notebook on the affected tenant. Our on-call engineer attempted to bypass the issue using PowerShell, but the required module was deprecated last Tuesday, and its replacement requires a P2 license. We then opened a Sev A ticket with Microsoft Premier Support. They responded asking us to run sfc /scannow and provide a screenshot. The prolonged recovery is primarily due to our change advisory board requiring a Teams meeting to authorize the rollback. The meeting organizer is stuck in the lobby, and the only person authorized to admit him was laid off in June. Current mitigation involves migrating affected users to Microsoft 365 E5, which should resolve the issue from a revenue perspective. We have identified the root cause as customer misconfiguration. Next update in 30 minutes, assuming someone can get Outlook working.

Click to expand...

TJT said:

Just going to point out here that this is an excellent example of "skeletons in the closet" so to speak. Every org has them and I would bet every penny I have that Microsoft, Google, AWS, and any other have stunning amounts of them. All kinds of shit like this where your entire production revenue generating infrastructure is relying on some ancient powershell script somebody wrote in 2004 and it's been running on a cron job in a long forgotten on prem server you had in the basement of an old auxiliary office. But you have not just one of such things, but thousands. I shit you not.

It's also why you don't fire all your employees and replace them with streetshitters.

I worked with a guy who was at Salesforce for like a decade. He found out that their internal revenue tracking system they used to write their legally required public reporting was running on some dudes extra machine he had at his desk (dude was long gone) and he just told them when he left that they probably shouldn't turn it off. It had been like that for like 5 years. It was also some Java app the dude made years prior.

Click to expand...