Here's a good one.
We monitor/manage over 100,000 devices. Switches, routers, firewalls, servers, voice gateways, DC blades/chassis, etc for about 2000 customers.
All of these these need passwords, obviously.
We had a pretty unsecure system for storing these, and we were generally missing about 10-15% of the passwords for the devices, which was a real pain, but for the rest, everything was fairly well organized, just insecure.
So we purchased a product to store passwords securely.
About 10% of the way into migrating the passwords over to this product, we realized the database search performance was awful and that searches chronically timed out and returned 0 results. We also had no plan for migrating the data in an organized manner, so much of it is just indexed really badly, like "Router 1, Router 2, Router 3, etc instead of "Dallas Core Router, Dallas Backup Router, Chicago Core Router, Chicago Backup Router." We threw a bunch of hardware at the product to try to improve performance but it did nothing.
Despite all these problems, the migrations continue. Passwords go in, but no one can retrieve them. So we've basically ransomwared our own passwords, which is then causing everyone to just open tickets with the OEMs and get the passwords reset, which is then invalidating all the passwords that were migrated into the system, and they still haven't come up with any kind of documentation guide for how to properly put passwords in the database... so the reset passwords aren't going into the database.
This has been going on for like 18 months now and no one has a plan to fix it.
And the best part is that we've realized that the third party company in India that we hired to do our tier 0 event management triage a) keeps locking out all the accounts because they log in with the wrong passwords and b) when they do successfully log in, they put the passwords into their own account notes for speedier access, which totally negates any security we "gained" in this whole process.
