I'm glad to hear you took my advice!
Sorry for the necro:
Going into a meeting in 30 minutes. I have a co-worker that has the same title as me, but his skill level is a couple notches below mine. He caused a major outage because of his laziness. (estimated a few million lost) During my RCA I figured out what and who the problem was. IF I call him out he might get fired. If I don't call him out I am blatantly lying, and my co-workers will know I am lying, because my RCA won't make sense. I then need to worry about someone going back to my VP with this analysis.
I do NOT like this guy for professional reasons. But he's a good person.
I call him out my team will see me as a sell out?
I am NOT a person to fuck with peoples careers or money this way. But I do my job when told. I try and be professional.
Quick thoughts?
This is why people hire us to manage this exact shit, so they can blame us and not have to fire anyone.*sigh*
That went as poorly as I could have imagined.
I went over the RCA. Quick explanation. We have paid millions for geo-redundancy. That was all out the window because someone "didn't know" information needed to be put in both locations manually. Verizon has an outage. Kicks over to secondary data center, where there is just test information entered for Verizon. Nothing works. Because the second data center with Verizon infrastructure is technically working, it can't roll over to our secondary provider Century Link. This same person also didn't know how to take Verizon SBC's offline to force it to Century Link.
Everyone knew instantly who the culprit was. He gets defensive before I even finish. Asks for documentation on when we were told to enter the info manually. I of course have those emails. He then questions the architecture which I am prepared for and answer every question. He then gets pissy with me as I answer his questions, and I ask him to let me finish my answers. That didn't happen. I then told him to STFU please and let me finish. In front of VP's and above. No one cares I said that it seemed, but I look like an asshole now because he did STFU, and it looks like I bullied him. This is all over Zoom, not in person...
I get a call immediately from my SVP asking me to explain to him again what happened, without my coworker on the phone. He asks for names and send him what I have. I apologize for the outburst and he doesn't say no problem it happens. He tells me to work on that. Especially considering who was on the phone. I look and feel the idiot.
Anyway...Is it lunch yet? I need a fuckin drink.
My question is why did you spend millions of dollars on redundancy but still manage it all in-house? You should send this out to a competent Avaya MSP, and pay them deal with VZ and CL too, so you have a single throat to choke. Then you can get rid of your analysts and the idiot that caused the outage.
We have both been trained soup to nuts and have been a year on the SIP platform
And that's why you hire a company that has NOC monkeys 24/7 who can log in and disable a SIP link or log into your VZ portal or an SBC and change the destination for an 800 number during an outage.THIS time I just happen to be off a few days.
I actually think you're projecting. You're the one that has a redundancy plan that requires manual failover but you don't have 24x7 coverage for performing said failover.You sound like the type of person that is given a solution to a problem, like a hot ass room, and doesn't fix the problem because you would have to do some work to fix said problem.
I actually think you're projecting. You're the one that has a redundancy plan that requires manual failover but you don't have 24x7 coverage for performing said failover.
I then told him to STFU please and let me finish.