How to handle a bad VP

Khane

Got something right about marriage
17,700
10,367
177d 5h 12m
I think you should Terry Tate him in front of everyone.

 

Brahma

Obi-Bro Kenobi-X
7,367
8,303
65d 20h 7m
*sigh*

That went as poorly as I could have imagined.

I went over the RCA. Quick explanation. We have paid millions for geo-redundancy. That was all out the window because someone "didn't know" information needed to be put in both locations manually. Verizon has an outage. Kicks over to secondary data center, where there is just test information entered for Verizon. Nothing works. Because the second data center with Verizon infrastructure is technically working, it can't roll over to our secondary provider Century Link. This same person also didn't know how to take Verizon SBC's offline to force it to Century Link.

Everyone knew instantly who the culprit was. He gets defensive before I even finish. Asks for documentation on when we were told to enter the info manually. I of course have those emails. He then questions the architecture which I am prepared for and answer every question. He then gets pissy with me as I answer his questions, and I ask him to let me finish my answers. That didn't happen. I then told him to STFU please and let me finish. In front of VP's and above. No one cares I said that it seemed, but I look like an asshole now because he did STFU, and it looks like I bullied him. This is all over Zoom, not in person...

I get a call immediately from my SVP asking me to explain to him again what happened, without my coworker on the phone. He asks for names and send him what I have. I apologize for the outburst and he doesn't say no problem it happens. He tells me to work on that. Especially considering who was on the phone. I look and feel the idiot.

Anyway...Is it lunch yet? I need a fuckin drink.
 
  • 3Like
Reactions: Hoss, Soygen and TJT

Phazael

Confirmed Beta Shitlord, Fat Bastard
<Gold Donor>
10,337
18,905
99d 6h 23m
If that other tech is your friend (you said you liked him personally), it sure seems like he tried to sell you up the river to save his own ass. Only mistake I can see is losing your cool, but you said yourself you were conflicted about your desire to do your job and desire to not ruin another man's career. Thing is, he made this mistake and you have the rest of the company's livelyhood (and not just that one guy) at stake here. And the fact that he tried to flip the script by making it about you pretty much tells me he will fuck up again at some point.

If you want to smooth things over with the higher ups, you can play it off as "I trusted him and felt betrayed by him trying to blame me" or just lay low and let it blow over. Either way, yeah go get a drink and mellow out. You losing composure is going to get lost in the shuffle of them raining shit on that other guy.
 

TJT

Adrenaline Junkie
<Silver Donator>
29,599
67,640
Sorry for the necro:

Going into a meeting in 30 minutes. I have a co-worker that has the same title as me, but his skill level is a couple notches below mine. He caused a major outage because of his laziness. (estimated a few million lost) During my RCA I figured out what and who the problem was. IF I call him out he might get fired. If I don't call him out I am blatantly lying, and my co-workers will know I am lying, because my RCA won't make sense. I then need to worry about someone going back to my VP with this analysis.

I do NOT like this guy for professional reasons. But he's a good person.
I call him out my team will see me as a sell out?
I am NOT a person to fuck with peoples careers or money this way. But I do my job when told. I try and be professional.

Quick thoughts?

If you lost a couple million dollars due to laziness at my company you'd be out the door the same day. Fuck that guy. I don't like to point fingers but if it was your direct responsibility and he just didn't tell leadership about issues or lied about doing them when he didn't its all on him.
 

Noodleface

A Mod Real Quick
37,960
14,470
62d 19h 51m
Probably shouldn't have lost your cool, because it sounded like that guy was starting to sink himself
 

Mist

I'm With HER ♀
22,824
9,699
167d 22h 45m
*sigh*

That went as poorly as I could have imagined.

I went over the RCA. Quick explanation. We have paid millions for geo-redundancy. That was all out the window because someone "didn't know" information needed to be put in both locations manually. Verizon has an outage. Kicks over to secondary data center, where there is just test information entered for Verizon. Nothing works. Because the second data center with Verizon infrastructure is technically working, it can't roll over to our secondary provider Century Link. This same person also didn't know how to take Verizon SBC's offline to force it to Century Link.

Everyone knew instantly who the culprit was. He gets defensive before I even finish. Asks for documentation on when we were told to enter the info manually. I of course have those emails. He then questions the architecture which I am prepared for and answer every question. He then gets pissy with me as I answer his questions, and I ask him to let me finish my answers. That didn't happen. I then told him to STFU please and let me finish. In front of VP's and above. No one cares I said that it seemed, but I look like an asshole now because he did STFU, and it looks like I bullied him. This is all over Zoom, not in person...

I get a call immediately from my SVP asking me to explain to him again what happened, without my coworker on the phone. He asks for names and send him what I have. I apologize for the outburst and he doesn't say no problem it happens. He tells me to work on that. Especially considering who was on the phone. I look and feel the idiot.

Anyway...Is it lunch yet? I need a fuckin drink.
This is why people hire us to manage this exact shit, so they can blame us and not have to fire anyone.

I'll tell you these kinds of outages happen every single weekend for dozens of companies. Most telecom SIP DR/GR designs are garbage the first few times something happens, and the implementations are always missing some link or some route somewhere at the very least. This stuff is actually pretty hard to handle every edge case, if you're doing all of this in-house. Even when done by experienced professionals who've done dozens of similar implementations, shit gets missed. I can't tell you how many times a certain major resort company has lost "millions of dollars" because their call centers couldn't take calls and therefore couldn't book vacations/rooms, and we don't get fired and they don't fire anyone internally either.

It's a dumb thing to fire people over. You go back and look at the design and try to make it better.

My question is why did you spend millions of dollars on redundancy but still manage it all in-house? You should send this out to a competent Avaya MSP, and pay them deal with VZ and CL too, so you have a single throat to choke. Then you can get rid of your analysts and the idiot that caused the outage.
 
Last edited:
  • 2Like
Reactions: Gurgeh and TJT

Brahma

Obi-Bro Kenobi-X
7,367
8,303
65d 20h 7m
My question is why did you spend millions of dollars on redundancy but still manage it all in-house? You should send this out to a competent Avaya MSP, and pay them deal with VZ and CL too, so you have a single throat to choke. Then you can get rid of your analysts and the idiot that caused the outage.

Why wouldn't we handle it in house? This isn't rocket science. I deal with Avaya/VZ/CL etc on a daily basis. That really isn't something you outsource for. I'm the one that stood up the system and session managers, he was in charge of the adaptation table data being placed across all system managers. He wrote that piece off as complete with the project manager.

Also I know it in and out. I THOUGHT my co-worker knew it in and out. We have both been trained soup to nuts and have been a year on the SIP platform with minor outages that I resolve in minutes. THIS time I just happen to be off a few days. This would have been a quick jury-rig if I was there by taking the SBC's for Verizon out of the picture. Or resporging over to CL via ATL. (we are our own resporg) Again minutes.

There is nothing at all wrong with the design. He missed a critical step.
 

Mist

I'm With HER ♀
22,824
9,699
167d 22h 45m
We have both been trained soup to nuts and have been a year on the SIP platform
sweet-summer-child.gif


In 3 or 4 you'll have all the bugs worked out, I promise, just in time for the next major upgrade.
THIS time I just happen to be off a few days.
And that's why you hire a company that has NOC monkeys 24/7 who can log in and disable a SIP link or log into your VZ portal or an SBC and change the destination for an 800 number during an outage.

Do you own your own SBCs or does VZ own and manage them?
 

Brahma

Obi-Bro Kenobi-X
7,367
8,303
65d 20h 7m
We own everything. I've been on both ends as a business partner and customer on SIP for almost 10 years my sweet summer child. I got this. ALL of this. Also this thread is where I said they got rid of my entire team and management. I doubt they are taking petitions to hire an outside provider for something I can and do on a daily basis.

You sound like the type of person that is given a solution to a problem, like a hot ass room, and doesn't fix the problem because you would have to do some work to fix said problem.
 
  • 2Worf
Reactions: Hateyou and Fucker

Mist

I'm With HER ♀
22,824
9,699
167d 22h 45m
You sound like the type of person that is given a solution to a problem, like a hot ass room, and doesn't fix the problem because you would have to do some work to fix said problem.
I actually think you're projecting. You're the one that has a redundancy plan that requires manual failover but you don't have 24x7 coverage for performing said failover.
 

Brahma

Obi-Bro Kenobi-X
7,367
8,303
65d 20h 7m
I actually think you're projecting. You're the one that has a redundancy plan that requires manual failover but you don't have 24x7 coverage for performing said failover.

There is no manual fail over Mist now. The information was missing that would have made it automatic. The guy responsible for that information said he put it in the adaptation tables but did not do his job. Hence this meeting today to discuss the RCA and solution. I have since added it said information, tested it and confirmed it works.
 

Unidin

Golden Knight of the Realm
589
181
17d 5h 8m
Ultimately, if he's going to lose his job over this, he probably had some other issues going on as well. Normally, companies don't fire people for one fuckup, even if it's a big one unless it's a integrity/harassment issue.
 

sakkath

Trakanon Raider
1,150
733
64d 23h 57m
I'm frequently put in similar situations where I have to explain why things things that wen't wrong happened. I almost always avoid naming names. I'll say (as an example) 'the power cable was accidentally pulled out' or 'backup of the data had not been confirmed before running the upgrade'. If it's a service that I'm responsible for, I would also add what change has been made to avoid it happening again in the future, such as 'backup of the data had not been confirmed before running the upgrade. We have amended our upgrade procedure to highlight the need to confirm data backup using procedure X before proceeding'.

If asked to name names, only then would I do it, but usually management doesn't care about names. They just want to know how, why, and will it happen again. There have been times I've resisted naming them when I don't feel like they need to suffer scrutiny, for one reason or another, like if they were following the procedure properly but the procedure itself is faulty.

In general if you get in the habit of naming names/blaming people, regardless of if you are correct or not, people notice and it may affect your reputation.
 
  • 1Like
  • 1Solidarity
Reactions: Flobee and Hateyou

Hoss

Make America's Team Great Again
<Gold Donor>
19,047
3,825
75d 14h 4m
I then told him to STFU please and let me finish.

Does zoom not have a way for the presenter to mute mics? Shoulda been like "sorry getting feedback, lemme mute everyone's mic so I can finish"
 

fris

Trakanon Raider
1,141
681
75d 6h 33m
There's always more

Find one of your guy ducks up? Time to dig and find the rest that he thought he got away with