We all make mistakes… right? Tell us about your worst outages!

I asked the group what the most catastrophic mistake they made at work was and, oh boy… we got some amazing replies. #ThenIWasFired

I’ve selected the Top 10 to feature in the blog, congratulations to those that have featured!

** All surnames have been redacted for privacy (and to ensure their managers don’t see this and make a certain spontaneous decision!). **

#10

Coming in at Number 10, the type of RAID configuration his customer also should’ve had (1+0)… congrats Gareth!

“I had an IDE 3.5 HDD enclosure, put in a drive to recover some important university documents for a customer… enclosure blew the drive.”

#9

At number 9, we have Ram, who made the mistake of pushing a new GPO right before lunch…

“I created a GPO that locks users to login to a specific PC only to discover I did the opposite I locked everyone outside their PCs.The cherry on the top was that i did this GPO and went for lunch outside the office.

Half hour drive distance….So yeah… I’m lucky to still have this job ”

#8

At one Byte (#8), we have Angel. Have you tried switching it off and uhhh… what was that last bit again?

“Sending a remote shutdown without a restart option… on 100 servers @ 2 AM on a Saturday”

#7

Lucky number 7, congrats to Cosma!

“Hit the little white button on the back on a Wang 5000, thinking it would reset the console. It rebooted the whole Mini and kicked a hundred or so users off. Apparently, it took 3 days for the PACE DB to recover. I was “asked” to leave site and not come back

That was about 25 years ago. They say that Good decisions come from experience and experience comes from bad decisions. I have never done that again ”

#6

Conrad is coming in at number 6, mainly because of how the whole situation unfolded so casually.

“It was late 1999, and I was with a friend as a church’s yard sale. They had some computer items for sale. One of which was a large stack of 3.5 floppies that I bought.They also had two laptops. These were a bit outdated even for 1999. So, I wondered if they were updated for Y2K. To find out, I went into the BIOS of one of them and changed the date to something beyond 2000. I then restarted the system. Next thing I know there is smoke coming out of the vents around the hinge.I cautiously and sneakily put the laptop back on a table and walked away.”

#5

We’re into the Top 5, congratulations Nick!

Nick was tasked to write an ASP page for users to change password. The SQL command used was “update usertable set password=@pass”. Yeah, I forgot to add a ” where userid=@userid” condition. We had tens of thousands of users…

Wasn’t fired though!

#4

Sitting comfortably at fourth place, we have Eric.

Eric wiped the OS off a $2 million system (their cost) that the software engineer spent 2 days installing. Luckily, it didn’t take too long to fix it…

#3

At third place we have Chris!

Chris removed the OSPF routing table from a core router of the state ambulance service. Oh boy were there some fireworks going off that day.

They should’ve deployed the backup service!

#2

Coming in at second place, we have Chris (a different Chris, I promise)!

Chris killed all cell phone service in Eastern Wyoming, USA, after dropping a socket into a UPS… oops.

#1

And the winner is… Joseph! Congratulations!

Joseph put concrete through a fibre line and took out an entire asphalt plant for a day, costing them several hundred thousand dollars and wasn’t fired.

I’d call that an achievement, right?

Thanks!

If you enjoyed this article and want to see more of this type of content, please let me know! You could feature in the next article!