Sharing my server horror story

Published on October 28, 2005 By just john In Business

After hearing all of the trials the folks over at SD had to go through, I thought I would share my own horror story.

*WARNING*
If you are a sys admin, net admin or anything related these pictures may seem a bit graphic.

It was a calm day in my world. I was just settling in for an afternoon of C&C. Our building has some sort of whisper effect. If you say it at one end of the hall you are bound to be heard at the other. So when I heard the sound of water, I just assumed it was the kitchen down the hall. Boy was I wrong.

The accounting staff is screaming. It was raining in their file room. Can you guess what is right next to the accounting file room? You got it, my servers. I open the door to two inches of water on the floor. Balls out; I rush in to start shutting off equipment. It is pouring in there. I am soaked and forget a graceful shutdown I am slamming power switches (I somehow thought that would be safer than pulling the cords out of the back of the servers.).

We call the building and they quickly determine that the water was coming from the A/C unit installed in the server room. How much water could an A/C make? Ours was fed by the same water system that the building was using for their A/C. So basically it was dumping hundreds of gallons of water a minute. The pressure was so strong that it cleaned all of the I-beams of their fire protection housing (3" thick).

Someone 14 years ago installed this A/C. They needed to cap one side of a T connector used in the installation. So they capped the copper pipe with a galvanized fitting. I am no metallurgist but I know water with chemicals and mixing of metals can really spell disaster. The cap popped off and the water ran free. (the cap was directly above my 42u server cabinet, which was right next to my PBX, which was right next to my switches)

Now everything in the room is wet, including me. All of the devices were powered down, but the UPS was still in the wall and still on. You really can't imagine this situation getting worse, but it does. The funny thing about batteries when they are connected together in mass and then you sprinkle a little water over them, they catch fire. Well they didn't catch fire, but they were arcing and caught all of the plastic parts of them on fire. So now the building maintenance people want the rack out of the building (we were on the 14th floor). I put the fire out as best as I can, reach over to the wall and grab this twist lock plug (110v 30amp) and pull it from the wall. Now I have to roll this into the service elevator and down to the loading dock.

Once I have it all downstairs, I have to begin removing the servers from the rack. There is nothing more depressing than picking up our main file server and having a gallon of water run out. We got it all back up stairs. Most of it was in good shape. All the resource cards in my PBX are covered with this chemically treated water. My servers all have some degree of the same water in them. We took everything apart, labeled it and left it over night to dry out. We couldn't do anything until it was dry.

The next morning I came in to find all of the water dried up, but this nice residue of a crystal powder. Salts and other chemicals were used in the treatment of the condenser water. Now all of this was in these servers. I wiped the PBX down and decided to power it up. It came up fine so I shut it down and moved it back to the rack.

The rest of the servers went to our outside support company. (we are a small staff so we use them for incremental support) These guys were great. They had new servers there that afternoon and they moved out disk arrays over to the new servers. Two days after we were flooded, we were back up with all critical services. We still had some issues, but we were back in business.

The statistics on a company’s survival after a catastrophic data loss are terrible. (Almost half don't make it a year) Fortunately, we lost NO data. Even more amazing is that we didn't even have any corrupt files in our COBOL accounting system. We have 20 accountants. I guess when the water came they stopped working.

This was a real trial. I think that first week and a half after I put in 124 hours. Now I didn't get in too much trouble because I risked my life, but our safety director (HSE for those that know) did fuss at me a bit for going in there so balls out. However, the company did cut me a check for all of my efforts and I did get a nice raise that year.

We made it through that trial and got a great story to tell. I'll never forget the call I made... "What's not under water is on fire, send me every free set of hands you have.

Now how about some more pics.

I hope I never have to go through that again.

BTW we moved a few months later and built out a whole new infrastructure that included redundant A/C units that were not located over the server room and an APC Symettra UPS with a remote kill switch that shuts the whole server room down at the touch of a button.

Comments

KarmaGirl

on Oct 28, 2005

All I can say is "Ack!" That looks like a world of pain. Quite impressive that you didn't have any corruption or loss of any of the drives. Man, that had to be one very bad day......

Dr Guy

on Oct 28, 2005

All I can say is Well done! And Ack! You are a comcrap shop!!

just john

on Oct 28, 2005

Man, that had to be one very bad day

yeah .. the IT director was out of town

I let our HR person call him to break the news.

just john

on Oct 28, 2005

You are a comcrap shop!!

Hey buddy! I like my compaq servers. I have a few Dells also, but they are a real pain in the ass.

Dr Guy

on Oct 28, 2005

Hey buddy! I like my compaq servers. I have a few Dells also, but they are a real pain in the ass.

We are solid Dell except for the VOIP setup. 10 years ago, I bought 40 Compaqs for the School district I was in to act as File servers in each Elementary school. Within a year, we had a 50% failure rate! 50%!

I wont touch them now.

Chaos Manager

on Oct 28, 2005

Gah. That qualifies as a REALLY bad day. Sheesh.

Nice reaction, though. Good thing nothing got corrupted.

just john

on Oct 28, 2005

Right now we have 8 production servers, 5 Compaq ... 3 Dell. THe one thing I really like about the Dells is that you can buy them dirt cheap on Ebay.

Dr Guy

on Oct 28, 2005

THe one thing I really like about the Dells is that you can buy them dirt cheap on Ebay.

You can buy them dirt cheap at the State Surplus sale too!

Welcome Guest! Please take the time to register with us.
There are many great features available to you once you register, including:

Richer content, access to many features that are disabled for guests like commenting on the forums.
Access to a great community, with a massive database of many, many areas of interest.
Access to contests & subscription offers like exclusive emails.
It's simple, and FREE!