Through my adventures in IT, I have come to the conclusion that server racks and data equipment will be housed in the very worst possible spot in a building. I can almost guarantee that your IT equipment is in a place that has live water pipes, possibly even sewage right above it.
I can also predict that the only place plumbing will fail is directly above said server rack.
In dealing with these types of losses there a few factors that can save equipment from total destruction and help get you back up and running quickly.
Don’t Panic (The exact opposite of what you are likely to do)
I know this is easier said than done, but trust me, it ain’t gonna help. People running around pulling equipment out of racks to protect it from water will probably do more harm than good. I’ll tell you why in a minute.
Make sure you are not walking into a room full of standing water with power flowing all around. If in doubt, kill the breaker for that room before you go in. Better to do a dirty shutdown than risk getting zapped.
Cover The Racks
Get a tarp or some sort of cover to move water around the rack. Put it on top or in front to deflect the water from directly hitting the equipment.
Don’t Move Anything (Yet)
Water is a liquid, right? Therefore it moves with great ease on any surface. Let’s say water got into the equipment through venting holes on top, but it didn’t hit the circuit board just yet. What do you think will happen when you tilt it to move it, especially if it’s still powered on? Yep. Zap!
Shut Down, Unplug and Dissipate
Do this BEFORE you try to move the equipment. Water on a de-energized board can do no harm as long as it is cleaned off properly before powering it on. How do you dissipate? Unplug from the power source and press the power button for several seconds.
Stop The Leaks
Pull The Equipment Apart And Look For Damage
Take the covers off and look for visual signs of water damage to the boards, inputs and outputs, cables and power cords… anything that could have been sitting in or near water.
Make Sure The Equipment Is Dry Before Testing For Power
This could take hours or days depending upon how much water got into the system. In the case of sprinkler water, it could be wet for weeks.
Clean Any Contamination As Quickly As Possible
Just because the equipment is dry does not mean it is good to go. Water leaves minerals behind that can corrode circuit boards. Proper cleaning can remove any trace of harmful contaminants from the equipment.
Test It Before Connecting To Other Equipment
Test the equipment alone first, make sure it will not affect other equipment negatively.
The goal is to assess what equipment has been damaged beyond cost effective repair (meaning get it replaced) and what can be brought back to life with a little effort.
- The quickest way to get up and running in any IT Emergency is to use the existing equipment.
- Waiting for new equipment, re-configuration and software installation takes time, which means business is down.
- It is much more cost effective to use existing equipment and will save business interruption cost.
This can be an overwhelming process for IT employees, who are usually handling day to day operations of a business and have little room to deal with ALL systems going down at once.