Crash: Data Center Horror Stories

This article talks about two data center failures that were caused by faults in the power distribution equipment coupled with design mistakes. Although the article is only about hardware faults, it contains some interesting lessons that should be considered in designing both hardware and software.

• First, anything can fail, even core infrastructure such as power-conducting equipment, and one undefended failure mode can bring down the entire system. One needs to ensure that there is no single point of failure in any aspect of the data center, not just obvious ones like machines or external power failing. One example story of a non-obvious "core component" failing in software is the EC2 outage earlier this year where a bit error induced in one machine led to erroneous information being propagated to all nodes in the system by the gossip protocol.
  
• It is important to test failure cases to make sure the problem cannot escalate. In general, and in software in particular, it seems very hard to design systems that fail gracefully - software never checks for "reasonable" values on timeouts, status messages, etc, and one cannot isolate software components by physical distance like one can with hardware.
• Finally, lesson three in the article - use datacenter-grade components only - may also be interesting to apply to software. Indeed, many data centers try to use either commercial software or proven open-source software. As researchers interested in data centers, however, we need to keep in mind how we plan to ensure that our work is datacenter-grade if we build new things. Again, I think this points to the importance of testing. It's shocking how little time is spent in systems, DB, etc courses on software testing, when testing is the main process through which bugs are kept in control in a production environment.