SSG works to tame temblors

Platforms protect servers from rumbling quakes

Deep inside the earth's crust, pressures are building that eventually will result in an earthquake of epic proportions. Deep inside Boeing, a few good people are busily tying everything down.

One of them, Doug Marsh, became a believer after the Kobe, Japan, quake in 1995. He vividly remembers seeing film footage of workers freezing at the onset of the temblor—only reacting as equipment started falling all around them. Having been in the Northwest during the 1965 Seattle shaker, he knew that 30 years was long enough for most people to get pretty relaxed about a potential large-scale earthquake.

"When I started talking about seismic mitigation in 1999, most people treated the subject without any particular sense of urgency," Marsh said. "To the company's credit, a disaster preparedness audit had just been completed that showed the need for more earthquake preparation."

Steve Guzek, senior manager of Computing Disaster Preparedness in SSG Information Technology Services, saw the connection to his then-new organization immediately. "After that audit, I became convinced that seismic mitigation was going to be a critical part of any serious companywide disaster preparedness program," Guzek said.

Guzek drew Marsh into his group. Marsh immediately began working with Boeing organizations to develop seismic mitigation plans for their computing assets.

"In the beginning, that boiled down to presenting customers with a pretty straightforward cost-benefit analysis," Dave Davis, Marsh's supervisor, explained. "A toppled high-end server tower can cost nearly $2 million to replace. For $5,000, you can make sure that accident never happens. Of course, after the Nisqually [Wash.] quake, convincing organizations to get on board became much easier."

Fortunately, by the time Seattle got its rolling wake-up call in February 2001, Marsh and Davis had completed the installation of nearly 1,200 seismic isolation platforms and had made almost 1,000 machines virtually quakeproof. As a testament to their work, none of the machines that they retrofitted failed in the Nisqually shaker.

California has long known the importance of seismic mitigation, but recently several other Boeing cities, such as Portland, have discovered that the ground beneath them is less than rock-solid. Some, like St. Louis, are long overdue for a shake-up.

The last significant earthquake in Missouri was an 8+ Richter event that rang church bells in Boston. And even though that 1811 temblor seems a long time gone, scientists estimate the chances of a 6+ earthquake happening in that region in the next 50 years at more than 90 percent.

A repeat of that kind of seismic disturbance is something that every server administrator fears. It's not just the potential of losing all the data since the last backup; it's the likely unavailability of replacement hardware when a tower topples. But when can they afford to take down the system to install safeguards?

Thanks to Marsh and his team, they don't have to. Working with the vendors who make the server isolation hardware, Marsh helped develop a number of new methods for installation and upgrade that operators can perform while the server is online. In fact, the step-by-step processes that the Computing Disaster Preparedness group wrote to accompany them have become the industry standard for seismic mitigation procedures.

"Boeing has become something of an industry bellwether in terms of seismic preparation," Guzek said. "But as we move further and further from the last significant quake, it is human nature to focus on other things. Organizations are less likely to put seismic preparation at the top of their 'to-do' list... "Until the ground moves again."