Planning for Disaster

I just bought a new pair of winter boots, guaranteed waterproof and warm to -20C; I stood in the store and swore to the sales clerk that I was not going to have cold, wet feet this year (I probably sounded a bit melodramatic, like Scarlett O’Hara declaring that she’d never be hungry again). For those of you who have never been to Toronto, you may not realize that some people make it through the winter without proper boots, just by avoiding the great outdoors on the few days when it is really cold or snowy. We only have a few weeks each winter as cold as -20; we only get a few big snowstorms; most of the snow usually melts within a day or two; and many days hover around the freezing mark so the bigger danger is cold slush leaking into your boots rather than the frigid air. However, every few years we have a colder-than-usual winter, or mounds of snow — like a few years back when a metre of the white stuff fell in two days, closing the city and causing sightings of cross-country skiers in the downtown financial district — and many people (including myself) aren’t properly prepared for it.

In my case, business still has to go on: being self-employed, I can’t just stay inside when the weather is foul, but have to get out there and continue with my day-to-day business of seeing clients and whatever other activities are on my schedule. In other words, the “weather event” occurs, and my business continues, although in a somewhat uncomfortable and restricted manner. There are many natural disasters that are a much greater challenge to business continuity, like the tsunamis, hurricanes and earthquakes that we’ve seen all over the world in the past year, in addition to manmade disasters and even biological events like a flu pandemic: a recent article in the Economist (subscription required) states that Gartner has advised their clients to consider the effect of 30% of their staff not showing up for work due to the flu, which would certainly fall into the “disaster” category for many businesses.

I spoke briefly about business continuity and BPM at a conference last week, and am doing a more comprehensive analysis for a client in the upcoming months. For me, it comes back to thinking about one of Tom Davenport’s nine steps to process innovation: geographical, or more specifically, location independence. BPM is one of the key technologies that may allow a process, or part of a process, to be located anywhere in the world, as long as the communications infrastructure and trained local staff exist. This has been a large driver behind the move to business process outsourcing, a controversial trend that is rejected outright by many organizations, but many people miss the fact that outsourcing also provides some level of business continuity: if you can move some of your business processes to a remote location, then you can just as easily have them at two locations so that there’s a fallback plan in the event of unforeseen events. I’m not talking about replicating systems here — that part’s relatively straightforward, although expensive — I’m talking about what is often forgotten by the IT disaster recovery team: people. If you have a single site where your human-facing business processes take place and something happens at that site, what’s your plan? Where do your people work in the advent of a physical site disaster? How do you reach them to coordinate them? Can you easily reroute client communications (phone, email, postal mail) to the new location? Are people trained at all locations to handle all processes? Can you reroute only part of the process if you have a partial failure at your main site?

Earthquakes are going to happen on the Pacific Rim; hurricanes are going to happen in the southern US, and it’s going to snow in Toronto. I’ve got my boots, are you ready?

Business discontinuity

I’ve been developing something recently for a customer on business continuity planning, and it put me in mind of how a former customer handled a disaster without the benefit of much BCP.

It was the ice storm of 1998, and this small financial company had their main processing site in downtown Montreal. Although the storm started on Monday (and continued for a week), power didn’t start failing until late in the week, and my customer didn’t lose their power until Friday afternoon. It quickly became obvious that the power was not coming back any time soon, which created a problem of unprocessed transactions: since they process mutual fund trades, many trades were already entered in the system, but would not be priced and processed until the overnight batch run. That weekend, the CIO and VP of IT decided to take action. They sneaked into the building (by now, the city was being patrolled by the military to deter looters), climbed 30 floors to their offices, disconnected the main server, and carried it on their backs down the 30 flights of stairs. They returned to one of their own homes, which still had power and telephone service, downloaded the pricing data and did the overnight batch run to process the trades. They then packed up the server in a van, drove it to Toronto — an interesting drive given that the main highways were all closed by now — and installed it in their small sales office there. They arranged for the toll-free customer service lines to be rerouted from their Montreal office to Toronto, added a few temporary and relocated staff to handle the phones, and were up and running by 6am on Monday. They were missing a few pieces, such as that nice imaging and workflow system that I had put in for them, but they were operational with effectively no interruption to their customers.

I remember laughing about this whole scenario when I heard it from the CIO shortly after that, and I definitely thought that these guys were heroes. In retrospect, if that same scenario had gone down today, someone would have been fired for a serious lack of business continuity planning. They suffered from a very common view of continuity planning that exists widely even today: a fatalistic sense that the risks are so low that it’s not worth planning for such a disaster. Given that the last ice storm of that magnitude to hit Montreal was almost 40 years before that, I can understand that view with regards to weather disasters, but there’s other ways to put your business out of commission; for example, Quebec’s history of domestic terrorism can’t be ignored in this post-9/11 world.

In other words, the potential for business discontinuity exists even if you don’t live on a fault line or in the path of major hurricanes: there are enough man-made disasters to keep us all awake at night. It’s no longer possible to ignore BCP, or claim that you can’t plan for something that might never happen. The question to ask yourself when budgeting for business continuity is “how long can we afford to be down?”