Business discontinuity

I’ve been developing something recently for a customer on business continuity planning, and it put me in mind of how a former customer handled a disaster without the benefit of much BCP.

It was the ice storm of 1998, and this small financial company had their main processing site in downtown Montreal. Although the storm started on Monday (and continued for a week), power didn’t start failing until late in the week, and my customer didn’t lose their power until Friday afternoon. It quickly became obvious that the power was not coming back any time soon, which created a problem of unprocessed transactions: since they process mutual fund trades, many trades were already entered in the system, but would not be priced and processed until the overnight batch run. That weekend, the CIO and VP of IT decided to take action. They sneaked into the building (by now, the city was being patrolled by the military to deter looters), climbed 30 floors to their offices, disconnected the main server, and carried it on their backs down the 30 flights of stairs. They returned to one of their own homes, which still had power and telephone service, downloaded the pricing data and did the overnight batch run to process the trades. They then packed up the server in a van, drove it to Toronto — an interesting drive given that the main highways were all closed by now — and installed it in their small sales office there. They arranged for the toll-free customer service lines to be rerouted from their Montreal office to Toronto, added a few temporary and relocated staff to handle the phones, and were up and running by 6am on Monday. They were missing a few pieces, such as that nice imaging and workflow system that I had put in for them, but they were operational with effectively no interruption to their customers.

I remember laughing about this whole scenario when I heard it from the CIO shortly after that, and I definitely thought that these guys were heroes. In retrospect, if that same scenario had gone down today, someone would have been fired for a serious lack of business continuity planning. They suffered from a very common view of continuity planning that exists widely even today: a fatalistic sense that the risks are so low that it’s not worth planning for such a disaster. Given that the last ice storm of that magnitude to hit Montreal was almost 40 years before that, I can understand that view with regards to weather disasters, but there’s other ways to put your business out of commission; for example, Quebec’s history of domestic terrorism can’t be ignored in this post-9/11 world.

In other words, the potential for business discontinuity exists even if you don’t live on a fault line or in the path of major hurricanes: there are enough man-made disasters to keep us all awake at night. It’s no longer possible to ignore BCP, or claim that you can’t plan for something that might never happen. The question to ask yourself when budgeting for business continuity is “how long can we afford to be down?”

One thought on “Business discontinuity

  1. Interesting story – I’m sure its being repeated more than many would like to know. Continuity of Operations and Critical Infrastructure aren’t topics restricted to the military, public safety and governmental operations any more. I’m getting a lot of questions not only about the IT aspect of continuity, but logistics, transporation, how to parallel key personnel, the works. Even some of the largest and otherwise most successful organizations haven’t thought through these issues, at least not in an integrated (i.e. truly cross-business) manner. Looks like you’re helping some folks with that – keep up the good work!

Leave a Reply