Disaster Recovery and Business Continuity Planning: Shelter from the Storm for IT

Weather(own)What do El Nino and IT have in common?

Yes, both are arguably unpredictable and volatile. But more seriously, lack of preparation for events such as the former can result in business-crippling disruption of the latter.

Two days after Thanksgiving, in the tri-state area at 11:00 a.m., the thermometer in my car registered 61 degrees Fahrenheit. On November 29, the headline on CNN.com is warning that a temperature increase of only two degrees Celsius will result in catastrophic warming in the climate and extreme weather events. In the Pacific Ocean, scientists are calling the El Nino brewing this year the scariest since 1998. The balmy weather on the east coast could soon change.

On the west coast of the U.S., historic drought conditions could be ameliorated with the storms that are expected to start in December. El Nino is expected to bring unprecedented levels of rain to the parched landscape of California, a state known for its mudslides. Texas has experienced flooding events brought on by rain of seven or more inches in a 24-hour period. California residents have been warned for months to take out flood insurance and replace their roofs, especially if they are aging or damaged. Those residents have also been repeatedly advised to take precautions well in advance of the inclement weather—precautions not unlike those advised in preparation for the next major earthquake.

Businesses should be taking the same precautions that savvy homeowners are taking. The business equivalent of putting in place a new rooftop is making sure that there is a plan in place to ensure business continuity. Downtime is not only disruptive, it’s expensive. In 2013, a Ponemon Institute study funded by Emerson Power found that data center downtime costs approximately $7,900 per minute. In 2014, a study conducted by Avaya found that outages cost an average of approximately $140,000 each, a cost that rises beyond a half-million dollars for companies in the financial sector.

Here are some steps to take to prepare for the impending extreme weather and keep the IT side of the business still running.

  1. Have a communication method that utilizes multiple channels to reach out to all employees to ensure their safety. If there are power outages, which could become protracted, cell phone tower back-up batteries and generators may run out of power. Put in place emergency communications channels by which employees can be reached, and provide a way for them to charge their portable devices. One or more small portable solar-powered chargers could prove an effective investment, if the outages last more than two days. For those who must make critical decisions in this scenario, organizations should consider using satellite phones. Empower employees to make local decisions, if the need arises and more senior decision makers can’t be reached.
  2. If employees are not able to get to work, implement strategies to enable workers to work remotely. Ensure there are cross-trained backups for all critical employees. Telecommuting may not be an option if employees have experienced an impact to their homes, so businesses should maintain a list of available office rental spaces that can be used as a backup work location for employees.
  3. If the business’s office space is affected, make sure that the backup and disaster recovery/business continuity (DR/BC) plan can be implemented effectively. When Hurricane Sandy hit, a vast number of businesses were disrupted simultaneously. This put a strain on local DR sites that forced them to fail over to multiple nearby locations. If your organization’s chosen DR site could be similarly disrupted, there should be back-up DR options. If multiple DR sites are not an option, consider putting critical business services in the cloud as a backup strategy. Also, ensure that each of your organization’s DR sites is at least 50 miles from the office location it supports, as recommended in many reputable DR/BC guidelines. All of this should be done keeping in mind the criticality of applications and business services, and the pre-determined recovery time objectives (RTO).
  4. Review insurance coverage, and keep multiple copies of relevant documents in multiple locations. If there is a total loss of a building, data center or other location, ensure that there is a full inventory of all IT assets on a master list. This master list will be needed to file an insurance claim, to replace what was damaged and to inform leasing vendors about their lost or destroyed assets.
  5. For the IT assets most critical to the business, consider purchasing additional quantities as spares and finding alternate suppliers in neighboring counties or states. For example, if there is demand for additional mobile and client devices, consider maintaining a pool of available hardware that can be lent to employees as needed. Establish and maintain contacts with other local businesses, and consider creating a buying consortium with them that could take advantage of co-location options or volume discounts from suppliers.
  6. Test your detailed DR/BC plans. Oftentimes, these plans are fairly high-level and don’t provide sufficient guidance about how the details should be handled. After the emergency has ended, review your DR plans and revise them as needed to reflect lessons learned. After Hurricane Sandy, many businesses in flood-prone areas realized that their data centers should not be located in basements. In tall office buildings, after the elevators were damaged by corrosive saltwater, businesses moved to lower levels that were easily accessible by stairs.

No plan will cover every eventuality. However, a good DR/BC plan will provide significant protection and accelerated recovery from any major disruptions, such as the storms, floods and extreme weather events that the US might encounter in the coming weeks. Whether El Nino does or does not live up to the concerns of many, the steps above are a solid foundation for such a plan.