Over the past 24 hours we have assisted a large number of customers with server disasters, mostly stemming from a series of Windows Updates that went horribly wrong. I blogged about the main issue on my personal blog today under the title, Windows Updates, Monitoring, and ASP.NET Oops!. If you have not yet read this article, I strongly recommend you check it out as well as continuing with this article. By most accounts that type of an issue would be considered a disaster with sites being down and resources unavailable to perform their regular duties. Today's excitement prompted a lot of questions regarding the concept of true disaster preparedness and what levels of protection a customer needs to ensure their assets are properly protected.
In this posting I will review the concept of Disaster Recovery Planning as we see it here at IowaComputerGurus. Our goal here is to educate, to ensure that in the unfortunate case of a disaster that minimal data, if any, is lost.
When to Plan and What to Plan For?
It is often that Disaster Recovery Planning, sometimes known as Business Continuity Planning is put off until there is time, or until that initial hiccup in the system prompts a more robust solution. However, this is often a situation that can expose your organization to a high level of risk and can result in Disaster. Yes, there are costs involved in setting up a plan, but the second that you roll out a solution no matter how large or small having a disaster recovery plan in place is key.
Another common question is what exactly do you want to plan for? There are a number of potential situations that can arise, identifying the key areas that are important to your organization is important to the final plan. What would you do if your web server crashes? What would you do if your hosting provider's backup solution failed? What would you do if your hosting provider for whatever reason is 100% unavailable? These are just a subset of situations that you might want to account for.
But I'm Already Covered
Many people think that just because they have a "Disaster Recovery" or "Managed Backups" from their hosting provider that they are properly covered. This isn't necessarily the case. Where is the hosting provider taking those backups? How are they taking the backups? What data do they get? Are their backups tested on a regular basis for integrity?
Regardless of the provider, relying on a single point of failure for your backup solution is typically not the best of ideas. A prime example was a situation that occurred today where a client's site after the Windows Update was deployed didn't come back, eventually a restore from backup was requested, but even that failed. Why? Well the backup the provider had was not a native SQL Server backup, but a "snapshot" of the database from a point in time. When restoring the database and reconfiguring it the provider wasn't aware of a specific configuration need at the database level. Should a normal, native SQL Server backup been completed the site would have been restored almost 4 hours earlier than it was.
This is just one example of how the type, nature, and location of your backups can make or break your reaction to a disaster.
What Does IowaComputerGurus Do?
We are often asked what we do when it comes to disaster recovery, and how to we plan. Well we have a multi-stage approach to our backups to ensure that we have the highest level of redundancy, just in case we have an issue.
Hosting Provider Backups
Our hosting provider does a "snapshot" backup every night at 1 AM. This is a whole drive snapshot and takes a point-in-time backup of the server, however, these backups are all stored locally to the hosting provider so it does not provide us protection from things such as site level outages, fires, or similar catastrophic events. We also have no ability to manually validate that this backup was taken or what the backup was accurate and complete.
Native SQL Server Backups
In addition to the "snapshot" backup of the server we have an additional process that takes a native backup of our SQL Server databases to the local server every morning. This file, which is also copied by the snapshot process allows us to quickly restore a single database to a point in time, without any concern on user or other configuration items.
Automatic Offsite Backups
In addition to the hosting provider backup of the server and our native SQL Server Backups. All critical files on the servers (Websites, Databases, etc.) are backed up automatically to the Amazon Cloud. This provides triple redundancy across multiple geographic regions and would protect us in the case of a true site level failure. Amazon S2 storage space is cheap and it is simple peace of mind should something happen to our host or their backups.
Manual Local Backups
Our last line of defense is a once monthly copy of all sites and databases to our corporate environment. The number of changes to our sites is minimal, with the exception of blog posts and comments so although these backups are a bit old towards the end of the life cycle they are better than nothing should a catastrophic event occur.
What We Recommend
We encourage all of our customers to at minimum have Disaster Recovery services from their hosting provider and some form of offsite backup. The hosting provider solution is key in many cases as it is the fastest, and typically the easiest way to get something back up and running. However, the limitations and risks associated with this necessitate at a bare minimum that regular, off-site backups that can be easily verified are completed.
We use JungleDisk and/or Cloudberry for our off-site backups and have recently become a reseller of backup solutions via Cloudberry.
Regardless of the solution, the goal is the same, protect those assets that are most important to you. Your website and the information contained within is critical to your business and should be treated as an asset and guarded just like you would guard your computer or keys. We hope this information has been helpful and feel free to contact us with questions or for more information on our reseller backup offerings.