The perils of being responsible for a big website

I maintain and develop a website for an insurance agency. All of our new business starts by someone coming to the website to fill out our online application. So naturally when something goes wrong with the site there's a lot of responsibility on my shoulders.

Now, for the past few days I've been preparing for a web server migration. We were going to move the website and SQL database to a new dedicated server in a datacenter. This morning was the set day to do the big move. I planned everything ahead of time very carefully. I wanted to minimize the downtime and this article was a big help: http://www.sqlservercentral.com/articles/Administering/2688/

The plan was this:
1.) Perform a complete backup of the database
2.) Restore the complete backup onto the new server with NORECOVERY
3.) Take down the online quoting process on the site for everyone except for my IP (for testing purposes), and change the connection string to point to the new server
4.) Perform a differential backup of the database
5.) Restore the differential backup with NORECOVERY
6.) Restore the differential backup with RECOVERY
7.) Test the site and if everything is OK then bring the site back online (still on the old server at this point)
8.) Change the DNS settings

I had tested all of this ahead of time and determined that we should be able to limit the downtime to just about 3 minutes. I don't migrate websites often so I thought I was the man for getting it down to just 3 minutes of downtime.

Well when it finally came down to it the migration didn't go as smoothly as I was hoping. Apparently the production server was set up to block all IP addresses except for those explicitly allowed in an exception list. And in the exception list was the IP I was testing from. So of course my testing worked but when it was pushed to production, it didn't. I didn't know the server had been configured that way and it took me awhile to figure that out. So as a result we had closer to two hours of downtime for the users that still had cached DNS. Realistically, most users wouldn't have had cached DNS so most users visiting the site during that window of time would have seen the site from the new web server. But still -- that sucked.

The good news is everything is configured properly now. The new web server is on a connection with more bandwidth than the old server. And it's a quad-core machine with 4GB of RAM. So it has the power to support us and then some. I also configured it to use HTTP compression (which we didn't have on the old server) and the impact of that can be pretty huge when it comes to download times.