Houston Data Center fire / explosion takes PTO off line.. June 1, 2008

**David** · #1 06-01-2008, 02:53 PM

While PTO is hosted up in Dallas (Colo4Dallas) at this time, a major outage at a Data Center in Houston (The Planet - Data Center 1) took us off line for a significant amount of time yesterday and today.

Since our hosting move back up to Dallas from Houston we have had our Domain Name Services handled by a server that unfortunately was in the data center in Houston which is now having major problems. From what I have been told they had a power transformer explode and take out the power in the data center. The fire department would not let them run their backup generators so no power = no servers online.

After I was notified that the data center had a major problem that took PTO offline, our server admin and I worked to transfer the DNS back to our server in Dallas. I initiated the change with my domain registrar around 1:45 this morning, however it can take up to 24-48 hours to completely propagate around the world, so while some people are able to access PTO, others may not. I can't access it with my Cable Internet still, but I can with my Verizon Wireless now..

I wish I could say all of this down time is due to upgrading the servers and moving them to the new data center, but unfortunately not.

I apologize for not making a post about this in the www.PrisonTalkBackup.com forum, but when I went to sleep last night I thought it would only take an hour at most, and I was exhausted. I forgot that this wasn't just a simple Name Server change...

Hopefully everyone will get service back asap!

Thanks,
David

~~Update 8PM~~

I had plans to go up to the North Side of Houston to visit family and decided to drop by the data center on the way and snap a few pictures... I only had a little point & shoot camera so they are not all that great, but better than nothing.

I don't think I have ever seen so many electric company vehicles (and people) even on a new construction project as they have there at this moment. There were at least 12 company trucks and about 5 large mobile generator units and a lot of activity. No surprise there. I'm just glad we were able to change our DNS and are now back online. Hopefully all the people affected by this will be as well soon. A number of years ago when we still had leased dedicated servers, PTO was housed in this very building and on the first floor where most of the damage took place.

**David** · #2 06-01-2008, 02:58 PM

Here is a news report yesterday about what took place.

Quote:

May 31, 2008 - Newsblaze

Tonight at 5:45 p.m. in Houston, Texas, a transformer in The Planet H1 datacenter caught fire. The fire department required them to take down all generators.

This significant outage is impacting around 9,000 servers and 7,500 customers.

Luckily, there were no injuries. It is now known that there was an explosion that knocked down three walls surrounding the electrical equipment room.

All servers in the H1 data center are down. This was the EV1servers old datacenter, that houses the main name servers for many of the dedicated hosting servers, we well as the ServerCommand customer portal, that legacy Ev1servers customers use to control their servers and add domain records.

The name servers and the ServerCommand system were still down, five hours later. ThePlanet staff said they are working to move the ServerCommand and the Name Servers to another data center.

According to staff, their management team and facilities staff are on site evaluating the situation.

"In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center."

According to customer support, one of their forum threads is being used to keep customers informed.

They report that their new Orbit system is not affected.

When NewsBlaze asked, they said "According to the last update, power should be restored in the data center tomorrow afternoon."

It is to be hoped they improve their disaster recovery plans after this. Having nameservers that control domains in other datacenters should not be a difficult problem to overcome.

NewsBlaze, Daily News