Network Issues 2007-06-13 (3:01 Central Time)

[QT]

We are currently experiencing a network issue in our Virginia Data Center at this time. Our engineers are currently investigating this issue and working to resolve it as quickly as possible.

I will continue to post updates as they are received. Thank you for your patience at this time.


-QT

[QT]

Currently there is a massive hail storm in Virginia which caused a power outage for our data center. This caused a failure in the emergency power system which is still being investigated.

The majority of the Virginia data center is online at this time, however there are still areas without power.

More information to come as I receive it.

[QT]

Please take note:

The time on the forums server is not correct. This outage took place at approximately 3:01 Central Time. I will time stamp each post that I continue to make.

[QT]

We are in the process of flying staff from Atlanta and San Antonio out to our Virginia data center to assist with the current issue taking place. I am still trying to find out more information to pass on to you.

Thank you again for your patience.

[QT]

I've just found out that the generators in the Virginia data center worked flawlessly and we are currently running on them now and have been since this issue began.

Currently, there are electrical engineers in our data center now investigating why we lost power in certain areas of our data center.

More to come as I find out.

[QT]

The electricians are still troubleshooting at this time in our Virginia data center. I will continue to post updates, even if it's to let you know nothing has changed.

[QT]

From what I understand, it seems that segments of a UPS have failed which is causing continued outages in certain PDU's. Our engineers are still investigating and I will post more information as I receive it.

Again, the majority of the data center is OK. It is just certain areas that are still without power.

[QT]

We are still running on generators at this time and city power has not been restored as of yet. The engineers are remaining on site and our employees are on a flight now headed that way to help where they can.

[QT]

What we've found out from our engineers:

There are 3 generators in our Virginia datacenter.

* Generator 1 - when the city power dropped, the generator did not come back on correctly.

* Generator 2 - started correctly without problem.

* Generator 3 - Started, ran for a few minutes and then failed (reason unknown).

2 out of 3 UPS's had issues and segments of the UPS's failed to distribute power where necessary, causing certain areas of the data center to be without power as I stated previously.

Engineers are still on site and working on a resolution. We are still on generator power and will continue to be throughout the night for precautionary measures due to another storm expected to take place tonight.

[QT]

Power has been restored. Our techs are now working to bring servers back online. If you have any further questions, please login to your MyServerBeach portal and submit a ticket so that one of our technicians can assist you.

I am heading home now and will see you in the morning.

Thank you again for all of your patience.