Network Issue - San Antonio 2013-02-12

[dbiers]

Good Evening,

We are currently experiencing a network issue in our San Antonio, TX data center. We are working quickly to resolve this issue. Further details to come as more information becomes available. We thank you for your patience.

ServerBeach Support

[dbiers]

We are currently following up with our underlying transport providers that connect our San Antonio Data Center to our Dallas POP. Both circuits are currently down and not able to pass network traffic.

[dbiers]

UPDATE:: 2:22am - 02/12/2013

The event started at approximately 12:00am and our initial
investigation showed that both of our transport circuits that connect
our San Antonio Data Center to our Dallas POP were down and not able
to pass traffic. We have escalated this issue to the Executive level
at both providers and are working diligently to restore network
services.
We apologize for this issue and are working to restore service as soon as possible.

[dbiers]

Update: 03:15am CST - 02/12/2013

We have identified that both of our transport carriers have a maintenance ongoing that is affecting our network connectivity between San Antonio and Dallas. The work is to be completed by 6AM CST and we are escalating through the Executive level to complete the maintenance window sooner in order to restore your service.

PEER 1/Serverbeach utilizes redundant providers and our transport circuits were originally provisioned on diverse routes to prevent incidents of this nature. We will be performing a full physical audit on our all of our circuits in light of this issue to determine why a commonality would exist between our two providers and how one maintenance could be affecting both paths.

We greatly appreciate your patience and are working dilligently to resolve this issue.

[dbiers]

UPDATE: 03:52am - 02/12/2013

Network connectivity has been restored to the San Antonio (SAT5) Data Center. We ask that you verify the connectivity to your server at this time and notify Support if you are not able to remotely access your server.

We appreciate your patience while we worked through this event and will be providing you with a RFO and timeline in the next few hours.

[dbiers]

On February 12 at 00:00 CT, PEER 1 Hosting’s Network Operations Center (NOC) was alerted to loss of connectivity on multiple transport circuits that connect our San Antonio Data Center to our node in Dallas. Network Engineers were immediately mobilized to investigate the issue locally, while in parallel the Network Operations Center began following up with our transport circuit providers for more information.

The transport circuits that connect Dallas and San Antonio are supplied by different providers, and are specified as running on diverse fiber paths. Continuing our investigation, the Network Operations Center was able to determine that the connectivity loss was the result of a fiber maintenance being performed by an underlying long-haul carrier, for which PEER 1 Hosting was not notified. At 03:30 CT, the first transport circuit came back online, which fully restored network connectivity to the San Antonio Data Center.

At the moment it is unclear how separate providers with previously diverse circuits paths would both be impacted by the same fiber work. We will be following up with our San Antonio transport providers and assessing how this scope of work had the impact that it did and then take appropriate action to ensure circuit redundancy is reinstituted. Additionally, PEER 1 Hosting will be doing an audit of the current fiber paths being used throughout our transport links to ensure full network redundancy is available in all locations.

PEER 1 sincerely apologizes for any inconvenience this has caused, and appreciates your understanding.