USA networrk situation post Sandy

[itzman]

For those of you in the USA - especially Western US, it seems from on-line chitchat that getting to Europe has been an issue.

 

This is culled from somewhere and may be useful to understand what is going on, and why

------------------------------------------------------------------------------------------------------------------------------------

 

You will have seen the news (having a general awareness is sort of mandatory for network operators) so I shouldn't need to tell you about Sandy, and the fairly major flooding to everything below ground in downtown NYC.

 

I've been keeping an eye on things. Outages are primarily divided into two:

 

1. Submarine and backhaul capacity - generally backhaul.

This has affected transatlantic capacity, particularly on TAT14 (one leg was undergoing maintenance and the other failed) and Yellow/AC2 (failure in backhaul from the landing station). I believe these have been restored. Not checked yet this morning

 

2. Datacentre power operations

Every major DC in NYC/NY is out of utility (mains) power and running on generators. Some have suffered problems where diesel tanks and pumps are in the basement (fire regs) and have suffered diesel contamination and pump failure due to flooding.

 

In particular, your timestamped problems coincide with generator problems at 111 8th, which is a massive building housing many colos including TelX and Equinix, resulting in power failures to various suites there. Most recent incident today was a fuel pump failure, resulting in a 40 minute failure to some suites.

 

However, in other locations, whole DC buildings have been offline for hours/days.

 

You are likely to see this continuing for days to come. Urban areas currently without utility power might not have it restored for some days.

 

 

--

Will Hargrave

Technical Director

LONAP

Edited November 6, 2012 by itzman

[Entropy]

Were there others with problems today?