/dev/zero

Nothing like sitting at work on the night shift and getting a call from a major customer.

Me: “${POE}. This is Andrew.”
Customer: (thick Aussie accent) “Hi. Are you having problems with your Savvis network right now?”
Me: “Uhm… no, not that I’m aware of.”
Customer: “We can’t reach any of our servers at your Savvis data center. Are you sure there’s no problems with the network? We can reach everywhere else, so we’re pretty sure it’s on your end.
Me: “We are having no network problems that we are aware of. Can you open a ticket with a traceroute from your location to your servers so we can look into your issue?”
Customer: “OK, we’ll do that. Thanks.”

I hang up the phone and 5 seconds later my fellow team lead who was trying to get my attention while on the phone informs me that we are completely offline. As in the rest of the world has ceased to exist from our NOC.

10 minutes later the customer called back and I confirmed that yes, we are indeed offline at this time and we are looking into it. Who is this by the way? Turns out to be one of our biggest customers.

We make all the appropriate phone calls to our internal networking group, the DC Operations Manager to notify him of the situation, even contact Savvis to basically ask “Wtf?”. Eventually we start getting some answers.

Savvis didn’t tell us they were going to do some network maintenance and when they did it the router tossed it’s cookies. They had to move us (and lots of other customers) to another router, try to restore the dead router, then move us back, which failed. We were offline for about 4 hours.

In the middle of this the DC Operations manager calls to inform us that no one is to go home until this incident is over, overtime is authorized (Which is another amusing tale for another post).

So while we have nothing to do but sit and wait and answer what tickets we can as they come in (since our ticket system is behind the outage, we’re not getting a whole lot, and our answers are having trouble getting back to customers.) I plug my iPod into my laptop, plug my headphones in and fire up some music. Now, while iTunes won’t do this on Windows (and presumably, on a Mac), Rythembox on Linux will see your iPod as a USB hard drive, recognize it as an iPod and find all your music and let you listen to it through your speakers, headphones, whatever. Easy way to listen to the iPod and keep it charged. So I plug it in, browse for something to listen to, click play… then wonder why everyone is staring at me. For some unknown reason the sound card on my laptop decided to play my music through both the headphones AND the external speakers. Quite loudly. Of course it would decide to do this when I pick “Art Music for the Highland Bagpipe”. Nothing like blasting the NOC with a piobaireachd.