Darkside Rainbow

If you’ve been trying to load this site for the past several hours (and missed today’s post), sorry everything’s been down. There was…a mess. A huge, nasty mess in the data center hosting our servers, so tangled and convoluted that it could only reasonably be called a clusterfsck. Only some sites were affected, but for those of us who were, it’s been a real pain in the petuckus waiting for everything to come back up again.

To make a long story short (basically paraphrasing the enormous statement from the data center that our Head Honcho copied over to the writer forum): Brief power outage. Generator runs for a little while, then transfers main load to the UPS. Main batteries fail. Backup batteries fail right after that. Generators go back online, troubleshooting starts. Backbone connectivity goes FTTZZZ because something went screwy with one of the switches. Half the backbones die, the other half start screaming in pain from carrying the full load. An access switch serving 16 servers died a grisly, brutal, screaming death. Oh, an APC failed too. Think we’re done? Oh, hell no. Some of the servers just tanked due to the power-related forced reboot, and have to be manually restored with an fsck (file system check, you pervs).

Or, to put it very simply, boom. Boom today, boom tomorrow. Sooner or later, always boom.

 
Yyyyeah. So when our data centers do FUBAR, they don’t half-arse it. To quote another 451 Press writer: “This definitely seems like the deluxe, full-ass version of FUBAR.”

We’re back up now, though.

What? What are you still staring at? The real content’s one post below. Go. Be outraged. Be pragmatic. Be apathetic. Be…something. I don’t know. I’m going to go be productive and write something. Later.

~Adri

omg holy crap BOOM.