Comment

NewsWireless apologises for being "off air" during emergency work at Red Bus

NewsWireless operates from a rack of servers in what we imagined was a standard co-location centre in London. It turns out that this co-lo centre has a faulty power supply. Yesterday, it crashed, wiping out connectivity for hundreds of web sites for several hours.

Normally, one tries to ignore little interruptions to the smooth running of life; you say: "Accidents will happen" and get on with it. Back in June last year, we shrugged, and got on with it. On 9th November we shrugged again, though less happily. And then yesterday everything went pear-shaped again.

From engineers on site, we discovered that not only had Red Bus lost all power to its whole centre, but that when the power went down, a huge power spike had ripped through all the equipment installed there. When the power came back on, all the equipment stayed dead, because it was dead; people had to rush around with screwdrivers, pulling power systems out of racks, installing new ones, testing, and re-starting them.

And guess what: when you switch a big, powerful computer system off suddenly, it turns out (who could have guessed?) that the software and data isn't necessarily in good shape, and re-starting takes time.

NewsWireless is just one tiny part of one of the servers in Red Bus, but if the other customers feel as frustrated and angry as we do today, it's not hard to explain the user revolt reported yesterday.

We had a pretty nearly wasted day as we tried to find the server, then started debugging all our own links into the server, then re-booted our own systems, switched to 3G data connections, and finally, found that even when the server was back, it wasn't up long enough to post stories.

Quote from another angry user: "They only have two things to do - keep the door locked and the power on. But even this seems beyond them."

The time to shrug shoulders is past: Red Bus should have fixed this problem back in June. Talk of spending a piffling two million now, eight months later, is just pissing on the ashes left after the fire - it doesn't smell right, it doesn't sound impressive, and it's not going to persuade anybody that there's any point in hanging around the place.