BBC Online Outage on Tuesday 29 March 2011
As many of you will have noticed (and reported on Twitter) the whole of BBC Online was down last night for an hour from 22:40 due to a major network incident. We would like to apologise to everyone that was unable to access BBC Online during this outage.
Our systems are designed to be sufficiently resilient (multiple systems, and multiple data centres) to make an outage like this extremely unlikely. However, I'm afraid that last night we suffered multiple failures, with the result that the whole site went down. Enough of the systems were restored to bring BBC Online pretty well back to normal by 23:45, and we were fully resilient again by 04:00 this morning.
For the more technically minded, this was a failure in the systems that perform two functions. The first is the aggregation of network traffic from the BBC's hosting centres to the internet. The second is the announcement of 'routes' onto the internet that allows BBC Online to be 'found.' With both of these having failed, we really were down!
We'll be taking a very hard look at what we need to do to make sure that this doesn't happen again.
Richard Cooper is Controller, Digital Distribution, BBC Future Media.