18 Months Of Blogs (Part 1)
Part One: History & Technical Challenges
The BBC Blogs Network has been up and running for 18 months. This milestone provides a good opportunity to give you some insight into what we've been doing, how we think our efforts measure up, and where we might be headed in the future, editorially and technically.
That's a lot to cover, so we're splitting the discussion in two. Today's post deals with the technical aspects of our Blogs Network and later this week, I'll return with a second post looking at editorial challenges.
So, first: some history. In early 2006, following several forays into blogging by the BBC, most notably BBC Scotland's Island Blogging, Ouch! and Nick Robinson's Newslog, the decision was made to customise and install an off-the-shelf blogging solution and create the BBC Blogs Network, which launched in April.
The graph below provides four snapshots, taken at six month intervals, of the unique visitors and visitor sessions for the BBC Blogs Network, starting with that first month and ending with October 2007.
The graph illustrates that the BBC blogs have, in aggregate, found a large and steadily growing audience. Additionally, it's worth noting that the average time of each visit has remained fairly static at just under four minutes.
The downside of this success has been our technical infrastructure becoming increasingly unstable. It was built quickly and involved a number of customisations to the software which effectively ended our ability to easily install software patches and bug fixes provided by the vendor in order to deal with some of the technical issues we've faced.
Our technical woes will have been invisible to most people visiting a BBC blog, but this will be of little comfort to those who have encountered them first-hand, including Newsnight editor Peter Barron who recently posted on The Editors: "Often I try to respond to a comment or complaint about the programme and end up gnawing my knuckles in frustration as the response either doesn't appear for many hours or fails to materialise at all. Hardly the best way to have a free flowing dialogue with our viewers."
It's not just those trying to comment who have ended up frustrated. An increasing number of live programmes, such as the recently launched iPM and World Have Your Say have begun to use their blogs to engage with their audiences directly. Inevitably, this leads to upset programme makers too.
Eddie Mair wrote a post at 7am on Saturday, the launch day of his new programme iPM: "I do want to say what a bang-up job the team have done. As you'll see from the iPM Blog, they have done a ton of great work - I find it really rewarding to have a rummage round the blog." Later that morning, he updated the post to say: "By the way we are aware that the 502 error messages are back. Great day for it to happen, no?"
We are aware of the problems and of how frustrating they are for anyone who takes the time and effort to compose a thoughtful comment only to see that effort result in a 502 error or an automatically generated message from the server saying, wrongly, that they've been banned. Here's where we're at in our efforts to diagnose and resolve them.
The early results of a full technical review we've commissioned from an outside agency - as well as our own investigations - indicate that these technical problems are almost certainly down to the bottleneck that exists between our three front-end web servers and the database server. The database is what ingests, stores, and allows the management (including moderation) of comments and then outputs those comments and the other data that make up the web pages you look at.
The graph below [click for large] shows the percentage of CPU usage on our blogs database server over a week. Clearly visible are the spikes, sometimes reaching as high as 100%. It's during these spikes that you are most likely to encounter problems commenting.
The primary cause of these spikes is spam attack. Worryingly, the amount of spam comments submitted is on the increase and, across the network, we can now get as many as 50,000 spam comments per week. In one particularly bad day last week, the comment cgi was hit well over 30,000 times.
Each time a spam comment is submitted, usually by a bot, the database CPU crunches the data. We're aware that the version of the application we use is less efficient in dealing with spam than newer versions but, as I mentioned above, customisations we made to the application have effectively ended our upgrade path, so we've been unable to take advantage of upgrades since made available by the vendor.
That's the bad news. The good news is that our new(ish) applications engineer, David, has been busy firefighting and implementing various anti-spam measures. We've also commissioned Headshift to conduct a full technical review of our blogging platform, the results of which will be delivered on November 19th. I don't want to pre-empt that report but I can say that, in addition to reviewing our existing platform, the report will also present us with a number of suggestions for improving our technical infrastructure and these are likely to include a mix of software and hardware upgrades or replacements.
My apologies if you've encountered problems when using the BBC's blogs. When the review is delivered next week, we'll begin the process of planning and implementing a robust solution over the following weeks and months. In the meantime, we'll carry on with our daily fire-fighting and will continue to provide as good a service as we can - although there are bound to be a few problems from time to time.
I'd be glad for any thoughts you might have and hope to see you on Wednesday for Part Two.
Robin Hamman is a Senior Community Producer.