Will You Comment On This Post?
If you do, you'll use a BBC system called DNA. It powers most of the comments, message boards, forums, blog comments, spaces and other social media applications across bbc.co.uk.
Just think of all the comments on 606, Robert Peston's blog, h2g2, the Archers message board, CBBC boards and many, many more...
On the morning of Thursday October 9th, there was a major code release - three months' worth of updates and fixes were scheduled to go live after extensive testing on the staging server. Usually, releases happen monthly, but due to the Olympics, we didn't release over summer.
The DNA team had planned on rolling out between 7 and 10am, but because of the popularity of Peston's Picks, the blog team felt we couldn't take any chances.
So the developers came in early and took down DNA between 4 and 6am. By 7am, everything was done, the software was back up and everything was working fine. By 10am, all was still well, until the central communities team picked up a minor little problem. Some threads on a single message board were being automatically closed. It wasn't a major problem; rather a minor problem that needed an urgent fix.
Paul and Jay from our communities team ran between their desks and the developer team to try and identify and to isolate the problem. Working closely with Mark and Mark on Martin's DNA developer team, they quickly identified the offending line of code.
Because DNA is used in so many different ways on so many different parts of the BBC website, it's very difficult to set up tests that exactly mimic all the possible different permutations. Very occasionally, a problem will slip through the net.
In less than an hour, the problem was identified and the one line of code was fixed. There were less than 500 threads affected, but they still needed to be reopened. Once again, Paul and the DNA team jumped in and put together a script implementing some of our moderation rules, sort of in reverse, to fix the problem. Shortly after 11am, this was run, and we were back to normal.
Or so we thought.
Sport contacted us saying that a lot of their threads were still closed - the fix had apparently not worked for them. The 606 service is by far the biggest single platform using DNA. Luckily, this turned out to be a caching issue, as we have to cache their boards because they're so big. The cache was cleared and 606 was back to its wild ways.
Not everyone was happy, though, as this unhappy 606 poster posted:
Why am I getting this message, I've been a member since 2006 not yesterday, how is that new 606 has been a complete mess today, sort it out
At the end of business on Thursday, it was also clear that we had another problem: the so-called "new user hole". Despite being around for a while, some users were identified as brand new users. This also meant their comments didn't show up immediately, as they were pre-modded in accordance to our house rules.
However, this turned out to affect only 24 users out of all the tens of thousands, and was fixed by close of business on Friday.
By noon on Thursday, members of the DNA team had already put in more than eight hours at the office. The central communities team had helped them to fix all the problems less than two hours after the first issues cropped up.
There was no data loss at any stage, everything was back to normal and the central communities team was just in time for a boring weekly team meeting.
Join in - leave a comment and become a DNA user yourself.
Tom Van Aardt is Communities Editor, bbc.co.uk.
© MMIX~RS~q~RS~~RS~z~RS~43~RS~)
Comments
Sign in or register to comment.
Some sites shut down (without warning) completely during maintenance and upgrades, so it is pretty good to keep the most of the site operational whilst this was being done.
Is possible to fix the most irritating issue, that of apostrophes, hyphens etc turning into question marks? (And pound symbols into diamonds.) I`ve just looked at the comments in Justin Webb`s blog and it still seems to happen e.g. # 110
Complain about this comment
Will You Comment On This Post?
Tom Van Aardt 15 Oct 08, 6:08 PM
Yes!
Michael Walsh 14 Oct 08 7:00 PM
Time travel not restricted to Doctor Who then?
Complain about this comment
Something very wierd happened there.
The post was saying it was not published in our system even though it had been.
Anyway, since it had been published I brought the date forwards.
Apologies for the time travel.
Nick Reynolds (editor, BBC Internet blog)
Complain about this comment
No need to apologise Nick - it just confirms that the BBC is just a bunch o' left-wing, time travellin', tree huggin' meeja luvvies!
Complain about this comment
@SheffTim (#1) I'll look into the special characters issue and get back to you.
@Michael-Walsh (#2) Under the "One BBC" policy there's a lot of sharing and openness between all the different divisions, so we get some pretty cool stuff from Dr Who, yes :-)
Complain about this comment
@Nick - Any reason why the Blogs homepage is has been stick in a time warp showing September stories for the last month?
http://www.bbc.co.uk/blogs/
All thse blogs featured in the top half have had posts long since 8 September ... yet they seem stuck around that date on the blog homepage.
Complain about this comment
Have you fixed the UTF–8 handling yet? Probably not…
Complain about this comment
Preview function?
Stop repetition
(accidental) of identical comments from the same user, often added when there are more than 500 comments and the comment doesn't appear to have been published on the first page.
Complain about this comment
Will You Comment On This Post?
No
Damn, I appear to have created a paradox.
Complain about this comment
Strange that you've updated DNA (after Douglas N Adams, of course), but have not switched the blogs to Barlesque layout (named after the fictional Nathan Barley).
Probably worth pointing out the former is on http://en.wikipedia.org/w/index.php?title=The_Long_Dark_Tea-Time_of_the_Soul_(radio_serial) and the latter's creator is doing a Big Brother with Zombies on E4!
Complain about this comment
Can you comment on this post?
"Yes"
Which one? I or _ or / or \ ?
I - It's still standing up.
I seem to have created a para Docs.
Complain about this comment
Maybe H2G2 could get a revamp and a bit of money and love spent on it, as the BBC's social media / community site. It always gets over looked, it was doing blogs long before anyone else. It'll be 10 years old next year!!
Complain about this comment
This comment was removed because the moderators found it broke the House Rules.
This comment was removed because the moderators found it broke the House Rules.
Tom, will DNA ever get the facility to edit posts. Many is the time when I've pressed Post Comment, only to spot a few seconds later that I've written "their" when I meant "there" or "whose" when I meant "who's". Not earth shattering I know, but it niggles me that I can't correct it.
Complain about this comment
There is a preview button Paul.
Complain about this comment
You've told me that before Nick, it hasn't helped. It's only when I get to the stage where I can't change them that I notice the errors, or else I've simply forgotten to press Preview. So an edit button would be nice, as incidentally would a spell-checker in Preview mode.
Complain about this comment
#17
"So an edit button would be nice"
Might be workable on a reactively or post moderated blog but on any pre-moderated blog the edited comment would have to be resubmitted for moderation, thus causing even more work to the mods and slowing the busy blogs up even more...
Nost poeple can accept typose on this sort of fast moving, interactive, socail media.
Complain about this comment
Pardon?
Complain about this comment
Boilerplated's reasons for why we don't allow people to edit comments after they've been published are good ones. Once it's published, it's published I'm afraid.
Complain about this comment
Nick, I was joking - I get the point. Cheers.
Complain about this comment
View these comments in RSS