« Previous | Main | Next »

Changes to BBC Message Boards: Update

Post categories:

David Williams | 09:20 UK time, Friday, 24 September 2010

I thought I'd just give a short update following my last post about message boards.

I will try and make this as free from jargon or 'techie' speak as possible, even though this is the internet blog, where we occasionally get a bit 'geeky'.

I'm not going to deny there are some bugs that need fixing or there are some extra features you'd like to see.

In the release deployed on Wednesday we delivered a number of fixes:

· Fixed an issue with link (url) translation.

· Increased security by preventing the submission of malicious code in posts.

· Tidied up some general formatting issues in posts.

This release is half way through our current sprint and there are more items being worked on presently.

I'd also like to point out my team isn't just working on messageboards. We're a small team that has many projects to work on. In addition to the messageboards we provide the comments on blogs, moderation tools and backend services to manage complaints, services for BBCiPlayer social features, 606 and h2g2 articles and forums.

We're also working on a significant project involving data mining/business intelligence technology that will allow us to have more detailed insights into how moderation works across all the BBC's social media services.

Additionally we are looking at various ways we can enable search across messageboards and blog comments. I'm not just talking about the Food or Archers messageboard, this is across all user generated text content across the whole BBC.

Originally we had this feature and it worked well when the scale of user contribution was small. As this has scaled up it has become an interesting challenge for us.

Indexing dynamic content compared to statically published pages is a real challenge. We have multiple levels of caching to prevent us being overwhelmed by search engines (bbc.co.uk receives 10% of UK internet traffic) and to stop them killing our ability to deliver content (and we also prevent spiders/web crawlers visiting us), so to enable search on our 250 GB database isn't a small feat.

It's something we are tackling though, and we need to find a solution that is cost effective for the license fee payer and can be supported across the various services live operation so it's as reliable as possible.

David WIlliams is Product Manager, BBC FM&T Social

Comments

  • Comment number 1.

    Could you please make sure that in the RSS feed of your blog summaries are available. Your RSS feed put the whole lenght of the text as summary in the RSS Feed reader, like Google Reader.

    Kind regards,
    Piet

  • Comment number 2.

    Indexing dynamic content compared to statically published pages is a real challenge ... so to enable search on our 250 GB database isn't a small feat.

    But, as no post is ever erased or moved on your system, surely it is closer to rapidly-accruing static content than dynamic content? What I mean is that you could take a snapshot of the database today, copy it onto a reasonable speed desktop computer and probably generate an index for it in a week or two. Then, you would port over the posts that had been added in that two weeks and index them - that might take an hour, and you would be in a position to port the index back across and start adding to it "live". Every time I have seen a member of BBC staff talk about indexing the legacy posts thay claim it would "slow the system down", but with very few additional resources (a loan of a decent PC!) this would not be the case.

  • Comment number 3.

    _ another glitch ? _ Hidden etc showing when it should not
    Just a comment I seem to see hidden or post closed unexpectedly at the moment sometimes.
    I noticed a comment from someone else on the food board about something similar.
    No time to investigate further right now, but maybe another problem is emerging or re-appearing.

    I note someone on the Food board suggested they used false links to highlight items, I guess they loose that now as 'malicious code'
    and that possibly latest changes finally nails down the coffin lid on the R4 & Archers boards use of unofficial search etc scripts

    __ RSS for this blog __
    I found this from this blog from food mb host's comment.
    My RSS subscription to InternetBlogs in FireFox is not yet showing this as a listed bookmark, the last one I see is "http://www.bbc.co.uk/blogs/bbcinternet/2010/09/improvements_to_wildlife_finde.html"
    until I actually open the InternetBlogs.



  • Comment number 4.

    David,

    I am glad you are mentioning specifically search engines, and searching and indexing messageboards in this blog.

    _ Arachnahobia_ :-) you ban spiders ?

    Maybe you could clarify something for the technically less knowledgeable users like myself. You write in the main blog above:

    and we also prevent spiders/web crawlers visiting us

    The messageboards are of course indexed by external search engines, and the BBC oficial guidance is to use external search engines. I would have thought that implies search engines do crawl your messageboards, or that you expose some sort of equivalent database to them. Maybe someone can explain what happens. (or other users post links to explanatory articles)

    _ Message Board Titles List _
    I have mentioned this before. A lack of Titles listing is a big drawback.

    It would help if users could access a list of thread titles.
    Especially if this could be something that in the absence of a BBC search facility could be indexed and searched with an external search engine.
    Eg
    if I wanr to search the food mb for posts containing the word
    eggs in the title
    Or I want to search the iPlayer message board for a named TV program or a software programme version
    Or for instance OS, Country, Platform, or Stream specific problems discussed on the iPlayer mb.

  • Comment number 5.

    _ Geeky_ BUT overkill when applied to messageboard content
    I read your linked ...'geeky' blog.

    Nice to know the great lengths Beeb goes to carefully processing data for indexing & linking. But you are mainly describing the news stories which are complex, content rich, dynamic, changing from second to second and obviously a challenge to index and link.

    A message board index or search that is a day or even a week out of date is often going to be of use.

    Users can easily look / scroll through the most recent posts. They display chronologically at present. And again at their own discussions.

    Back to the example in my last post re 'EGGS'
    Your news system needs to cope with, Dinosaur eggs, Hens Eggs, eggs & IVF, Easter eggs (chocolate, decorative none edible & Software Easter Eggs) etc etc

    A user on a food board only needs to search the title list for what they want & or the mb text itself & can customise as they see fit when search engine results come up, & may put in say
    duck egg, egg & bacon or mayonnaise
    The user does not need a fully automated sophisticated inteligent system, they just need something better than we have at present, which is nothing other than

    we suggest typing key words such as BBC iPlayer + your issue into your preferred search engine. {BBC iP Help]

    Sure it would be great if you could feed the mb text into your existing geeky system used for handling news stories

    ? _ DISCUSSIONS STARTED _
    As a closing point;
    It may be useful to have a list of discussions a user has started, not just a list of discussions participated in. As usual something common on many existing standard none BBC mb.
    ( Not something I personally find essential, but some prolific posters would I suspect find it very useful)

  • Comment number 6.

    David,

    Are you going to expand a bit more on the three issues you have fixed yesterday ? ie (my numbering)

    In the release deployed on Wednesday we delivered a number of fixes:

    1) · Fixed an issue with link (url) translation.

    2) · Increased security by preventing the submission of malicious code in posts.

    3)· Tidied up some general formatting issues in posts.
    .

    Are these fixes applicable only to the food mb ?
    Presumably the fixes will work also on any other mb as it is improved; is that correct ?

    The fix I numbered 1) above
    Sounds like it could relate i to what I christened the gobeldegook bug. or see (last #28) although I see that bug is still corrupting posts made today
    Compare
    links ok, and deliberately truncated by BBC mb
    With
    Links displayed incorrectly as a text string

    Fix 3)
    What sort of formatting issues, what sort of differences are users expected to see

    The food boards 'My Discussions' listing is still obviously broken
    - last contribution 'link'does not link, it is only plain text
    - post number does not compute correctly displaying what has already been described as'negative farcical millions'

  • Comment number 7.

    I thought your plan was to close them all one by one

  • Comment number 8.

    -John99 - I'm going to check 1) you mentioned. These issues appear related but sometimes have different underlying causes.

    Links appearing truncated is deliberate so it helps the formatting of the page. I'm sure there will be a post about it soon - but there are plenty of url shortening tools now. I personally can't recommend one over another but http://en.wikipedia.org/wiki/URL_shortening has some information about this.

    3) We're working through a list of feedback from you all and will address the most significant problems as soon as we can. We try to find the right balance between fixing problems and bringing new features in, this is in addition to the other projects my team is involved in.

    As for last contributions - we're restoring that soon and the post number problem.

    -lightoftruth - that is not my call. I'm on the technical team not editorial.

    @All - thanks for your feedback on search. As I said we're working on a solution. I (or a colleague) will be writing a post about it once we're in a position to deploy it.

    On the spidering - yes I was a bit general in saying we prevent all our dynamic content being spidered/web crawled. Some of it is currently indexed by various search engines.

  • Comment number 9.

    David,
    Thank you for replying.

    __Truncated Links
    I was hoping you had fixed one of the bugs.

    Links appearing truncated is deliberate so it helps ...
    Yes understood. Please do not misunderstand my comments. I was not complaining. The iP mb for instance changed earlier this year to using truncated links.

    In my comment #6 above if you follow the two links you will see they are to exactly the same post. It is obviously an example of a mb error. One version is with truncated links the other with plain text. usually in such cases tags are also seen. The example in comment #6 being from a post made Today, proving the problem still exists.
    An older example would be good & bad.
    Why do these two forms exist anyway. And why are only some posts affected by this messageboard bug.
    Just conjecture I wondered sometimes whether they tend to be occurring in threads that are subject to moderation reviews.

    __3) Format improvements already made on Wednesday
    You now say
    3) We're working through a list of feedback from you all and will address ...
    But in the blog above wrote
    Tidied up some general formatting issues in posts.
    I was wondering what actual changes we should already be seeing as improvements on the food board. That being presumably the board you are working on.

    _ Search & Bugs
    Hope all goes well in fixing the bugs, and look forward to future comments about messageboard searchability. Any idea on timescale for that. As in when you or someone else is likely to comment further, next month, next year, or is it a case of if and when money is available.

    It is appreciated that you are taking the trouble to try to advise us about what is happening to our messageboards.

  • Comment number 10.

    Thanks Peet for your comments; #2 Above, and your comments on the Archers board where you discussed my concerns about messageboard bugs.

  • Comment number 11.

    I'm still at a loss about this problem providing a search facility on the food message board. There's a search available on the food homepage, and there used to be a working search engine for the message board, which was built and maintained in his own time, at his own expense, by a community member - so that makes it quite hard to understand why it would be beyond the resources of the BBC.

    Sadly that facility has not been available for a while now due to a dispute with the host of that message board and part of her new policy.

  • Comment number 12.

    A search option would be nice

  • Comment number 13.

    Please can we have a list of the problems with the new Food Board that ARE scheduled to be rectified, so that we can see if all the problems we perceive are at least known to you.

    eg the lack of a link on "Your Discussions" to one's own last contribution

    and the minus millions post counts

    and the eye-straining font size - surely curable in minutes?

    TIA

    Steve

  • Comment number 14.

    Search has been promised for 5 years. It isn't going to happen if the BBC continue to be hide-bound by their design restrictions.

    Your 250GB database for message boards can be deleted today... if no-one can search it, it might as well be deleted. OR, as many have suggested (and indeed as Saccarin on the Food boards has done for the past 3 years) take a snapshot of the boards and search that instead.

    OR strip all the boards apart so we only search a specific board. The design ideal of seamless movement /login / search of all BBC boards together is flawed - I would speculate that only 2% of your users move between different boards every day, and of those, hardly anyone would need to do a BBC wide search. Logging in for each board is hardly a chore..especially as login is flaky anyway.

    So all the design weaknesses of the BBC boards (and blogs) have no realworld benefit but completely hobble its functionality.... as we al know.

    OR -scrap the whole bloated, expensive and clunky platform and buy something better... probably MUCH cheaper in the long run and the users will be much happier to be using a full feature set that is tested and works.

  • Comment number 15.

    Hi Organoleptic Icon,

    Over on the POV boards we have been through the process of listing our top 20 issues - ad nauseum. It was a 100% fruitless exercise and we ended up with the list being completely ignored. 2 years later we have seen board closure, 8 here today gone tomorrow hosts, draconian rules and not a single practical improvement. The users left in droves and the acrimony lingers long after the perpetrator has moved on.

  • Comment number 16.

    OfficerDibble - the specific changes to the Points of View message boards have been discussed at length on this blog and happened more than a year ago. Therefore I'm ruling them off topic.

    Please stay on topic.

    Thanks

  • Comment number 17.

    I felt the proposed changes (and listing of issues) of the food boards were a very close parallel to those promises of change on other boards - our universal experience is very pertinent to Organoleptic's post 13. I wanted to assist in managing his expectations from this current process.

    Search is a very important issue. 6 years ago it was promised as being "in process" and that promise has been made numerous times since then, right upto hosts (including you Nick) stating that "it is being looked at". I wanted to post a reality check for anyone expecting change for the better. History suggests otherwise.

  • Comment number 18.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 19.

    Can we keep the conversation on topic and civil please.

    Thanks

  • Comment number 20.

    To rephrase, past changes to boards are clearly relevent when considering new changes. It is hard to see why they would be considered off topic by an impartial observer.

  • Comment number 21.

    @Nick

    Having watched some of the arguments on the Food board from the sidelines, I'm just curious - why *aren't* "Subscribe/Unsubscribe" links a standard part of the "Vanilla" skin?

  • Comment number 22.

    RedRedRobin - the changes to the Points of View message boards were primarily editorial and about hosting. David's post is not about these things, rather it's about technical improvements. The POV changes have also been discussed at great length on this blog and on the boards themselves. I don't want this post to turn into yet another off topic discussion about them.

    Thanks

  • Comment number 23.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 24.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 25.

    All our requests for improvements were technical. I am sure the Food board users will identify with our requests for search, editing, quoting, profiles, subscription, peer rating, threading, real names, etc- as Nick says, none of the changes were technical.

  • Comment number 26.

    and of course the Food Board would also benefit from Stickies, hot topic filtering, image embedding, and URL linking. In summary - all the facilities we have all experienced elsewhere since the late 90's.

  • Comment number 27.

    New bug report:

    In the list of 'my discussions', the value for the number of new posts in a particular thread is currently corrupted, e.g.:

    'New posts: -96621191'
    'New posts: -101529706'

    This bug is beginning to appear from approx 14 00 today, when the Radio4 blueys were switched over. The bug is now spreading to the iPlayer board, but as yet older style boards, e.g. Archers, are currently displaying correct DNA values.

    Russ

  • Comment number 28.

    Update: new bug has now spread to older style boards, where new post number indications are all locked to 'New posts: 0'.

    Russ

  • Comment number 29.

    Bug analysis: I think the entry for the 'number of new posts' is perhaps citing the number of the permalink of the latest post rather than the number of new posts?

    Russ

  • Comment number 30.

    Russ - these bugs seem to be illustrating how even minor changes are introduced without proper testing - and all for no perceived benefit. Other message boards don't have their users as guinea pigs. The changes that are required are root and branch, not tinkering with a flawed platform. Professional boards sorting out post count bugs back in the 90's. How much does all this re-inventing the wheel cost the BBC?

  • Comment number 31.

    I do hope you are reading and taking note of the feedback on The Choice Is Yours board. If you are, but do not consider it worth acting upon, perhaps the board could be renamed The Choice Is NOT Yours?

  • Comment number 32.

    I have also noticed the 'my discussions' bug, but this only appears to be happening on an Archers thread.

  • Comment number 33.

    The Choice is Yours: Text lines are too close together. And a former nice touch(?) seems to be not working now, that buttons on a page from a multiple page thread to go to the first/previous/next/last page were visually changed when they were not operative because you were -on- the first or last page with nowhere to go. However, the current page number (1, 2, 3) appears as a different coloured button (and likewise only decorative... oh, no, it actually works, that may be pointless but keep it).

  • Comment number 34.

    ...Hullo. I normally leave two spaces after a full stop. Not just one like this. Is there a problem here with that, I seem to have no spaces? Or did I get it wrong in previous post? This time it looks okay in preview.

  • Comment number 35.

    Robert... lost spaces - it is a bug. They have known about it for a long time.

  • Comment number 36.

    Bug update: the 'Latest reply' field for thread listings on some old-style boards is starting to show incorrect data.

    Russ

  • Comment number 37.

    The bug reported in my #36 now appears to be corrected.

    Russ

  • Comment number 38.

    Sitrep: all Radio4 blueys continue to be in lockdown, with no new threads being allowed. Other new style boards (Nature UK, iPlayer, and Ouch!) are not currently affected.

    Russ

  • Comment number 39.

    Nearly two weeks since the blog post, any more updates yet ?

    The Archers board has quite a few comments on it even though it has not yet been improved eg thread And the latest board to be improved the R4 board has multiple threads started by the BBC host, who has now gone silent with a thread approaching 300 comments after only a couple of days

  • Comment number 40.

    It would be nice too if someone would have the courtesy to tell the unhappy folk on the WoM board what is happening about this - see http://www.bbc.co.uk/dna/mbradio4/NF2766781?thread=7799328.

  • Comment number 41.

    What's the story with the font size on the new style messageboards? Are we able to increase it? I'm short-sighted and wear glasses but find it difficult to read. What accessibilty standards were used? Thanks

  • Comment number 42.

    The delivery of the changes to the R4 messageboards has been a disaster. For Goodness sake, back the whole thing out, test it properly and then................bin it. Even without the bugs, it is a dog's breakfast that is difficult to read and navigate.
    Two days later it is still not possible to start new threads. There has been no apology, explanation or status report from the hosts, who appear to have gone into hiding. The whole thing is a disgrace and highly unprofessional. Ok, mistakes happen, but the response from the BBC to this shambles verges on contempt.

  • Comment number 43.

    It sounds like there is a consistency in the user responses to changes throughout the platform. Perhaps an off the shelf platform would be a more professional way to provide the service. Scale down your ideals and bring them in line with a realistic appraisal of the ACTUAL needs of the users, not some global strategy for a homogenous over-arching "community" system that won't give any benefits when it works properly, and in practice is just a stitched together bodge that has split any semblance of community.

    The irony is, the community (for what there is) only really unites and shows affinity when working towards a common goal - to protest against apparent inadequacies of the platform and rail-roaded change.

  • Comment number 44.

    @ Atilla the Pun#40
    You linked to the general thread.
    I note there was also a comment saying:
    I've found another bug -- perhaps the weirdest yet. {-- click link

  • Comment number 45.

    Absolutely rumbaba! msg 42! grrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

  • Comment number 46.

    Should the BBC dig themselves out of this BBC generated quagmire it may be worth remembering this blog post was actually mentioning something useful about messageboards.

    Something users have repeatedly said they want
    Something that the BBC has repeatedly promised.

    I quote directly from the blog post above:
    Additionally we are looking at various ways we can enable search across messageboards and blog comments. I'm not just talking about the Food or Archers messageboard, this is across all user generated text content across the whole BBC.

    Lets hope that is implemented soon and more successfully than the current changes.

  • Comment number 47.

    I don't disagree with you, John, but until the BBC can return to a state where its DNA database can be interrogated correctly and pagecode generated correctly from it, together with their correct link translations etc (all of which are failing in some or whole part at the moment), I can't see any point in discussing secondary aspirations such as search.

    Russ

  • Comment number 48.

    Personally I don't care about search across ALL BBC content - that is why it currently doesn't work.. the database is too big - and that fact has hobbled the platform for 6 years - and every new BBC techie promises they'll look into it and it always goes quiet for a year or two. So forget the top down ideals... give us board by board search - that will satisfy 90% of needs. (how many want to search "Cruickshank" on the R'n'B and Parenting boards?)

    Perhaps we should also address why the search seems selective... I can't find a way to discover all the threads on Blogs about DOGs and IPPs. I wonder why the search is slightly blind?

  • Comment number 49.

    A search would be a nice feature, but more important at the mo are working Radio 4 message boards. They've been broken since the "upgrade" and nobody had bothered to inform the message board users when or if they are going to be fixed.

  • Comment number 50.

    Stop the 'upgrade' of the message boards, please. It's absolute rubbish. Please 'downgrade' immediately.
    It wasn't broke; so why waste time trying to fix it?
    Or at the very least, if you must have a revamp, despite the waste of money that it is, try employing somebody who is capable of effecting it properly..

  • Comment number 51.

    Can someone reassure us that the changes will not be rolled out to the remainder of the boards until all the glitches are fixed? Thanks.

  • Comment number 52.

    Why are the R4 boards still not functioning properly? Forget the 'style issues', it has been a week since it was possible to actually start a new thread on and basic functionality like the 'new posts' counter, the display of the correct time and date is still wrong and the boards are opening an hour late withour comment or explanation. The BBC continue to treat us with absolute contempt. Ok, maybe the delivery was a complete disaster, you have no resources to fix it and we will have to be patient but tell us that, if it is the case. At least let's have an apology and some sort of explanation.

  • Comment number 53.

    Or, as any professional organisation would have done, revert to the previous working platform overnight.

  • Comment number 54.

    I don't use blogs much, how are they supposed to work? Is someone suppose to read these comments and then respond? If so, can someone please respond to the request to know what the current status of the R4 boards is? Believe me, my expectation levels are rock bottom, I'm not expecting you to actually fix anything, I just want someone at the BBC to (a) acknowledge that there are major problems (like not being able to start a new thread since 5 October)(b) Maybe apologise and (c) Tell us what the current status is. Come guys, that's not too much to ask, is it?

  • Comment number 55.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 56.

    Um, hello? Is anybody there?

    Anybody going to answer our questions?

  • Comment number 57.

    Sitrep: 'Start a new discussion' functionality now restored to R4 blueys.

    Russ

  • Comment number 58.

    Hello - I've asked the DNA team for an update. They are aware of the bugs you mention and they are working a release for early next week which should hopefully fix most of them.

    Thanks

  • Comment number 59.

    Thanks Nick.

    Will all the bugs be fixed before the new design is rolled out to any more messageboards?

  • Comment number 60.

    .. and the unpopular design / readability / layout issues... will they be fixed? No, thought not.

  • Comment number 61.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 62.

    Why have most of the comments that were posted on the 9th and 10th been deleted? Does nobody have the courage to answer our questions? Seems not, so they just delete them!

  • Comment number 63.

    There seems to be another bug here.... when two posts are deleted (for mentioning a different bug) you are only notified by email once. Can you explain this Nick? and the other questions?

  • Comment number 64.

    OfficerDibble - please explain what you mean. If you are talking about "bugs" on this blog remember that we are talking about message boards, not blogs.

    Thanks

  • Comment number 65.

    According to Ramona, the food host, the new changes will be rolled out to other boards from now to the end of the month. So, that leaves roughly two weeks to sort everything out, and eradicate the many bugs. Will it happen? Let's hope so. There's still quite a lot of people who aren't even vaguely aware this is on its way. Best to get the bugs sorted or there'll be a mini-meltdown (again)

  • Comment number 66.

    Well the evidence is that the Message Board bugs will not be fixed in time for launch as evidenced by all the Bugs on the Blogs here... they have been evident on a live platform for months and no-one has managed to fix them.

    Secondly, a lot of the message board criticism is of the design.. how it is difficult to read, less easy to navigate, to speed read, to browse, and of course the crazy waste of display real estate - all to make it consistent with a homogenous design.

    All this is a side issue as no-one wanted the new facelift -nor the bugs that come with it, and the long list of USER generated needs have been ignored again.

    I am curious about Dave Williams' comment that this revamp was a result of feedback - he didn't answer who that feedback came from. I doubt it came from the non-staff users. Anyone?

  • Comment number 67.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 68.

    This comment was removed because the moderators found it broke the house rules. Explain.

  • Comment number 69.

    #58 Nick
    They are aware of the bugs you mention and they are working a release for early next week which should hopefully fix most of them.

    The R4 messageboard seems to have bugs with both page numbering and the reply to message permalink. Subtly different from the bug on the Food mb that was or was not fixed recently.

    No doubt such bugs will be introduced to other boards.

    What is the latest on other boards getting improved, the Food Host did say you are still expecting all boards to be changed this month. but the Archers host did hope to be able to warn users in advance or any change

  • Comment number 70.

    Hi - for a list of things that will be fixed next week see this new blog post from David.

    Thanks

 

More from this blog...

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.