Now Pilot Technical Architecture

Tuesday 21 May 2013, 08:00

David Wynne David Wynne Technical Architect

Tagged with:

Hello there my name is David Wynne and I’m a technical architect and founder of Red Badger, a creative software workshop that has been working with the BBC to develop features such as BBC Now.

Through BBC Connected Studio we’ve been working closely with Eleni Sharp, BBC Homepage product manager and her team to design, test and develop the BBC Now pilot and you can read more about the feature in Eleni’s recent blog post.


The Challenge

Our challenge with BBC Now was to capture all the content being published both on and on other platforms and to bring it all together into a single unified stream of real-time push updates that are instantly delivered live to the BBC Homepage.

Broadly speaking this broke down into four technical challenges:

  • Monitoring for new content.
  • Contextualising new content against a BBC brand (such as Top Gear).
  • Pushing content out to the BBC Homepage in real-time.
  • Aggregating common terms within the content to establish trending topics.

The Solution

Each type of datasource (RSS, Twitter, BBC iPlayer content etc) is monitored by a separate Content Processor. Each Content Processor identifies new content, transforms it into a standard structure, associates it to a BBC brand and publishes it to the Firehose.

The Firehose is an unadulterated, unified stream of all new content from every Content Processor. The Firehose allows for multiple subscribers who can use or transform the data and produce their own streams of content.

The primary subscriber to the Firehose is the BBC Homepage which surfaces each item of content as an update in the Fast Lane. The Fast Lane is a column on the Homepage that receives push updates, from the Firehose and displays them in chronological order.

The Trend Analysis module also subscribes to the Firehose and receives the same set of content. The Trend Analysis module extracts common terms from the published content and aggregates that content using those terms, creating categorised ‘buckets’ of content.

When the buckets are sorted the Trend Analysis Module produces a list of the most common terms appearing in content over a given period and facilitates content filtering.

now_tech_1024.jpg Flow diagram of BBC Now

The Tech

The heart of BBC Now is built using Node.js, a relatively new technology but one which has picked up huge momentum and for good reason. Node.js is a technology born of the web, for the web. It uses Google's V8 JavaScript engine on the server and an event loop to drive code execution. It allows scalable, real-time web applications, quickly.

BBC Now is built out of flat, independent modules making it easy to maintain and scale out. The thin layer of glue that holds them together is Redis. Redis is an advanced key-value store that, like Node, uses an event loop to drive execution and is both fast and scalable.

Redis also offers a fast and scalable implementation of the Publish/Subscribe (or pub/sub) messaging pattern. A pub/sub implementation allows publishers (in this case our Content Processors) to publish data without needing to know how it is delivered to other parts of the application.

Conversely the subscribers (in this case the BBC Homepage & Trend Analysis module) can subscribe to topics of data without needing to know which other part of the application produced it. They simply react to new data as it’s published.

This separation of concerns encourages each part of the application to concentrate on their single responsibility, which in turn means the system as a whole becomes easier to scale and reconfigure.

In essence, BBC Now is all about subscribing to data and publishing it elsewhere making Redis a perfect partner.

In designing the real-time push updates to the BBC homepage we had a few options.

This most traditional method of polling, whereby the client (your browser) periodically asks to the server (the BBC homepage) for updates, seemed like a poor choice. Most of the work we had put in was to convert passive data sources into active ones, returning them to a passive data source seemed to be heading in the wrong direction.

WebSockets, a relatively new technology that allows a constant two-way conversation between client and server (think of an instant message conversation), seemed like an obvious choice. But would have perhaps been a bit overkill given we only needed one-way communication.

Server-sent Events (SSE) provided the perfect lightweight middle ground. SSE are expressly designed for providing push notifications from a server to a browser client and have wide browser support (apart from IETM). SSE operate over HTTP (unlike Websockets) which means that JavaScript polyfills can be employed to bridge the gap for older browsers which don’t natively support the mechanism.

The Future

Besides further feature development (more data sources, personalisation, mobile) we’d love to dovetail our work with that of the BBC Linked Data Platform team which would provide a wealth of contextualised data that would really push us onto the next level.

I’d love to chat more about BBC Now – please leave a comment below or you can catch me on Twitter @dwynne or via the #bbcnow hashtag.

David Wynne is a technical architect and founder of Red Badger.

Tagged with:


Jump to comments pagination
  • rate this

    Comment number 1.

    I hope you remove the Twitter updates from the new BBC Now Homepage. Twitter means the feed updates far to often to be useful. The twitter conversations are disconnected and therefore incomprehensible and the information content is so broad as to be meaningless. The last two iterations of the homepage have been about providing a wide range of content to the audience. However, with such broad content availability on the BBC, a high degree of customisability is important to make the page useful. This is why the conversion to the carousel page was a failure, it removed the customisation, bombarded the user with irrelevant dynamic information and therefore the readers bypassed the homepage and went directly to the pages that comprise their interest, therefore ironically reducing the content being placed in front of them. (In my case I go directly to the news page, missing the wide range of material I used to get). I think BBC NOW could be a good product if it removed the Twitter feed, therefore concentrating on news, sport, sci/tech and programme information rather than a stream of irrelevant twitter responses to a conversation I have joined half way through.


This entry is now closed for comments

Share this page

More Posts

BBC Online Briefing: Your questions

Monday 20 May 2013, 07:01

Building the Knowledge & Learning Beta

Wednesday 22 May 2013, 07:00

About this Blog

Staff from the BBC's online and technology teams talk about BBC Online, BBC iPlayer, BBC Red Button and the BBC's digital and mobile services. The blog is reactively moderated. Your host is Nick Reynolds.

Blog Updates

Stay updated with the latest posts from the blog.

Subscribe using:

What are feeds?

Links about BBC Online

BBC Internet blog Archive

owl-plain-112.jpg 2012 ι 2011 ι 2010 ι 2009 ι 2008 ι 2007

Tags for archived posts