Tuesday 21 May 2013, 08:00
Through BBC Connected Studio we’ve been working closely with Eleni Sharp, BBC Homepage product manager and her team to design, test and develop the BBC Now pilot and you can read more about the feature in Eleni’s recent blog post.
Our challenge with BBC Now was to capture all the content being published both on bbc.co.uk and on other platforms and to bring it all together into a single unified stream of real-time push updates that are instantly delivered live to the BBC Homepage.
Broadly speaking this broke down into four technical challenges:
Each type of datasource (RSS, Twitter, BBC iPlayer content etc) is monitored by a separate Content Processor. Each Content Processor identifies new content, transforms it into a standard structure, associates it to a BBC brand and publishes it to the Firehose.
The Firehose is an unadulterated, unified stream of all new content from every Content Processor. The Firehose allows for multiple subscribers who can use or transform the data and produce their own streams of content.
The primary subscriber to the Firehose is the BBC Homepage which surfaces each item of content as an update in the Fast Lane. The Fast Lane is a column on the Homepage that receives push updates, from the Firehose and displays them in chronological order.
The Trend Analysis module also subscribes to the Firehose and receives the same set of content. The Trend Analysis module extracts common terms from the published content and aggregates that content using those terms, creating categorised ‘buckets’ of content.
When the buckets are sorted the Trend Analysis Module produces a list of the most common terms appearing in content over a given period and facilitates content filtering.Flow diagram of BBC Now
BBC Now is built out of flat, independent modules making it easy to maintain and scale out. The thin layer of glue that holds them together is Redis. Redis is an advanced key-value store that, like Node, uses an event loop to drive execution and is both fast and scalable.
Redis also offers a fast and scalable implementation of the Publish/Subscribe (or pub/sub) messaging pattern. A pub/sub implementation allows publishers (in this case our Content Processors) to publish data without needing to know how it is delivered to other parts of the application.
Conversely the subscribers (in this case the BBC Homepage & Trend Analysis module) can subscribe to topics of data without needing to know which other part of the application produced it. They simply react to new data as it’s published.
In essence, BBC Now is all about subscribing to data and publishing it elsewhere making Redis a perfect partner.
In designing the real-time push updates to the BBC homepage we had a few options.
This most traditional method of polling, whereby the client (your browser) periodically asks to the server (the BBC homepage) for updates, seemed like a poor choice. Most of the work we had put in was to convert passive data sources into active ones, returning them to a passive data source seemed to be heading in the wrong direction.
WebSockets, a relatively new technology that allows a constant two-way conversation between client and server (think of an instant message conversation), seemed like an obvious choice. But would have perhaps been a bit overkill given we only needed one-way communication.
Besides further feature development (more data sources, personalisation, mobile) we’d love to dovetail our work with that of the BBC Linked Data Platform team which would provide a wealth of contextualised data that would really push us onto the next level.
I’d love to chat more about BBC Now – please leave a comment below or you can catch me on Twitter @dwynne or via the #bbcnow hashtag.
David Wynne is a technical architect and founder of Red Badger.
Join the discussion...