BBC Online - 2019 in review
Lead Technical Architect, D&E
A large number of teams within the BBC work on various aspects of of our online services. Responsibilities span the entire stack, quite literally everything from low-level networking all the way through to audience engagement. In this post, we’ll give you a brief overview of what some of our teams have been working on in 2019. Each contributing team has written an introduction to explain their role and responsibilities, along with some highlights of their work this year. If anything in particular catches your interest, please leave a comment and we’ll get some follow up deep-dive blog posts written to give more detail.
The programmes team is based in Broadcast Centre, White City, London, and manages /programmes, /blogs and /schedules, as well as tooling for ourselves and the editorial staff who publish to these platforms.
- We finished off our migration from the BBC’s on-premises infrastructure to AWS, including a complete rewrite of the codebase. This has resulted in render times being slashed down to 115ms on average.
- To increase the reliability of /programmes we've implemented a stale-while-revalidate cache using Redis that compresses and stores all of our queries for at least 6 hours. This means that our application will be able to survive extended downtime from our data providers without a major interruption in service.
- We moved from a Jenkins CI/CD environment we shared with a couple of other teams to our own Codebuild environment fully under our control in AWS. Build times and our automation tests generally are faster without having to queue to be run.
- We added structured data to enable search engines to understand the complicated nature of /programmes content better, thus helping our end users find the right content faster through their favourite search engine.
Online Technology Group (OTG)
The Online Technology Group is responsible for the provision and management of the BBC’s core online services such as networking, DNS, CDNs, audience-facing TLS, load balancing, caching, routing and high-availability facilities to content origins, our on-premise hosting platforms, and media analytics.
Website traffic management
- Launched our new, in-house website traffic management (TLS termination, layer 7 routing, caching, origin HA) service in two UK PoPs (our largest audience) as part of our worldwide rollout, alongside the existing mainland Europe and Ireland PoPs.
- Gradually enabling TLS1.3 on our website traffic managers, as platform technology allows.
- Built “bq-logs”, a large-scale, low cost, auto-scaling, reliable, extensible system to receive, enrich and import our traffic management access logs (or any structured or semi-structured data) into Google BigQuery. This system requires no day-to-day management, currently processes around 1TB of logs every day and enables access to all BBC online teams to the data it contains, maintaining access control over personal data.
- Built a very simple, fast, low cost, auto-scaling Report URI and Reporting API endpoint which feeds reports into bq-logs (above) for CSP, NEL and similar reports.
- Began deploying a consistent baseline of HTTP security headers, such as CSP, NEL, X-Content-Type-Options, X-Frame-Options, Expect-CT.
- Deployed CAA records on bbc.co.uk, bbci.co.uk and bbc.com.
- Enabled HSTS on the bbc.co.uk and bbc.com apexes.
- Along with our colleagues in R&D, Media Services and iPlayer, successfully delivered the FA Cup Finals and all of Wimbledon 2019 in glorious live streaming UHD, to compatible smart TVs.
- BIDI (our in-house media CDN) reached a milestone of serving 1 Terabit of iPlayer video on-demand traffic per second in 2019. BIDI is now running from 78 servers embedded within 15 data centres, across 4 ISPs.
- New BIDI cache hardware model designed and validated, which provides 119% more throughput, 316% increased programme storage capacity and is 88% more power efficient at peak throughput. All whilst providing an enhanced quality of service to the audience and at a lower cost per unit to the BBC.
- Radix, our in-house media content origin platform has been refreshed. The new platform provides a better quality of service and increased capacity to meet the demands of iPlayer's on-demand catalogue for the next three years.
- Software improvements helped reduce SSD drive write rate on cache servers by 75%, improving latency and increasing drive longevity.
Media Services is the team responsible for audio and video workflows across the BBC Online. We transcode and manage access to media, deliver video to linear playout systems, and develop the media players for Web and native iOS and Android. We've been making some subtle but important changes under the hood. Here are the highlights:
- We improved our Audio Description workflow so that we can deliver audio described programmes to iPlayer before they are broadcast on linear TV. This meant that we could deliver the entire second series of Killing Eve with audio description, so visually impaired viewers could enjoy the whole series before broadcast like anybody else. One of our developers, Marcel, wrote a blog post with all the details here.
- We integrated a new transcoding service from Bitmovin into our workflow that dramatically speeds up the delivery of content to iPlayer. Last year, for example, fans of Strictly Come Dancing had to wait hours after broadcast for the show to appear on some platforms: it took over 2 hours to transcode each episode. This year, we reduced that time to just 10 minutes by splitting the programme into small chunks and processing them in parallel. This is just one of several transcoding services we use to optimise our media transcode function for different scenarios.
- "Content Enrichment" is a term that's being used a lot in planning meetings. If we can better understand our own content, we'll be able to provide better recommendations to the audience. One way we can do this is by creating transcripts of the audio and video we produce. Media Services has now launched a managed transcription service using BBC R&D's tool “BBC Kaldi” that will make it easier for other BBC teams to transcribe their content at a fraction of the cost of other speech-to-text providers on the market. So far we’ve transcribed 75,000 hours of archive audio!
- The Playback team have been busy. The web team rebuilt the ‘continuous play’ experience so that programmes and clips will automatically play in sequence. Alongside that, they commenced a major programme to rebuild the web media player from scratch in order to take advantage of new technologies available in browsers, such as Media Source Extensions, Progressive Web Apps and Service Workers. This also enable us to leave behind older or legacy technologies like Flash, iFrames and suchlike (noting that we will still support older browsers for as long as they are needed even as we move to the new player sometime next year). The new player will also sport the new design which we’re excited to roll out.
- The Mobile playback team have been equally busy; we onboarded several new apps over the last year and have been working with them to enable full use of the players. Again, much work ‘under the hood’ is ongoing to maintain an uninterrupted delivery of BBC content to those apps with a robust and reliable service.
iPlayer Business Layer is “The Beating Heart of iPlayer”. If content is the lifeblood of iPlayer, iBL is the team responsible for controlling its flow through the product. We achieve that by gathering the metadata, curation, business logic and algorithms in one place: we are iPlayer as an API.
Some of the highlights of what we worked on in 2019:
- Curator DSL - In order to help our editorial teams with their work, we have developed a DSL for selecting a subset of episodes on iPlayer, which can be refreshed automatically rather than needing manual updating of groups. This is using existing commissioning information such as genre, but also analyses and labels episodes based on descriptions and other data - see also the content enrichment work Media Services is working on above.
- Personalised Bundles - On our journey for iPlayer to be Total TV, we have introduced more personalised “Bundles” of content on the iPlayer homepage, such as the “If you Liked” bundle, which suggests shows based on your recent watching history and communicates that reason to you.
- Content to Content Recommendations - We have taken some of the concepts BBC R&D have developed for their Recommender System into production and running some of iPlayer’s recommendations using this system. Stay tuned for more in this space in the future!
- Live Events - 2019 saw the return of Glastonbury after a fallow year, and the BBC is committed to show much more live coverage across the board, including sports. This year, we have revamped our live event workflows after some user testing: From dynamically updating the titles and images on the playback screen, and on the homepage, based on what’s actually happening on stage or tennis court right now to grouping content into sensible slices, we’ve made many improvements to iPlayer, and won two UX UK Awards 🏆.
- Full Series - Following recent Ofcom regulation changes we are expending the number of series available on iPlayer significantly, and are improving the ways in which iPlayer handles this. We allow browsing by series now, use better imagery and promotional materials for a whole programme rather than individual episodes, and we keep track of users’ watching progress and current episode.
- Experiments - We have run dozens of API driven experiments in the live product this year, many of which resulted in changes to the audience experience. From re-ordering categories for an U13 (under 13) audience to a more useful way we generate recommendations for new users who don’t have a watching history, we hope that these under-the-hood improvements have made some people’s lives easier.
- Scotland - Scotland got its own TV channel, and obviously this is featured on iPlayer.
- Programme Publication Workflow - With the increasing size of the iPlayer catalogue, we’re undergoing a migration to an event-driven workflow to update available episodes to ensure everybody gets the latest Episode of Ru Paul’s Drag Race UK as quickly as possible.
- Better client workflows - Our GraphQL APIs uses Persisted Queries, which means that if any iPlayer client would like to update what they query from the API, they need to register their new query first, we need to review it, and then add it to our API. This used to be a manual process, which we have now automated, allowing clients to update their queries in minutes rather than hours or days.
- Operational Visibility - All the new services to deliver all the new features above have introduced some changes to our operational workflows. We have worked with the BBC’s operations teams to allow them to handle most actionable events without needing to call out an engineer. Across the whole department and organisation we have also started to share a lot more background on incidents and how we can work together to learn from them.
The iPlayer Web team are responsible for building and maintaining the face of iPlayer on the web, across thousands of different types of device.
- Analytics & Data - With audiences at the heart of everything we do, we need to ensure we can capture data around user behaviour, so that we can fully understand how users use the product. We have therefore increased our focus on understanding user data and insight, with more advanced implementations of analytics.
- Childrens’ experience - iPlayer has lots of programmes but not all are suitable for our younger audiences! We have started creating a safeguarded and child-friendly experience in iPlayer Web, for those users who are signed in with an Under 13’s BBC Account. We have run several experiments on this experience, for example, simplifying the user interface to see if young people find it easier to discover content. There are many more improvements to come, but to see our progress so far please check out how to register for a child account
- End-of-playback experience - We have run over 10 experiments this year, around the end-of-playback experience. We’ve been trialling various changes to the user interface, to assist users in discovering new programmes that they’d like to watch.
- Live Events - As mentioned by iBL, we designed and launched a brand new live events experience on iPlayer, making it easier to discover the various live streams available for users to watch at various live events such as Glastonbury and Wimbledon. We even won 2 UXUK awards for this new experience!
- React Migration - We finished moving all of our custom client-side code to React components - the last things to move were the main iPlayer navigation and footer. This makes for a more consistent codebase, allowing quicker development and experimentation across the product.
- End-to-end test infrastructure - We started making improvements to our end-to-end acceptance tests and infrastructure, with an aim of making tests quicker, more stable and run more frequently. By upgrading our testing framework and running tests in parallel, we saw time to run a full regression of the playback experience more than half, making it even quicker to ship new features to the audience!
Technology Strategy & Architecture
Technology Strategy & Architecture (TS&A) is a collaborative group which brings together expertise from across BBC Design & Engineering. It’s the team that looks ahead and assesses what technology the BBC will need in five years' time, and maps out a plan of how to get there; and ensures that we have the right priorities in place so we can digitally transform the BBC.
- The Blue Room ran a successful conference and fair in October with a focus on responsible use of Artificial Intelligence. 300+ internal and external guests took part and were offered a range of discussions on Machine Learning developments, creativity, regulation and ethics and invited to explore their hands-on technology fair. A summary of the conference, and an exhibition walk-through can be found here.
- BBC Datalab facilitated a pan-BBC initiative to create the BBC’s first Machine Learning Ethics Guidelines - specifically for the BBC Recommender Engine project but with the expectation that they’ll be rolled out further to other machine learning initiatives.
- Our substantial work on developing personalised recommender systems has started to take effect with the first in-house recommenders being deployed. You can read more here.
- Following publication last year of a blogpost on the BBC Capability Framework, developed by TS&A, the EBU asked to share the framework to support the creation of an EBU-owned model which could be adopted by its members. This July, the EBU technical committee formally approved the work to create an EBU media industry capability model. Read our follow-up blogpost here.
- The BBC Machine Learning Fireside Chats are now available as a podcast on Soundcloud. The ML Fireside Chats are friendly bi-monthly events to discuss and share knowledge in the ML space. To find out more, join our MeetUp group. To join the conversation follow us on Twitter: @BBC_MLFireside
RMS is the service that powers the BBC Sounds experience across Mobile and Web. Our API provides a view into the content, and guides the experience that clients receive when using Sounds. Some of the work we’ve done (or are working on) this year includes:
- Local Radio Stations in the Listen Live Rail - To help make Sounds more personalised and relevant to the audience and increase user engagement, we started displaying a Local Radio suggestion to a user. If a user has a postcode set and has enabled personalisation, we will find the nearest Local Radio Station to that user’s postcode and display it at the end of the Listen Live Rail, along with the National Radio Stations.
- Shared, distributed cache to support Personalised Content - We’re investigating caching at different layers, and sharing the cache with other instances of the same service. In order to support more personalised content, we have started a long term investigation into new caching strategies. RMS serves hundreds of thousands of requests per day, and historically, most of those were for static or infrequently changing content that every user could consume. With the move towards more personalised content, we will have to change our caching strategy, as each user will receive a different experience. The variance of pages that could be cached grows massively with each piece of personalised content, and it is not practical to cache this in the way we’ve been doing it before.
Research & Development
R&D activity at the BBC has been going on for almost 100 years. A specific department was established when the BBC Research Department was formed in 1930. Doing research and development activity is Clause 65 of the BBC Charter. In 2011 the BBC Executive Committee’s paper interpreted Clause 65 to mean the following for the corporation’s research and development activity:
- Drive Innovation in support of BBC strategy and public purposes
- Horizon scan, acting as early warning radar in the face of technical change
- Provide an expert team to solve problems or maximise opportunities
- Add tangible value to the wider media industry
- Maintain BBC partnerships, and with them, primary position in the wider ecosystem
Our 2019 highlights include:
- BBC Click used our Object-Based Media tool Storyformer to make their 1000th episode.
- We introduced our personal data store prototype BBC Box designed to give audiences control over their personal information.
- Casualty audiences benefited from our new trial on Taster which made it easier for people with a hearing impairment to enjoy BBC dramas the feature allowed audiences to set the audio mix to suit their needs.
- R&D tested the use of 4G/5G broadcast technology to deliver live radio services to members of the public in Orkney. The island was chosen because it has limited fixed broadband, little or no mobile coverage and poor digital radio coverage.
- Connected Studio, part of R&D launched Makerbox the destination for developers, producers and storytellers to use cutting edge BBC R&D and third party originated tools to help them create new ways of telling stories to audiences.
- We continued our High Dynamic Range trials and increased the jeopardy by tackling the 2019 FA Cup live.
- The BBC Proms was used to test Notes our new web app that sends you extra information about what you’re seeing and hearing to your device, as it happens.
- We launched our Diversity & Inclusion policy on what speaker opportunities we will be involved with to help to improve diversity in the tech industry.
News and World Service (Web)
BBC News and World Service online bring BBC news content to the web both for our domestic UK audience and for our international audiences, around the world.
This year our main focus has been on moving large parts of our website to the cloud and to our new technology stack (Including the ~40 world service language sites). As well as delivering coverage of the December general election, and improving our existing web products.
- Released a new open-source World Service platform that is “performance-first” to ensure that our users around the world are getting a first-class experience that caters for different needs
- Continued to migrate from our legacy monolithic applications to a more microservices-oriented approach (mostly made up of Node.js/React and Ruby microservices on AWS).
- Added support for personalised content on the News front page with top stories tailored to each user's preferred nation.
- Developed a new Elixir-based “preflight” service that is responsible for the resiliency and availability of our web platforms.
- Upgrading our existing Elections infrastructure and creating new components for general election coverage in the UK, India, Nigeria, Thailand, Indonesia, and Sri Lanka amongst others.
- Using AWS Fargate ECS clusters to render images with R and publish directly to Twitter and other platforms.
- Used natural language selection templates to create semi-automated editorial content in English and Welsh for the general election.
- Moving our iframe services to Shadow Dom.
- Working with our data science teams to deliver an API to provide content recommendations based on users recent article reading history.
The BBC Weather website serves an average of 8 million users per week, offering 14-days of hourly forecast information, warnings, interactive maps, and the latest editorial analysis and videos from the BBC Weather Centre.
In 2018, we relaunched a significant portion of the BBC Weather website on the cloud, expanding our use of AWS lambda to transform and render incoming weather data. Several features, however, remained on the BBC’s outgoing on-premise platform. Our primary aim for 2019, therefore, was to complete our migration to the cloud, and in the process deliver a visually consistent and performant experience across the entire site.
- Automated UK Weather Warnings and Flood Warnings pages.
- A refreshed Monthly Outlook.
- Migration of our article, video and photo galleries to the cloud, here’s an example article.
- Incorporating the BBC’s new typeface, Reith, in line with our new TV graphics.
- Showcasing Weather Watchers photos (user-generated content) on our UK forecast pages
- New and improved Tide Tables information
In early 2020, we’ll release a new look for the remaining section of the site: our specialised forecasts covering Coast and Sea.
The Search team runs the back-end services that power most of the search features on BBC Online including the website, mobile, TV and voice applications. We’ve been busy completing our migration from the on-premise “Forge” platform to AWS, enabling improvements to our search features going forward.
- Migrated search engines, services and data feeds to AWS
- Created a content enrichment pipeline and associated editorial tooling on AWS
- Started migration of search results pages to AWS
- Invested in Relevancy Engineering skills
The Navigation service provides the BBC blocks, navigation links and footer on the website, as well as delivering the cookies/GDPR banners, analytics integration, the BBC Reith font, etc.
- Supported migration to the new BBC website analytics system
- Created a new EU cookie banner to allow our audience to opt in or out of the various cookies we use in BBC Online
- Supported insertion of GDPR banner in BBC CoreServices/Webcore products
- Improved click-per-view tracking support, enabling a feedback loop for improvements to Search etc
- Created a replacement geolocation service
Application Security (AppSec)
- Conducted a plethora of security architecture surgeries and threat modelling sessions in collaboration with BBC Online teams as they develop or change their services
- Re-launched the Security Champions forum in January 2019 for our online teams
- Introduced an initial training pathway for BBC Security Champions where new members have to take secure coding training, threat modelling training and data protection training in order to qualify as a formal BBC Security Champion
- Ran a pilot of a computer based training secure coding training tool and migrated to that from the previous classroom based course. This was successful so the platform was opened up to the entire BBC Security Champion community as a part of their training pathway
- Launched a newly developed threat modelling course and have trained 120 BBC Security Champions across the year
- Ran a red team testing event against BBC iPlayer and BBC Sounds in order to identify vulnerabilities in their codebase and to help productionise our red teaming capability
- Hosted an OWASP Manchester vs Secure Code Warrior event at BBC MediaCityUK
- Launched a proof of concept deployment of OWASP DefectDojo, a vulnerability consolidation tool for BBC Online teams and InfoSec
- Launched a proof of concept deployment of OWASP Juice-Shop, fronted by Juicy-CTF, for the BBC Security Champions to test their knowledge and skills in a safe environment.
So there you have it, a snapshot of what some of the teams in BBC Online have been working on in 2019. To the best of my knowledge, this is the first time we’ve published a roundup and whilst not all our teams were able to contribute this year, I hope we’ll continue this each year and improve as we go.
As I mentioned earlier, if there’s something which particularly interests you or a question you’d like to ask, please leave a comment and we’ll do our best to cover them in follow-up blog posts.
Thank you to all the BBC Online teams who contributed. All that’s left to say is that we hope you had a great 2019 and wish you all the best for 2020!