What we've done

The concept prototype we have developed builds on the original work initiated in R&D on analysing audio/video stream to determine the pace and mood of the programme (WHP 232 ). The algorithm has been refined and integrated with a system we created to automatically generate video/audio clips of highlights within a football or tennis match.

The system also integrates subtitles providing contextual information on what type of highlight each clip is - e.g. goal, penalty, break point, etc. The initial design demonstrates how users can view the list of highlights clips, select/reject specific ones allowing for fine-tuning/curation of results and select the desired clip to re-watch a highlight.

Why it matters

Highlights are currently manually identified, selected and extracted by production teams while the programme is being broadcast. In an already fast-paced environment, identifying accurate and relevant highlights requires extra effort and focus during big events, particularly sport, where multiple live streams are broadcast live.

The project builds on research and technology already available and is related to new form of content in relation to being able to extract key moments in a programme and presenting it back to the audience in the most appropriate order and duration to fit their different needs.

There is the potential to create a system to be integrated in the current production pipeline of live events and identify new TV audience-facing user experiences.

Automated extraction of interesting moments in a programme could also have great potential in speeding the process of reviewing archive footage. It could enable archive editors to process hours of footage in a fraction of the time.

Our goals

We aim at evaluating the validity/accuracy of the algorithm by using the prototype to generate automatic highlights during the World Cup and Wimbledon by working in partnership with Multimedia Sport Online production team. The automatically generated results will be verified against the highlights created through manual curation.

How it works

The system is able to identify the general feel of a programme, for example if it is generally funny or quite serious. The extremes in tone and pace of TV programmes are also used to identify what happens within a programme, for example, when a fast action sequences occur, or when a slow serious scene takes place.

he Highlights system is composed of several independent components:

  • a media features analyser which is able to identify the general feel of a programme
  • a realtime file analyser which is in charge of processing media features
  • an HTTP API which responsibilities are to receive, to enrich and to expose Highlights, generate a thumbnail for each Highlight timecode and display the subtitles covering the duration of the Highlight
  • a Web interface to enable users to browse, to visualise and to playback found Highlights

A more detailed description of the end-to-end process, the challenges encountered and solutions adopted can be found in this dedicated blog post.


The MVP at this stage of the research is the deployment of an end-to-end automated system forms the basis/building blocks of future use cases, applications and evolution of the system.

The validity/accuracy of the pace and tone algorithm has been evaluated at key stages of the development carrying out quantitative studies. A quantitative assessment resulted in 81% precision and 81%recall for the algorithm based on the ground truth 2010 England Vs Germany football match which was manually tagged by 4 Assistant Producers (APs).

An ethnographic study has been conducted to learn about TV audiences' understanding of what they identify as key moments within a programme in parallel to continuous improvement of the algorithm, development and integration of the prototype. The results from this study are also a useful source of information in relation to audience preferences on different genres and how their perception of an interesting moment shifts according to the type of programme they watch.

We have also investigated the processes, environment and tools used by the production teams for capturing the highlights during live events by shadowing key members of the team during a simulated live football match.

As part of the software development, we released a couple of JavaScript open source components in the BBC R&D's GitHub organisation. They are all installable through the npm registry:

  • slayer.
    A JavaScript time series spike detection for Node.js; like the Octave findpeaks function. The module returns all the local maxima within a flat array or an array of objects.
  • media-sequence.
    It is an HTML5 media sequenced playback module. It offers the opportunity to play one or multiple sequences of a same audio or video with plain JavaScript. Its architecture enables to play a precise chunk of video, some or all of the defined chunks of an HTMLMediaElement. Consider it as an unofficial JavaScript API for the Media Fragments.


People & Partners

Project Team