Posted by Bruce Weir on , last updated

With developments such as Bamzooki, the Piero Sports Graphics system and the BBC Election Swingometer, BBC R&D has a long history of developing new ways of mixing video with graphics in order to offer new ways of presenting interesting content to the viewer. An earlier version of Venue Explorer, our experimental web application, was trialled at the Glasgow Commonwealth Games and we are now trying it out with a series of concert sessions and a special episode of Radio 2's Friday Night is Music Night.

Radio 2 - Friday Night is Music Night Remixed Videos

BBC Philharmonic - Red Brick Sessions Video Streams

Traditionally, graphics have been added to the video by the broadcaster and then sent directly to the viewer. However, in a world of object-based media, there is another way - send the video, graphics data (and audio) as separate streams and allow the viewer to reconstruct a programme by mixing the different elements as they choose. This gives the user greater control over how they enjoy the content. Venue Explorer gives a programme maker the tools to make this happen.

We first trialled the concept at the Glasgow Commonwealth Games in 2014, and we have since been working with the BBC Philharmonic and BBC Concert orchestras to develop the technology further. As well as offering extra information to the viewer in the form of graphical overlays describing the performance, both of these recordings also include audio ‘zooming’. As you zoom your view into the video you will notice the audio mix changing, highlighting the playing of the section of the orchestra which you are viewing.

Introducing The Red Brick Sessions

Technical Details

The browser application is written using Javascript, HTML5 & CSS. Graphical overlays are animated (using GreenSock JS) according to a timeline which can either be created using our own editing tool, or generated live via a remote control interface (if we are producing a live programme). Video and audio streams are delivered using MPEG-DASH and real-time updates to the overlays can be delivered via a WebSocket connection to a Venue Explorer server instance. The encoding and delivery chain imposes some latency to the system, and so, if there is a live audience watching the performance in the venue who would also like to view the live graphical overlays, we have also created a web page for smartphones which displays the overlays as they are triggered. For the Philharmonic recordings, this was used to present live programme notes to the audience.

The server uses NodeJS and Express to deliver the web application, and SocketIO to act as the real-time messaging interface between the editing tool, control interfaces, server and client browsers. We currently deploy instances of the service using Amazon EC2.

This technology stack provides us with a simple deployment route, as well as the ability to handle both recorded and on-demand video services. The real-time comms channel via SocketIO/WebSocket allows live control over the service - triggering, adding or editing the graphical overlays during a streamed performance for example, or displaying live data from an external source, such as a musical score follower.

BBC Concert Orchestra playing for Friday Night is Music Night

Audio Zooming

To produce the audio remixing effect, we capture multiple audio channels from the BBC outside broadcast truck.  Rather than streaming the stereo broadcast mix to the browser we split the audio into its component parts which when combined (or summed) produce the stereo mix. This means that at any time we can be streaming up to 24 channels of audio to the browser. The streams are generally separated by instrument group, for example, Piano, First Violins, Harp, Woodwind...

When you zoom in or pan the video we track the zoom-level and pan-position in the video scene, and modify the loudness and spatial position of the audio sources or objects in response. Static sources such as the overall sound of the venue are louder when zoomed out and quieter when zoomed in. Conversely, dynamic sources such as individual instrument groups are louder when zoomed in and quieter when zoomed out; and louder when closer to the pan-position of the user. The level of the dynamic sources in the mix is adjusted to reflect the distance of the sound source from your pan position in the scene.

Audio processing is all carried out in the browser using the Web Audio API.

Try for Yourself

Friday Night is Music Night Remixed
Recently, Venue Explorer has been used to create a demo with the BBC Concert Orchestra, called Friday Night is Music Night Remixed. Three pieces of music were recorded for it (Star Wars, Jaws and Jurassic Park) and all are available to view.

BBC Philharmonic - The Red Brick Sessions
We have been working with the BBC Philharmonic to offer enhancements to the Red Brick Sessions concert series. Recordings are available for 30 days on our concert server.

Tweet This - Share on Facebook

BBC Radio 2 - Friday Night Is Music Night (Full Episode)

BBC Music

BBC R&D - How the Commonwealth Games is defining the future of broadcasting

BBC R&D - Augmented Video Player

This post is part of the Immersive and Interactive Content section