Posted by Jon Francombe on , last updated
Regular readers of this blog will know that the Audio Team have been experimenting for a while with device orchestration - the concept of using any available devices to reproduce or augment a media experience.
Using our Audio Orchestrator production tool and supporting software framework, we've released several trial productions. These have mainly been audio dramas and a music trial, but our research suggests that there are many more use cases for this technology - and sport is one that we've been keen to try for a while. (In fact, BBC Research & Development has done previous work in this area: in the 2-IMMERSE project, developing a prototype MotoGP experience for multiple screens was developed.)
So we were happy to help out when Cathy Robinson and the Digital Marketing team at BBC Wales got in touch and suggested making an immersive version of a Six Nations match. This felt particularly fitting as fans are not allowed into the stadiums for the tournament - so anything we can do to try to give people some of the sense of immersion of being part of a crowd and right next to the action is a step in the right direction. Also, as an orchestrated audio experience with accompanying video, this was a nice use case for driving our software framework's development.
We decided to keep things relatively simple for our first attempt at orchestrated audio for sport. We designed an experience based on the Scrum V highlights package of the Scotland v Wales Six Nations match on Saturday 13th February 2021 (no spoilers, but it was a close match with loads of action).
On the main device, the listener can watch the video and hear the crowd noise and commentary (which can be toggled on and off). When extra devices are connected, the listener can hear additional crowd noise to give an immersive experience and can also choose to add sound effects track consisting of the referee's microphone and pitch sounds (e.g. ball kicks and players shouting).
Screenshots of the experience showing the connection screen on a laptop with two phones connected, playing additional crowd and referee noise.
Our previous audio orchestrator trials:
There were a few technical challenges in pulling this project together. First, we needed to create the content. Being able to send different parts of the audio to different connected devices relies on having access to the separate audio tracks, not just the complete mix. We got hold of the highlights video with the commentary separated from the crowd noise but only had access to the referee and sound effects as a separate recording from the whole match - so it was necessary to find and extract the highlight clips. This was achieved with a process that was part-manual, part-automated. The approximate time of the highlight in the full match was found by hand, then a signal processing technique called cross-correlation (which works by determining the similarity between two audio signals) was used to find the exact clip.
Digital audio workstation session with audio tracks and start points for each highlight clip.
With the content in place, we needed to expand our orchestration framework to replay video. To do this, we modified the BBC Standard Media Player plugin that was used to drive the synchronisation in BBC Together (our experimental tool for remote shared viewing) so that we could hook it up to the audio device orchestration template. We also made some changes to the user experience, for example, providing a prompt to let viewers know that they can connect extra devices, a button to show the connection instructions, and a switch for turning the commentary on and off.
Standard media player with synchronisation plugin and user interface components for connecting extra devices and switching audio tracks on and off.
We're interested to see how this trial of orchestrated audio for sports content is received. If you get a chance to try it out, please take a minute to complete the short BBC Taster survey to let us know about your experience.
We're only just scratching the surface of what we could possibly do with sports audio and device orchestration. We're using quite a simple sound scene for this project - crowd noise, commentary, and sound effects/referee microphone. But with more access to different microphone feeds, there would be increased possibilities for creating really immersive experiences. Of course, this will be easier to do once stadiums are full of fans again.
It would also be great to try out different sports. How about Formula 1, where you could listen to a team's radio communications on your mobile? Or a tennis match where you can really hear the ball as it bounces on each side of the court.
One of the joys of watching sport is that it's happening live, and the outcome is unknown - so trialling this technology with a live match rather than highlights would be another good step. There will be more technical challenges to overcome for synchronised multi-device playback of live content, so testing out the user experience with 'on-demand' playback is a good start.
This post is part of the Immersive and Interactive Content section