Just over a year ago, the Research & Development audio team in collaboration with the S3A Project launched The Vostok-K Incident, a short science-fiction audio drama. The piece works on its own as a stereo production but also lets users connect personal devices as additional loudspeakers. Parts of the sound scene (as well as additional content) are automatically routed to these extra devices. By doing this, we can flexibly create immersive spatial audio experiences without being "locked-in" to particular loudspeaker layouts or technology providers.
We'd been investigating this idea of device orchestration for a while, publishing a conference paper and a journal paper on the user experience in a demo system. We decided to commission and create The Vostok-K Incident to develop the production process and also see how the idea of using all available devices played out in the real world.
Last year, we summarised the production process for The Vostok-K Incident. Since then, we've published three papers describing the production, delivery, and evaluation in more detail. These are now all freely available as BBC R&D White Papers and below is a summary of each paper, as well as giving a brief update on our next steps. Additionally, two more papers on research related to device orchestration (output from a PhD project that we support) have also been made available.
Producing audio drama content for an array of orchestrated personal devices
In October 2018, Jon presented an engineering brief at the 145th Audio Engineering Society Convention in New York. Audio content that can adapt to an unknown number of connected devices spread out in unknown positions in the room is a significant departure from the norm of assuming a standard layout of stereo loudspeakers. Consequently, the production process was involved and time-consuming. In this paper, we outlined the bespoke production environment that was set up at BBC R&D for creating The Vostok-K Incident. We introduced the ruleset that was used to decide which parts of the audio scene should be sent to each connected device, and then reviewed some of the challenges of writing, recording, and producing this kind of content.
Also at the Audio Engineering Society Convention
In addition to the work on device orchestration happening in the audio team, we support a PhD project at the University of Surrey. Craig Cieciura is investigating the development of rules for automatically generating orchestrated mixes, and he presented two engineering briefs at the AES in New York. The first (AESeBrief 456, republished as BBC R&D White Paper 363) describes a survey looking into the types of loudspeaker devices that listeners have at home as well as how they consume media. The results showed that there is significant ownership of wireless and smart loudspeakers, and a low proportion of listeners have surround sound systems. The second (AESeBrief 463, republished as BBC R&D White Paper 364) describes a library of audio and audio-visual object-based content created for use in experiments investigating device orchestration.
Plenty more work by the R&D audio team was represented at the convention. Lauren Ward presented an engineering brief about using object-based audio for delivering accessible mixes. Her narrative importance control works by allowing the producer to tag different sounds with their importance level, then adjusting the object volumes according to whether the listener wants an immersive experience or a clear narrative. This work has since been implemented in a trial with BBC One's hospital drama series, Casualty. Additionally, Jon chaired a panel of industry and academic experts discussing best practice for recruiting and training participants for listening experiments and was also on a panel talking about S3A Project work on evaluating the quality of audio source separation.
Framework for web delivery of immersive experiences using device orchestration
Once The Vostok-K Incident had been produced, we needed a way to deliver it over the internet. Kristian's paper at TVX 2019 (closer to home in the building next door to our Salford office) summarised the web delivery framework that he built. The short paper was written to accompany a demonstration and starts by placing our work in the context of previous orchestrated experiences. It goes on to introduce the framework, including audio processing, delivery, and routing; synchronisation of devices (using a cloud-based service developed in the 2-IMMERSE project); and user interface considerations.
Evaluation of an immersive audio experience using questionnaire and interaction data
Finally, after delivering The Vostok-K Incident on BBC Taster, we wanted to evaluate it. One way of doing this is to look at how people interacted with the trial. This was the subject of a paper that Jon presented in September 2019 at the International Congress on Acoustics in Aachen, Germany. We used interaction logs to look at the time that listeners spent with The Vostok-K Incident, the number of extra devices they connected, and how they used the different options on the interface. We also analysed the results of a short questionnaire that listeners could complete after the experience. The results suggested that there is value in this approach to delivering audio drama: 79% of respondents loved or liked using phones as speakers, and 85% would use the technology again. The interaction results gave helpful pointers for future experiences, suggesting that we should aim to get the greatest possible benefit out of few connected devices, ensure that content is impactful right from the start, and explore different types of user interaction.
The paper also discussed the pros and cons of using a large scale public trial for evaluation, concluding that it is a useful technique that should be performed alongside more carefully controlled user testing. We have subsequently performed such a user test, conducting a small number of telephone interviews with listeners. The early results from these look useful and will be published in due course.
What's next for our device orchestration research?
After evaluating The Vostok-K Incident, we designed and delivered several workshops aimed to better understand audience and producer requirements for the device orchestration technology. These workshops are ongoing, and we're aiming to write up our findings towards the end of 2019. Alongside this, we've been working on a software tool that makes it much easier to create prototype orchestrated experiences. Watch this space for more information! We're also involved in longer-term research projects through industrial supervision of PhD students. Craig's project detailed above is ongoing, and we're shortly embarking on another PhD project at the University of York (as part of the XR Stories project), looking into the creative affordances of device orchestration.
This post is part of the Immersive and Interactive Content section