Research & Development

Posted by Jon Francombe on , last updated

Have you ever wanted to get a taste of what it might be like to conduct your own Prom? To sit in the middle of the BBC Philharmonic orchestra? Or to listen in really closely to what just one of the musicians is playing? We're using our audio orchestration technology to give listeners that opportunity.

This week, we’ve launched Pick A Part on BBC Taster in collaboration with the BBC Philharmonic. With Pick A Part, you can connect your devices (laptops, phones, and tablets) together and choose which instruments they play. Everything’s synchronised as closely as possible using our audio orchestration software.

The BBC Philharmonic Pick A Part web application

Pick A Part is the latest in a series of orchestrated trial productions that started with audio drama The Vostok-K Incident in 2018 and recently saw the release of an orchestrated episode of 1927's Decameron Nights. You can read more about how we worked with 1927 to make Decameron Nights.

BBC Taster - Try Pick A Part for yourself

Adapting to lockdown

We've been discussing audio orchestration with the BBC Philharmonic for a while now - it's something that we've been interested in the potential of, but it's been hard to actually put together. One of the challenges is that orchestral music is normally recorded with all of the musicians playing together at the same time, whilst audio orchestration required object-based audio - that's having access to all of the individual parts that make up the piece.

That all changed when the country went into lockdown earlier this year, and musicians at the BBC Philharmonic had to adapt to making music from their homes. Suddenly, we had unprecedented access to recordings of individual musicians playing their parts to be edited together later. So we jumped on the opportunity to see if our idea of applying device orchestration to orchestral music would work.

Simon Highfield, a technical producer at the BBC Philharmonic, sent over the recordings from the musicians so we could start experimenting. And we were immediately very excited by how well it worked. Not only was this an immersive listening experience just by connecting one or two phones, but it was also novel to be able to explore a piece of music in such detail.  

Simon Webb, director of the BBC Philharmonic:

There are so many potential applications for Pick-a-Part; this is the BBC at its very best, collaborating, innovating and sharing. Whether in an educational setting exploring the inner workings of a string quartet, exploring different audio mixes of a recording, or playing along with professional musicians in your own home, this is a brilliant piece of kit.

How we made Pick A Part

Pick A Part is the latest production that we've made with our new production tool, Audio Orchestrator. With Audio Orchestrator, you can import your audio files, set up rules that determine how they're assigned to connected devices, and export a prototype application.

Audio Orchestrator

To make Pick A Part, we modified the template application that is produced by Audio Orchestrator to add some new features, including:

  • showing a picture of the selected instrument on the connected devices;
  • automatically assigning an instrument once a device has been connected; and
  • displaying a representation of the instruments assigned to each connected device.

We're using trial productions like this to experiment with new features and see what we might want to feed back into the Audio Orchestrator software.

If you'd like to find out more about Audio Orchestrator, you can request access through BBC MakerBox, read our blog post, or have a look at the full documentation.

A screenshot of the Audio Orchestrator production tool being used to add rules that determine how the audio objects are allocated to connected devices.

The Audio Orchestrator production tool being used to add rules that determine how the audio objects are allocated to connected devices.

Keeping instruments in sync

I mentioned above that one barrier to using audio orchestration with classical music was that we needed access to the individual instruments. The second challenge that we needed to address was that music requires very accurate synchronisation between devices. The synchronisation framework that we use is accurate to within around 10–20 milliseconds, but there's sometimes additional delay added by devices, which can reach up to 200 milliseconds. That's OK for the types of sounds in the drama productions that we'd previously released, but not good enough for music.

To get around this, we developed a manual calibration stage so that listeners can easily correct the time-alignment for their own devices. Danial Haddadi, an industrial placement student with BBC R&D, ran an experiment to determine how well listeners can align two devices, and what method is easiest for doing this. Based on the experimental results, he implemented the calibration tool that we used in Pick A Part. The same speech content is played from the main device and an aux device, and the listener is asked to adjust the time delay on the aux device until the two devices are in sync.

Pick A Part is the first orchestrated music experience that we've released, so we're interested to find out about how well this works and how easy it is to use the calibration tool.

An illustration showing calibration mode being used on a smartphone.

Calibration mode on a connected device - use the slider to match the audio playback on two devices

What's next?

We've got a few more trial productions made with Audio Orchestrator in the pipeline, including a spooky Hallowe'en drama with the BBC Writersroom. Keep an eye on BBC Taster if you'd like to hear more. Hundreds of creators have requested access to Audio Orchestrator, and we're really looking forward to see the range of ideas that they develop. This will help us to evaluate the production tool, decide on new features to implement and assess the potential for audio device orchestration. We're also involved in research collaborations at the universities of Surrey and York looking at technical and creative aspects of device orchestration.

Tweet This - Share on Facebook

BBC MakerBox - Audio Orchestrator

The Telegraph - BBC technology that turns your gadgets into speakers could be used for the Proms, says Radio 3 controller

BBC R&D - Vostok K Incident - Immersive Spatial Sound Using Personal Audio Devices

BBC R&D - Vostok-K Incident: Immersive Audio Drama on Personal Devices

Financial Times - BBC pioneers immersive sound using phone speakers

The Telegraph - BBC develops system to turn household objects into surround sound

What Hi-Fi - BBC 3D audio technology aims to revolutionise home entertainment

Aberdeen Evening Express - Nowhere safe from Daleks thanks to new surround sound technology

BBC R&D - Evaluation of an immersive audio experience

BBC R&D - Exploring audio device orchestration with audio professionals

BBC R&D - Framework for web delivery of immersive audio experiences using device orchestration

S3A Project - The Vostok-K Incident

BBC R&D - The Mermaid's Tears

BBC R&D - ORPHEUS

BBC R&D - Talking with Machines

BBC R&D - Responsive Radio

This post is part of the Immersive and Interactive Content section

Topics