Posted by Jon Francombe on , last updated
Have you ever wanted to get a taste of what it might be like to conduct your own Prom? To sit in the middle of the BBC Philharmonic orchestra? Or to listen in really closely to what just one of the musicians is playing? We're using our audio orchestration technology to give listeners that opportunity.
This week, we’ve launched Pick A Part on BBC Taster in collaboration with the BBC Philharmonic. With Pick A Part, you can connect your devices (laptops, phones, and tablets) together and choose which instruments they play. Everything’s synchronised as closely as possible using our audio orchestration software.
The BBC Philharmonic Pick A Part web application
Pick A Part is the latest in a series of orchestrated trial productions that started with audio drama The Vostok-K Incident in 2018 and recently saw the release of an orchestrated episode of 1927's Decameron Nights. You can read more about how we worked with 1927 to make Decameron Nights.
Adapting to lockdown
We've been discussing audio orchestration with the BBC Philharmonic for a while now - it's something that we've been interested in the potential of, but it's been hard to actually put together. One of the challenges is that orchestral music is normally recorded with all of the musicians playing together at the same time, whilst audio orchestration required object-based audio - that's having access to all of the individual parts that make up the piece.
That all changed when the country went into lockdown earlier this year, and musicians at the BBC Philharmonic had to adapt to making music from their homes. Suddenly, we had unprecedented access to recordings of individual musicians playing their parts to be edited together later. So we jumped on the opportunity to see if our idea of applying device orchestration to orchestral music would work.
Simon Highfield, a technical producer at the BBC Philharmonic, sent over the recordings from the musicians so we could start experimenting. And we were immediately very excited by how well it worked. Not only was this an immersive listening experience just by connecting one or two phones, but it was also novel to be able to explore a piece of music in such detail.
Simon Webb, director of the BBC Philharmonic:
There are so many potential applications for Pick-a-Part; this is the BBC at its very best, collaborating, innovating and sharing. Whether in an educational setting exploring the inner workings of a string quartet, exploring different audio mixes of a recording, or playing along with professional musicians in your own home, this is a brilliant piece of kit.
How we made Pick A Part
Pick A Part is the latest production that we've made with our new production tool, Audio Orchestrator. With Audio Orchestrator, you can import your audio files, set up rules that determine how they're assigned to connected devices, and export a prototype application.
To make Pick A Part, we modified the template application that is produced by Audio Orchestrator to add some new features, including:
- showing a picture of the selected instrument on the connected devices;
- automatically assigning an instrument once a device has been connected; and
- displaying a representation of the instruments assigned to each connected device.
We're using trial productions like this to experiment with new features and see what we might want to feed back into the Audio Orchestrator software.
The Audio Orchestrator production tool being used to add rules that determine how the audio objects are allocated to connected devices.
Keeping instruments in sync
I mentioned above that one barrier to using audio orchestration with classical music was that we needed access to the individual instruments. The second challenge that we needed to address was that music requires very accurate synchronisation between devices. The synchronisation framework that we use is accurate to within around 10–20 milliseconds, but there's sometimes additional delay added by devices, which can reach up to 200 milliseconds. That's OK for the types of sounds in the drama productions that we'd previously released, but not good enough for music.
To get around this, we developed a manual calibration stage so that listeners can easily correct the time-alignment for their own devices. Danial Haddadi, an industrial placement student with BBC R&D, ran an experiment to determine how well listeners can align two devices, and what method is easiest for doing this. Based on the experimental results, he implemented the calibration tool that we used in Pick A Part. The same speech content is played from the main device and an aux device, and the listener is asked to adjust the time delay on the aux device until the two devices are in sync.
Pick A Part is the first orchestrated music experience that we've released, so we're interested to find out about how well this works and how easy it is to use the calibration tool.
Calibration mode on a connected device - use the slider to match the audio playback on two devices
We've got a few more trial productions made with Audio Orchestrator in the pipeline, including a spooky Hallowe'en drama with the BBC Writersroom. Keep an eye on BBC Taster if you'd like to hear more. Hundreds of creators have requested access to Audio Orchestrator, and we're really looking forward to see the range of ideas that they develop. This will help us to evaluate the production tool, decide on new features to implement and assess the potential for audio device orchestration. We're also involved in research collaborations at the universities of Surrey and York looking at technical and creative aspects of device orchestration.
This post is part of the Immersive and Interactive Content section