Posted by Zillah Watson on , last updated
In a land that never was and a time that could never be, a boy stared into the eyes of a fantastical creature. Around them, a magical forest, in front of them, a magical journey.
The Turning Forest is a sound-based real-time CGI VR fairytale for people young and old--inviting audiences into a magical space of imagination, where rustling leaves of an autumn forest are also the footsteps of something familiar, yet strange.
The work will premiere in April at the Tribeca Film Festival Storyscapes Exhibition, which focuses on cutting edge artworks that explore new uses of media, highlighting innovation.
The dynamic binaural soundtrack for The Turning Forest showcases the BBC’s vision for new listening experiences over headphones and how this can be used in virtual reality. We’ll write more about the technical aspects of the sound and the development into a virtual reality experience later. But since this project started its life as an audio piece, first here’s the story of how the extraordinary soundtrack came to be made.
The studio recording sessions occurred last year in Salford
The soundtrack for The Turning Forest was commissioned under the EPSRC Programme Grant S3A: Future Spatial Audio for an Immersive Listener Experience at Home (EP/L000539/1) and the BBC as part of the BBC Audio Research Partnership.
Frank Melchior, BBC R&D’s Lead Technologist for audio says:
"Our work is all about enabling the best story telling in the world. Sound provides one of the most powerful and direct routes to our audiences. Our team is laying the technical foundations for amazing sound that will immerse people in new listening experience. Whether it’s for telling our stories using virtual reality headsets, car radios, your mobile phone or your living room – you cannot close your ears."
The starting point for The Turning Forest was a great story. The S3A project presented writer Shelley Silas with the challenge of creating a short drama specifically designed to demonstrate the possibilities of spatial audio.
Shelley says: “I normally start writing with characters and then story and give very little attention to the technical side, as that comes last, and I don't really have much to do with it. The challenge for The Turning Forest was to begin with the specific technical and audio requirements and finding characters and story that would work well organically within those parameters without forcing them to work. Rather than the brief being limiting, I found it extremely liberating and was surprised the whole way through the process. It was a challenge I wish I'd had many years ago. It has given me a new way to write and exciting possibilities.”
The team captured 3D field recordings for the production
Working with BBC R&D and S3A researchers, sound designer Eloise Whitemore then embraced the challenge of creating 3D soundscapes and creatures, and unusually dealing with height and distance for sound.
Eloise says, “Suddenly the way our creature moved became important. The boy running needed to be realistic. The creature needed to leap over our heads. The audience needed to hear the heaviness of the body, the vast size of the creature, the depth of birds in the forest, the different layers of ice and water in our icy landscape. Musical notes needed to escape from between our creature’s teeth and float away. The sound needed to encompass the listener.
The way we recorded was also a different approach. The process was object based, so it was about capturing individual sounds and then placing them in a 3D space until the world contained 100s of sound objects. Overall, it was about storytelling, and creating a believable, magical world.”
Composer Jon Nichols created the music for the piece, components of which will become interactive in the VR version.
The original spatial sound mix was created in BBC R&D’s listening room on a multichannel speaker array. The story takes place around you, immersing you in the sounds of the forest. The soundtrack was created using object-based techniques, which means the content can be replayed on different sound reproduction systems without a complete remix. The VR version to be shown at Tribeca will use dynamic binaural sound, reproduced on headphones.
The BBC R&D listening room where The Turning Forest was first mixed on a 3D loudspeaker rig
BBC R&D’s binaural expert Chris Pike explains:
“Binaural techniques allow us to create a spatial sound experience for headphone listeners. We’ve been working on this topic for several years and have built up a good understanding of the technology and production techniques. Recently we produced special binaural mixes for an evening of spooky Halloween plays on Radio 4.
But with VR systems the spatial sound experience has a new dimension, we can update the binaural sound processing in real-time as the listener changes orientation, just as is done with the video content. This creates a soundscape around that listener that you can freely rotate within and really enhances the feeling of immersion in VR. This is what we call dynamic binaural sound.
With the software tools that we’ve built we’re able to produce a rich 3D soundscape and present it on headphones in dynamic binaural sound. In this case a virtual reality headset provides us with the listener tracking as well as displaying the imagery.”
In many VR projects sound is not the first consideration, and compromises have to be made. The challenge we set Oscar Raby was to start with this unique sound track and to visualise it – creating a magical fairy tale world. You will see and hear the results in April.
More on Virtual Reality and 360 Video:
This post is part of the Immersive and Interactive Content section