Posted by Zillah Watson on , last updated
Fire, plague... and bees! A recap of the 360 Video work we've carried out so far which is helping us to explore a new way of telling stories and make films. By Zillah Watson and Peter Boyd Maclean.
Peter Boyd Maclean 360 filming at Crossrail
360-degree video is being widely accepted as one form of VR experience, when viewed through a VR headset, alongside “true VR” - computed generated (CG) experiences created in game engines. 360 virtual reality films offer enormous new possibilities to both filmmakers and their audiences. But to realise them demands a rethink of everything we know about film - a whole new approach to storytelling, to films’ narrative structure, to filming techniques (including camera position, sound and post-production) as well as to the viewer’s experience. In order to be successful and to exploit the new technology fully, compelling VR films cannot simply be conventional films with an added dimension: they have to use an entirely new grammar of film-making. We have begun to develop this in four films.
When the first 360 VR films showing extreme sports and “experiences” were released in late 2014, we realised it also offered opportunities for new ways to tell factual stories - ones that put the viewer at the heart of the action. Working out how to do this, and creating a new grammar for film-making in 360, has preoccupied us for the last 18 months. Throughout this process we have been guided and assisted by Dr Peter Passmore, senior lecturer in Computer Science at Middlesex University. Our quest began with simple news reports, including “What’s it like in the Jungle” (June 2015) and “Inside London’s China Town” (Nov 2015) and later progressed to more complex documentaries.
We have now created four short films to showcase different aspects of factual storytelling with live action 360 and to inspire future formats and thinking. One of the films in particular, Fire Rescue 360, which uses reconstruction to show what it’s like to be a firefighter, clearly points to the value of 360 films for public awareness campaigns and for training scenarios. Another, The Tragic Story of Betty Corrigall, demonstrates the power of a direct performance to a camera in 360, and the possibilities of visualising poems and music with creative 360 footage.
These films are designed to be viewed through a VR headset with headphones to fully immerse the viewer in the worlds they evoke. For this reason, they are not all publically available yet, but you can watch clips below, but they are being shown at events and festivals (including the Future of Storytelling Festival in NYC and IDFA).
Creating films optimised for tablet/mobile viewing (rather than the more immersive experience provided by VR headsets) demands different production considerations to maintain viewer interest : for example, the pace would be quicker, and because nausea isn’t an issue, they allow for more camera movements.
VR Theater at Festival of Future Storytelling, New York, September 2016
We’ve approached this project as film-makers, and our focus was on story, rather than achieving technical perfection in each shot. The films are not steroscopic (3D) - they were shot on a standard Freedom 360 rig with six GoPros,and post-production budgets were limited, so the stitching is not always perfect.
Here are some the things we have learnt:
- Story is all. We tried to select stories that showed off location-based/spatial storytelling and engendered presence to show off the possibilities of 360. We coined the phrase “spherical thinking” in our earlier blog.
- We still believe that the unique value of 360 films is to take people to places they cannot otherwise go.
- Simple narratives still work best for now. For example, the Resistance of Honey is a profile of a beekeeper with a quirk.
- Because these films are intended to watch through a VR headset we have generally avoided camera movements to prevent nausea for viewers. Any camera movements must be approached with care, generally only moving forward and at a slow and even pace, with a clear, stable horizon in view.
- To “direct attention” in these 360 films, we have generally kept the focal point of shots in front of the viewer, to make watching more comfortable. But there are other ways to direct attention, using movement, lighting or sound cues.
- You can create sequences of 360 shots (without fading to black) if you carefully cut from focal point to focal point. But transitioning between “worlds” needs to be done with care, or you’ll create a jarring experience for the viewer.
- Pacing is very different for 360 immersive films - it needs to be slow. To give people a chance to look around each shot needs to be at least 30 seconds long. You also need to give people time to allow the user to become acquainted with the environment before you begin the story.
- Having a voice-over seems to allow people to look around more than having a presenter/reporter talking to you. (We have some early audience research to back this up - see Maxine Glancy’s blog on 360 viewing scenarios)
- Shot composition - if you can’t get the camera in the right place you are not going to achieve powerful 360. You also need to evaluate how difficult the shot will be to stitch when you are shooting (eg a drone shot over a graveyard in The Tragic Story of Betty Corrigall is a great shot for the story, but on our budget we couldn’t achieve perfect stitching on our budget. So you need to make a judgement call.) Planning and good recces are essential.
- The secret of good 360 often lies in post-production - the time spent “stitching” footage together and grading it. Our workflow was as follows: we used Kolor Autopano to rough stitch and Premiere Pro to edit. Then, once we had a picture lock we went back and fixed the stitching and began object removal (for tripods etc) in Premiere After Effects. Then the footage was graded. Then we worked on sound.
- You don’t have to hide the crew to keep them out of shot as much as we thought. For news films, we left the crew in shot because it felt more transparent to do so. In the Resistance of Honey we hid behind trees, or dressed up as beekeepers. But as we progressed we used object removal to erase the crew and tripod from shots. We used the Mettle SkyBox suite and tried the beta Mocha VR plug-in too on some tricky moving shots which made this easier. How much digital manipulation of footage is done in post-production may give rise to trust issues in some genres.
Sound designer Catherine Robinson hides behind a rock while filming in Orkney
- First person narrative - we haven’t found this to work in the powerful way some people imagine. Sticking a camera on the head of the fireman would not make you feel like the fireman - it could leave you feeling sick and confused. But you can show what it’s like to be with them,as we have done in Fire Rescue 360. We’ve generally placed the camera on a tripod at the centre of the action to achieve this.
- Imaginative use of effects can enable you to achieve things you couldn’t actually film in 360. For example, we created an artistic representation of the inside of a beehive in the Resistance of Honey. In Fire Rescue 360 we reconstructed a house fire with effects. Similar editorial issues arise in relation to journalistic responsibility and reconstruction as those arising in CGI based VR, discussed in a blog by Frontline and Emblematic.
- Sound - this is a whole new creative field for 360 which still hasn’t been explored properly. For most films use stereo mixes. For Betty Corrigall we used spatialized sound, using R&D’s sound production system, previously used for The Turning Forest - a CGI based VR experience with binaural sound.
Breathe the smoke and smell the fear as two London firefighters put their lives on the line to rescue six children from a blazing house fire. Experience what it’s like to be a firefighter in a life and death emergency in this true story with 360 reconstruction.
Filming interview with firefighter of the year, Paul Rich, at Southall Fire Station
The film combines interviews with the firemen, filmed at Southall Fire Station, with a reconstruction of the fire. London Fire Brigade Commissioner, Ron Dobson, was impressed with the results.
"This film is the most realistic representation of being at a real fire that I have seen. We never been able to show people the dangers and difficulties our firefighters are faced with before."
Ron Dobson, London Fire Brigade Commissioner
Our challenge was to see if we could use the qualities of 360 film-making to take the viewer into the shoes of a firefighter. This was a technical challenge as well as an exercise in using empathy to see the world from another’s point of view.
Experimenting with 360 face cameras in Fire Rescue 360
We experimented with putting the camera on the face of the firefighter to get his point of view and make the viewer feel in the moment. But we couldn’t make this work, because sudden movement that you are not in control of as a viewer is confusing at best, and at worst creates motion sickness. However we did use a POV (point of view) shot to show the fireman going up the stairs - we used a short shot and slowed it down to avoid nausea.
With clever use of special effects, we were able to create a strong sense of what it would be like to be alongside the firefighters through reconstructing a fire scene in 360. This film was made with Transmission TX.
Filming with fire special effects team at the Fire Training College in Gloucestershire
The fire reconstruction is created through multi-layer compositing. To create the fire effects we filmed multiple shots, for example on the stairs in a real house - clean, with smoke and with the firefighter. We then re-shot the scene at the Fire Service College in Gloucestershire, where we could add the fire effects safely. We used a fire special effects team with a gas controlled fire which burned on different areas on the stairs so we could composite the scene later. The final film had over 500 layers of effects.
Crossrail 360: The Musical
Travel deep below the streets of London to the Crossrail tunnels - Europe's biggest construction project - for an unforgettable musical tour in 360. From the ghosts of the plague pit to the platforms of future stations, this provides a unique glimpse of the tunnels that will carry 200 million people across London in 2018.
Our challenge here was to demonstrate how 360 could give you a unique tour of places that you wouldn’t otherwise be able to visit. And for us, this meant dressing up in lots of safety gear. It’s a short film, intended also to work for audiences watching it on YouTube with Google Cardboard.
The song about Crossrail was composed by Matthew Robins and brilliantly strings together a series of engineering facts about the construction project. This allowed us to cut a range of shots to the music and create a sense of story, which included shots of the plague pit being excavated. We’d originally attempted to create a tour of the tunnels led by an apprentice engineer, but didn’t feel a straight 360 tour was strong enough to engage viewers.
We deliberately introduced a low camera angle to film the train driving over the camera in an homage to the Lumiere brothers and to create a dramatic effect. Although low angle shots in 360 are generally avoided because it makes the viewer feel like a giant, here it works because of the context. Some viewers have reported that it’s given them the sense of being ‘tied to the tracks’.
We also experimented with using controlled camera movements. If the camera is moving directly forward at an even pace, and the horizon doesn’t change, then sickness can generally be avoided. But individual reactions can vary.
Filming in the tunnels was more straightforward. The camera was on a tripod, which we later removed in post-production.
The Resistance of Honey
Step inside the world of Bioni Samp, an urban beekeeper who makes honey - and music - from his bees. This immersive 360 profile takes you deep inside the beehive.
This film demonstrated how a 360 profile could help you understand someone’s world, and was our technical playground for creating sequences of 360 shots. It has been featured at iDocs, the Future of Storytelling Festival, IDFA and other documentary festivals.
The Tragic Story of Betty Corrigall
One of the saddest sights on the Scottish island of Orkney must be the lonely grave of Betty Corrigall. Gather round the smoky peat fire to hear storyteller Tom Muir recall her tragic fate as virtual reality meets the centuries old tradition of oral storytelling.
Our challenge here was to use binaural sound effectively and to create a more passive viewing experience.
Storyteller Tom Muir creates 360 performance of George Mackay Brown’s story of Betty Corrigall in an Orkney crofter’s cottage
The itself film experiments with two 360 techniques. The first calls upon the Orcadian tradition of oral storytelling in which people gathered around peat fires to tell stories passed down for centuries. The storyteller, Tom Muir introduces the story of Betty Corrigall (by Orkney poet George Mackay Brown) by addressing the viewer directly as you stand beside him in a crofter’s cottage by the fire. It’s an intense experience - a personal performance by a master storyteller.
But as the story begins, the viewer is transported to the landscapes of the island of Orkney, which are used to evoke this emotionally intense tale. The story continues in voiceover and works upon the viewer’s imagination (rather like a radio drama). The film uses compositing and dissolves and fades for artistic effect - for example the waves from the sea are layered over the grass.
Evoking the landscapes of Orkney in 360
We aimed to show how landscape, colour and sound can transport the viewer to the story world. It’s proved a powerful technique, and is one that could be repeated to create visual immersive formats for poetry and music.
Peter Boyd Maclean filming 360 by Betty Corrigall’s grave in Orkney
Sound designer Catherine Robinson has created a unique dynamic binaural soundtrack using BBC Research & Development's software, and will describe this process in more detail elsewhere.
Nearly two years on from creating our first advanced 360 film - Living the Dream (a profile of Martin Mossop, a printer), we have achieved many of the challenges we set out to understand. We’ve created engaging 360 stories with flowing sequences of shots, without fading to black. And we’ve begun to explore the possibilities of spatial audio. But there is still much more work to be done to realise the full possibilities of this new medium. The on-going developments in technology - such as 360 cameras, and production work tools - will not only increase the quality of 360 films, but allow new storytelling techniques. It’s an area that will continue to develop rapidly.
The films we’ve created have been the starting point for several audience research projects - notably by Maxine Glancy (BBC), Peter Passmore (Middlesex University) and Andrew MacQuarrie (UCL), which will soon be published.
Meanwhile, other BBC R&D research (for example by Alia Sheikh, Maxine Glancy and Andrew Brown) continues to explore other long term possibilities of 360 including performance in 360 and 360 for Web VR. This research has continued apace as the wider BBC has also begun to explore the more immediate editorial possibilities of 360 films too - many of which can be seen on the BBC’s innovation website Taster.
Over the next few years we can expect better stereoscopic 360 video, improved spatial sound and more fusion with CGI based techniques allowing a range of fantastic effects. We’ll also see more volumetric video (video with depth) and more interactivity built into 360 experiences. But story will remain at the heart of what makes these experiences truly memorable.
We would like to thank all the people who have help us on our journey to understand storytelling in 360. Especially Peter Passmore from Middlesex University, Andrew McQuarrie from UCL, and Kim Wigger, Chris Pike and Kofi Sekyere from the BBC. And to everyone from BBC R&D, News Labs and Connected Studio who has helped support us. And to Nonny de la Pena for inspiring and advising us never to stop thinking about the story.
More on Virtual Reality and 360 Video: