R&D Production Lab: Testing QC Live


Tagged with:

My name is Alia Sheikh, I am a technologist at BBC Research & Development (R&D) and have also trained as a director.

BBC R&D often develops new technologies intended for production environments such as new types of camera equipment or studio systems. These need to be tested but it can be expensive and risky to try them out on real life productions.

Instead I run a project called Production Labs which allows R&D to run tests without compromising live productions. 

The QC Live tool detects a variety of errors in audio and video streams

Every Lab is designed specifically for the prototype being tested giving us the flexibility to create exactly the right conditions for the test. Or to put it another way, we can test things until they break, something that would normally be unthinkable on a production.

For example, to test a content capture system intended for use in drama productions I would organise a self-contained mini shoot staffed by professionals - a director, actors, camera operators etc - allowing us to gather honest real-world feedback from the people who would be using the technology day to day.

My colleague Jonathan McKinnell recently blogged about the QC Live tool he developed and I’d like to give you a bit more detail about how it was tested in a Production Lab.

Jonathan McKinnell and Mark Glanville setting up the QC Live Tool at the Production Lab

The QC Live tool sits between the captured footage and the video presented to the director, in essence it ‘previews’ the content to find errors in it. If we were to test the tool in a live production there would be no guarantee that the errors we were interested in would occur, certainly not at the rate we wished to test them, and if they did we’d still be unaware of which errors the tool failed to spot.

Jonathan and his colleague Mark wanted to know:

  • Whether the tool detected the various errors it was trained to notice in audio and video streams.
  • Whether the tool warned of those errors in a timely and helpful fashion.

We decided to carry out the test at Summer Saturnalia, an annual music festival in Skipton which is not usually filmed. The Production Lab guaranteed us an abundance of helpful errors, because we intended to create them ourselves.

Filming a test performance outside our OB vehicle

The first step was to work closely with Jonathan and Mark to make sure I understood which functions of QC Live they wanted tested. The next step was to run pre-tests in our Television Centre test studio (to check our equipment worked with the festival installation) and then organise the equipment and facilities we needed on location.

I worked with the festival organisers to ensure our tests caused zero disruption to the festival and arranged kit, crew and the necessary health and safety checks.

However, the one part of the organisation I was explicitly not involved with was setting up the schedule of the tests. Jonathan and Mark did this in complete secrecy from me, and for a very good reason. For this Production Lab I would be taking part in the test as the director. I therefore needed to avoid any prior knowledge of the experimental procedure that might have affected my interactions with the QC live tool.


Me in Room one of our OB Vehicle, as the director I was filmed as part of the test

This Production Lab was, to most intents and purposes, an outside broadcast. We arrived with an American Airstream trailer previously kitted out as a radio station. Helpfully for us it had two rooms.

In the first room we set up a gallery with space for a director, a vision mixer and screens showing previews from our cameras as well as screens showing the warnings delivered by the QC Live tool.

There was a camera filming the director and vision mixer so that we could observe their interaction with the tool we were testing. We also recorded the talkback between the director and the camera operators.


Brandon Butterworth filming the main stage of Summer Saturnalia

In the second room, we set up Ingex – BBC R&D's tapeless content capture system – to record the audio and video streams, an additional screen showing a split-view of all the cameras and the QC Live hardware.

We decided to film some performances error-free (at least without any errors that we were introducing). This would allow us to see what errors might occur naturally. All the other performances we filmed included a series of audio and video errors that Jonathan and Mark explicitly introduced.

Jonathan Fletcher in room two operating the Ingex Tapeless Capture System

During each performance the camera operators took instructions from me about shot framing or camera moves but Jonathan and Mark were also giving the crew instructions which I did not know about.

For example

“Go out of focus two minutes into the performance.”
“Deliberately leave your ND filter on for this set.”
“Mess up your white balance settings as instructed now.”
“Pull out the audio cable.”
“Use a broken SDI cable for this performance.”

This created predetermined errors for the QC Live tool to spot and also allowed us to see whether the director (me!) was able to make good use of the warning messages from the QC Live tool.

The schedule of errors

As the director I found more often than not that I was able to make use of the warnings delivered by the QC Live tool and there were even some interesting occurrences of the tool delivering timely warnings for errors that we had not deliberately put into the captured footage.

The three day test highlighted a number of useful insights. Happily the QC tool behaved as expected, spotting all the errors we threw at it and some we didn’t. However we found that the director’s concentration was unhelpfully split between monitoring the tool and doing their job.

Although the tool worked, it could not be of use if the director forgot or didn’t have time to check it. The test highlighted that the QC Live user interface needed improving so that the warnings showed up as an overlay to the preview from each camera, a feature we subsequently added.

Error warnings delivered to the director and vision mixer

An additional benefit of this Production Lab was that we now own a dataset of errored and error-free live music footage with which we can test subsequent iterations of QC Live and indeed any other BBC R&D algorithms.

We are interested to investigate at which point in the production chain the various functions of the QC Live tool could best be put to use. For example, it is clear that some of the error-spotting features it offers might be of use in-camera as an aid for the camera operator.

It would also be useful to run the tool across pre-recorded footage before an edit to highlight potential ‘trouble spots’ which an editor would be wise to reject from a final cut.

Although this work is still at a relatively early stage in its development we found that testing the tool in a ‘live’ environment allowed us to make improvements that were both faster and more responsive to real-world needs than if we had simply run tests in a lab.

If you have any questions about QC Live or Production Labs Jonathan and I would be happy to answer them. Please leave a comment below or on Jonathan’s previous post.

Alia Sheikh is a technologist in BBC Research & Development.

Tagged with:


More Posts