As a public service broadcaster the BBC strives to continually improve the quality of the services it provides to its audience. One of the keys to making improvements to any service is the ability to measure these improvements. This paper will take an overview of the roles of three different approaches to measurement; surveys, user research and monitoring. This paper discusses work at BBC R&D in this context and how it may shape future improvements to subtitling

Surveys: the paper looks at previous surveys that have been published on subtitle quality and looks at how we can take this approach forward using the BBC’s audience research resources.

User Research: the paper reviews previous published user research on various aspects of subtitles which have an impact on quality, both qualitative and quantitative. It contrasts the use of objective measurement of subtitle performance against the subjective quality metric more commonly used to measure broadcasting and telecommunications systems. The paper will show how the relationship between issues such as subtitle delay, errors and perceived quality can be quantified.

Automatic Monitoring: since the advent of digital television broadcasting around 15 years ago BBC R&D has been trying to ensure that all the components of the digital television signal reach the audience. The paper looks at previous monitoring approaches, and the possibility of building new tools which not only check that the subtitles are reaching the audience, but also provide measurements of various aspects of the subtitles related to their perceived quality.

This document was originally published at the conference, Subtitling: A Collective Approach at the university of Nottingham.