The Hitchhiker's Guide to Encoding: Mostly Testing (Or how to set up an encoder test)
WednesdaySo what testing do we do and how do we do it?
Test results can only be seen to be accurate if as many variables as possible have been removed or at least minimised, and the methodology produces repeatable results.
Our picture quality assessment is based on a combination of Peak Signal to Noise Ratio (PSNR) measurement and expert viewing. Obviously, if done correctly PSNR measurements are accurate and repeatable within a very small tolerance range but the results are in the end only numbers. Eyeballs on the other hand, even expert ones, are slightly less predictable but can still produce repeatable results within an agreed tolerance but eyeballs are essential to judging overall picture quality of any device.
Ideally the only variable in a test should be the device actually being tested. In a delivery chain with domestic devices from many manufacturers this is not as easy as it sounds and testing every combination of is just not possible.
To minimise the number of variables we use:
Test material - EBU test sequences used by all members for subjective picture quality testing.
Test Path - this is a duplicate of the actual transmission chain and includes the playout server and the continuity path. The playout server can also be bypassed to simulate live studio programmes.
Encoder - the bit we are testing!
Receivers - this is slightly more difficult because processing in set top boxes varies between different manufacturers and different models from the same manufacturer. For encoder comparison testing though we can use broadcast quality receivers. We also use domestic set top boxes from several different manufacturers to make sure the domestic receiver technology does not cause unexpected results.
Displays - this is even more difficult than dealing with receivers. We should use a graded broadcast monitor (a display where all the parameters are know and are adjustable to give a stable accurate image). There is still some debate around flat screen Grade 1 displays - until recently CRT was the only option and they're not easy to get hold of now!
In practice we have to think about what people are actually watching at home so we use 42" Plasma displays for comparisons and Plasma and LCD for quality assessment.
A 42" display is slightly bigger than the current highest selling TV size so this gives us a bit of head room during the assessments. The results will remain valid until the TV size passes 42" or the panel display technology changes dramatically (e.g. OLED or Laser etc.) and a new one takes over as the primary panel technology.
Viewing distance - The ITU has set out criteria for viewing distance in the document ITU-R BT.500 "Methodology for the subjective assessment of the quality of television pictures"
In the section "General viewing conditions for subjective assessments in home environment" it suggests preferred viewing distance (PVD) measured in screen heights (h) for assessment of picture quality.
The PVD suggested for 16:9 screens are:
The BBC HD Channel expert viewing is done at 4h (four times the height of the display used) so slightly closer than the ITU recommendations. This is based on the premise that the average domestic viewing distance is somewhere between 4 and 6h. Some interesting work done by BBC Research (WHP090) suggests the most common viewing distance is actually 2.7m no matter what the screen size. This is not a "serious" piece of work, more an observation of current viewing habits!
How to calculate your viewing distance?
There are two ways to calculate the height of a 16:9 television:
1. Televisions size always refers to the diagonal so first calculate the angle of the diagonal (Θ°):
Θ°= 29.36˚ All 16:9 televisions should have the same angle no matter what the screen size.
Using the angle calculate the height (h) of the television knowing the screen size:
so (h) is near enough half the (Screen Size). Multiply this by 4 to get the viewing distance.
Unfortunately this assumes all televisions are actually 16:9 which they are not - so just in case method 2 might be more accurate!
2. Use a tape measure
Or just use the table below!
When the 4 - 6h criterion was originally devised, screen sizes around 42" would have been considered extremely unusual. Now 42" screens are common and could soon be the highest selling set size. Due to the distance between the television and the back wall of the room you watch it in; WHP090 may prove to be spot on!
In the quality viewing area at BBC Research there is a mix of set top boxes and display types. Viewing distance is set at 4h.
Eye sight - before the BBC HD test channel started, a few of us spent several days in a very hot room above a shop by Oxford Circus doing viewer testing of HD and SD. We had two top end (at that time) 40" flat screens and a 28" CRT, the highest selling TV size at the time. We also had a selection of BBC HD test material as this was before the EBU had agreed test sequences.
The pictures were coded in HD using the encoder profile we proposed to go on air with and also in SD using the BBC 1 encoder and bit rate. The HD signal was fed to one LCD and the SD signal to the other LCD and the CRT.
After the first day we had a number of results that suggested no difference between the SD and HD LCD - quite confusing because even viewing at a considerable distance there was a marked difference between the two LCD images.
The next day we took a standard eye test chart and asked anyone who couldn't see a difference between the two LCD screens if they would mind taking an eye test. Everyone who took the test should not have been driving without glasses and many of them didn't event realise they should see an optician!
Anyway the expert viewers do have eyesight within the normal sight range (with glasses if they usually use them for watching TV).
This sets out the criteria for visual picture quality assessment but as I said earlier, testing uses a combination of visual assessment and a Peak Signal to Noise Ratio (PSNR) measurement.
Tomorrow will be all about the picture quality tests and the PSNR test results.
Andy Quested is Principal Technologist, HD, BBC Future Media and Technology.
- Read part 1 of Andy Quested's HD guide: The Hitchhiker's Guide to Encoding: Before we start
- Read part 2: The Hitchhiker's Guide to Encoding: Life, Encoders and Everything (Or a brief history of HD encoding)
- Part 3: The Hitchhiker's Guide to Encoding: So Many Tests, and Thanks for All the Recommendations (Or the BBC and the EBU)