The results and an extensive analysis of the formal subjective verification tests of the H.265/HEVC video compression standard are published in the IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), January 2016. BBC R&D's video coding research team focused on evaluations of UHD content and definition of analytics as part of standardisation process and presented in this paper. The full text of the paper can be accessed here.
The High Efficiency Video Coding (HEVC) standard has been developed jointly by the two standardisation bodies ITU and ISO (as has been the practice with all major video coding standards in the past 3 decades); in this instance as the Joint Collaborative Team on Video Coding (JCT-VC). The main goal of the process was to provide significantly improved video compression compared with its predecessors; H.264/AVC being the most recent. HEVC standard was ratified in 2013 as H.265 by the ITU-T and as MPEG-H Part 2 by ISO/IEC. The associated formal subjective verification tests leading to the results presented here were carried out on behalf of the JCT-VC in Feb/March 2014.
The purpose of the subjective tests was to verify using human viewers the compression gains of the new video coding standard that had previously been estimated using objective metrics (e.g. Peak Signal to Noise Ratio – PSNR). The PSNR based computations had shown up to about 50% bit rate savings in HEVC compared to the previous standard of H.264/AVC, for similar video quality. The subjective tests used a carefully selected set of coded video sequences at four different picture sizes: UHD (3840x2160 and 4096x2048), 1080p (1920x1080), 720p (1280x720) and 480p (832x480), at frame rates of 30Hz, 50Hz, or 60Hz. The video content was chosen to represent diverse spatial and temporal characteristics, and then coded using HEVC and AVC standards at a wide span of bit rates producing a variety of quality levels. The tests for UHD and 720p picture sizes were conducted at the BBC R&D labs under controlled viewing environments complying with the ITU-R BT.500 and ITU-T P.910 recommendations on visual quality assessment. Degradation Category Rating (DCR) test method was used for this test under the schema of evaluation of the quality (and not of the impairment) with a basic test cell (BTC) show below. The viewers were shown BTCs containing HEVC and AVC coded videos in random order.
The tests confirmed the significant compression efficiency improvements achieved in HEVC, verifying the results previously reported using objective quality metrics (PSNR based methods). In fact the compression gains of HEVC compared to AVC were noted to be significantly higher when the subjective metrics (Mean Opinion Scores – MOS) were considered compared to the same considering objective metrics (PSNR). The overall average bit rate saving achieved by HEVC compared to AVC for the same subjective quality was found to be 59% as supposed to the 44% gain shown with objective quality metrics. It was also noted that the bit rate savings for larger picture sizes were higher than smaller picture sizes, which is a very encouraging sign for future UHD deployments.
Detailed results of this subjective experiment are published in this paper, including an extensive set of statistical analyses to help interpret the observations, including the variations in compression performance in distinct test categories. For example, a student’s t-test was used to show that in 92.5% of the test cases, HEVC achieved a comparable or better subjective quality at half the bit rate of AVC.
Detailed introductions to the video quality evaluation methods and statistical analysis methods are also included in the paper to help in appreciating the full extent of the analysis of results.