Testing mobiles: Mobile Compatibility Program
Hello, I'm Paul Rutter, Test Manager for POD Test in Mobile Platforms, Programmes and On-Demand, BBC Future Media.
In my last blog post, "Testing BBC iPlayer for Android Downloads" I mentioned how our Android products are installable on over 3000 different devices. In addition to this vast range of devices, there’s also different firmware versions and operator customisations to consider. For iOS we too have hardware and firmware fragmentation to consider. That’s for native applications. For responsive web products the hardware/firmware considerations are even greater.
More and more people are using mobiles and the BBC has many products that audiences enjoy on an ever-increasing range of mobile handsets and tablets. We need to better understand, internally, what are our supported devices across our range of products and also to better inform our audience on what devices they can expect to get the best experience with on our products.
In addition, we’ve always worked with manufacturers to evaluate new devices that are yet to hit the market to make sure that they are technically capable of supporting our products and that the experience of our products on the device will be acceptable. We want to make this a more robust and formalised process.
To address these issues, we’ve rolled out a Device Compatibility Testing program in my team. We are calling it the BBC Mobile Compatibility Program.
What are we trying to achieve?
The program is expected to provide the following:
• A better understanding of what our supported Mobile devices are across BBC mobile native applications and responsive web products
• Evaluation of pre-production devices for compatibility in meeting technical requirements for BBC mobile products
• Evaluation of pre-production operating system updates
• Advice on classification of devices for product specific features
• Maintained results, fedback to manufacturers and the audience
• Initial focus is on Android, in particular BBC iPlayer, BBC Media Player, BBC iPlayer Radio and BBC Sport
We hope the program will cover all BBC mobile products in the future.
How devices are chosen:
In order to ensure the cost effectiveness of committing to support a device, we have set fair criteria each must meet to determine whether they should enter the BBC Mobile Compatibility Program.
Further information on how devices are selected for BBC iPlayer features (including the downloads service) is available on the BBC iPlayer help pages.
Once we have run the MCP on all devices that meet the selection criteria, we will continue to work our way through other devices in order of popularity in the UK and BBC usage.
How is the Testing performed?
We are trying to run as much of the testing outside of our products. We have developed an Audio-Video (AV) Test Harness application that allows us to select and play representative video content, at different bitrates, on the device to perform our video playback evaluation.
We are working to integrate our DRM solution and add samples of our audio streams into the AV Test Harness. This will allow us to evaluate DRM video downloads support and audio playback evaluation outside of the respective products (BBC iPlayer and BBC iPlayer Radio).
Longer term we are hoping to offer access of the AV Test Harness to manufacturers so they can perform some level of compatibility testing themselves before submitting a device to the BBC.
In addition to the AV Test Harness evaluation, we also perform smoke testing of core functionality of the BBC iPlayer, BBC iPlayer Radio, BBC Media Player and BBC Sport products.
As mentioned above, longer term we hope to extend this to the full range of BBC native applications as well as responsive web products.
Once testing is complete we update our results matrix and create a short report per device. This can then provide any or all of the following:
• Advice to Product Owners around device compatibility
• Recommendations to enable/disable particular product features
• Product defects to be reviewed by a product owner and prioritised within a development team
• Defects relating to the device and/or operating system itself which are fed back to the manufacturer
I hope this insight has been interesting and if you have any questions please feel free to comment below.
Paul Rutter is Test Manager, POD Test in Mobile Platforms, Programmes and On-Demand, BBC Future Media