Are the results of an online survey really trustworthy?

Professor Peter Kinderman

Surveys are a blunt tool for understanding mental health, and the people who take part in experiments on the BBC website are not necessarily representative of the public as a whole. Professor Peter Kinderman explains how we can still get valid results.

Most of the reliable ways of assessing mental health problems are complex and time-consuming

Scientists always like to use accurate, valid and reliable tests. That’s especially true in psychology, where people’s experiences and accounts are very variable indeed. Most of the reliable ways of assessing mental health problems are complex and time-consuming. They are often very specific to particular types of problems and tend to avoid the simple questionnaire-based approaches that are easy to do online.

Another issue is that people may react differently when questions are asked online, instead of face to face, or in another way. That means we can’t compare our findings from this experiment with studies that weren’t an online survey. Everything is different and you can’t compare apples with pears.

Another potential problem with the data from the BBC Stress Test, is how open the experiment was. Everyone with access to the internet and twenty minutes spare time was able to take part. And many did – we had over 30,000 participants from more than 170 different countries. Can we really conclude anything valid and reliable about stress, mental health and well-being as a result of this study, given that we haven’t carefully selected our participants to be representative of the UK population?

Be careful what you claim

All this means that we can’t confidently make and claims about general levels of stress.

However, we can be much more confident about looking at the differences between people who all did the BBC Stress Test. The measures we used are all well-designed assessment tools. They were all adapted from measures that have been widely used in previous research, and data from this study demonstrates their reliability.

Apples and pears

When comparing samples, you need to be sure you are measuring the same thing

So while we can’t say that a score of, say, 12 on our measure of mental health problems is comparable to a score of 12 on another team’s research, we can say that a score of 12 by one person on our survey is genuinely and meaningfully different to a score of 6 by someone else who did the same survey.

Our strategy was to analyse the data in this way. We looked for the relationships between scores, not their absolute levels. For this reason, it was important that, when designing the BBC Stress Test, we chose a research question for which this type of analysis was suitable.

Not just a fishing expedition

We can’t now go back on our word and change the predictions we published in 2005

Most importantly, we weren’t just on a ‘fishing expedition’. We didn’t simply gather as much data as we could in the hope that something interesting would emerge. Instead we were testing a theory. Back in 2005, we published a series of papers which set out – before we did the experiment – what pattern of relationships we expected to find. This is a key point. We can’t now go back on our word and change the predictions we published in 2005. If a different pattern of results had emerged, we would have had to rethink our ideas. Of course, we would have learned from that too.

However, the data we gathered did confirm what we predicted that it might have done. You can read more about what we learned here.

So, yes: people are right to ask questions about any ‘survey’ style results. However, because we are comparing sub-groups of the people we surveyed, rather than making any claims about what the population as a whole is like – we can be much more confident in our results.

The data from the BBC Stress Test was analysed by Professor Peter Kinderman and Eleanor Pontin from the University of Liverpool, Dr Sara Tai from the University of Manchester and Dr Matthias Schwannauer from the University of Edinburgh.

All in the Mind

Claudia Hammond presents the programme exploring the limits and potential of the human mind.

Also on BBC Lab UK

If you're worried...

If you are worried about your mental health, there are plenty of places where you can go for help and support.

  • BBC mental health site

    A good starting point for anyone who wants to know more about mental health

  • www.samaritans.org

    Samaritans provide provides confidential non-judgmental emotional support. Lines are open 24 hours a day

The BBC is not responsible for the content of external websites.