Self-selecting surveys and GB News

How do you spot a self-selecting survey?

Opinion polling is important for understanding what the public believe. It is also important for readers to recognise quality in survey estimates.

Can you repeat the question? (Image: WebFX)

Voodoo polling in the digital era

Self-selecting surveys allow people to put themselves in the sample. Sir Robert Worcester used the term voodoo polls to describe open access surveys.People could phone or send text messages to news organisations to take part. The digital era heralded two more types of self-selecting survey:

  • Social media surveys: Users generate their own questions and response options. People with accounts on those platforms can take part.
  • Clickable website surveys: Users can click a button to vote in a survey on a website page.

Many problems arise in these kinds of surveys.

There are often few limits on participation. Social media surveys can limit votes to one per account. People can have more than one account, and vote more than once. Clickable website surveys also may limit votes to one per browser cookie. You can vote many times on different browsers and different devices. Also, people outside the intended population could vote.

People on social media and websites are not representative of the whole population. Social media users in Britain tend to be younger, with higher educational attainment.

The nature of voluntary responses causes unknown errors. The statistical problem is self-selection bias. People who choose to take part are more likely to feel animated about the question. Extreme responses get over-represented.

How to spot a self-selecting survey

Alastair Stewart (a presenter for GB News) brings attention to a Daily Express article:

It is not a representative poll: it is a self-selecting survey. How can we tell?

Here are some clues to the nature of the survey:

The article refers to a website, rather than a market research company.

Ahead of the launch of the new channel on June 13, ran a poll asking readers if they will switch to GB News show ‘The Great British Breakfast’.

The article includes no information about the survey participants.

For a representative poll, we would expect to see — at least — the number of respondents and the target population. There is no such information in this article.

Sometimes, journalists reporting on these surveys may emphasise the number of respondents. Self-selecting surveys do not use random selection. A larger sample does not lead to more precise estimates.

The survey has a very high share choosing one of the responses.

The article suggests:

In a resounding result, 93 percent voted they will switch to the new breakfast show on the channel.

Self-selecting surveys are not reliable measures of public opinion. We can make no inferences about the population from these surveys.

The Market Research Society and IMPRESS guide to reporting on polls states:

A questionnaire/voodoo poll/ straw poll/online vote/text vote all try to obtain a view by contacting as many people as possible to answer questions. The sample will always be self-selecting, and the numbers will have no statistical significance.

News reports should not include results from self-selecting surveys.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store