Cycling and Self-Selection Bias

The Mail on Sunday reports on a self-selecting survey.

A Mail on Sunday article starts a headline with:

We will ditch Boris Johnson over bike lanes, say 27 percent of Tory voters

The claim derives from a self-selecting survey, run by FairFuelUK. The questions are likely to be leading for respondents. Recent surveys suggest around 2% of GB adults think transport is an important issue.

Self-selection bias

FairFuelUK is a campaign group, managed by motoring journalist Quentin Wilson.

It is not an independent market research company. Campaign groups running surveys often have two difficulties:

  • Non-representative samples: Market research companies take great care to seek representative samples of the target population. Campaign groups may not take such care.
  • Imbalanced questions: Writing questions is difficult. Research companies seek to measure public opinion, through balanced questions. Campaign groups may write questions which lead respondents to particular answers.

Representative survey samples look like the target population. Market research companies use quota and weights to ensure their samples are representative. There are also extra transparency rules through the British Polling Council.

The FairFuelUK Annual Road User Opinion Survey was an open access survey. The article notes anyone could complete the survey:

The online survey by motoring campaign group FairFuelUK was free to complete and open to all road users.

Self-selecting surveys comprise those animated enough to vote. People choose to click on the survey link and take part. People selecting themselves into a survey causes an error. The error is unknown — and can be very large.

Self-selection bias is the name in survey research for this error. It is also called voluntary response bias.

There are no controls on who can respond. The same person could respond across many accounts. There are no guarantees that responses come from the intended population.

The page says the survey received “over 20,000” responses. The Mail on Sunday article said the survey was of “almost 11,000 drivers and cyclists”. Howard Cox (FairFuelUK) said the campaign removed “mickeymouse email addresses and abusive responses”.

There is no sampling error for self-selecting samples. By their nature, open access surveys do not select from a list. People must choose to take part — they select themselves into the survey sample.

The sample size of a self-selecting survey does not matter. The FairFuelUK Twitter account said this was the “largest ever road user survey”.

This is irrelevant. We cannot make inferences about the views of road users from this survey.

Imbalanced questions

Writing a good questionnaire is a difficult exercise. Researchers need to write questions that are clear, neutral and give accurate measures.

Different wordings and orders can lead to different responses. Question wordings can offer too much or too little information. Questions can also lead respondents to give certain answers.

An example from the Pew Research Center is:

In addition to the previous United Nations scandal involving mismanagement of the Oil-for-Food program, there are now indications of corruption in the handling of peacekeeping activities. Do you think the United States should stop paying dues to the United Nations until its problems of mismanagement are cleaned up?

This is how some questions can lead respondents to certain answers. (Image: YouTube/Pew Research Center)

As Anthony Wells (YouGov) highlights, some questions appear to lead respondents.

Most questions had “Yes” and “No” as the main response options. This can lead to acquiescence bias. Some people would click ‘Yes’ to be agreeable.

A question about increasing fuel duty “to pay for the significant cost of COVID” features:

  • A table showing the UK at the top for ‘average fuel tax take’.
  • A long quote from an economic consultancy on the “10 years freeze”.
  • Three options for ‘No’, but only two for ‘Yes’.
The question has an imbalanced stem and imbalanced response options. (Image: FairFuelUK)

A question on fuel duty for haulage vehicles contains the statement:

The commercial heartbeat of any economy is its network of distribution. An average haulier can spend 50% of its income on fuel costs. the Majority of which is tax. Lowering fuel duty here would help lower inflation and the cost of products we buy.

Good things come to those who click Yes. (Image: FairFuelUK)

These questions do not show balance, and may lead respondents.

More or less?

The Mail on Sunday article writes:

More than one in four people — 27 per cent — who voted Conservative in last year’s General Election say they will not do so next time due to the congestion and disruption caused in towns and cities across the country.

This is very unlikely to be right.

Asking about how a particular issue affects a person’s vote primes that respondent. The response becomes about expressing importance, rather than an accurate reflection of a person’s vote intention. It is also difficult to think in probabilistic terms about your own future actions.

YouGov have a regular tracker survey, which asks 1,600–3,300 GB adults in each wave. The company asks:

Which of the following do you think are the most important issues facing the country at this time? Please tick up to three.

The share of respondents saying transport was a top issue had fluctuated around 2%.

The Mail on Sunday reported on a self-selecting survey. The campaign group appears to have removed a large number of responses.

Question wordings and responses showed imbalances, and may have led respondents. We cannot infer the views of road users from this survey.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store