No Deal for Self-Selecting Surveys

Some comments were published on hyper-partisan websites and social media concerning a self-selecting survey about leaving the European Union without a withdrawal agreement.

This article covers the problem of self-selection bias, and looks at polling about leaving the European Union without a withdrawal agreement.

In short

Self-selecting surveys are not reliable: If the propensity to entirely choose to answer a survey is related to the matter of study, self-selection bias arises.

Implied second ranking: In a hypothetical three-way referendum, No Deal could faces a close contest for second with Deal. Asking directly with a specified choice, it is more popular than the government’s deal, but less popular than remaining in the EU.

Ordered preferences: Ordered preferences suggest No Deal is an unpopular second option amongst those who would firstly prefer Remain.

Self-selection bias

A self-selecting survey (or open access survey) is one where participation in the survey itself is entirely down to the person involved. Common examples of self-selecting surveys are phone-in newspaper surveys, or clickable surveys on social media or websites.

People can choose to participate in the survey, rather than being invited by a market or social research organisation.

Self-selecting surveys typically collect no additional information about respondents, meaning no weighting procedures are possible.

There are also no controls against a person answering multiple times, or being gamed by automated accounts. Self-selecting surveys were labelled ‘voodoo polls’ by MORI founder Sir Robert Worcester.

There is also ‘self-selection bias’. This is the name given to the arising bias when the likelihood of participating in the survey is related to giving a particular response. User-generated questions may also have leading wordings.

No inferences about the overall population can be made from these surveys.

The ‘poll’

Martin Lewis, founder of MoneySavingExpert.com, hosted a Facebook survey, asking if they favoured a second referendum or leaving the EU without a withdrawal agreement (though the descriptions are insufficient.)

This self-selecting survey was seized upon by Westmonster, a hyper-partisan website:

An unscientific but extremely large online poll has shown 61% of people would back a No Deal Brexit over a second referendum.

If the survey is “unscientific”, then the sample size is irrelevant.

The ‘BNN News’ website also wrote:

What I believe makes this poll different is the fact that the majority of the people that voted in the poll are unlikely to be a part of the Westminster establishment.

Alternately, this ‘poll’ is self-selecting. Scientific polls get their samples through either random digit dialling of phones, a random selection of participants in an internet panel, or random people in random households.

Twitter accounts makes similar errors:

What actual polling says

The Westmonster article does cite two polls by British Polling Council members. Let us consider the available evidence of how popular ‘No Deal’ is, against the other options of leaving under the government’s agreement (Deal) or remaining in the European Union (Remain).

There are three ways to measure this:

  1. Out of all three options, ask which one they most prefer;
  2. Ask people to order their preferences, and then compare two options;
  3. Ask people in separate questions which option they prefer from a pair.

The first way has been asked six times by BPC members in November and December 2018. The ‘Don’t Know’ responses have been excluded:

In the three-option referendum, leaving without a deal was the third most popular option in one poll, and is always substantially less popular than remaining in the European Union.

Ordering preferences

The second method has been undertaken by two companies: Deltapoll and YouGov. They asked people to order their preferences, and then put two options against one another. They show similar outcomes: Remain against No Deal is near-evenly split, but Deal is substantially more popular than Remain.

In both surveys, Deal wins the pairwise contests (barely in YouGov’s case), through being a popular second preference. Respecting a political scientist, Deal is called the Condorcet winner.

In YouGov’s ordering survey, No Deal is the second preference for 30% of respondents, and third preference for 43%. In the Deltapoll sample, those figures are 37% and 35% respectively.

Deal or No Deal

The last method involves asking people which option of two they prefer.

In direct questioning, leaving the EU without a deal is the preferred option against the government’s deal in two polls (asked by Survation):

There are two different wordings for a hypothetical Remain-No Deal referendum, by two companies. However, YouGov and Survation both show remaining in the EU having a large lead over No Deal.

Lastly, YouGov asked a similar question to Martin Lewis, on 15th November 2018. Holding another membership referendum was preferred by 54%:

Self-selecting surveys are not a reliable measure of public opinion.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.