Response Options and Exiting the EU II

Does a recent poll suggest, as the polling company chairman asserts, “an increase in the 2016 Leave margin of victory”?

No. This article examines why.

Change the options, change the question

In a Telegraph Premium article, Andrew Hawkins (ComRes) claims:

As becomes apparent from that first paragraph, the question ComRes asked on 2nd — 14th October 2019 was not a European Union membership referendum redux:

This question provides substantive options, and is about people’s “preferred outcome”. This is different to the standard question about a hypothetical referendum — a binary question of principle.

It cannot be assumed people saying they would prefer one of the two ‘Leave’ options would necessarily choose Leave in a new EU membership referendum.

Changing the options changes the question: leaving the European Union is not a scale. Think: how might the 2016 referendum have gone differently if three options were on the ballot paper?

What responses differ?

In a standard referendum question, we typically see similar rates of persistence: the proportion of Remain voters say they would vote Remain again are usually similar to the proportion of Leave voters backing the same option once more.

For instance, Prof Curtice averaged polls from late April to early June 2019, and found very similar rates of persistence among those recalling voting Remain and Leave in the 2016 referendum:

Image for post
Image for post
Public opinion is still finely divided. (Image: What UK Thinks EU)

These standard referendum polls generally find modest Remain leads, principally driven by the views of those did not cast a vote in 2016. The latest six polls estimate, once undecided people and non-voters are excluded, Remain to be around 52%.

Image for post
Image for post
There has been some fluctuation. (Image: What UK Thinks EU)

In the ComRes sample, what differs is that rate in persistence: 78% of 2016 Remain voters chose the ‘Remain’ option, compared to 87% of 2016 Leave voters choosing either of the two ‘Leave’ options — a difference of nine points.

It is some of those who recall voting Remain in 2016 who are giving divergent answers: preferring to leave the EU with a withdrawal agreement, whilst other surveys suggest they would re-support Remain in a hypothetical referendum.

Image for post
Image for post

In such questions about preferred outcomes, it is typical to see the ‘remaining in the EU’ option receive plural support, but more respondents back a combination of ‘Leave’ choices:

Image for post
Image for post
In the latest reading of this YouGov question, one third of respondents chose having a new referendum and remaining in the EU after all. (Image: What UK Thinks EU)

This is also true in the survey. Survation asked their internet panel on 17th — 18th October about their preferred outcome (of three) and how they would vote in a binary referendum. 49% of Survation’s sample backed Remain in a hypothetical ballot (excluding non-voters), and 45% said remaining in the EU was their preferred outcome.

Does a large sample ensure accuracy?

Another notable aspect of the ComRes survey was its large sample size: 26,000 respondents. Hawkins writes:

The stated ‘margin of error’ relates to an assumption the survey was a simple random sample.

As Pew Research Center notes:

For internet panels, large samples reduce sampling variability: they do not eliminate systematic differences between the opt-in panel and the public.

Image for post
Image for post
This is why polling companies have to ensure diverse participation in their internet panels. (Image: Pew Research Center)

If such a question was not intended “to be a formal modelling of a Referendum re-run”, then commentary should not imply that it is. These initial intentions are not respected by Hawkins later in the same article:

Survey researchers should take great care to explain limitations of their own polls, and use their voice to accurately represent their research.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store