Response Options and Exiting the EU II

Does a recent poll suggest, as the polling company chairman asserts, “an increase in the 2016 Leave margin of victory”?

No. This article examines why.

Change the options, change the question

At first sight, headline public opinion does not seem to have moved much in three years: the poll gives a combined Leave lead of 50% over Remain’s 42%, with a Brexit deal (30%) more popular than leaving without one (20%). Eight percent declined to offer a view.

This was not intended to be a formal modelling of a Referendum re-run but, if we exclude don’t knows, it nonetheless points to an increase in the 2016 Leave margin of victory, with 54% for Leave and 46% for Remain, while the country remains profoundly divided by region, age and political preference (of which more in a moment).

As becomes apparent from that first paragraph, the question ComRes asked on 2nd — 14th October 2019 was not a European Union membership referendum redux:

Q1. Thinking about the UK leaving the EU, which of the following would be your preferred outcome?

The UK remaining in the EU;
The UK leaving the EU with a withdrawal deal;
The UK leaving the EU on a No Deal basis;
Don’t know

This question provides three substantive options, and is about people’s “preferred outcome”. This is different to the standard question about a hypothetical referendum — a binary question of principle.

It cannot be assumed people saying they would prefer one of the two ‘Leave’ options would necessarily choose Leave in a new EU membership referendum.

Changing the options changes the question: leaving the European Union is not a scale. Think: how might the 2016 referendum have gone differently if three options were on the ballot paper?

What responses differ?

For instance, Prof Curtice averaged polls from late April to early June 2019, and found very similar rates of persistence among those recalling voting Remain and Leave in the 2016 referendum:

Public opinion is still finely divided. (Image: What UK Thinks EU)

These standard referendum polls generally find modest Remain leads, principally driven by the views of those did not cast a vote in 2016. The latest six polls estimate, once undecided people and non-voters are excluded, Remain to be around 52%.

There has been some fluctuation. (Image: What UK Thinks EU)

In the ComRes sample, what differs is that rate in persistence: 78% of 2016 Remain voters chose the ‘Remain’ option, compared to 87% of 2016 Leave voters choosing either of the two ‘Leave’ options — a difference of nine points.

It is some of those who recall voting Remain in 2016 who are giving divergent answers: preferring to leave the EU with a withdrawal agreement, whilst other surveys suggest they would re-support Remain in a hypothetical referendum.

In such questions about preferred outcomes, it is typical to see the ‘remaining in the EU’ option receive plural support, but more respondents back a combination of ‘Leave’ choices:

In the latest reading of this YouGov question, one third of respondents chose having a new referendum and remaining in the EU after all. (Image: What UK Thinks EU)

This is also true in the same survey. Survation asked their internet panel on 17th — 18th October about their preferred outcome (of three) and how they would vote in a binary referendum. 49% of Survation’s sample backed Remain in a hypothetical ballot (excluding non-voters), and 45% said remaining in the EU was their preferred outcome.

Does a large sample ensure accuracy?

The ComRes poll was conducted online across 12 days earlier in October with a gargantuan sample size of some 26,000 UK adults — equivalent to 13 of our regular voting intention polls. This makes the margin of error an impressive +/-0.61%.

The stated ‘margin of error’ relates to an assumption the survey was a simple random sample.

As Pew Research Center notes:

Very large sample sizes do not fix the shortcomings of online opt-in samples. While an online opt-in survey with 8,000 interviews may sound more impressive than one with 2,000, this study finds virtually no difference in accuracy.

For internet panels, large samples reduce sampling variability: they do not eliminate systematic differences between the opt-in panel and the public.

This is why polling companies have to ensure diverse participation in their internet panels. (Image: Pew Research Center)

If such a question was not intended “to be a formal modelling of a Referendum re-run”, then commentary should not imply that it is. These initial intentions are not respected by Hawkins later in the same article:

However, the numbers switching from Remain to Leave since 2016 has been more than sufficient to outweigh these demographic changes: eight percent of 2016 Leave voters now support Remain, while more than twice that proportion of 2016 Remain voters, 18%, now support Leave.

Survey researchers should take great care to explain limitations of their own polls, and use their voice to accurately represent their research.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store