Response Options and Exiting the EU II
Does a recent poll suggest, as the polling company chairman asserts, “an increase in the 2016 Leave margin of victory”?
No. This article examines why.
Change the options, change the question
In a Telegraph Premium article, Andrew Hawkins (ComRes) claims:
At first sight, headline public opinion does not seem to have moved much in three years: the poll gives a combined Leave lead of 50% over Remain’s 42%, with a Brexit deal (30%) more popular than leaving without one (20%). Eight percent declined to offer a view.
This was not intended to be a formal modelling of a Referendum re-run but, if we exclude don’t knows, it nonetheless points to an increase in the 2016 Leave margin of victory, with 54% for Leave and 46% for Remain, while the country remains profoundly divided by region, age and political preference (of which more in a moment).
As becomes apparent from that first paragraph, the question ComRes asked on 2nd — 14th October 2019 was not a European Union membership referendum redux:
Q1. Thinking about the UK leaving the EU, which of the following would be your preferred outcome?
The UK remaining in the EU;
The UK leaving the EU with a withdrawal deal;
The UK leaving the EU on a No Deal basis;
Don’t know
This question provides three substantive options, and is about people’s “preferred outcome”. This is different to the standard question about a hypothetical referendum — a binary question of principle.
It cannot be assumed people saying they would prefer one of the two ‘Leave’ options would necessarily choose Leave in a new EU membership referendum.
Changing the options changes the question: leaving the European Union is not a scale. Think: how might the 2016 referendum have gone differently if three options were on the ballot paper?
What responses differ?
In a standard referendum question, we typically see similar rates of persistence: the proportion of Remain voters say they would vote Remain again are usually similar to the proportion of Leave voters backing the same option once more.
For instance, Prof Curtice averaged polls from late April to early June 2019, and found very similar rates of persistence among those recalling voting Remain and Leave in the 2016 referendum:
These standard referendum polls generally find modest Remain leads, principally driven by the views of those did not cast a vote in 2016. The latest six polls estimate, once undecided people and non-voters are excluded, Remain to be around 52%.
In the ComRes sample, what differs is that rate in persistence: 78% of 2016 Remain voters chose the ‘Remain’ option, compared to 87% of 2016 Leave voters choosing either of the two ‘Leave’ options — a difference of nine points.
It is some of those who recall voting Remain in 2016 who are giving divergent answers: preferring to leave the EU with a withdrawal agreement, whilst other surveys suggest they would re-support Remain in a hypothetical referendum.
In such questions about preferred outcomes, it is typical to see the ‘remaining in the EU’ option receive plural support, but more respondents back a combination of ‘Leave’ choices:
This is also true in the same survey. Survation asked their internet panel on 17th — 18th October about their preferred outcome (of three) and how they would vote in a binary referendum. 49% of Survation’s sample backed Remain in a hypothetical ballot (excluding non-voters), and 45% said remaining in the EU was their preferred outcome.
Does a large sample ensure accuracy?
Another notable aspect of the ComRes survey was its large sample size: 26,000 respondents. Hawkins writes:
The ComRes poll was conducted online across 12 days earlier in October with a gargantuan sample size of some 26,000 UK adults — equivalent to 13 of our regular voting intention polls. This makes the margin of error an impressive +/-0.61%.
The stated ‘margin of error’ relates to an assumption the survey was a simple random sample.
As Pew Research Center notes:
Very large sample sizes do not fix the shortcomings of online opt-in samples. While an online opt-in survey with 8,000 interviews may sound more impressive than one with 2,000, this study finds virtually no difference in accuracy.
For internet panels, large samples reduce sampling variability: they do not eliminate systematic differences between the opt-in panel and the public.
If such a question was not intended “to be a formal modelling of a Referendum re-run”, then commentary should not imply that it is. These initial intentions are not respected by Hawkins later in the same article:
However, the numbers switching from Remain to Leave since 2016 has been more than sufficient to outweigh these demographic changes: eight percent of 2016 Leave voters now support Remain, while more than twice that proportion of 2016 Remain voters, 18%, now support Leave.
Survey researchers should take great care to explain limitations of their own polls, and use their voice to accurately represent their research.
