Question Wording and Data Sourcing

This article briefly considers two issues: question wording in public opinion polling, and statements about where data comes from.

A Question of Wording

This phenomenon was recently illustrated when the same company, surveying people in the same time period, asked two different questions on holding an EU referendum and received two different responses.

Between 14–15th March 2019, YouGov had two separate polls of its internet panel: sponsored by the People’s Vote campaign and The Times.

At least, that is clear. (Image: Twitter)

Consider the full wording of these questions:

  • People’s Vote: If Britain looks set to leave the EU without a deal, would you support or oppose a public vote on whether Britain should leave without a deal or stay as a member of the EU?
  • The Times: Do you think there should or should not be a new referendum held on whether Britain should leave the European Union or remain a member?

There are three differences between these questions:

  • Condition: the former question sets the premise that Britain is about to leave the EU without a deal, whilst the latter does not;
  • Means: the former question specifies that this referendum should have leaving without a deal on the ballot paper;
  • A Public Vote: the former question uses the term ‘a public vote’, whilst the latter uses the term ‘referendum’.

As Prof Curtice demonstrated in UK in a Changing Europe’s January report on public opinion:

It seems that some Leave voters, at least, begin to warm to the idea of another ballot when it is implied that its purpose would be to enable voters (like themselves) to decide the fate of the Brexit deal that the government has negotiated. However, this support largely disappears when it is made clear it might result in a reversal of the decision to leave the EU.

Instead of repeating the same question, polling companies are often asking a large number of different questions with the same aim. Claims about a clamour of support for a new EU referendum should be treated cautiously.

Data Sourcing

We’ll wait for this to all blow over. (Source: Facebook/Right to Vote)

There is a curious statement about where this data comes from:

Source: Right to Vote — March 19

This is unhelpful. I asked their press officer what the data source actually was, and received the following reply (which is part of their press releases):

Analysis by FocalData from multiple research vendors, including Opinium, 2 January-4 March 2019. Sample: 14,046 adults from 632 constituencies across Britain. Findings based on preferences expressed.

Findings derived using multilevel regression and post-stratification (MRP) methodology, which is proven to produce very accurate constituency level findings. FocalData is a consumer analytics platform that uses machine learning to predict public opinion.

MRP does not, inherently, produce “very accurate constituency level findings”. It simply provides more accurate and robust estimates than looking at constituency sub-samples, which are often tiny.

Given the variety of question wordings, caution should be shown to specific claims of estimated support for a new referendum in each constituency.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store