Prompts and Political Polling

In survey research, how people answer questions can depend on how those questions are worded, what order the questions appear, and what response options are offered.

This article looks at polling performance in the recent European Elections and the different methods employed by UK polling companies.

Overestimation of the Brexit Party

This is a strange accusation, given most polling companies overestimated Brexit Party support in the recent European Elections.

The two polling companies with the smallest five party mean absolute errors were Ipsos MORI and YouGov.

YouGov overestimated the Brexit Party’s support in that election by five points.

How do different companies prompt?

For Westminster vote intention, most polling companies provide a reasonably long list of options for people to choose: Conservatives, Labour, Liberal Democrats, UK Independence Party, Green, the Brexit Party, and Change UK. In addition, Welsh respondents are typically offered Plaid Cymru and Scottish people are able to choose the Scottish National Party.

Companies may offer a shorter initial list, taking two approaches to people wanting to vote for another party. Five examples are given, correct as of 1st June:

  • Secondary prompting [YouGov, BMG Research, Survation]: The first YouGov list contains Conservatives, Labour and the Liberal Democrats. If people select they wish to vote for another party, a second list appears. This secondary list currently contains UKIP, the Brexit Party, Greens, UKIP and the two national parties. For BMG Research, the primary list is the Conservatives, Labour, Liberal Democrats, and SNP/Plaid Cymru. Their secondary list is the Brexit Party, Change UK, Green and UKIP. Survation places the SNP — but not Plaid Cymru — in its primary prompt.
  • Spontaneous response [Ipsos MORI, Kantar]: The shorter list of parties contains Conservative, Labour, Liberal Democrats, national parties, Green and UKIP. People intending to vote for Change UK and the Brexit Party must spontaneously respond to that question, by saying or typing.

Polling companies may use different lists for different types of elections, and may change these approaches in future.

Why can prompting lead to overestimation?

Back in 2014, UKIP won the most votes in the European Elections. Even in YouGov’s secondary list, UKIP were receiving around 15% in opinion polls across that year.

Individual polls by YouGov had an estimated UKIP vote intention share between 10% and 19%. (Photo: YouGov)

Starting in 2015, UKIP were then elevated to the primary list. It should be noted that the party’s estimated vote intention share fell slightly after this change. Whilst YouGov’s UKIP vote intention share was reasonably accurate in 2015, the primary status persisted.

Following the calling of the 2017 General Election — with the Conservative government initiating the UK’s withdrawal from the European Union — the UKIP vote intention share immediately fell.

Calling the General Election causes an immediate drop for UKIP, and Labour’s fortunes were reversed. (Photo: Polling Observatory)

In the snap election, UKIP candidates only covered 378 constituencies (out of 650). People may have intended to vote for this party, but found themselves unable to do so. After votes were counted, there was some overestimation of UKIP’s vote share in pre-election polling.

The goals of survey research are accuracy and insight, not pandering to politicians. It is for social research companies to test methodological changes, and determine the most accurate means of measuring support for new (or reincarnated) parties.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store