Prompts, polls, and parliaments

How does choice of party options affect polling estimates?

Wording, order, and offered response options affect how people answer questions. Survey estimates can differ based on available or prompted answers.

Open and closed

A common political question is: what is the most important issue to you (and your vote choice)? Survey researchers could give people a list to choose from. That is a closed-ended question. Another way is to allow people to write what they think of: an open-ended question.

Pew Research Center looked at the difference between these two response types. After the US election in 2008, the company split respondents for their telephony poll. One group could give an unprompted response to the question:

What one issue matter most to you in deciding how you voted for president?

The second group had a list: the economy, the war in Iraq, health care, terrorism, and energy policy. Respondents could also give an answer absent from that closed list.

Among open responses, the economy was the choice of 35%. For the closed list, the economy was the most important issue for 58%.

There was a major difference for ‘other’ answers too. In open responses, 43% offered an answer outside of that limited list. For closed response options, only 8% replied with a non-listed answer.

(Image: Pew Research Center)

Prompting questions

A challenging application is for vote intention surveys.

There are hundreds of registered political parties. Many do not stand candidates for election. Many go unnoticed. Anthony Wells (YouGov) writes:

More to the point, including smaller parties in the polling question may risk overstating their support.

Around the end of May 2019, there was not a large effect of different response types:

This is not a full experiment: as the polls were at different times. (Image: YouGov)

Another example is for the Senedd 2021 vote intention polling.

YouGov conducted an internet panel survey on 18th — 21st April 2021. The company gathered answers from 1,142 Welsh adults (aged 16 and over). For the Senedd constituency vote, their Labour share estimate was 35%. The Conservative party estimate was 24%. That survey was on behalf of ITV Cymru Wales and Cardiff University.

Opinium, another internet panel company, did a survey on 9th — 19th April. The poll for Sky Wales was of 2,005 Welsh adults (aged 18 and over). Their estimate of Labour’s vote intention share in Senedd constituencies was 40%. For the Conservatives, the estimated share was 30%.

There was a large difference in party estimates between different companies. (Image: R Pubs)

As Prof Roger Awan-Scully (Cardiff) says:

Opinium’s results certainly paint a very different picture to those from the Barometer poll.

YouGov’s vote question gives a primary list of parties. That list was: Conservative, Labour, Liberal Democrats, Plaid Cymru, Reform UK, and Green. There are also ‘Other’, which opens a secondary list. That second list includes: UKIP, Abolish, Gwlad, Propel, Freedom Alliance, and ‘Other’.

Opinium offered a single, short list. That list was: Conservative, Labour, Plaid Cymru, Liberal Democrats, and ‘some other party’.

In the Opinium poll, other parties constituted 7% of likely voters that chose a party. In the YouGov survey, parties outside the main four had about 14%.

Researchers balance respondent burden versus giving people available options. That is a difficult choice.

Today, there are national elections for the Senedd and Holyrood parliaments. There are also local, mayoral and Police & Crime Commissioner elections in England today.

The R code for the graph is available on R Pubs and GitHub.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store