Off Target

Did weightings induce a polling error for one company in 2019?

ICM Unlimited conducted voting intention polls for the 2019 election. Their final election poll underestimated the Conservative lead by six points. In Great Britain, the actual lead was around 12 points.

This article considers if their weighting targets induced an error.

Final polls

ICM Unlimited collect internet panel responses, for Great British adults. Their last vote intention poll was for 6–9th December 2019. In final polling, the mean average estimated Conservative lead was nine points.

Image for post
Image for post
Savanta ComRes estimated the smallest lead of all companies. (Image: Google Sheets)

There are house effects. Relative to the average company, ICM Unlimited’s Conservative estimates were lower. ICM Unlimited estimates for Labour were higher. Among final polls, their central Labour vote intention estimate was joint-highest, at 36%.

ICM Unlimited were consistent — showing a narrower lead than the average company. There were method changes too, during the campaign.

The importance of age

ICM Unlimited are transparent about their methods. As the British Polling Council requires, the company states:

The data has been weighted to the profile of all adults aged 18+ in Great Britain and is weighted by age, gender, social grade, household tenure, work status, and region. The data is also weighted by 2017 general election vote and 2016 EU referendum vote.

What are weights? Representative surveys should look like small versions of their target population. If a national poll had answers from 500 women and 700 men, we would adjust (or ‘weigh’) the responses. The ‘weighted’ responses from women count more and men less. This weighting means the sample reflects the population.

The sources of weighting targets are in the data tables:

Weighting targets are based on 2011 census data from the Office for National Statistics (ONS) and data from the National Readership Survey.

The population’s age structure changes over time. People are born and die. People arrive and leave the country. People get older.

Image for post
Image for post
Time marches on. (Image: ONS)

The choice of targets rewinds the GB population back to mid-2011. In 2019, there was a strengthened relationship between age and vote choice.

Image for post
Image for post
Age has become important for vote choice across elections. (Image: Ipsos MORI)

An indicative rake

The 2011 census targets mean there are too many younger adults, and too few older adults. As age matters for vote choice, these targets affect vote intention estimates.

I use the 2019 mid-year population estimates. I calculate an increased lead in the ICM Unlimited final poll of 1–3 points. This is a range, due to rounding.

The Conservative calculated estimate was 43%, up one point from the published result. The new Labour estimate was 35%, down one point.

This is indicative. It is only based on applying new weights for age groups to the summary statistics.

We need respondent micro-data to do this right. These figures should not be considered the ‘correct’ estimates.

Weighting targets appears to be a partial explanation of ICM’s total survey error. Surveys provide estimates, subject to many sources of potential error. That includes sampling error.

Across the industry, polling in 2019 was broadly accurate.

I published a Google Sheet page, showing the calculations.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store