Polling Medians and Accuracy

Past elections do not suggest the polling median is “highly likely to be accurate”.

As part of their amended article on BBC election night coverage and bias, Prof Norris and Prof Dunleavy maintain:

Any political scientist could have told the BBC that the median result here was highly likely to be accurate on national vote share.

This article examines that claim.

Predictive Power?

Numerous market research companies attempt to estimate the opinion of the British public, and ask people how they intend to vote in upcoming elections.

We can look at the performance of final polls from British Polling Council companies, in the last four General Elections:

“Highly likely” to be accurate?

The median is the middle value in a sorted list. The arithmetic mean sums up all the values and divides by how many values there are. It is notable that the mean and median of vote intention estimates are similar for the Conservatives, Labour and the Liberal Democrats.

Whilst the polls were broadly accurate in the 2019 General Election, this was not the case in either 2010, 2015 or 2017.

General Election 2010

There was a modest underestimate of the Conservatives and Labour, coupled with a major overestimate of the Liberal Democrats.

In July 2015, Mellon and Prosser (Manchester & British Election Study team) suggest a possible explanation for this Liberal Democrat overestimate: the missing non-voters problem.

People who are less engaged in political matters are less likely to answer surveys. Their ‘slots’ in quota samples get filled by people with the same demographic characteristics (such as younger people), but who are more politically engaged and more likely to vote.

General Election 2015

This is considered to be the industry’s “annus horribilis” (horrible year). The Conservative and Labour vote intention estimates to be appear ‘neck and neck’. In reality, the Conservatives won by over six points in Great Britain.

The two main parties appeared very close. (Image: NCRM)

An inquiry was called, led by Prof Patrick Sturgis (then NCRM; now LSE Methodology). The report finds that the cause is systemic:

Our conclusion is that the primary cause of the polling miss in 2015 was unrepresentative samples. The methods the pollsters used to collect samples of voters systematically over-represented Labour supporters and under-represented Conservative supporters. The statistical adjustment procedures applied to the raw data did not mitigate this basic problem to any notable degree. The other putative causes can have made, at most, only a small contribution to the total error.

The suggestion that ‘shy Tories’ told market research companies one thing but did another in the secrecy of the polling booth is broadly rejected.

The British Election Study and the British Social Attitudes surveys were able to find Conservative voters. (Image: NCRM)

Turnout models are highlighted as a potential weakness in future elections.

General Election 2017

Turnout models become a major issue in the election. Two companies begin ‘ascribing’ turnout probabilities based on demographic factors — ignoring the self-reported likelihood of voting.

Whilst the Conservative vote share was estimated with broad accuracy, Labour were underestimated by five points.

Error is not seen across the industry: Survation and Kantar Public demonstrate moderate accuracy.

Error does not have a universal cause: turnout models appear to have induced severe issues in some companies, but ad hoc adjustments reduced the accuracy of other companies. An adjustment meant YouGov’s traditional polling and MRP national estimates diverged — the latter being more accurate.

A link to the past

Was the median vote intention estimate “highly likely to be accurate” in the past? No.

As part of the 2015 polling miss inquiry, the final report showed the historic record of net errors on the Conservative lead, with the mean average shown:

The 1992 election was also followed by a polling inquiry. (Image: NCRM)

Between 1945 and 2017, the difference between Conservative and Labour vote shares was correctly estimated within four points by the mean polling average in 10 out of 20 elections.

Whilst polling gives good sketches of public opinion, errors do occur.

A political scientist would be recommended to urge caution in their interpretation. Based on historical record, it is false to say British opinion polls are “highly likely to be accurate” ahead of a General Election. Accuracy of vote intention surveys can only be judged after the votes have been counted.

Median and mean vote intention estimates are calculated from the British Polling Council lists of final polls in 2010, 2015, 2017 and 2019. General Election results for 2010–2017 and 2019 are drawn from the House of Commons Library. The Google Sheet web page is available to view.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store