Knowing What The Question Says

There is an issue with interpreting survey data: misunderstanding what the question and responses say.

Asa Bennett, the “Brexit commissioning editor” at The Telegraph, mistakenly conflates feelings of knowledge with knowledge itself in a recent article.

In short

Deltapoll asked about feelings: The main question responses all began with “I felt”. The question was about recalled feelings of knowledge.

Transparency and clarity: The exact wording of questions by research companies should be present in all news and comment articles.

“The Lion’s Share”

The headline asserts:

Actually, Brexiteers knew what they were voting for — but Remainers had less of a clue

The headline is reflective of the article, as Asa Bennett writes:

But how firm a grasp did voters have on what they were voting for? Fresh polling by Deltapoll shows that the lion’s share of Leave voters (80 per cent) had an understanding of the implications of Brexit.

The provided table appears to justify that position:

There are some words missing, revealing flaws in the article’s interpretation of this survey.

What the question actually said

Deltapoll interviewed 1,017 GB adults via an internet panel, between 24th and 26th October 2018. The poll was sponsored by the Daily Mirror.

The exact question wording and responses were:

Thinking back to the time of the EU Referendum in June 2016, which of the following comes closest to your own view?

  • I felt fully informed of the implications of Brexit at the time;
  • I felt I knew quite a lot about the implications of Brexit at the time;
  • I felt I only had a basic understanding of the implications of Brexit at the time;
  • I felt I knew nothing about the implications of Brexit at the time.

This question is about recalled feelings of knowledge, not knowledge itself.

Feeling informed is not the same as being informed. There is also the problem of false recall: respondents may not accurately state how they felt over two years ago.

Exact response wordings are only tenuously implied at some points in the article (“admit to having known nothing”, “to say they were fully informed”). The article erroneously suggests otherwise elsewhere, such as:

These findings should make Remainers pause for thought as they insist that the British people should have to vote again on Brexit because of how little they knew about the implications of the referendum.

There are studies, such as those conducted by the Policy Institute at King’s College London and Ipsos MORI, on British public knowledge relating to the European Union.

Comparing measures of public perceptions to reality is very different to measuring self-reported and recalled beliefs about people’s own knowledge.

The Telegraph article does link to the research company’s data tables, which is to be strongly encouraged.

All news and comment articles should contain the exact question and response wordings.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store