The Value of Survey Information

We often want to know what people think. Surveys offer this mirror onto society. Articles published in newspapers and through digital platforms may cite survey data.

Alongside survey results, the name of the polling company is ideally included. Additionally, journalists when and how the polling was conducted, and how many people were surveyed.

What happens when this critical information is omitted? This article looks at two recent examples.

Good companies lose out

Publishing opinion polls serves to enlighten public debate — to offer that mirror to society — and demonstrate the capabilities of the research company.

Let us look at a recent example involving a now-rare constituency opinion poll. In the 2010–15 Parliament, numerous constituency surveys were paid for Lord Ashcroft.

Number Cruncher Politics (which are not members of the British Polling Council) conducted an internet river sample of 509 people living in Brecon & Radnorshire.

The constituency of interest is in Wales. (Image: Wales Online/ONS)

This constituency will hold a by-election on 1st August — the second of its kind induced by the Recall of MPs Act 2015.

The data tables for this poll are available to read.

The survey itself was innovative. This is one of the first published vote intention polls of a British sub-national area to use internet river sampling. This type of sampling involves finding people as they visit websites and mobile apps, and inviting them via ads or banners to complete your survey. Like fish, respondents are caught in the river and then thrown back.

However, some news organisations chose to report on this poll in a very limited fashion — sometimes not including reference to the research company.

A recent editorial in The Guardian read:

One poll suggests the party will capture Brecon and Radnorshire in a by-election, helped by other remain parties standing down.

A provided link initially went to the Left Foot Forward blog. This link has now been edited to go to the Number Cruncher Politics website.

Both the Huffington Post UK and the Financial Times originally had articles which did not even refer to the research company. The respective articles have since been amended. (I am grateful to the journalists involved for remedying this problem.)

Constituency polling can be properly conducted through digital means. If unnamed, the research company loses out on recognition and future business. Omitting critical information also does not aid the reader. Failing to attribute is poor journalistic practice.

Questionable results spread without checks

52% of respondents admitted to kissing their dog more than their partner.

This figure has been repeated in many other articles. In each article, there is only one piece of information provided: Riley’s Organics — an organic dog treats company — conducted this survey.

Their website does not refer to any press release containing the results. No article provides any additional information so we can judge the survey’s value — not sampling, mode, sample size, fieldwork dates, weightings, question wording, order or response formats. Nothing.

“I love dogs!” I also dislike poor statistics. (GIF: Keiley Leatherwood/Pinterest)

Without this information, we cannot judge the probable accuracy of the supposed survey estimate.

I have contacted Riley’s Organics, but received no reply.

If all surveys can be written about in this way, what distinguishes a self-selecting Twitter ‘poll’ from a scientific random survey?

Polling results may be detached from critical information like the company, survey mode and fieldwork dates.

When using statistics in political debate, it is important to have proper sourcing. ‘A poll says’ — without naming the company, methods or dates — is unhelpful.

(The R code for the graph is available to read on R Pubs.)

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store