I previously wrote about how a survey — commissioned by e2save and undertaken by Atomik Research — apparently showed that “52% of Brits think the Moon landings from 1969–1972 were faked”.
This claim led to a debate on national television between a cosmologist and a man who does not believe the Moon is solid.
Thanks to the Market Research Society and Atomik Research (for which I am grateful), I now understand how a survey gave this particular result.
The commentary was not agreed upon: The survey interpretation was made by e2save: without the agreement of Atomik Research. The company then failed to get retractions of the press reports.
Check-boxes: There were many coded answers to a question of what people believe in, such as ghosts, aliens and unicorns. It is plausible some respondents misinterpreted what the response (‘The moon landing’) meant.
‘Tick all that apply’
In self-administered survey questions, a respondent may be offered various possible responses, in check-boxes. The person ticks those boxes to indicate their answer.
Between 25th and 29th May 2016, Atomik Research interviewed 1,003 UK adults via an internet panel, through systematic sampling.
The data was unweighted. With permission, I have uploaded the data tables.
The question was:
Which of the following do you believe in? Tick all that apply.
Now, for most responses, the percentage ticking those boxes are similar to other surveys.
In October 2014, a YouGov survey estimated 34% of GB adults believed ghosts ‘definitely’ or ‘probably exist’. In the same survey, 4% said the same about vampires. The unweighted Atomik Research data places these figures at 30% and 5%, respectively.
However, it is noticeable that two responses (‘The moon landing’ and ‘Dinosaurs’) have much lower percentages than we would expect, especially compared to other surveys.
It is plausible, given the nature of the other possible responses, that some respondents misinterpreted what was meant by the ‘The moon landing’ and ‘Dinosaurs’ options.
Some respondents may have interpreted ticking those boxes to mean the Moon Landings were faked, or the existence of dinosaurs was fictional.
According to the researcher at Atomik Research I conversed with, e2save then interpreted the unweighted survey data literally (52% must not believe the Moon Landings happened — which in of itself, is flawed). The press release was published without the consent and agreement of Atomik Research.
The company then failed to get retractions from either e2save or news organisations who published articles based on the press release.
It is important to remember the answer options, question wording and question order can all affect survey responses.