How a survey question is put can affect the responses received. This is important for how surveys are interpreted, and what inferences about the population (such as the general public) can be made.
This article looks at compound questions, acquiescence bias, and alternatives to agree-disagree question formats.
On 24th September 2019, ComRes interviewed 1,027 British adults via an internet panel, about the Supreme Court ruling and other matters.
One question asked:
Do you agree or disagree with the following statements?
“Parliament has had plenty of time to debate Brexit and we should just get on with leaving the EU.”
This is called a double-barrelled (or compound) question. There are two parts (“Parliament has had plenty of time…” and “we should just get on…”) but respondents can only agree, disagree or say they do not know.
What if you agree that Parliament has had “plenty” of debate time, but disagree that we should “just get on with” exiting the EU? In that situation, how a respondent answers is indeterminate.
The sample’s agreement with that statement was 60% — higher than a similar wording to one component: only 49% clicked they agreed that: “I don’t care how Brexit happens, we just need to get on with it now”.
There are other examples of such compound questions, such as from BMG Research in March 2018:
Do you agree or disagree with the following statement?
“The Government should get on with implementing the result of the referendum to take Britain out of the EU and in doing so take back control of our borders, laws, money and trade.”
50% of the sample either strongly or tended to agree with that statement.
Acquiescence response bias is the general tendency for respondents to agree with statements. According to Krosnick and Presser’s 2009 handbook on question design:
In 7 other studies, an average of 22% of the respondents agreed with both a statement and its reversal, whereas only 10% disagreed with both. Thus, taken together, these methods suggest an acquiescence effect averaging about 10%.
There are many suggested reasons as to why this effect occurs: some respondents are polite, wishing to defer to the interviewer. The question itself could justify agreement with the statement, with no corresponding reason to disagree. Some people are ‘satisficing’ — advancing through the survey, giving minimal answers, just to finish.
In the ComRes survey, reversing the statement (“The Supreme Court was right to rule that Boris Johnson’s prorogation of parliament was unlawful”) to (“The Supreme Court was wrong…”) elicits a near-mirror response.
50% agreed with the “right” statement and 49% disagreed the “wrong version”. Large effects of acquiescence bias are not always present. This is a general tendency, rather than an iron law.
In self-administered surveys likes internet panel polls, acquiescence response bias may not be as pronounced as interviewer surveys. Nay-saying — where respondents disagree with statements — exists too.
Constructing an alternative
Given the issue of acquiescence bias, Krosnick and Presser argued:
Therefore, a better approach to eliminating acquiescence is to avoid using agree/disagree, true/false, and yes/no questions altogether.
A common alternative to the agree-disagree question is a forced-choice format, where you ask what is closest to their views.
We can take one such agree-disagree question and look for alternatives.
I don’t care how Brexit happens, we just need to get on with it now.
This question is seeking to measure time pressure relating to the UK’s exit from the European Union.
Why might someone disagree with that statement?
Firstly, they could care exactly how the UK leaves the EU, and be willing to wait for their preferred exit. Secondly, they could wish for the UK to remain in the EU. Thirdly, they may have no strong opinions about the issue.
Consequently, we could ask a question (such as “which is closest to your view?”) with four different response options:
- I don’t care how Britain leaves the EU, we should leave as soon as possible;
- I don’t care how long it takes, we should leave the EU on best possible terms;
- I want Britain to remain in the EU, no matter what the exit deal is;
- None of the above.
This thought experiment constructed a question very similar to one asked by Opinium:
Acquiescence bias is a problem in asking people if they agree or disagree with statements. Such question formats are often used in academic surveys when constructing scales — such as measuring the tendency towards social liberalism and social conservatism.
Wider use of such question types has seemingly developed due to the ease of question writing and coding responses. It is crucial to design survey questions which are neutral, clear and provide an accurate reading of public opinion.