Surveying Labour Members
YouGov and Survation have published polls of Labour members for the leadership contest. Voting will close on 2nd April, with the new leader announced two days later.
A LabourList article stated that the race “appears to be narrowing”. Their article compares the results of YouGov’s earlier poll and Survation’s survey. This post looks at why we should have avoided such a conclusion.
Four Labour MPs are standing to replace current leader Jeremy Corbyn: Rebecca Long-Bailey, Lisa Nandy, Sir Keir Starmer, and Emily Thornberry. Clive Lewis also sought nominations from his fellow MPs, and later withdrew. Jess Phillips received enough parliamentary nominations, but quit the race on 21st January.
The contest itself runs under the Alternative Vote system. Members return ballots via the internet or posts. Members state their preferences.
This system eliminates candidates with the least votes in each round. A fallen candidate’s votes redistribute to the next stated preference. The winner is the one candidate to exceed 50% of cast votes.
There have currently been three published surveys of Labour members.
YouGov (ESRC Party Members Project)
YouGov run surveys from their own opt-in internet panel, sending invites by email. This panel is particularly large. It contains enough self-identified members of the main parties to draw samples.
The first survey was of 1,059 Labour party members, conducted on 20–30th December 2019. The ESRC Party Members Project sponsored this poll. Queen Mary University of London and Sussex University runs this project.
The Party Members Project uses the YouGov panel for its research of party members. The 2019 book, Footsoldiers, compiles some of this research.
Around Christmas, it was not known who would be standing for leadership. The YouGov survey included Jess Phillips, Clive Lewis, and Yvette Cooper as options.
On first preferences — excluding undecided members — the survey estimated Starmer at 36%. Long-Bailey received 23% of first preferences in this survey.
The survey calibrates by age, gender and region. This is the same approach in YouGov’s polling for the 2015 Labour leadership race.
Survation (on behalf of LabourList)
Survation conducted their survey of 3,835 Labour members, on 8th — 13th January 2019. The company drew their sample from Labour members within LabourList’s email database. The sample was also weighted by age, gender and region.
At the time of the survey, Jess Phillips and Clive Lewis were still running for leadership. Survation’s poll offered those two candidates as options.
After excluding undecided members, the survey estimated Long-Bailey had 42% of first preferences. Starmer had an estimated 37%.
Appearances can be deceiving
Announcing the Survation poll results, LabourList published an article claiming:
Labour’s leadership race appears to be narrowing with Rebecca Long-Bailey coming out just ahead of her opponents on the basis of first preferences in a new poll of party members — with Keir Starmer close behind.
Appearances can be deceiving.
These are different polls from different companies. We should compare a poll with other surveys from the same company and method.
Compare like with like. Remember, surveys can differ due to sampling variability too.
Polling companies make contrasting methodological choices. Their survey results may show higher shares for some parties or candidates. These consistent differences are often called ‘house effects’.
In this case, the main divergence is the sampling frame: the list of people to draw samples from.
Survation are using LabourList registrants who are also Labour members. These people could be different to other members. For example, LabourList readers may have higher average political engagement. These samples may be unrepresentative of the wider membership.
These Survation estimates may suffer from a large frame error.
LabourList registrants appear more supportive of Long-Bailey than YouGov panellists. Journalists and others should highlight Survation’s survey is of LabourList registrants.
There are other differences too. Survation asked respondents about candidate familiarity before asking their vote intention. Weighting targets or techniques are slightly distinct.
We should not compare these two surveys and conclude the race is “narrowing”. These results were the first such survey from each company.
A second YouGov survey
By this point, Cooper and Lewis were not running. Both front-runners increased their share of first preferences. Considering the central estimate, Starmer’s lead widened by one point — rather than narrowed.
Surveying small populations
There are various methods that social research organisations use to survey small populations. These approaches include:
- Existing frame: another organisation may have an incomplete list of the small population.
- Filtering survey: an initial survey of the general population finds members of the small population.
- Specialist panel: through recruitment, members of that population join a special panel and take part in polls.
- High concentration: if the small population concentrates in geographic areas, send fieldworkers there.
- Snowball sampling: find ‘seed’ members of the small population, who then pass on the survey to others in the same group.
- Colossal panel: building a very large panel of the general population, containing enough numbers from this small group.
It is important to experiment in survey research. Survation are trying LabourList’s email database as an existing frame.
Generally, we should compare polls against other surveys using the same company. Additionally, highlight any changes in methods. This will help track changes in electoral contests and public beliefs.