Vaccines, big surveys, and paradoxes

In statistics, data quality matters more than data quantity.

Surveys act as a dusty mirror on society. These tools are crucial to understanding attitudes, opinions, and reported behaviours in populations.

Surveys provide estimates, differing from true values for many potential reasons. One such reason is sampling error: the difference due to picking random members to survey. That random selection may not reflect the population, even within specific groups. There are other reasons, inducing systematic differences — or statistical biases — in survey estimates.

Big surveys with big differences

Social media site Facebook partnered with Delphi Research Center at Carnegie Mellon University. With the University of Maryland, this survey collects data from global active users. After weighting, the United States user survey seeks to represent all adults in the US. The Delphi-Facebook survey had about 4.5m responses across 19 waves.

The Census Bureau runs the Household Pulse Survey, in collaboration with federal agencies. This is an experimental survey, using a master address file as the sampling frame. Potential respondents must have a mobile phone number or email address on this file. The Census Bureau then sends invites to their online survey by text message or email. Between January and May 2021, the Household Pulse Survey had over 600,000 responses. This is about 75,000 responses for each…