Vaccine Efficacy Estimates

What does a ‘95% effective’ vaccine mean?

News reports shared trial findings of different vaccinations against COVID-19. For example, a BBC headline on 16th November was:

Moderna: Covid vaccine shows nearly 95% protection

This article looks at what statements about ‘95% protection’ and similar claims mean. This is another way of phrasing vaccine efficacy. It also shows how to calculate approximate uncertainty around those central estimates.

What is vaccine efficacy?

Vaccine efficacy is: the relative change in having a disease in the vaccinated group. The comparison is against non-vaccinated people in the trial.

By its nature, researchers estimate vaccine efficacy through trials. The optimal condition is a randomised control trial. This is where trial volunteers are in groups. For example, one group receives the vaccine; the other receives a placebo.

Image for post
Image for post
Elisa Granato is a volunteer in the Oxford vaccine trial. (Image: BBC)

Due to random assignment, researchers can estimate how good the vaccine is. Researchers look at what proportions of people have a disease in each group, and compare them. Different research projects may use different definitions of a case. A case is a person having the disease, or virus. They could use different tests, or confirm in labs after different periods of time.

We can work through an example: the Pfizer & BioNTech phase III study. There were 170 COVID-19 cases. Eight of these were in the vaccine group, and 162 in the placebo group. In this study, there were “over 43,000” participants. For this calculation, assume there were exactly 43,000 participants . These people split into two even groups.

  • Attack rate in the vaccinated group (ARV): Eight divided by 21,500 is 0.04%.
  • Attack rate in the non-vaccinated group (ARU): 162 divided by 21,500 is 0.75%.

The reduction from 0.75% to 0.04% is then 95%. This is our central estimate of vaccine efficacy.

What is the uncertainty around this estimate?

In 1988, Hightower, Orenstein and Martin wrote about approximations. These approximations were for vaccine efficacy. In our situation, the approximate intervals may be too wide — or too conservative.

We can calculate the approximate confidence interval for vaccine efficacy. This is at the 95% confidence level.

Image for post
Image for post
Vaccine efficacy is 1 minus the relative risk, shown here times by 100. (Image: WHO)

We can go through the arithmetic here:

  • Relative risk: 0.04% divided by 0.75%, which equals 0.05.
  • Error term: 1.96 times the square root of the sum of 0.9996/8 and 0.9925/162. This is equal to 0.71.
  • Upper bound of the relative risk interval: The exponent of the logarithm of the relative risk plus 0.71. That is 0.10.
  • Lower bound: The same calculation, except it is minus 0.71. That is 0.02.

Now, we can construct the confidence interval for vaccine efficacy:

  • Central estimate: 1 minus the relative risk estimate: 95%.
  • Lower bound: 1 minus the upper bound of the relative risk interval: 90%.
  • Upper bound: 1 minus the lower bound of the relative risk interval: 98%.

That gives us an interval of 90% — 98%.

We could use Bayesian probability: updating our prior beliefs with new information. Prof Rogers (Phastar) writes:

success at the final analysis was defined as a posterior probability that the VE was greater than 30% being greater than 98.6% (P(VE>30%|data) > 0.986)

Against this criterion, the boundary for success was 52.3% — which the trial exceeded.

It is important to remember there is uncertainty around estimates. It could be somewhat higher or lower than the central estimate.

What is vaccine effectiveness?

Despite the similar name, vaccine effectiveness is a different concept to efficacy. This is about how well a vaccine does outside of optimal conditions. For example, researchers may want to know how much a vaccine reduces hospitalisations.

Image for post
Image for post

A randomised control trial might exclude some people from volunteering. Such trials may exclude children, or those who with compromised immune systems. Participation affects how well efficacy estimates translate to the general population.

Observational studies can suffer from statistical biases. Vaccine programmes may give at-risk groups the vaccine first. They could be less healthy than the wider population. Alternately, healthy people — conscious of the disease — could take the vaccine.

There are many factors which affect how effective a vaccine is. Those include individual factors, such as age. Also, the characteristics of the vaccine itself — cost, storage, delivery — matter.

Efficacy is not comparable to the ‘survival rate’

There are social media posts comparing vaccine efficacy to the COVID-19 ‘survival rate’. These posts are often of the form: ‘95% vaccine effectiveness versus 99% COVID survival rate’.

These figures are not comparable. The latter number is an estimated proportion of infected people who do not die. This is the reverse of the infection fatality rate.

This rate varies by country, and may improve over the course of the pandemic. Collated WHO estimates suggest an infection fatality rate between 0.5% and 1.0%. Surviving COVID-19 may mean suffering from long-term health conditions. The prevalence of lingering symptoms and complications is under study. This is a novel virus, so much still unknown about long-term effects.

The former number is vaccine efficacy. This is an estimated reduction via vaccination in the chance of having COVID-19. This is not a survival ratio. In the Pfizer trial, the most common severe side effects were fatigue and headaches.

This is a simplified calculation. Suppose there are two populations: one million people with a vaccination and one million without. Assume the vaccine efficacy is 90%, and the infection fatality rate is 1%. The virus spreads, infecting 10% in the non-vaccinated population. That is 100,000 infected people — of which 1,000 die.

In the vaccinated population, only 10,000 people get the disease. Of those, 100 die. In our hypothetical example, vaccination reduced deaths by 90%.

Vaccine efficacy is the change in having a disease amongst those vaccinated. The comparison is versus non-vaccinated people in the same trial. These trials produce estimates — true efficacy could be somewhat higher or lower. Many factors affect the real-world effectiveness of vaccines.

This blog looks at the use of statistics in Britain and beyond. It is written by RSS Statistical Ambassador and Chartered Statistician @anthonybmasters.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store