Challenges in estimating the effectiveness of COVID-19 vaccination using observational data

William J Hulme, Elizabeth Williamson, Elsie M F Horne, Amelia Green, Helen I McDonald, Alex J Walker, Helen J Curtis, Caroline E Morton, Brian MacKenna, Richard Croker, Amir Mehrkar, Seb Bacon, David Evans, Peter Inglesby, Simon Davy, Krishnan Bhaskaran, Anna Schultze, Christopher T Rentsch, Laurie Tomlinson, Ian J DouglasStephen J W Evans, Liam Smeeth, Tom Palmer, Ben Goldacre, Miguel A Hernán, Jonathan A C Sterne

Research output: Contribution to journalArticlepeer-review


The COVID-19 vaccines were developed and rigorously evaluated in randomized trials during 2020. However, important questions, such as the magnitude and duration of protection, their effectiveness against new virus variants, and the effectiveness of booster vaccination, could not be answered by randomized trials and have therefore been addressed in observational studies. Analyses of observational data can be biased because of confounding and because of inadequate design that does not consider the evolution of the pandemic over time and the rapid uptake of vaccination. Emulating a hypothetical "target trial" using observational data assembled during vaccine rollouts can help manage such potential sources of bias. This article describes 2 approaches to target trial emulation. In the sequential approach, on each day, eligible persons who have not yet been vaccinated are matched to a vaccinated person. The single-trial approach sets a single baseline at the start of the rollout and considers vaccination as a time-varying variable. The nature of the confounding depends on the analysis strategy: Estimating "per-protocol" effects (accounting for vaccination of initially unvaccinated persons after baseline) may require adjustment for both baseline and "time-varying" confounders. These issues are illustrated by using observational data from 2 780 931 persons in the United Kingdom aged 70 years or older to estimate the effect of a first dose of a COVID-19 vaccine. Addressing the issues discussed in this article should help authors of observational studies provide robust evidence to guide clinical and policy decisions.

Original languageEnglish
Pages (from-to)685-693
Number of pages8
JournalAnnals of Internal Medicine
Issue number5
Early online date2 May 2023
Publication statusE-pub ahead of print - 2 May 2023
Externally publishedYes


Dive into the research topics of 'Challenges in estimating the effectiveness of COVID-19 vaccination using observational data'. Together they form a unique fingerprint.

Cite this