Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data

William J. Hulme, Elizabeth Williamson, Elsie M.F. Horne, Amelia Green, Helen I. McDonald, Alex J. Walker, Helen J. Curtis, Caroline E. Morton, Brian MacKenna, Richard Croker, Amir Mehrkar, Seb Bacon, David Evans, Peter Inglesby, Simon Davy, Krishnan Bhaskaran, Anna Schultze, Christopher T. Rentsch, Laurie Tomlinson, Ian J. DouglasStephen J.W. Evans, Liam Smeeth, Tom Palmer, Ben Goldacre, Miguel A. Hernán, Jonathan A.C. Sterne

Research output: Contribution to journalArticlepeer-review

4 Citations (SciVal)


The COVID-19 vaccines were developed and rigorously evaluated in randomized trials during 2020. However, important questions, such as the magnitude and duration of protection, their effectiveness against new virus variants, and the effectiveness of booster vaccination, could not be answered by randomized trials and have therefore been addressed in observational studies. Analyses of observational data can be biased because of confounding and because of inadequate design that does not consider the evolution of the pandemic over time and the rapid uptake of vaccination. Emulating a hypothetical “target trial” using observational data assembled during vaccine roll-outs can help manage such potential sources of bias. This article describes 2 approaches to target trial emulation. In the sequential approach, on each day, eligible persons who have not yet been vaccinated are matched to a vaccinated person. The single-trial approach sets a single baseline at the start of the rollout and considers vaccination as a time-varying variable. The nature of the confounding depends on the analysis strategy: Estimating “per-protocol” effects (accounting for vaccination of initially unvaccinated persons after baseline) may require adjustment for both baseline and “time-varying” confounders. These issues are illustrated by using observational data from 2 780 931 persons in the United Kingdom aged 70 years or older to estimate the effect of a first dose of a COVID-19 vaccine. Addressing the issues discussed in this article should help authors of observational studies provide robust evidence to guide clinical and policy decisions.

Original languageEnglish
Pages (from-to)685-694
Number of pages10
JournalAnnals of Internal Medicine
Issue number5
Early online date2 May 2023
Publication statusPublished - 16 May 2023
Externally publishedYes

Bibliographical note

Financial Support: This study was supported by the COVID-19 Longitudinal Health and Wellbeing National Core Study, funded by UK Research and Innovation (UKRI) Medical Research Council (MRC) (grant reference MC_PC_20059), and the COVID-19 Data and Connectivity National Core Study, led by Health Data Research UK in partnership with the Office for National Statistics and funded by UKRI MRC (MC_PC_20058). The OpenSAFELY Platform is supported by grants from the Wellcome Trust (222097/Z/20/Z); UKRI MRC (MR/V015757/1, MR/W016729/1); National Institute for Health and Care Research (NIHR135559, COV-LT2-0073), and Health Data Research UK (HDRUK2021.000, 2021.0157). TPP provided technical expertise and infrastructure within its data center pro bono in the context of a national emergency. Drs. Horne and Sterne are funded by the NIHR Bristol Biomedical Research Centre. Dr. Sterne is funded by Health Data Research UK South-West. The funders had no role in the design of the study; collection, analysis, or interpretation of the data; writing of the report; or the decision to submit the manuscript for publication.

ASJC Scopus subject areas

  • Internal Medicine


Dive into the research topics of 'Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data'. Together they form a unique fingerprint.

Cite this