Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data

William J. Hulme; Elizabeth Williamson; Elsie M.F. Horne; Amelia Green; Helen I. McDonald; Alex J. Walker; Helen J. Curtis; Caroline E. Morton; Brian MacKenna; Richard Croker; Amir Mehrkar; Seb Bacon; David Evans; Peter Inglesby; Simon Davy; Krishnan Bhaskaran; Anna Schultze; Christopher T. Rentsch; Laurie Tomlinson; Ian J. Douglas; Stephen J.W. Evans; Liam Smeeth; Tom Palmer; Ben Goldacre; Miguel A. Hernán; Jonathan A.C. Sterne

doi:10.7326/M21-4269

Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data

William J. Hulme, Elizabeth Williamson, Elsie M.F. Horne, Amelia Green, Helen I. McDonald, Alex J. Walker, Helen J. Curtis, Caroline E. Morton, Brian MacKenna, Richard Croker, Amir Mehrkar, Seb Bacon, David Evans, Peter Inglesby, Simon Davy, Krishnan Bhaskaran, Anna Schultze, Christopher T. Rentsch, Laurie Tomlinson, Ian J. DouglasStephen J.W. Evans, Liam Smeeth, Tom Palmer, Ben Goldacre, Miguel A. Hernán, Jonathan A.C. Sterne

Research output: Contribution to journal › Article › peer-review

9 Citations (SciVal)

Abstract

The COVID-19 vaccines were developed and rigorously evaluated in randomized trials during 2020. However, important questions, such as the magnitude and duration of protection, their effectiveness against new virus variants, and the effectiveness of booster vaccination, could not be answered by randomized trials and have therefore been addressed in observational studies. Analyses of observational data can be biased because of confounding and because of inadequate design that does not consider the evolution of the pandemic over time and the rapid uptake of vaccination. Emulating a hypothetical “target trial” using observational data assembled during vaccine roll-outs can help manage such potential sources of bias. This article describes 2 approaches to target trial emulation. In the sequential approach, on each day, eligible persons who have not yet been vaccinated are matched to a vaccinated person. The single-trial approach sets a single baseline at the start of the rollout and considers vaccination as a time-varying variable. The nature of the confounding depends on the analysis strategy: Estimating “per-protocol” effects (accounting for vaccination of initially unvaccinated persons after baseline) may require adjustment for both baseline and “time-varying” confounders. These issues are illustrated by using observational data from 2 780 931 persons in the United Kingdom aged 70 years or older to estimate the effect of a first dose of a COVID-19 vaccine. Addressing the issues discussed in this article should help authors of observational studies provide robust evidence to guide clinical and policy decisions.

Original language	English
Pages (from-to)	685-694
Number of pages	10
Journal	Annals of Internal Medicine
Volume	176
Issue number	5
Early online date	2 May 2023
DOIs	https://doi.org/10.7326/M21-4269
Publication status	Published - 16 May 2023
Externally published	Yes

Bibliographical note

Financial Support: This study was supported by the COVID-19 Longitudinal Health and Wellbeing National Core Study, funded by UK Research and Innovation (UKRI) Medical Research Council (MRC) (grant reference MC_PC_20059), and the COVID-19 Data and Connectivity National Core Study, led by Health Data Research UK in partnership with the Office for National Statistics and funded by UKRI MRC (MC_PC_20058). The OpenSAFELY Platform is supported by grants from the Wellcome Trust (222097/Z/20/Z); UKRI MRC (MR/V015757/1, MR/W016729/1); National Institute for Health and Care Research (NIHR135559, COV-LT2-0073), and Health Data Research UK (HDRUK2021.000, 2021.0157). TPP provided technical expertise and infrastructure within its data center pro bono in the context of a national emergency. Drs. Horne and Sterne are funded by the NIHR Bristol Biomedical Research Centre. Dr. Sterne is funded by Health Data Research UK South-West. The funders had no role in the design of the study; collection, analysis, or interpretation of the data; writing of the report; or the decision to submit the manuscript for publication.

ASJC Scopus subject areas

Internal Medicine

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.7326/M21-4269

Cite this

Hulme, W. J., Williamson, E., Horne, E. M. F., Green, A., McDonald, H. I., Walker, A. J., Curtis, H. J., Morton, C. E., MacKenna, B., Croker, R., Mehrkar, A., Bacon, S., Evans, D., Inglesby, P., Davy, S., Bhaskaran, K., Schultze, A., Rentsch, C. T., Tomlinson, L., ... Sterne, J. A. C. (2023). Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data. Annals of Internal Medicine, 176(5), 685-694. https://doi.org/10.7326/M21-4269

Hulme, WJ, Williamson, E, Horne, EMF, Green, A, McDonald, HI, Walker, AJ, Curtis, HJ, Morton, CE, MacKenna, B, Croker, R, Mehrkar, A, Bacon, S, Evans, D, Inglesby, P, Davy, S, Bhaskaran, K, Schultze, A, Rentsch, CT, Tomlinson, L, Douglas, IJ, Evans, SJW, Smeeth, L, Palmer, T, Goldacre, B, Hernán, MA & Sterne, JAC 2023, 'Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data', Annals of Internal Medicine, vol. 176, no. 5, pp. 685-694. https://doi.org/10.7326/M21-4269

@article{de9369ba568b4fd397b9ab7a8eb4ea93,

title = "Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data",

abstract = "The COVID-19 vaccines were developed and rigorously evaluated in randomized trials during 2020. However, important questions, such as the magnitude and duration of protection, their effectiveness against new virus variants, and the effectiveness of booster vaccination, could not be answered by randomized trials and have therefore been addressed in observational studies. Analyses of observational data can be biased because of confounding and because of inadequate design that does not consider the evolution of the pandemic over time and the rapid uptake of vaccination. Emulating a hypothetical “target trial” using observational data assembled during vaccine roll-outs can help manage such potential sources of bias. This article describes 2 approaches to target trial emulation. In the sequential approach, on each day, eligible persons who have not yet been vaccinated are matched to a vaccinated person. The single-trial approach sets a single baseline at the start of the rollout and considers vaccination as a time-varying variable. The nature of the confounding depends on the analysis strategy: Estimating “per-protocol” effects (accounting for vaccination of initially unvaccinated persons after baseline) may require adjustment for both baseline and “time-varying” confounders. These issues are illustrated by using observational data from 2 780 931 persons in the United Kingdom aged 70 years or older to estimate the effect of a first dose of a COVID-19 vaccine. Addressing the issues discussed in this article should help authors of observational studies provide robust evidence to guide clinical and policy decisions.",

author = "Hulme, {William J.} and Elizabeth Williamson and Horne, {Elsie M.F.} and Amelia Green and McDonald, {Helen I.} and Walker, {Alex J.} and Curtis, {Helen J.} and Morton, {Caroline E.} and Brian MacKenna and Richard Croker and Amir Mehrkar and Seb Bacon and David Evans and Peter Inglesby and Simon Davy and Krishnan Bhaskaran and Anna Schultze and Rentsch, {Christopher T.} and Laurie Tomlinson and Douglas, {Ian J.} and Evans, {Stephen J.W.} and Liam Smeeth and Tom Palmer and Ben Goldacre and Hern{\'a}n, {Miguel A.} and Sterne, {Jonathan A.C.}",

note = "Financial Support: This study was supported by the COVID-19 Longitudinal Health and Wellbeing National Core Study, funded by UK Research and Innovation (UKRI) Medical Research Council (MRC) (grant reference MC_PC_20059), and the COVID-19 Data and Connectivity National Core Study, led by Health Data Research UK in partnership with the Office for National Statistics and funded by UKRI MRC (MC_PC_20058). The OpenSAFELY Platform is supported by grants from the Wellcome Trust (222097/Z/20/Z); UKRI MRC (MR/V015757/1, MR/W016729/1); National Institute for Health and Care Research (NIHR135559, COV-LT2-0073), and Health Data Research UK (HDRUK2021.000, 2021.0157). TPP provided technical expertise and infrastructure within its data center pro bono in the context of a national emergency. Drs. Horne and Sterne are funded by the NIHR Bristol Biomedical Research Centre. Dr. Sterne is funded by Health Data Research UK South-West. The funders had no role in the design of the study; collection, analysis, or interpretation of the data; writing of the report; or the decision to submit the manuscript for publication.",

year = "2023",

month = may,

day = "16",

doi = "10.7326/M21-4269",

language = "English",

volume = "176",

pages = "685--694",

journal = "Annals of Internal Medicine",

issn = "0003-4819",

publisher = "American College of Chest Physicians",

number = "5",

}

TY - JOUR

T1 - Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data

AU - Hulme, William J.

AU - Williamson, Elizabeth

AU - Horne, Elsie M.F.

AU - Green, Amelia

AU - McDonald, Helen I.

AU - Walker, Alex J.

AU - Curtis, Helen J.

AU - Morton, Caroline E.

AU - MacKenna, Brian

AU - Croker, Richard

AU - Mehrkar, Amir

AU - Bacon, Seb

AU - Evans, David

AU - Inglesby, Peter

AU - Davy, Simon

AU - Bhaskaran, Krishnan

AU - Schultze, Anna

AU - Rentsch, Christopher T.

AU - Tomlinson, Laurie

AU - Douglas, Ian J.

AU - Evans, Stephen J.W.

AU - Smeeth, Liam

AU - Palmer, Tom

AU - Goldacre, Ben

AU - Hernán, Miguel A.

AU - Sterne, Jonathan A.C.

N1 - Financial Support: This study was supported by the COVID-19 Longitudinal Health and Wellbeing National Core Study, funded by UK Research and Innovation (UKRI) Medical Research Council (MRC) (grant reference MC_PC_20059), and the COVID-19 Data and Connectivity National Core Study, led by Health Data Research UK in partnership with the Office for National Statistics and funded by UKRI MRC (MC_PC_20058). The OpenSAFELY Platform is supported by grants from the Wellcome Trust (222097/Z/20/Z); UKRI MRC (MR/V015757/1, MR/W016729/1); National Institute for Health and Care Research (NIHR135559, COV-LT2-0073), and Health Data Research UK (HDRUK2021.000, 2021.0157). TPP provided technical expertise and infrastructure within its data center pro bono in the context of a national emergency. Drs. Horne and Sterne are funded by the NIHR Bristol Biomedical Research Centre. Dr. Sterne is funded by Health Data Research UK South-West. The funders had no role in the design of the study; collection, analysis, or interpretation of the data; writing of the report; or the decision to submit the manuscript for publication.

PY - 2023/5/16

Y1 - 2023/5/16

N2 - The COVID-19 vaccines were developed and rigorously evaluated in randomized trials during 2020. However, important questions, such as the magnitude and duration of protection, their effectiveness against new virus variants, and the effectiveness of booster vaccination, could not be answered by randomized trials and have therefore been addressed in observational studies. Analyses of observational data can be biased because of confounding and because of inadequate design that does not consider the evolution of the pandemic over time and the rapid uptake of vaccination. Emulating a hypothetical “target trial” using observational data assembled during vaccine roll-outs can help manage such potential sources of bias. This article describes 2 approaches to target trial emulation. In the sequential approach, on each day, eligible persons who have not yet been vaccinated are matched to a vaccinated person. The single-trial approach sets a single baseline at the start of the rollout and considers vaccination as a time-varying variable. The nature of the confounding depends on the analysis strategy: Estimating “per-protocol” effects (accounting for vaccination of initially unvaccinated persons after baseline) may require adjustment for both baseline and “time-varying” confounders. These issues are illustrated by using observational data from 2 780 931 persons in the United Kingdom aged 70 years or older to estimate the effect of a first dose of a COVID-19 vaccine. Addressing the issues discussed in this article should help authors of observational studies provide robust evidence to guide clinical and policy decisions.

AB - The COVID-19 vaccines were developed and rigorously evaluated in randomized trials during 2020. However, important questions, such as the magnitude and duration of protection, their effectiveness against new virus variants, and the effectiveness of booster vaccination, could not be answered by randomized trials and have therefore been addressed in observational studies. Analyses of observational data can be biased because of confounding and because of inadequate design that does not consider the evolution of the pandemic over time and the rapid uptake of vaccination. Emulating a hypothetical “target trial” using observational data assembled during vaccine roll-outs can help manage such potential sources of bias. This article describes 2 approaches to target trial emulation. In the sequential approach, on each day, eligible persons who have not yet been vaccinated are matched to a vaccinated person. The single-trial approach sets a single baseline at the start of the rollout and considers vaccination as a time-varying variable. The nature of the confounding depends on the analysis strategy: Estimating “per-protocol” effects (accounting for vaccination of initially unvaccinated persons after baseline) may require adjustment for both baseline and “time-varying” confounders. These issues are illustrated by using observational data from 2 780 931 persons in the United Kingdom aged 70 years or older to estimate the effect of a first dose of a COVID-19 vaccine. Addressing the issues discussed in this article should help authors of observational studies provide robust evidence to guide clinical and policy decisions.

UR - http://www.scopus.com/inward/record.url?scp=85159732185&partnerID=8YFLogxK

U2 - 10.7326/M21-4269

DO - 10.7326/M21-4269

M3 - Article

C2 - 37126810

SN - 0003-4819

VL - 176

SP - 685

EP - 694

JO - Annals of Internal Medicine

JF - Annals of Internal Medicine

IS - 5

ER -

Challenges in Estimating the Effectiveness of COVID-19 Vaccination Using Observational Data

Abstract

Bibliographical note

ASJC Scopus subject areas

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this