Reducing bias in secondary data analysis via an Explore and Confirm Analysis Workflow (ECAW): A proposal and survey of observational researchers

Robert T. Thibault; Marton Kovacs; Tom E. Hardwicke; Alexandra Sarafoglou; John P.A. Ioannidis; Marcus R. Munafò

doi:10.1098/rsos.230568

Reducing bias in secondary data analysis via an Explore and Confirm Analysis Workflow (ECAW): A proposal and survey of observational researchers

Robert T. Thibault
, Marton Kovacs
, Tom E. Hardwicke
, Alexandra Sarafoglou
, John P.A. Ioannidis
, Marcus R. Munafò

Vice Chancellor's Office

Stanford University
University of Bristol
Eötvös Loránd University
University of Melbourne
Vrije Universiteit Amsterdam
Berlin Institute of Health
MRC Integrative Epidemiology Unit

Research output: Contribution to journal › Article › peer-review

3 Citations (SciVal)

Abstract

Background: Although preregistration can reduce researcher bias and increase transparency in primary research settings, it is less applicable to secondary data analysis. An alternative method that affords additional protection from researcher bias, which cannot be gained from conventional forms of preregistration alone, is an Explore and Confirm Analysis Workflow (ECAW). In this workflow, a data management organization initially provides access to only a subset of their dataset to researchers who request it. The researchers then prepare an analysis script based on the subset of data, upload the analysis script to a registry, and then receive access to the full dataset. ECAWs aim to achieve similar goals to preregistration, but make access to the full dataset contingent on compliance. The present survey aimed to garner information from the research community where ECAWs could be applied - employing the Avon Longitudinal Study of Parents and Children (ALSPAC) as a case example.

Methods: We emailed a Web-based survey to researchers who had previously applied for access to ALSPAC's transgenerational observational dataset.

Results: We received 103 responses, for a 9% response rate. The results suggest that - at least among our sample of respondents - ECAWs hold the potential to serve their intended purpose and appear relatively acceptable. For example, only 10% of respondents disagreed that ALSPAC should run a study on ECAWs (versus 55% who agreed). However, as many as 26% of respondents agreed that they would be less willing to use ALSPAC data if they were required to use an ECAW (versus 45% who disagreed).

Conclusion: Our data and findings provide information for organizations and individuals interested in implementing ECAWs and related interventions.

Preregistration. https://osf.io/g2fw5 Deviations from the preregistration are outlined in electronic supplementary material A.

Original language	English
Article number	230568
Journal	Royal Society Open Science
Volume	10
Issue number	10
Early online date	11 Oct 2023
DOIs	https://doi.org/10.1098/rsos.230568
Publication status	Published - 11 Oct 2023

Data Availability Statement

Data, data dictionaries, analysis script and materials related to this study are publicly available at https://osf.io/gmd3n/ [18]. The cleaned dataset used in our analyses is available at https://osf.io/vn695. The study protocol and materials were registered on 24 September 2022 at https://osf.io/g2fw5. Discrepancies between this paper and the registered protocol are outlined in electronic supplementary material A. To facilitate reproducibility, this paper was written by interleaving regular prose and analysis code using R Markdown. The relevant files are available in a Code Ocean container (https://doi.org/10.24433/CO.1558762.v1) [19] which recreates the software environment in which the original analyses were performed. This container allows this paper to be reproduced from the data and code with a single button press.

Electronic supplementary material is available online [20].

Acknowledgements

We thank Nicholas Timpson and Kate Northstone for invaluable input throughout this research project. We thank the ALSPAC Executive for their collaboration and willingness to run this study. We thank all the participants for their responses.

Funding

R.T.T. is supported by a general support grant awarded to METRICS from Arnold Ventures and a postdoctoral fellowship from the Canadian Institutes of Health Research. A.S. was supported by an Amsterdam Brain and Cognition project grant (grant ref. ABC PG 22 January 2022) ‘From rigid theory to cognitive models: a framework to study individual differences in meaning representations’. The UK Medical Research Council and Wellcome (grant ref. 217065/Z/19/Z) and the University of Bristol provide core support for ALSPAC. This publication is the work of the authors and R.T.T. will serve as guarantor for the contents of this paper. The funders have no role in the preparation of this paper or the decision to publish.

Keywords

ALSPAC
blind data analysis
Explore and Confirm Analysis Workflow (ECAW)
meta-research
open science
preregistration

ASJC Scopus subject areas

General

Access to Document

10.1098/rsos.230568Licence: CC BY

Cite this

@article{79b91b9cc76f4cf2a9d3a352bf2ea10e,

title = "Reducing bias in secondary data analysis via an Explore and Confirm Analysis Workflow (ECAW): A proposal and survey of observational researchers",

abstract = "Background: Although preregistration can reduce researcher bias and increase transparency in primary research settings, it is less applicable to secondary data analysis. An alternative method that affords additional protection from researcher bias, which cannot be gained from conventional forms of preregistration alone, is an Explore and Confirm Analysis Workflow (ECAW). In this workflow, a data management organization initially provides access to only a subset of their dataset to researchers who request it. The researchers then prepare an analysis script based on the subset of data, upload the analysis script to a registry, and then receive access to the full dataset. ECAWs aim to achieve similar goals to preregistration, but make access to the full dataset contingent on compliance. The present survey aimed to garner information from the research community where ECAWs could be applied - employing the Avon Longitudinal Study of Parents and Children (ALSPAC) as a case example. Methods: We emailed a Web-based survey to researchers who had previously applied for access to ALSPAC's transgenerational observational dataset. Results: We received 103 responses, for a 9\% response rate. The results suggest that - at least among our sample of respondents - ECAWs hold the potential to serve their intended purpose and appear relatively acceptable. For example, only 10\% of respondents disagreed that ALSPAC should run a study on ECAWs (versus 55\% who agreed). However, as many as 26\% of respondents agreed that they would be less willing to use ALSPAC data if they were required to use an ECAW (versus 45\% who disagreed). Conclusion: Our data and findings provide information for organizations and individuals interested in implementing ECAWs and related interventions. Preregistration. https://osf.io/g2fw5 Deviations from the preregistration are outlined in electronic supplementary material A.",

keywords = "ALSPAC, blind data analysis, Explore and Confirm Analysis Workflow (ECAW), meta-research, open science, preregistration",

author = "Thibault, \{Robert T.\} and Marton Kovacs and Hardwicke, \{Tom E.\} and Alexandra Sarafoglou and Ioannidis, \{John P.A.\} and Munaf{\`o}, \{Marcus R.\}",

year = "2023",

month = oct,

day = "11",

doi = "10.1098/rsos.230568",

language = "English",

volume = "10",

journal = "Royal Society Open Science",

issn = "2054-5703",

publisher = "The Royal Society",

number = "10",

}

TY - JOUR

T1 - Reducing bias in secondary data analysis via an Explore and Confirm Analysis Workflow (ECAW)

T2 - A proposal and survey of observational researchers

AU - Thibault, Robert T.

AU - Kovacs, Marton

AU - Hardwicke, Tom E.

AU - Sarafoglou, Alexandra

AU - Ioannidis, John P.A.

AU - Munafò, Marcus R.

PY - 2023/10/11

Y1 - 2023/10/11

N2 - Background: Although preregistration can reduce researcher bias and increase transparency in primary research settings, it is less applicable to secondary data analysis. An alternative method that affords additional protection from researcher bias, which cannot be gained from conventional forms of preregistration alone, is an Explore and Confirm Analysis Workflow (ECAW). In this workflow, a data management organization initially provides access to only a subset of their dataset to researchers who request it. The researchers then prepare an analysis script based on the subset of data, upload the analysis script to a registry, and then receive access to the full dataset. ECAWs aim to achieve similar goals to preregistration, but make access to the full dataset contingent on compliance. The present survey aimed to garner information from the research community where ECAWs could be applied - employing the Avon Longitudinal Study of Parents and Children (ALSPAC) as a case example. Methods: We emailed a Web-based survey to researchers who had previously applied for access to ALSPAC's transgenerational observational dataset. Results: We received 103 responses, for a 9% response rate. The results suggest that - at least among our sample of respondents - ECAWs hold the potential to serve their intended purpose and appear relatively acceptable. For example, only 10% of respondents disagreed that ALSPAC should run a study on ECAWs (versus 55% who agreed). However, as many as 26% of respondents agreed that they would be less willing to use ALSPAC data if they were required to use an ECAW (versus 45% who disagreed). Conclusion: Our data and findings provide information for organizations and individuals interested in implementing ECAWs and related interventions. Preregistration. https://osf.io/g2fw5 Deviations from the preregistration are outlined in electronic supplementary material A.

AB - Background: Although preregistration can reduce researcher bias and increase transparency in primary research settings, it is less applicable to secondary data analysis. An alternative method that affords additional protection from researcher bias, which cannot be gained from conventional forms of preregistration alone, is an Explore and Confirm Analysis Workflow (ECAW). In this workflow, a data management organization initially provides access to only a subset of their dataset to researchers who request it. The researchers then prepare an analysis script based on the subset of data, upload the analysis script to a registry, and then receive access to the full dataset. ECAWs aim to achieve similar goals to preregistration, but make access to the full dataset contingent on compliance. The present survey aimed to garner information from the research community where ECAWs could be applied - employing the Avon Longitudinal Study of Parents and Children (ALSPAC) as a case example. Methods: We emailed a Web-based survey to researchers who had previously applied for access to ALSPAC's transgenerational observational dataset. Results: We received 103 responses, for a 9% response rate. The results suggest that - at least among our sample of respondents - ECAWs hold the potential to serve their intended purpose and appear relatively acceptable. For example, only 10% of respondents disagreed that ALSPAC should run a study on ECAWs (versus 55% who agreed). However, as many as 26% of respondents agreed that they would be less willing to use ALSPAC data if they were required to use an ECAW (versus 45% who disagreed). Conclusion: Our data and findings provide information for organizations and individuals interested in implementing ECAWs and related interventions. Preregistration. https://osf.io/g2fw5 Deviations from the preregistration are outlined in electronic supplementary material A.

KW - ALSPAC

KW - blind data analysis

KW - Explore and Confirm Analysis Workflow (ECAW)

KW - meta-research

KW - open science

KW - preregistration

UR - https://www.scopus.com/pages/publications/85175178778

U2 - 10.1098/rsos.230568

DO - 10.1098/rsos.230568

M3 - Article

AN - SCOPUS:85175178778

SN - 2054-5703

VL - 10

JO - Royal Society Open Science

JF - Royal Society Open Science

IS - 10

M1 - 230568

ER -

Reducing bias in secondary data analysis via an Explore and Confirm Analysis Workflow (ECAW): A proposal and survey of observational researchers

Abstract

Data Availability Statement

Acknowledgements

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this