Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel; Barnabas Szaszi; Gustav Nilsonne; Olmo R. Van Den Akker; Casper J. Albers; Marcel Alm Van Assen; Jojanneke A. Bastiaansen; Daniel Benjamin; Udo Boehm; Rotem Botvinik-Nezer; Laura F. Bringmann; Niko A. Busch; Emmanuel Caruyer; Andrea M. Cataldo; Nelson Cowan; Andrew Delios; Noah Nn Van Dongen; Chris Donkin; Johnny B. Van Doorn; Anna Dreber; Gilles Dutilh; Gary F. Egan; Morton Ann Gernsbacher; Rink Hoekstra; Sabine Hoffmann; Felix Holzmeister; Juergen Huber; Magnus Johannesson; Kai J. Jonas; Alexander T. Kindel; Michael Kirchler; Yoram K. Kunkels; D. Stephen Lindsay; Jean Francois Mangin; Dora Matzke; Marcus R. Munafò; Ben R. Newell; Brian A. Nosek; Russell A. Poldrack; Don Van Ravenzwaaij; Jörg Rieskamp; Matthew J. Salganik; Alexandra Sarafoglou; Tom Schonberg; Martin Schweinsberg; David Shanks; Raphael Silberzahn; Daniel J. Simons; Barbara A. Spellman; Samuel St-Jean; Jeffrey J. Starns; Eric Luis Uhlmann; Jelte Wicherts; Eric Jan Wagenmakers

doi:10.7554/eLife.72185

Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel, Barnabas Szaszi, Gustav Nilsonne, Olmo R. Van Den Akker, Casper J. Albers, Marcel Alm Van Assen, Jojanneke A. Bastiaansen, Daniel Benjamin, Udo Boehm, Rotem Botvinik-Nezer, Laura F. Bringmann, Niko A. Busch, Emmanuel Caruyer, Andrea M. Cataldo, Nelson Cowan, Andrew Delios, Noah Nn Van Dongen, Chris Donkin, Johnny B. Van Doorn, Anna DreberGilles Dutilh, Gary F. Egan, Morton Ann Gernsbacher, Rink Hoekstra, Sabine Hoffmann, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Kai J. Jonas, Alexander T. Kindel, Michael Kirchler, Yoram K. Kunkels, D. Stephen Lindsay, Jean Francois Mangin, Dora Matzke, Marcus R. Munafò, Ben R. Newell, Brian A. Nosek, Russell A. Poldrack, Don Van Ravenzwaaij, Jörg Rieskamp, Matthew J. Salganik, Alexandra Sarafoglou, Tom Schonberg, Martin Schweinsberg, David Shanks, Raphael Silberzahn, Daniel J. Simons, Barbara A. Spellman, Samuel St-Jean, Jeffrey J. Starns, Eric Luis Uhlmann, Jelte Wicherts, Eric Jan Wagenmakers

Vice Chancellor's Office

Research output: Contribution to journal › Article › peer-review

32 Citations (SciVal)

Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Original language	English
Article number	e72185
Journal	eLife
Volume	10
Early online date	9 Nov 2021
DOIs	https://doi.org/10.7554/eLife.72185
Publication status	Published - 30 Nov 2021

Data Availability Statement

All anonymized data as well as the survey materials are publicly shared on the Open Science Framework page of the project: https://osf.io/4zvst/. Our methodology and data-analysis plan were preregistered. The preregistration document can be accessed at: https://osf.io/dgrua.

Funding

Netherlands Organisation for Scientific Research (406-17-568) - Alexandra Sarafoglou; Natural Sciences and Engineering Research Council of Canada (BP-546283-2020) - Samuel St-Jean; Fonds de Recherche du Québec - Nature et Technologies (290978) - Samuel St-Jean; European Research Council (726361) Jelte Wicherts and Olmo R van den Akker; European Research Council (681466) - Yoram K Kunkels; VIDI fellowship organisation (016.Vidi.188.001) - Don van Ravenzwaaij; VENI fellowship grant (Veni 191G.037) - Laura F Bringmann; National Science Foundation (1760052) - Matthew J Salganik; Weizmann Institute of Science (Israel National Postdoctoral Award Program for Advancing Women in Science) - Rotem Botvinik-Nezer; John Templeton Foundation, Templeton World Charity Foundation, Templeton Religion Trust, and Arnold Ventures - Brian A Nosek; Institut Européen d'Administration des Affaires - Eric Luis Uhlmann; European Research Council (640638) - Noah NN van Dongen.

ASJC Scopus subject areas

General Neuroscience
General Biochemistry,Genetics and Molecular Biology
General Immunology and Microbiology

Access to Document

10.7554/eLife.72185Licence: CC BY

Cite this

Aczel, B., Szaszi, B., Nilsonne, G., Van Den Akker, O. R., Albers, C. J., Van Assen, M. A., Bastiaansen, J. A., Benjamin, D., Boehm, U., Botvinik-Nezer, R., Bringmann, L. F., Busch, N. A., Caruyer, E., Cataldo, A. M., Cowan, N., Delios, A., Van Dongen, N. N., Donkin, C., Van Doorn, J. B., ... Wagenmakers, E. J. (2021). Consensus-based guidance for conducting and reporting multi-analyst studies. eLife, 10, Article e72185. https://doi.org/10.7554/eLife.72185

Aczel, B, Szaszi, B, Nilsonne, G, Van Den Akker, OR, Albers, CJ, Van Assen, MA, Bastiaansen, JA, Benjamin, D, Boehm, U, Botvinik-Nezer, R, Bringmann, LF, Busch, NA, Caruyer, E, Cataldo, AM, Cowan, N, Delios, A, Van Dongen, NN, Donkin, C, Van Doorn, JB, Dreber, A, Dutilh, G, Egan, GF, Gernsbacher, MA, Hoekstra, R, Hoffmann, S, Holzmeister, F, Huber, J, Johannesson, M, Jonas, KJ, Kindel, AT, Kirchler, M, Kunkels, YK, Lindsay, DS, Mangin, JF, Matzke, D, Munafò, MR, Newell, BR, Nosek, BA, Poldrack, RA, Ravenzwaaij, DV, Rieskamp, J, Salganik, MJ, Sarafoglou, A, Schonberg, T, Schweinsberg, M, Shanks, D, Silberzahn, R, Simons, DJ, Spellman, BA, St-Jean, S, Starns, JJ, Uhlmann, EL, Wicherts, J & Wagenmakers, EJ 2021, 'Consensus-based guidance for conducting and reporting multi-analyst studies', eLife, vol. 10, e72185. https://doi.org/10.7554/eLife.72185

@article{497b4549fff64f80ae0821499a30d8fa,

title = "Consensus-based guidance for conducting and reporting multi-analyst studies",

abstract = "Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.",

author = "Balazs Aczel and Barnabas Szaszi and Gustav Nilsonne and {Van Den Akker}, {Olmo R.} and Albers, {Casper J.} and {Van Assen}, {Marcel Alm} and Bastiaansen, {Jojanneke A.} and Daniel Benjamin and Udo Boehm and Rotem Botvinik-Nezer and Bringmann, {Laura F.} and Busch, {Niko A.} and Emmanuel Caruyer and Cataldo, {Andrea M.} and Nelson Cowan and Andrew Delios and {Van Dongen}, {Noah Nn} and Chris Donkin and {Van Doorn}, {Johnny B.} and Anna Dreber and Gilles Dutilh and Egan, {Gary F.} and Gernsbacher, {Morton Ann} and Rink Hoekstra and Sabine Hoffmann and Felix Holzmeister and Juergen Huber and Magnus Johannesson and Jonas, {Kai J.} and Kindel, {Alexander T.} and Michael Kirchler and Kunkels, {Yoram K.} and Lindsay, {D. Stephen} and Mangin, {Jean Francois} and Dora Matzke and Munaf{\`o}, {Marcus R.} and Newell, {Ben R.} and Nosek, {Brian A.} and Poldrack, {Russell A.} and Ravenzwaaij, {Don Van} and J{\"o}rg Rieskamp and Salganik, {Matthew J.} and Alexandra Sarafoglou and Tom Schonberg and Martin Schweinsberg and David Shanks and Raphael Silberzahn and Simons, {Daniel J.} and Spellman, {Barbara A.} and Samuel St-Jean and Starns, {Jeffrey J.} and Uhlmann, {Eric Luis} and Jelte Wicherts and Wagenmakers, {Eric Jan}",

year = "2021",

month = nov,

day = "30",

doi = "10.7554/eLife.72185",

language = "English",

volume = "10",

journal = "eLife",

issn = "2050-084X",

publisher = "eLife Sciences Publications Ltd",

}

TY - JOUR

T1 - Consensus-based guidance for conducting and reporting multi-analyst studies

AU - Aczel, Balazs

AU - Szaszi, Barnabas

AU - Nilsonne, Gustav

AU - Van Den Akker, Olmo R.

AU - Albers, Casper J.

AU - Van Assen, Marcel Alm

AU - Bastiaansen, Jojanneke A.

AU - Benjamin, Daniel

AU - Boehm, Udo

AU - Botvinik-Nezer, Rotem

AU - Bringmann, Laura F.

AU - Busch, Niko A.

AU - Caruyer, Emmanuel

AU - Cataldo, Andrea M.

AU - Cowan, Nelson

AU - Delios, Andrew

AU - Van Dongen, Noah Nn

AU - Donkin, Chris

AU - Van Doorn, Johnny B.

AU - Dreber, Anna

AU - Dutilh, Gilles

AU - Egan, Gary F.

AU - Gernsbacher, Morton Ann

AU - Hoekstra, Rink

AU - Hoffmann, Sabine

AU - Holzmeister, Felix

AU - Huber, Juergen

AU - Johannesson, Magnus

AU - Jonas, Kai J.

AU - Kindel, Alexander T.

AU - Kirchler, Michael

AU - Kunkels, Yoram K.

AU - Lindsay, D. Stephen

AU - Mangin, Jean Francois

AU - Matzke, Dora

AU - Munafò, Marcus R.

AU - Newell, Ben R.

AU - Nosek, Brian A.

AU - Poldrack, Russell A.

AU - Ravenzwaaij, Don Van

AU - Rieskamp, Jörg

AU - Salganik, Matthew J.

AU - Sarafoglou, Alexandra

AU - Schonberg, Tom

AU - Schweinsberg, Martin

AU - Shanks, David

AU - Silberzahn, Raphael

AU - Simons, Daniel J.

AU - Spellman, Barbara A.

AU - St-Jean, Samuel

AU - Starns, Jeffrey J.

AU - Uhlmann, Eric Luis

AU - Wicherts, Jelte

AU - Wagenmakers, Eric Jan

PY - 2021/11/30

Y1 - 2021/11/30

N2 - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

AB - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

UR - http://www.scopus.com/inward/record.url?scp=85120181656&partnerID=8YFLogxK

U2 - 10.7554/eLife.72185

DO - 10.7554/eLife.72185

M3 - Article

C2 - 34751133

AN - SCOPUS:85120181656

SN - 2050-084X

VL - 10

JO - eLife

JF - eLife

M1 - e72185

ER -

Consensus-based guidance for conducting and reporting multi-analyst studies

Abstract

Data Availability Statement

Funding

ASJC Scopus subject areas

Access to Document

Other files and links

Cite this