The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Martin Schweinsberg; Nikhil Madan; Michelangelo Vianello; S. Amy Sommer; Jennifer Jordan; Warren Tierney; Eli Awtrey; Luke Lei Zhu; Daniel Diermeier; Justin E. Heinze; Malavika Srinivasan; David Tannenbaum; Eliza Bivolaru; Jason Dana; Clintin P. Davis-Stober; Christilene du Plessis; Quentin F. Gronau; Andrew C. Hafenbrack; Eko Yi Liao; Alexander Ly; Maarten Marsman; Toshio Murase; Israr Qureshi; Michael Schaerer; Nico Thornley; Christina M. Tworek; Eric Jan Wagenmakers; Lynn Wong; Tabitha Anderson; Christopher W. Bauman; Wendy L. Bedwell; Victoria Brescoll; Andrew Canavan; Jesse J. Chandler; Erik Cheries; Sapna Cheryan; Felix Cheung; Andrei Cimpian; Mark A. Clark; Diana Cordon; Fiery Cushman; Peter H. Ditto; Thomas Donahue; Sarah E. Frick; Monica Gamez-Djokic; Rebecca Hofstein Grady; Jesse Graham; Jun Gu; Adam Hahn; Brittany E. Hanson; Nicole J. Hartwich; Kristie Hein; Yoel Inbar; Lily Jiang; Tehlyr Kellogg; Deanna M. Kennedy; Nicole Legate; Timo P. Luoma; Heidi Maibuecher; Peter Meindl; Jennifer Miles; Alexandra Mislin; Daniel C. Molden; Matt Motyl; George Newman; Hoai Huong Ngo; Harvey Packham; Philip S. Ramsay; Jennifer L. Ray; Aaron M. Sackett; Anne Laure Sellier; Tatiana Sokolova; Walter Sowden; Daniel Storage; Xiaomin Sun; Jay J. Van Bavel; Anthony N. Washburn; Cong Wei; Erik Wetter; Carlos T. Wilson; Sophie Charlotte Darroux; Eric Luis Uhlmann

doi:10.1016/j.jesp.2015.10.001

The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Martin Schweinsberg, Nikhil Madan, Michelangelo Vianello, S. Amy Sommer, Jennifer Jordan, Warren Tierney, Eli Awtrey, Luke Lei Zhu, Daniel Diermeier, Justin E. Heinze, Malavika Srinivasan, David Tannenbaum, Eliza Bivolaru, Jason Dana, Clintin P. Davis-Stober, Christilene du Plessis, Quentin F. Gronau, Andrew C. Hafenbrack, Eko Yi Liao, Alexander LyMaarten Marsman, Toshio Murase, Israr Qureshi, Michael Schaerer, Nico Thornley, Christina M. Tworek, Eric Jan Wagenmakers, Lynn Wong, Tabitha Anderson, Christopher W. Bauman, Wendy L. Bedwell, Victoria Brescoll, Andrew Canavan, Jesse J. Chandler, Erik Cheries, Sapna Cheryan, Felix Cheung, Andrei Cimpian, Mark A. Clark, Diana Cordon, Fiery Cushman, Peter H. Ditto, Thomas Donahue, Sarah E. Frick, Monica Gamez-Djokic, Rebecca Hofstein Grady, Jesse Graham, Jun Gu, Adam Hahn, Brittany E. Hanson, Nicole J. Hartwich, Kristie Hein, Yoel Inbar, Lily Jiang, Tehlyr Kellogg, Deanna M. Kennedy, Nicole Legate, Timo P. Luoma, Heidi Maibuecher, Peter Meindl, Jennifer Miles, Alexandra Mislin, Daniel C. Molden, Matt Motyl, George Newman, Hoai Huong Ngo, Harvey Packham, Philip S. Ramsay, Jennifer L. Ray, Aaron M. Sackett, Anne Laure Sellier, Tatiana Sokolova, Walter Sowden, Daniel Storage, Xiaomin Sun, Jay J. Van Bavel, Anthony N. Washburn, Cong Wei, Erik Wetter, Carlos T. Wilson, Sophie Charlotte Darroux, Eric Luis Uhlmann

Research output: Contribution to journal › Article › peer-review

83 Citations (SciVal)

Abstract

This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

Original language	English
Pages (from-to)	55-67
Number of pages	13
Journal	Journal of Experimental Social Psychology
Volume	66
DOIs	https://doi.org/10.1016/j.jesp.2015.10.001
Publication status	Published - 1 Sept 2016

Funding

The first, second, and last authors contributed equally to the project. Eric Luis Uhlmann designed the Pipeline Project and wrote the initial project proposal. Martin Schweinsberg, Nikhil Madan, Michelangelo Vianello, Amy Sommer, Jennifer Jordan, Warren Tierney, Eli Awtrey, and Luke (Lei) Zhu served as project coordinators. Daniel Diermeier, Justin E. Heinze, Malavika Srinivasan, David Tannenbaum, Eric Luis Uhlmann, and Luke Zhu contributed original studies for replication. Michelangelo Vianello, Jennifer Jordan, Amy Sommer, Eli Awtrey, Eliza Bivolaru, Jason Dana, Clintin P. Davis-Stober, Christilene du Plessis, Quentin F. Gronau, Andrew C. Hafenbrack, Eko Yi Liao, Alexander Ly, Maarten Marsman, Toshio Murase, Israr Qureshi, Michael Schaerer, Nico Thornley, Christina M. Tworek, Eric-Jan Wagenmakers, and Lynn Wong helped analyze the data. Eli Awtrey, Jennifer Jordan, Amy Sommer, Tabitha Anderson, Christopher W. Bauman, Wendy L. Bedwell, Victoria Brescoll, Andrew Canavan, Jesse J. Chandler, Erik Cheries, Sapna Cheryan, Felix Cheung, Andrei Cimpian, Mark A. Clark, Diana Cordon, Fiery Cushman, Peter Ditto, Thomas Donahue, Sarah E. Frick, Monica Gamez-Djokic, Rebecca Hofstein Grady, Jesse Graham, Jun Gu, Adam Hahn, Brittany E. Hanson, Nicole J. Hartwich, Kristie Hein, Yoel Inbar, Lily Jiang, Tehlyr Kellogg, Deanna M. Kennedy, Nicole Legate, Timo P. Luoma, Heidi Maibuecher, Peter Meindl, Jennifer Miles, Alexandra Mislin, Daniel Molden, Matt Motyl, George Newman, Hoai Huong Ngo, Harvey Packham, Philip S. Ramsay, Jennifer Lauren Ray, Aaron M. Sackett, Anne-Laure Sellier, Tatiana Sokolova, Walter Sowden, Daniel Storage, Xiaomin Sun, Christina M. Tworek, Jay J. van Bavel, Anthony N. Washburn, Cong Wei, Erik Wetter, and Carlos T. Wilson carried out the replications. Adam Hahn, Nicole Hartwich, Timo Luoma, Hoai Huong Ngo, and Sophie-Charlotte Darroux translated study materials from English into the local language. Eric Luis Uhlmann and Martin Schweinsberg wrote the first draft of the paper and numerous authors provided feedback, comments, and revisions. Correspondence concerning this paper should be addressed to Martin Schweinsberg, Boulevard de Constance, 77305 Fontainebleau, France, [email protected] , or to Eric Luis Uhlmann, INSEAD Organizational Behavior Area, 1 Ayer Rajah Avenue, 138676 Singapore, [email protected] . The Pipeline Project was generously supported by an R&D grant from INSEAD.

Keywords

Crowdsourcing science
Meta-science
Methodology
Replication
Reproducibility
Research transparency

ASJC Scopus subject areas

Social Psychology
Sociology and Political Science

Access to Document

10.1016/j.jesp.2015.10.001Licence: CC BY-NC-ND

Cite this

Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., Awtrey, E., Zhu, L. L., Diermeier, D., Heinze, J. E., Srinivasan, M., Tannenbaum, D., Bivolaru, E., Dana, J., Davis-Stober, C. P., du Plessis, C., Gronau, Q. F., Hafenbrack, A. C., Liao, E. Y., ... Uhlmann, E. L. (2016). The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline. Journal of Experimental Social Psychology, 66, 55-67. https://doi.org/10.1016/j.jesp.2015.10.001

Schweinsberg, M, Madan, N, Vianello, M, Sommer, SA, Jordan, J, Tierney, W, Awtrey, E, Zhu, LL, Diermeier, D, Heinze, JE, Srinivasan, M, Tannenbaum, D, Bivolaru, E, Dana, J, Davis-Stober, CP, du Plessis, C, Gronau, QF, Hafenbrack, AC, Liao, EY, Ly, A, Marsman, M, Murase, T, Qureshi, I, Schaerer, M, Thornley, N, Tworek, CM, Wagenmakers, EJ, Wong, L, Anderson, T, Bauman, CW, Bedwell, WL, Brescoll, V, Canavan, A, Chandler, JJ, Cheries, E, Cheryan, S, Cheung, F, Cimpian, A, Clark, MA, Cordon, D, Cushman, F, Ditto, PH, Donahue, T, Frick, SE, Gamez-Djokic, M, Grady, RH, Graham, J, Gu, J, Hahn, A, Hanson, BE, Hartwich, NJ, Hein, K, Inbar, Y, Jiang, L, Kellogg, T, Kennedy, DM, Legate, N, Luoma, TP, Maibuecher, H, Meindl, P, Miles, J, Mislin, A, Molden, DC, Motyl, M, Newman, G, Ngo, HH, Packham, H, Ramsay, PS, Ray, JL, Sackett, AM, Sellier, AL, Sokolova, T, Sowden, W, Storage, D, Sun, X, Van Bavel, JJ, Washburn, AN, Wei, C, Wetter, E, Wilson, CT, Darroux, SC & Uhlmann, EL 2016, 'The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline', Journal of Experimental Social Psychology, vol. 66, pp. 55-67. https://doi.org/10.1016/j.jesp.2015.10.001

@article{5f7dd16677a34a5fb594198446ca5a60,

title = "The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline",

abstract = "This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.",

keywords = "Crowdsourcing science, Meta-science, Methodology, Replication, Reproducibility, Research transparency",

author = "Martin Schweinsberg and Nikhil Madan and Michelangelo Vianello and Sommer, {S. Amy} and Jennifer Jordan and Warren Tierney and Eli Awtrey and Zhu, {Luke Lei} and Daniel Diermeier and Heinze, {Justin E.} and Malavika Srinivasan and David Tannenbaum and Eliza Bivolaru and Jason Dana and Davis-Stober, {Clintin P.} and {du Plessis}, Christilene and Gronau, {Quentin F.} and Hafenbrack, {Andrew C.} and Liao, {Eko Yi} and Alexander Ly and Maarten Marsman and Toshio Murase and Israr Qureshi and Michael Schaerer and Nico Thornley and Tworek, {Christina M.} and Wagenmakers, {Eric Jan} and Lynn Wong and Tabitha Anderson and Bauman, {Christopher W.} and Bedwell, {Wendy L.} and Victoria Brescoll and Andrew Canavan and Chandler, {Jesse J.} and Erik Cheries and Sapna Cheryan and Felix Cheung and Andrei Cimpian and Clark, {Mark A.} and Diana Cordon and Fiery Cushman and Ditto, {Peter H.} and Thomas Donahue and Frick, {Sarah E.} and Monica Gamez-Djokic and Grady, {Rebecca Hofstein} and Jesse Graham and Jun Gu and Adam Hahn and Hanson, {Brittany E.} and Hartwich, {Nicole J.} and Kristie Hein and Yoel Inbar and Lily Jiang and Tehlyr Kellogg and Kennedy, {Deanna M.} and Nicole Legate and Luoma, {Timo P.} and Heidi Maibuecher and Peter Meindl and Jennifer Miles and Alexandra Mislin and Molden, {Daniel C.} and Matt Motyl and George Newman and Ngo, {Hoai Huong} and Harvey Packham and Ramsay, {Philip S.} and Ray, {Jennifer L.} and Sackett, {Aaron M.} and Sellier, {Anne Laure} and Tatiana Sokolova and Walter Sowden and Daniel Storage and Xiaomin Sun and {Van Bavel}, {Jay J.} and Washburn, {Anthony N.} and Cong Wei and Erik Wetter and Wilson, {Carlos T.} and Darroux, {Sophie Charlotte} and Uhlmann, {Eric Luis}",

year = "2016",

month = sep,

day = "1",

doi = "10.1016/j.jesp.2015.10.001",

language = "English",

volume = "66",

pages = "55--67",

journal = "Journal of Experimental Social Psychology",

issn = "0022-1031",

publisher = "Elsevier Academic Press Inc",

}

TY - JOUR

T1 - The pipeline project

T2 - Pre-publication independent replications of a single laboratory's research pipeline

AU - Schweinsberg, Martin

AU - Madan, Nikhil

AU - Vianello, Michelangelo

AU - Sommer, S. Amy

AU - Jordan, Jennifer

AU - Tierney, Warren

AU - Awtrey, Eli

AU - Zhu, Luke Lei

AU - Diermeier, Daniel

AU - Heinze, Justin E.

AU - Srinivasan, Malavika

AU - Tannenbaum, David

AU - Bivolaru, Eliza

AU - Dana, Jason

AU - Davis-Stober, Clintin P.

AU - du Plessis, Christilene

AU - Gronau, Quentin F.

AU - Hafenbrack, Andrew C.

AU - Liao, Eko Yi

AU - Ly, Alexander

AU - Marsman, Maarten

AU - Murase, Toshio

AU - Qureshi, Israr

AU - Schaerer, Michael

AU - Thornley, Nico

AU - Tworek, Christina M.

AU - Wagenmakers, Eric Jan

AU - Wong, Lynn

AU - Anderson, Tabitha

AU - Bauman, Christopher W.

AU - Bedwell, Wendy L.

AU - Brescoll, Victoria

AU - Canavan, Andrew

AU - Chandler, Jesse J.

AU - Cheries, Erik

AU - Cheryan, Sapna

AU - Cheung, Felix

AU - Cimpian, Andrei

AU - Clark, Mark A.

AU - Cordon, Diana

AU - Cushman, Fiery

AU - Ditto, Peter H.

AU - Donahue, Thomas

AU - Frick, Sarah E.

AU - Gamez-Djokic, Monica

AU - Grady, Rebecca Hofstein

AU - Graham, Jesse

AU - Gu, Jun

AU - Hahn, Adam

AU - Hanson, Brittany E.

AU - Hartwich, Nicole J.

AU - Hein, Kristie

AU - Inbar, Yoel

AU - Jiang, Lily

AU - Kellogg, Tehlyr

AU - Kennedy, Deanna M.

AU - Legate, Nicole

AU - Luoma, Timo P.

AU - Maibuecher, Heidi

AU - Meindl, Peter

AU - Miles, Jennifer

AU - Mislin, Alexandra

AU - Molden, Daniel C.

AU - Motyl, Matt

AU - Newman, George

AU - Ngo, Hoai Huong

AU - Packham, Harvey

AU - Ramsay, Philip S.

AU - Ray, Jennifer L.

AU - Sackett, Aaron M.

AU - Sellier, Anne Laure

AU - Sokolova, Tatiana

AU - Sowden, Walter

AU - Storage, Daniel

AU - Sun, Xiaomin

AU - Van Bavel, Jay J.

AU - Washburn, Anthony N.

AU - Wei, Cong

AU - Wetter, Erik

AU - Wilson, Carlos T.

AU - Darroux, Sophie Charlotte

AU - Uhlmann, Eric Luis

PY - 2016/9/1

Y1 - 2016/9/1

N2 - This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

AB - This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

KW - Crowdsourcing science

KW - Meta-science

KW - Methodology

KW - Replication

KW - Reproducibility

KW - Research transparency

UR - http://www.scopus.com/inward/record.url?scp=84977144634&partnerID=8YFLogxK

U2 - 10.1016/j.jesp.2015.10.001

DO - 10.1016/j.jesp.2015.10.001

M3 - Article

AN - SCOPUS:84977144634

SN - 0022-1031

VL - 66

SP - 55

EP - 67

JO - Journal of Experimental Social Psychology

JF - Journal of Experimental Social Psychology

ER -

The pipeline project: Pre-publication independent replications of a single laboratory's research pipeline

Abstract

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this