Abstract
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
| Original language | English |
|---|---|
| Article number | aac4716 |
| Journal | Science |
| Volume | 349 |
| Issue number | 6251 |
| DOIs | |
| Publication status | Published - 28 Aug 2015 |
ASJC Scopus subject areas
- General
Fingerprint
Dive into the research topics of 'Estimating the reproducibility of psychological science'. Together they form a unique fingerprint.Cite this
- APA
- Standard
- Harvard
- Vancouver
- Author
- BIBTEX
- RIS
In: Science, Vol. 349, No. 6251, aac4716, 28.08.2015.
Research output: Contribution to journal › Article › peer-review
}
TY - JOUR
T1 - Estimating the reproducibility of psychological science
AU - Aarts, Alexander A.
AU - Anderson, Joanna E.
AU - Anderson, Christopher J.
AU - Attridge, Peter R.
AU - Attwood, Angela
AU - Axt, Jordan
AU - Babel, Molly
AU - Bahník, Štěpán
AU - Baranski, Erica
AU - Barnett-Cowan, Michael
AU - Bartmess, Elizabeth
AU - Beer, Jennifer
AU - Bell, Raoul
AU - Bentley, Heather
AU - Beyan, Leah
AU - Binion, Grace
AU - Borsboom, Denny
AU - Bosch, Annick
AU - Bosco, Frank A.
AU - Bowman, Sara D.
AU - Brandt, Mark J.
AU - Braswell, Erin
AU - Brohmer, Hilmar
AU - Brown, Benjamin T.
AU - Brown, Kristina
AU - Brüning, Jovita
AU - Calhoun-Sauls, Ann
AU - Callahan, Shannon P.
AU - Chagnon, Elizabeth
AU - Chandler, Jesse
AU - Chartier, Christopher R.
AU - Cheung, Felix
AU - Christopherson, Cody D.
AU - Cillessen, Linda
AU - Clay, Russ
AU - Cleary, Hayley
AU - Cloud, Mark D.
AU - Conn, Michael
AU - Cohoon, Johanna
AU - Columbus, Simon
AU - Cordes, Andreas
AU - Costantini, Giulio
AU - Alvarez, Leslie D.Cramblet
AU - Cremata, Ed
AU - Crusius, Jan
AU - DeCoster, Jamie
AU - DeGaetano, Michelle A.
AU - Penna, Nicolás Delia
AU - Den Bezemer, Bobby
AU - Deserno, Marie K.
AU - Devitt, Olivia
AU - Dewitte, Laura
AU - Dobolyi, David G.
AU - Dodson, Geneva T.
AU - Donnellan, M. Brent
AU - Donohue, Ryan
AU - Dore, Rebecca A.
AU - Dorrough, Angela
AU - Dreber, Anna
AU - Dugas, Michelle
AU - Dunn, Elizabeth W.
AU - Easey, Kayleigh
AU - Eboigbe, Sylvia
AU - Eggleston, Casey
AU - Embley, Jo
AU - Epskamp, Sacha
AU - Errington, Timothy M.
AU - Estel, Vivien
AU - Farach, Frank J.
AU - Feather, Jenelle
AU - Fedor, Anna
AU - Fernández-Castilla, Belén
AU - Fiedler, Susann
AU - Field, James G.
AU - Fitneva, Stanka A.
AU - Flagan, Taru
AU - Forest, Amanda L.
AU - Forsell, Eskil
AU - Foster, Joshua D.
AU - Frank, Michael C.
AU - Frazier, Rebecca S.
AU - Fuchs, Heather
AU - Gable, Philip
AU - Galak, Jeff
AU - Galliani, Elisa Maria
AU - Gampa, Anup
AU - Garcia, Sara
AU - Gazarian, Douglas
AU - Gilbert, Elizabeth
AU - Giner-Sorolla, Roger
AU - Glöckner, Andreas
AU - Goellner, Lars
AU - Goh, Jin X.
AU - Goldberg, Rebecca
AU - Goodbourn, Patrick T.
AU - Gordon-McKeon, Shauna
AU - Gorges, Bryan
AU - Gorges, Jessie
AU - Goss, Justin
AU - Graham, Jesse
AU - Grange, James A.
AU - Gray, Jeremy
AU - Hartgerink, Chris
AU - Hartshorne, Joshua
AU - Hasselman, Fred
AU - Hayes, Timothy
AU - Heikensten, Emma
AU - Henninger, Felix
AU - Hodsoll, John
AU - Holubar, Taylor
AU - Hoogendoorn, Gea
AU - Humphries, Denise J.
AU - Hung, Cathy O.Y.
AU - Immelman, Nathali
AU - Irsik, Vanessa C.
AU - Jahn, Georg
AU - Jäkel, Frank
AU - Jekel, Marc
AU - Johannesson, Magnus
AU - Johnson, Larissa G.
AU - Johnson, David J.
AU - Johnson, Kate M.
AU - Johnston, William J.
AU - Jonas, Kai
AU - Joy-Gaba, Jennifer A.
AU - Kappes, Heather Barry
AU - Kelso, Kim
AU - Kidwell, Mallory C.
AU - Kim, Seung Kyung
AU - Kirkhart, Matthew
AU - Kleinberg, Bennett
AU - Knežević, Goran
AU - Kolorz, Franziska Maria
AU - Kossakowski, Jolanda J.
AU - Krause, Robert Wilhelm
AU - Krijnen, Job
AU - Kuhlmann, Tim
AU - Kunkels, Yoram K.
AU - Kyc, Megan M.
AU - Lai, Calvin K.
AU - Laique, Aamir
AU - Lakens, Daniël
AU - Lane, Kristin A.
AU - Lassetter, Bethany
AU - Lazarević, Ljiljana B.
AU - Le Bel, Etienne P.
AU - Lee, Key Jung
AU - Lee, Minha
AU - Lemm, Kristi
AU - Levitan, Carmel A.
AU - Lewis, Melissa
AU - Lin, Lin
AU - Lin, Stephanie
AU - Lippold, Matthias
AU - Loureiro, Darren
AU - Luteijn, Ilse
AU - MacKinnon, Sean
AU - Mainard, Heather N.
AU - Marigold, Denise C.
AU - Martin, Daniel P.
AU - Martinez, Tylar
AU - Masicampo, E. J.
AU - Matacotta, Josh
AU - Mathur, Maya
AU - May, Michael
AU - Mechin, Nicole
AU - Mehta, Pranjal
AU - Meixner, Johannes
AU - Melinger, Alissa
AU - Miller, Jeremy K.
AU - Miller, Mallorie
AU - Moore, Katherine
AU - Möschl, Marcus
AU - Motyl, Matt
AU - Müller, Stephanie M.
AU - Munafo, Marcus
AU - Neijenhuijs, Koen I.
AU - Nervi, Taylor
AU - Nicolas, Gandalf
AU - Nilsonne, Gustav
AU - Nosek, Brian A.
AU - Nuijten, Michèle B.
AU - Olsson, Catherine
AU - Osborne, Colleen
AU - Ostkamp, Lutz
AU - Pavel, Misha
AU - Penton-Voak, Ian S.
AU - Perna, Olivia
AU - Pernet, Cyril
AU - Perugini, Marco
AU - Pipitone, R. Nathan
AU - Pitts, Michael
AU - Plessow, Franziska
AU - Prenoveau, Jason M.
AU - Rahal, Rima Maria
AU - Ratliff, Kate A.
AU - Reinhard, David
AU - Renkewitz, Frank
AU - Ricker, Ashley A.
AU - Rigney, Anastasia
AU - Rivers, Andrew M.
AU - Roebke, Mark
AU - Rutchick, Abraham M.
AU - Ryan, Robert S.
AU - Sahin, Onur
AU - Saide, Anondah
AU - Sandstrom, Gillian M.
AU - Santos, David
AU - Saxe, Rebecca
AU - Schlegelmilch, René
AU - Schmidt, Kathleen
AU - Scholz, Sabine
AU - Seibel, Larissa
AU - Selterman, Dylan Faulkner
AU - Shaki, Samuel
AU - Simpson, William B.
AU - Sinclair, H. Colleen
AU - Skorinko, Jeanine L.M.
AU - Slowik, Agnieszka
AU - Snyder, Joel S.
AU - Soderberg, Courtney
AU - Sonnleitner, Carina
AU - Spencer, Nick
AU - Spies, Jeffrey R.
AU - Steegen, Sara
AU - Stieger, Stefan
AU - Strohminger, Nina
AU - Sullivan, Gavin B.
AU - Talhelm, Thomas
AU - Tapia, Megan
AU - Te Dorsthorst, Anniek
AU - Thomae, Manuela
AU - Thomas, Sarah L.
AU - Tio, Pia
AU - Traets, Frits
AU - Tsang, Steve
AU - Tuerlinckx, Francis
AU - Turchan, Paul
AU - Valášek, Milan
AU - Van't Veer, Anna E.
AU - Van Aert, Robbie
AU - Van Assen, Marcel
AU - Van Bork, Riet
AU - Van De Ven, Mathijs
AU - Van Den Bergh, Don
AU - Van Der Hulst, Marije
AU - Van Dooren, Roel
AU - Van Doorn, Johnny
AU - Van Renswoude, Daan R.
AU - Van Rijn, Hedderik
AU - Vanpaemel, Wolf
AU - Echeverría, Alejandro Vásquez
AU - Vazquez, Melissa
AU - Velez, Natalia
AU - Vermue, Marieke
AU - Verschoor, Mark
AU - Vianello, Michelangelo
AU - Voracek, Martin
AU - Vuu, Gina
AU - Wagenmakers, Eric Jan
AU - Weerdmeester, Joanneke
AU - Welsh, Ashlee
AU - Westgate, Erin C.
AU - Wissink, Joeri
AU - Wood, Michael
AU - Woods, Andy
AU - Wright, Emily
AU - Wu, Sining
AU - Zeelenberg, Marcel
AU - Zuni, Kellylynn
PY - 2015/8/28
Y1 - 2015/8/28
N2 - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
AB - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
UR - https://www.scopus.com/pages/publications/84940513037
U2 - 10.1126/science.aac4716
DO - 10.1126/science.aac4716
M3 - Article
C2 - 26315443
AN - SCOPUS:84940513037
SN - 0036-8075
VL - 349
JO - Science
JF - Science
IS - 6251
M1 - aac4716
ER -