Modelling the effects of subjective and objective decision making in scientific peer review

In-Uck Park, M W Peacey, Marcus Munafo

Research output: Contribution to journalLetter

51 Citations (Scopus)

Abstract

The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists' decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers' behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.
Original languageEnglish
Pages (from-to)93-96
Number of pages4
JournalNature
Volume506
Issue number7486
Early online date4 Dec 2013
DOIs
Publication statusPublished - 6 Feb 2014

Fingerprint

peer review
scientific community
decision making
gatekeeper
science
available information
aggregation
disposition
subjectivity

Cite this

Modelling the effects of subjective and objective decision making in scientific peer review. / Park, In-Uck; Peacey, M W; Munafo, Marcus.

In: Nature, Vol. 506, No. 7486, 06.02.2014, p. 93-96.

Research output: Contribution to journalLetter

Park, In-Uck ; Peacey, M W ; Munafo, Marcus. / Modelling the effects of subjective and objective decision making in scientific peer review. In: Nature. 2014 ; Vol. 506, No. 7486. pp. 93-96.
@article{5ae8948f7b3d49b0b9c6a917d25b50e9,
title = "Modelling the effects of subjective and objective decision making in scientific peer review",
abstract = "The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists' decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers' behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.",
author = "In-Uck Park and Peacey, {M W} and Marcus Munafo",
year = "2014",
month = "2",
day = "6",
doi = "10.1038/nature12786",
language = "English",
volume = "506",
pages = "93--96",
journal = "Nature",
issn = "0028-0836",
publisher = "Nature Research",
number = "7486",

}

TY - JOUR

T1 - Modelling the effects of subjective and objective decision making in scientific peer review

AU - Park, In-Uck

AU - Peacey, M W

AU - Munafo, Marcus

PY - 2014/2/6

Y1 - 2014/2/6

N2 - The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists' decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers' behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.

AB - The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists' decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers' behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.

UR - http://www.scopus.com/inward/record.url?scp=84893811448&partnerID=8YFLogxK

UR - http://dx.doi.org/10.1038/nature12786

UR - http://www.nature.com/nature/journal/vaop/ncurrent/full/nature12786.html

U2 - 10.1038/nature12786

DO - 10.1038/nature12786

M3 - Letter

VL - 506

SP - 93

EP - 96

JO - Nature

JF - Nature

SN - 0028-0836

IS - 7486

ER -