Replication Validity of Initial Association Studies: A Comparison between Psychiatry, Neurology and Four Somatic Diseases

Estelle Dumas-Mallet, Katherine Button, Thomas Boraud, Marcus Munafo, François Gonon

Research output: Contribution to journalArticle

  • 7 Citations

Abstract

CONTEXT: There are growing concerns about effect size inflation and replication validity of association studies, but few observational investigations have explored the extent of these problems.

OBJECTIVE: Using meta-analyses to measure the reliability of initial studies and explore whether this varies across biomedical domains and study types (cognitive/behavioral, brain imaging, genetic and "others").

METHODS: We analyzed 663 meta-analyses describing associations between markers or risk factors and 12 pathologies within three biomedical domains (psychiatry, neurology and four somatic diseases). We collected the effect size, sample size, publication year and Impact Factor of initial studies, largest studies (i.e., with the largest sample size) and the corresponding meta-analyses. Initial studies were considered as replicated if they were in nominal agreement with meta-analyses and if their effect size inflation was below 100%.

RESULTS: Nominal agreement between initial studies and meta-analyses regarding the presence of a significant effect was not better than chance in psychiatry, whereas it was somewhat better in neurology and somatic diseases. Whereas effect sizes reported by largest studies and meta-analyses were similar, most of those reported by initial studies were inflated. Among the 256 initial studies reporting a significant effect (p<0.05) and paired with significant meta-analyses, 97 effect sizes were inflated by more than 100%. Nominal agreement and effect size inflation varied with the biomedical domain and study type. Indeed, the replication rate of initial studies reporting a significant effect ranged from 6.3% for genetic studies in psychiatry to 86.4% for cognitive/behavioral studies. Comparison between eight subgroups shows that replication rate decreases with sample size and "true" effect size. We observed no evidence of association between replication rate and publication year or Impact Factor.

CONCLUSION: The differences in reliability between biological psychiatry, neurology and somatic diseases suggest that there is room for improvement, at least in some subdomains.

LanguageEnglish
Pagese0158064
JournalPLoS ONE
Volume11
Issue number6
DOIs
StatusPublished - 23 Jun 2016

Fingerprint

inflation
Neurology
Psychiatry
Meta-Analysis
Economic Inflation
Sample Size
sampling
Pathology
risk factors
Publications
image analysis
brain
Biological Psychiatry
Brain
Imaging techniques
Neuroimaging
Observational Studies

Cite this

Replication Validity of Initial Association Studies : A Comparison between Psychiatry, Neurology and Four Somatic Diseases. / Dumas-Mallet, Estelle; Button, Katherine; Boraud, Thomas; Munafo, Marcus; Gonon, François.

In: PLoS ONE, Vol. 11, No. 6, 23.06.2016, p. e0158064.

Research output: Contribution to journalArticle

Dumas-Mallet, Estelle ; Button, Katherine ; Boraud, Thomas ; Munafo, Marcus ; Gonon, François. / Replication Validity of Initial Association Studies : A Comparison between Psychiatry, Neurology and Four Somatic Diseases. In: PLoS ONE. 2016 ; Vol. 11, No. 6. pp. e0158064.
@article{940a743e859f4c44ba621258ce15dc4c,
title = "Replication Validity of Initial Association Studies: A Comparison between Psychiatry, Neurology and Four Somatic Diseases",
abstract = "CONTEXT: There are growing concerns about effect size inflation and replication validity of association studies, but few observational investigations have explored the extent of these problems.OBJECTIVE: Using meta-analyses to measure the reliability of initial studies and explore whether this varies across biomedical domains and study types (cognitive/behavioral, brain imaging, genetic and {"}others{"}).METHODS: We analyzed 663 meta-analyses describing associations between markers or risk factors and 12 pathologies within three biomedical domains (psychiatry, neurology and four somatic diseases). We collected the effect size, sample size, publication year and Impact Factor of initial studies, largest studies (i.e., with the largest sample size) and the corresponding meta-analyses. Initial studies were considered as replicated if they were in nominal agreement with meta-analyses and if their effect size inflation was below 100{\%}.RESULTS: Nominal agreement between initial studies and meta-analyses regarding the presence of a significant effect was not better than chance in psychiatry, whereas it was somewhat better in neurology and somatic diseases. Whereas effect sizes reported by largest studies and meta-analyses were similar, most of those reported by initial studies were inflated. Among the 256 initial studies reporting a significant effect (p<0.05) and paired with significant meta-analyses, 97 effect sizes were inflated by more than 100{\%}. Nominal agreement and effect size inflation varied with the biomedical domain and study type. Indeed, the replication rate of initial studies reporting a significant effect ranged from 6.3{\%} for genetic studies in psychiatry to 86.4{\%} for cognitive/behavioral studies. Comparison between eight subgroups shows that replication rate decreases with sample size and {"}true{"} effect size. We observed no evidence of association between replication rate and publication year or Impact Factor.CONCLUSION: The differences in reliability between biological psychiatry, neurology and somatic diseases suggest that there is room for improvement, at least in some subdomains.",
author = "Estelle Dumas-Mallet and Katherine Button and Thomas Boraud and Marcus Munafo and Fran{\cc}ois Gonon",
year = "2016",
month = "6",
day = "23",
doi = "10.1371/journal.pone.0158064",
language = "English",
volume = "11",
pages = "e0158064",
journal = "PLoS ONE",
issn = "1932-6203",
publisher = "Public Library of Science (PLOS)",
number = "6",

}

TY - JOUR

T1 - Replication Validity of Initial Association Studies

T2 - PLoS ONE

AU - Dumas-Mallet, Estelle

AU - Button, Katherine

AU - Boraud, Thomas

AU - Munafo, Marcus

AU - Gonon, François

PY - 2016/6/23

Y1 - 2016/6/23

N2 - CONTEXT: There are growing concerns about effect size inflation and replication validity of association studies, but few observational investigations have explored the extent of these problems.OBJECTIVE: Using meta-analyses to measure the reliability of initial studies and explore whether this varies across biomedical domains and study types (cognitive/behavioral, brain imaging, genetic and "others").METHODS: We analyzed 663 meta-analyses describing associations between markers or risk factors and 12 pathologies within three biomedical domains (psychiatry, neurology and four somatic diseases). We collected the effect size, sample size, publication year and Impact Factor of initial studies, largest studies (i.e., with the largest sample size) and the corresponding meta-analyses. Initial studies were considered as replicated if they were in nominal agreement with meta-analyses and if their effect size inflation was below 100%.RESULTS: Nominal agreement between initial studies and meta-analyses regarding the presence of a significant effect was not better than chance in psychiatry, whereas it was somewhat better in neurology and somatic diseases. Whereas effect sizes reported by largest studies and meta-analyses were similar, most of those reported by initial studies were inflated. Among the 256 initial studies reporting a significant effect (p<0.05) and paired with significant meta-analyses, 97 effect sizes were inflated by more than 100%. Nominal agreement and effect size inflation varied with the biomedical domain and study type. Indeed, the replication rate of initial studies reporting a significant effect ranged from 6.3% for genetic studies in psychiatry to 86.4% for cognitive/behavioral studies. Comparison between eight subgroups shows that replication rate decreases with sample size and "true" effect size. We observed no evidence of association between replication rate and publication year or Impact Factor.CONCLUSION: The differences in reliability between biological psychiatry, neurology and somatic diseases suggest that there is room for improvement, at least in some subdomains.

AB - CONTEXT: There are growing concerns about effect size inflation and replication validity of association studies, but few observational investigations have explored the extent of these problems.OBJECTIVE: Using meta-analyses to measure the reliability of initial studies and explore whether this varies across biomedical domains and study types (cognitive/behavioral, brain imaging, genetic and "others").METHODS: We analyzed 663 meta-analyses describing associations between markers or risk factors and 12 pathologies within three biomedical domains (psychiatry, neurology and four somatic diseases). We collected the effect size, sample size, publication year and Impact Factor of initial studies, largest studies (i.e., with the largest sample size) and the corresponding meta-analyses. Initial studies were considered as replicated if they were in nominal agreement with meta-analyses and if their effect size inflation was below 100%.RESULTS: Nominal agreement between initial studies and meta-analyses regarding the presence of a significant effect was not better than chance in psychiatry, whereas it was somewhat better in neurology and somatic diseases. Whereas effect sizes reported by largest studies and meta-analyses were similar, most of those reported by initial studies were inflated. Among the 256 initial studies reporting a significant effect (p<0.05) and paired with significant meta-analyses, 97 effect sizes were inflated by more than 100%. Nominal agreement and effect size inflation varied with the biomedical domain and study type. Indeed, the replication rate of initial studies reporting a significant effect ranged from 6.3% for genetic studies in psychiatry to 86.4% for cognitive/behavioral studies. Comparison between eight subgroups shows that replication rate decreases with sample size and "true" effect size. We observed no evidence of association between replication rate and publication year or Impact Factor.CONCLUSION: The differences in reliability between biological psychiatry, neurology and somatic diseases suggest that there is room for improvement, at least in some subdomains.

UR - http://dx.doi.org/10.1371/journal.pone.0158064

U2 - 10.1371/journal.pone.0158064

DO - 10.1371/journal.pone.0158064

M3 - Article

VL - 11

SP - e0158064

JO - PLoS ONE

JF - PLoS ONE

SN - 1932-6203

IS - 6

ER -