Multiple Holdouts With Stability: Improving the Generalizability of Machine Learning Analyses of Brain–Behavior Relationships

NeuroScience in Psychiatry Network (NSPN) Consortium, Janaina Mourao-Miranda

Research output: Contribution to journalArticlepeer-review

36 Citations (SciVal)

Abstract

Background: In 2009, the National Institute of Mental Health launched the Research Domain Criteria, an attempt to move beyond diagnostic categories and ground psychiatry within neurobiological constructs that combine different levels of measures (e.g., brain imaging and behavior). Statistical methods that can integrate such multimodal data, however, are often vulnerable to overfitting, poor generalization, and difficulties in interpreting the results. Methods: We propose an innovative machine learning framework combining multiple holdouts and a stability criterion with regularized multivariate techniques, such as sparse partial least squares and kernel canonical correlation analysis, for identifying hidden dimensions of cross-modality relationships. To illustrate the approach, we investigated structural brain–behavior associations in an extensively phenotyped developmental sample of 345 participants (312 healthy and 33 with clinical depression). The brain data consisted of whole-brain voxel-based gray matter volumes, and the behavioral data included item-level self-report questionnaires and IQ and demographic measures. Results: Both sparse partial least squares and kernel canonical correlation analysis captured two hidden dimensions of brain–behavior relationships: one related to age and drinking and the other one related to depression. The applied machine learning framework indicates that these results are stable and generalize well to new data. Indeed, the identified brain–behavior associations are in agreement with previous findings in the literature concerning age, alcohol use, and depression-related changes in brain volume. Conclusions: Multivariate techniques (such as sparse partial least squares and kernel canonical correlation analysis) embedded in our novel framework are promising tools to link behavior and/or symptoms to neurobiology and thus have great potential to contribute to a biologically grounded definition of psychiatric disorders.

Original languageEnglish
Pages (from-to)368-376
Number of pages9
JournalBiological Psychiatry
Volume87
Issue number4
Early online date10 Dec 2019
DOIs
Publication statusPublished - 15 Feb 2020

Funding

This work was supported by a Wellcome Trust Strategic Award (No. 095844 [to IMG, RD, ETB, PBJ, and PF]) that provides core funding for the NSPN . Scanning at the Wellcome Centre for Human Neuroimaging was funded under Grant No. 203147/Z/16/Z. AM, MJR, and JM-M were funded by the Wellcome Trust under Grant No. WT102845/Z/13/Z. FSF was funded by a Ph.D. scholarship awarded by Fundacao para a Ciencia e a Tecnologia (No. SFRH/BD/ 120640 /2016). MM received support from the University College London Hospitals (UCLH) National Institute for Health Research (NIHR) Biomedical Research Centre (BRC). RAA was supported by a Medical Research Council (MRC) Skills Development Fellowship (Grant No. MR/ S007806 /1). PF was in receipt of an NIHR Senior Investigator Award (Grant No. NF-SI-0514-10157) and was in part supported by the NIHR Collaboration for Leadership in Applied Health Research and Care (CLAHRC) North Thames at Barts Health NHS Trust. The views expressed are those of the authors and not necessarily those of the National Health Service, the NIHR, or the Department of Health. ETB is employed half-time by the University of Cambridge and half-time by GlaxoSmithKline, and he holds stock in GlaxoSmithKline. The other authors report no biomedical financial interests or potential conflicts of interest.

Keywords

  • Adolescence
  • Brain–behavior relationship
  • Depression
  • Framework
  • RDoC
  • SPLS

ASJC Scopus subject areas

  • Biological Psychiatry

Fingerprint

Dive into the research topics of 'Multiple Holdouts With Stability: Improving the Generalizability of Machine Learning Analyses of Brain–Behavior Relationships'. Together they form a unique fingerprint.

Cite this