Deep Learning with Convolutional Neural Networks for Motor Brain-Computer Interfaces based on Stereo-electroencephalography (SEEG)

Xiaolong Wu; Shize Jiang; Guangye Li; Shengjie Liu; Benjamin Metcalfe; Liang Chen; Dingguo Zhang

doi:10.1109/JBHI.2023.3242262

Deep Learning with Convolutional Neural Networks for Motor Brain-Computer Interfaces based on Stereo-electroencephalography (SEEG)

Xiaolong Wu, Shize Jiang, Guangye Li, Shengjie Liu, Benjamin Metcalfe, Liang Chen, Dingguo Zhang

Research output: Contribution to journal › Article › peer-review

9 Downloads (Pure)

Abstract

Objective: Deep learning based on convolutional neural networks (CNN) has achieved success in brain-computer interfaces (BCIs) using scalp electroencephalography (EEG). However, the interpretation of the so-called ‘black box’ method and its application in stereo-electroencephalography (SEEG)-based BCIs remain largely unknown. Therefore, in this paper, an evaluation is performed on the decoding performance of deep learning methods on SEEG signals. <italic>Methods:</italic> Thirty epilepsy patients were recruited, and a paradigm including five hand and forearm motion types was designed. Six methods, including filter bank common spatial pattern (FBCSP) and five deep learning methods (EEGNet, shallow and deep CNN, ResNet, and a deep CNN variant named STSCNN), were used to classify the SEEG data. Various experiments were conducted to investigate the effect of windowing, model structure, and the decoding process of ResNet and STSCNN. <italic>Results:</italic> The average classification accuracy for EEGNet, FBCSP, shallow CNN, deep CNN, STSCNN, and ResNet were 35 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 6.1%, 38 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 4.9%, 60 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 3.9%, 60 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 3.3%, 61 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 3.2%, and 63 <inline-formula><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 3.1% respectively. Further analysis of the proposed method demonstrated clear separability between different classes in the spectral domain. <italic>Conclusion:</italic> ResNet and STSCNN achieved the first- and second-highest decoding accuracy, respectively. The STSCNN demonstrated that an extra spatial convolution layer was beneficial, and the decoding process can be partially interpreted from spatial and spectral perspectives. <italic>Significance:</italic> This study is the first to investigate the performance of deep learning on SEEG signals. In addition, this paper demonstrated that the so-called ‘black-box’ method can be partially interpreted.

Original language	English
Pages (from-to)	1-12
Number of pages	12
Journal	IEEE Journal of Biomedical and Health Informatics
Early online date	6 Feb 2023
DOIs	https://doi.org/10.1109/JBHI.2023.3242262
Publication status	Published - 3 Apr 2023

Keywords

Brain modeling
brain-computer interface (BCI)
Convolution
Convolutional neural networks
convolutional neural networks (CNN)
Decoding
Deep learning
deep learning
Electrodes
forearm and hand motion
stereo-electroencephalography (SEEG)
Temporal lobe

ASJC Scopus subject areas

Computer Science Applications
Health Informatics
Electrical and Electronic Engineering
Health Information Management

Access to Document

10.1109/JBHI.2023.3242262

Deep_Learning_with_Convolutional_Neural_Networks_for_Motor_Brain-Computer_Interfaces_based_on_Stereo-electroencephalography_SEEG
© 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works. https://doi.org/10.1109/JBHI.2023.3242262
Accepted author manuscript, 5.87 MB

Cite this

@article{8f52f6c51ed9481c8057f54f10cca0cd,

title = "Deep Learning with Convolutional Neural Networks for Motor Brain-Computer Interfaces based on Stereo-electroencephalography (SEEG)",

abstract = "Objective: Deep learning based on convolutional neural networks (CNN) has achieved success in brain-computer interfaces (BCIs) using scalp electroencephalography (EEG). However, the interpretation of the so-called {\textquoteleft}black box{\textquoteright} method and its application in stereo-electroencephalography (SEEG)-based BCIs remain largely unknown. Therefore, in this paper, an evaluation is performed on the decoding performance of deep learning methods on SEEG signals. Methods: Thirty epilepsy patients were recruited, and a paradigm including five hand and forearm motion types was designed. Six methods, including filter bank common spatial pattern (FBCSP) and five deep learning methods (EEGNet, shallow and deep CNN, ResNet, and a deep CNN variant named STSCNN), were used to classify the SEEG data. Various experiments were conducted to investigate the effect of windowing, model structure, and the decoding process of ResNet and STSCNN. Results: The average classification accuracy for EEGNet, FBCSP, shallow CNN, deep CNN, STSCNN, and ResNet were 35 $\pm$ 6.1%, 38 $\pm$ 4.9%, 60 $\pm$ 3.9%, 60 $\pm$ 3.3%, 61 $\pm$ 3.2%, and 63 $\pm$ 3.1% respectively. Further analysis of the proposed method demonstrated clear separability between different classes in the spectral domain. Conclusion: ResNet and STSCNN achieved the first- and second-highest decoding accuracy, respectively. The STSCNN demonstrated that an extra spatial convolution layer was beneficial, and the decoding process can be partially interpreted from spatial and spectral perspectives. Significance: This study is the first to investigate the performance of deep learning on SEEG signals. In addition, this paper demonstrated that the so-called {\textquoteleft}black-box{\textquoteright} method can be partially interpreted.",

keywords = "Brain modeling, brain-computer interface (BCI), Convolution, Convolutional neural networks, convolutional neural networks (CNN), Decoding, Deep learning, deep learning, Electrodes, forearm and hand motion, stereo-electroencephalography (SEEG), Temporal lobe",

author = "Xiaolong Wu and Shize Jiang and Guangye Li and Shengjie Liu and Benjamin Metcalfe and Liang Chen and Dingguo Zhang",

note = "Funding EPSRC New Horizons Grant of UK 10.13039/501100001809-National Natural Science Foundation of China 10.13039/501100002858-China Postdoctoral Science Foundation (Grant Number: 20Z102060158) Medical & Engineering Cross Foundation of SJTU (Grant Number: AH0200003)",

year = "2023",

month = apr,

day = "3",

doi = "10.1109/JBHI.2023.3242262",

language = "English",

pages = "1--12",

journal = "IEEE Journal of Biomedical and Health Informatics",

issn = "2168-2194",

publisher = "IEEE",

}

TY - JOUR

T1 - Deep Learning with Convolutional Neural Networks for Motor Brain-Computer Interfaces based on Stereo-electroencephalography (SEEG)

AU - Wu, Xiaolong

AU - Jiang, Shize

AU - Li, Guangye

AU - Liu, Shengjie

AU - Metcalfe, Benjamin

AU - Chen, Liang

AU - Zhang, Dingguo

N1 - Funding EPSRC New Horizons Grant of UK 10.13039/501100001809-National Natural Science Foundation of China 10.13039/501100002858-China Postdoctoral Science Foundation (Grant Number: 20Z102060158) Medical & Engineering Cross Foundation of SJTU (Grant Number: AH0200003)

PY - 2023/4/3

Y1 - 2023/4/3

N2 - Objective: Deep learning based on convolutional neural networks (CNN) has achieved success in brain-computer interfaces (BCIs) using scalp electroencephalography (EEG). However, the interpretation of the so-called ‘black box’ method and its application in stereo-electroencephalography (SEEG)-based BCIs remain largely unknown. Therefore, in this paper, an evaluation is performed on the decoding performance of deep learning methods on SEEG signals. Methods: Thirty epilepsy patients were recruited, and a paradigm including five hand and forearm motion types was designed. Six methods, including filter bank common spatial pattern (FBCSP) and five deep learning methods (EEGNet, shallow and deep CNN, ResNet, and a deep CNN variant named STSCNN), were used to classify the SEEG data. Various experiments were conducted to investigate the effect of windowing, model structure, and the decoding process of ResNet and STSCNN. Results: The average classification accuracy for EEGNet, FBCSP, shallow CNN, deep CNN, STSCNN, and ResNet were 35 $\pm$ 6.1%, 38 $\pm$ 4.9%, 60 $\pm$ 3.9%, 60 $\pm$ 3.3%, 61 $\pm$ 3.2%, and 63 $\pm$ 3.1% respectively. Further analysis of the proposed method demonstrated clear separability between different classes in the spectral domain. Conclusion: ResNet and STSCNN achieved the first- and second-highest decoding accuracy, respectively. The STSCNN demonstrated that an extra spatial convolution layer was beneficial, and the decoding process can be partially interpreted from spatial and spectral perspectives. Significance: This study is the first to investigate the performance of deep learning on SEEG signals. In addition, this paper demonstrated that the so-called ‘black-box’ method can be partially interpreted.

AB - Objective: Deep learning based on convolutional neural networks (CNN) has achieved success in brain-computer interfaces (BCIs) using scalp electroencephalography (EEG). However, the interpretation of the so-called ‘black box’ method and its application in stereo-electroencephalography (SEEG)-based BCIs remain largely unknown. Therefore, in this paper, an evaluation is performed on the decoding performance of deep learning methods on SEEG signals. Methods: Thirty epilepsy patients were recruited, and a paradigm including five hand and forearm motion types was designed. Six methods, including filter bank common spatial pattern (FBCSP) and five deep learning methods (EEGNet, shallow and deep CNN, ResNet, and a deep CNN variant named STSCNN), were used to classify the SEEG data. Various experiments were conducted to investigate the effect of windowing, model structure, and the decoding process of ResNet and STSCNN. Results: The average classification accuracy for EEGNet, FBCSP, shallow CNN, deep CNN, STSCNN, and ResNet were 35 $\pm$ 6.1%, 38 $\pm$ 4.9%, 60 $\pm$ 3.9%, 60 $\pm$ 3.3%, 61 $\pm$ 3.2%, and 63 $\pm$ 3.1% respectively. Further analysis of the proposed method demonstrated clear separability between different classes in the spectral domain. Conclusion: ResNet and STSCNN achieved the first- and second-highest decoding accuracy, respectively. The STSCNN demonstrated that an extra spatial convolution layer was beneficial, and the decoding process can be partially interpreted from spatial and spectral perspectives. Significance: This study is the first to investigate the performance of deep learning on SEEG signals. In addition, this paper demonstrated that the so-called ‘black-box’ method can be partially interpreted.

KW - Brain modeling

KW - brain-computer interface (BCI)

KW - Convolution

KW - Convolutional neural networks

KW - convolutional neural networks (CNN)

KW - Decoding

KW - Deep learning

KW - deep learning

KW - Electrodes

KW - forearm and hand motion

KW - stereo-electroencephalography (SEEG)

KW - Temporal lobe

UR - http://www.scopus.com/inward/record.url?scp=85148434239&partnerID=8YFLogxK

U2 - 10.1109/JBHI.2023.3242262

DO - 10.1109/JBHI.2023.3242262

M3 - Article

AN - SCOPUS:85148434239

SN - 2168-2194

SP - 1

EP - 12

JO - IEEE Journal of Biomedical and Health Informatics

JF - IEEE Journal of Biomedical and Health Informatics

ER -

Deep Learning with Convolutional Neural Networks for Motor Brain-Computer Interfaces based on Stereo-electroencephalography (SEEG)

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this