Deep multimodal representation learning for noninvasive neural speech decoding

Decoding speech directly from brain activity is a rapidly developing research area with the potential to improve communication for those unable to speak. Traditionally, neural recording of speech processes has relied on unimodal data acquisition and signal decoding. Recent advances facilitating multimodal recording of brain activity require multimodal techniques for decoding that activity. Multimodal deep learning offers the possibility of learning joint representations of speech from distinct modalities trained in an end-to-end fashion. In this chapter, we present a multimodal neural network developed to decode speech from simultaneously recorded EEG and fNIRS. The network consists of two convolutional subnets designed to learn signal-specific features corresponding to overt and imagined speech, which are combined to enable the learning of a joint representation. Our model is compared with the unimodal baselines and exhibits enhanced decoding performance, with maximum decoding scores of 87.18% and 53.0% for overt and imagined speech, respectively. This novel architecture can be used to improve speech decoding using non-invasive neural signals.