Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models

Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a com...

Full description

Saved in:
Bibliographic Details
Published in:Journal of neural engineering 2023-10, Vol.20 (5), p.56010
Main Authors: Berezutskaya, Julia, Freudenburg, Zachary V, Vansteensel, Mariska J, Aarnoutse, Erik J, Ramsey, Nick F, van Gerven, Marcel A J
Format: Article
Language:English
Subjects:
audio reconstruction
brain
Brain-Computer Interfaces
Communication
Deep Learning
deep neural networks
electrocorticography
Electrocorticography - methods
Humans
neural decoding
Sensorimotor Cortex
Speech
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field. In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task. We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%-100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech. These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.
ISSN:1741-2560
1741-2552
DOI:10.1088/1741-2552/ace8be