Performance of data-driven inner speech decoding with same-task EEG-fMRI data fusion and bimodal models

Decoding inner speech from the brain signal via hybridisation of fMRI and EEG data is explored to investigate the performance benefits over unimodal models. Two different bimodal fusion approaches are examined: concatenation of probability vectors output from unimodal fMRI and EEG machine learning m...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-06
Main Authors: Wilson, Holly, Wellington, Scott, Liwicki, Foteini Simistira, Gupta, Vibha, Saini, Rajkumar, De, Kanjar, Abid, Nosheen, Rakesh, Sumit, Eriksson, Johan, Watts, Oliver, Chen, Xi, Golbabaee, Mohammad, Proulx, Michael J, Liwicki, Marcus, O'Neill, Eamonn, Metcalfe, Benjamin
Format: Article
Language:English
Subjects:
Data integration
Electroencephalography
Machine learning
Magnetic resonance imaging
Speech
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Decoding inner speech from the brain signal via hybridisation of fMRI and EEG data is explored to investigate the performance benefits over unimodal models. Two different bimodal fusion approaches are examined: concatenation of probability vectors output from unimodal fMRI and EEG machine learning models, and data fusion with feature engineering. Same task inner speech data are recorded from four participants, and different processing strategies are compared and contrasted to previously-employed hybridisation methods. Data across participants are discovered to encode different underlying structures, which results in varying decoding performances between subject-dependent fusion models. Decoding performance is demonstrated as improved when pursuing bimodal fMRI-EEG fusion strategies, if the data show underlying structure.
ISSN:2331-8422