Toward a universal decoder of linguistic meaning from brain activation

Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences...

Full description

Saved in:
Bibliographic Details
Published in:Nature communications 2018-03, Vol.9 (1), p.963-13, Article 963
Main Authors: Pereira, Francisco, Lou, Bin, Pritchett, Brianna, Ritter, Samuel, Gershman, Samuel J., Kanwisher, Nancy, Botvinick, Matthew, Fedorenko, Evelina
Format: Article
Language:English
Subjects:
631/378/116/2394
639/705/1042
Adult
Brain
Brain Mapping
Decoding
Female
Humanities and Social Sciences
Humans
Imaging
Magnetic Resonance Imaging
Male
Mathematical analysis
multidisciplinary
Neuroimaging
Physical Stimulation
Representations
Science
Science (multidisciplinary)
Semantics
Sentences
Stimuli
Training
Vectors (mathematics)
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to generalize to new meanings from limited imaging data. To validate this approach, we train the system on imaging data of individual concepts, and show it can decode semantic vector representations from imaging data of sentences about a wide variety of both concrete and abstract topics from two separate datasets. These decoded representations are sufficiently detailed to distinguish even semantically similar sentences, and to capture the similarity structure of meaning relationships between sentences. Previous work decoding linguistic meaning from imaging data has generally been limited to a small number of semantic categories. Here, authors show that a decoder trained on neuroimaging data of single concepts sampling the semantic space can robustly decode meanings of semantically diverse new sentences with topics not encountered during training.
ISSN:2041-1723
2041-1723
DOI:10.1038/s41467-018-03068-4