Brain2Pix: Fully convolutional naturalistic video frame reconstruction from brain activity

Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here, we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance imagin...

Full description

Saved in:
Bibliographic Details
Published in:Frontiers in neuroscience 2022-11, Vol.16, p.940972
Main Authors: Le, Lynn, Ambrogioni, Luca, Seeliger, Katja, Güçlütürk, Yağmur, van Gerven, Marcel, Güçlü, Umut
Format: Article
Language:English
Subjects:
Brain mapping
decoding
Deep learning
fMRI
Functional magnetic resonance imaging
Image processing
Mapping
Nervous system
Neural networks
Neuroimaging
Neuroscience
Receptive field
Retina
Semantics
Topography
vision
Visual field
Visual perception
visual reconstruction
Visual system
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here, we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance imaging data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent in the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments, we show that our method offers a significant improvement over existing video reconstruction techniques.
ISSN:1662-4548
1662-453X
1662-453X
DOI:10.3389/fnins.2022.940972