3D View Prediction Models of the Dorsal Visual Stream

Deep neural network representations align well with brain activity in the ventral visual stream. However, the primate visual system has a distinct dorsal processing stream with different functional properties. To test if a model trained to perceive 3D scene geometry aligns better with neural respons...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-09
Main Authors: Sarch, Gabriel, Hsiao-Yu, Fish Tung, Wang, Aria, Prince, Jacob, Tarr, Michael
Format: Article
Language:English
Subjects:
Artificial neural networks
Brain
Prediction models
Recurrent neural networks
Three dimensional models
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Deep neural network representations align well with brain activity in the ventral visual stream. However, the primate visual system has a distinct dorsal processing stream with different functional properties. To test if a model trained to perceive 3D scene geometry aligns better with neural responses in dorsal visual areas, we trained a self-supervised geometry-aware recurrent neural network (GRNN) to predict novel camera views using a 3D feature memory. We compared GRNN to self-supervised baseline models that have been shown to align well with ventral regions using the large-scale fMRI Natural Scenes Dataset (NSD). We found that while the baseline models accounted better for ventral brain regions, GRNN accounted for a greater proportion of variance in dorsal brain regions. Our findings demonstrate the potential for using task-relevant models to probe representational differences across visual streams.
ISSN:2331-8422