Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision

• Published in: Human brain mapping 2018-05, Vol.39 (5), p.2269-2282
• Main Authors: Shi, Junxing, Wen, Haiguang, Zhang, Yizhen, Han, Kuan, Liu, Zhongming
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Brain; Brain Mapping; Computation; Computational neuroscience; Cortex (temporal); deep learning; Distributed memory; Eye Movements; Feature extraction; Feature recognition; Female; Humans; Image Processing, Computer-Assisted; Information processing; Learning; Magnetic Resonance Imaging; Male; Memory; Memory - physiology; Models, Neurological; natural vision; neural encoding; Neural networks; Oxygen - blood; process memory; Recognition, Psychology; recurrent neural network; Recurrent neural networks; Representations; temporal receptive window; Vision; Vision, Ocular - physiology; Visual cortex; Visual observation; Visual Pathways - diagnostic imaging; Visual Pathways - physiology; Visual perception; Visual stimuli; Visual task performance
• ISSN: 1065-9471; 1097-0193
• DOI: 10.1002/hbm.24006

The human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas, especially those along the dorsal stream. As a fully observable model of visual processing, the RNN also revealed a cortical hierarchy of temporal receptive window, dynamics of process memory, and spatiotemporal representations. These results support the hypothesis of process memory, and demonstrate the potential of using the RNN for in‐depth computational understanding of dynamic natural vision.