Estimating heading from optic flow: Comparing deep learning network and human performance

• Published in: Neural networks 2022-10, Vol.154, p.383-396
• Main Authors: Maus, Natalie, Layton, Oliver W.
• Format: Article
• Language: English
• Subjects: Deep learning; Heading; Optic flow; Self-motion; Vision
• ISSN: 0893-6080; 1879-2782
• DOI: 10.1016/j.neunet.2022.07.007

Convolutional neural networks (CNNs) have made significant advances over the past decade with visual recognition, matching or exceeding human performance on certain tasks. Visual recognition is subserved by the ventral stream of the visual system, which, remarkably, CNNs also effectively model. Inspired by this connection, we investigated the extent to which CNNs account for human heading perception, an important function of the complementary dorsal stream. Heading refers to the direction of movement during self-motion, which humans judge with high degrees of accuracy from the streaming pattern of motion on the eye known as optic flow. We examined the accuracy with which CNNs estimate heading from optic flow in a range of situations in which human heading perception has been well studied. These scenarios include heading estimation from sparse optic flow, in the presence of moving objects, and in the presence of rotation. We assessed performance under controlled conditions wherein self-motion was simulated through minimal or realistic scenes. We found that the CNN did not capture the accuracy of heading perception. The addition of recurrent processing to the network, however, closed the gap in performance with humans substantially in many situations. Our work highlights important self-motion scenarios in which recurrent processing supports heading estimation that approaches human-like accuracy. •We characterized heading estimation in neural networks under controlled conditions.•Convolutional neural network did not account for human heading perception.•Addition of recurrent processing generally resulted in more human-like performance.•RNN performed better than CNN on sparse flow, moving objects, and realistic scenes.