DeepEfference: Learning to predict the sensory consequences of action through deep correspondence

• Main Authors: Shamwell, E. Jared, Nothwang, William D., Perlis, Donald
• Format: Conference Proceeding
• Language: English
• Subjects: Cameras; Convolutional codes; Image reconstruction; Robot sensing systems; Two dimensional displays; Visualization
• ISSN: 2161-9484
• DOI: 10.1109/DEVLRN.2017.8329823

As the human eyeball saccades across the visual scene, humans maintain egocentric visual positional constancy despite retinal motion identical to an egocentric shift of the scene. Characterizing the underlying biological computations enabling visual constancy can inform methods of robotic localization by serving as a model for intelligently integrating complimentary, heterogeneous information. Here we present DeepEfference, a bio-inspired, unsupervised, deep sensorimotor network that learns to predict the sensory consequences of self-generated actions. DeepEfference computes dense image correspondences [1] at over 500 Hz and uses only a single monocular grayscale image and a low-dimensional extra-modal motion estimate as data inputs. Designed for robotic applications, DeepEfference employs multi-level fusion via two parallel pathways to learn dense, pixel-level predictions and correspondences between source and target images. We present quantitative and qualitative results from the SceneNet RGBD [2] and KITTI Odometry [3] datasets and demonstrate an approximate runtime decrease of over 20,000% with only a 12% increase in mean pixel matching error compared to DeepMatching [4] on KITTI Odometry.