An Embodied Multi-Sensor Fusion Approach to Visual Motion Estimation Using Unsupervised Deep Networks

• Published in: Sensors (Basel, Switzerland) Switzerland), 2018-05, Vol.18 (5), p.1427
• Main Authors: Shamwell, E Jared, Nothwang, William D, Perlis, Donald
• Format: Article
• Language: English
• Subjects: deep learning; Deformation; Hypotheses; Motion simulation; Multisensor fusion; Odometers; optical flow; Pyramids; sensor fusion; Sensors; State estimation
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s18051427

Aimed at improving size, weight, and power (SWaP)-constrained robotic vision-aided state estimation, we describe our unsupervised, deep convolutional-deconvolutional sensor fusion network, Multi-Hypothesis DeepEfference (MHDE). MHDE learns to intelligently combine noisy heterogeneous sensor data to predict several probable hypotheses for the dense, pixel-level correspondence between a source image and an unseen target image. We show how our multi-hypothesis formulation provides increased robustness against dynamic, heteroscedastic sensor and motion noise by computing hypothesis image mappings and predictions at 76⁻357 Hz depending on the number of hypotheses being generated. MHDE fuses noisy, heterogeneous sensory inputs using two parallel, inter-connected architectural pathways and (1⁻20 in this work) multi-hypothesis generating sub-pathways to produce global correspondence estimates between a source and a target image. We evaluated MHDE on the KITTI Odometry dataset and benchmarked it against the vision-only DeepMatching and Deformable Spatial Pyramids algorithms and were able to demonstrate a significant runtime decrease and a performance increase compared to the next-best performing method.