Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion

Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model - usually the standard pinhole ge...

Full description

Saved in:
Bibliographic Details
Main Authors: Vasiljevic, Igor, Guizilini, Vitor, Ambrus, Rares, Pillai, Sudeep, Burgard, Wolfram, Shakhnarovich, Greg, Gaidon, Adrien
Format: Conference Proceeding
Language:English
Subjects:
Adaptation models
Calibration
Cameras
Solid modeling
Three-dimensional displays
Training
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model - usually the standard pinhole geometry - leading to failure when applied to imaging systems that deviate significantly from this assumption (e.g., catadioptric cameras or underwater imaging). In this work, we show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the camera model. Inspired by the geometric model of Grossberg and Nayar, we introduce Neural Ray Surfaces (NRS), convolutional networks that represent pixel-wise projection rays, approximating a wide range of cameras. NRS are fully differentiable and can be learned end-to-end from unlabeled raw videos. We demonstrate the use of NRS for self-supervised learning of visual odometry and depth estimation from raw videos obtained using a wide variety of camera systems, including pinhole, fisheye, and catadioptric.
ISSN:2475-7888
DOI:10.1109/3DV50981.2020.00010