Learning Optical Flow, Depth, and Scene Flow Without Real-World Labels

Self-supervised monocular depth estimation enables robots to learn 3D perception from raw video streams. This scalable approach leverages projective geometry and ego-motion to learn via view synthesis, assuming the world is mostly static. Dynamic scenes, which are common in autonomous driving and hu...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters 2022-04, Vol.7 (2), p.3491-3498
Main Authors: Guizilini, Vitor, Lee, Kuan-Hui, Ambrus, Rares, Gaidon, Adrien
Format: Article
Language:English
Subjects:
Algorithms
Computer architecture
Computer vision
Deep learning for visual perception
Estimation
Human engineering
Image motion analysis
Machine learning
Optical flow
Optical flow (image analysis)
Projective geometry
Robots
Space perception
Task analysis
Three dimensional motion
Three-dimensional displays
Triangulation
Video data
visual learning
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Self-supervised monocular depth estimation enables robots to learn 3D perception from raw video streams. This scalable approach leverages projective geometry and ego-motion to learn via view synthesis, assuming the world is mostly static. Dynamic scenes, which are common in autonomous driving and human-robot interaction, violate this assumption. Therefore, they require modeling dynamic objects explicitly, for instance via estimating pixel-wise 3D motion, i.e. scene flow. However, the simultaneous self-supervised learning of depth and scene flow is ill-posed, as there are infinitely many combinations that result in the same 3D point. In this letter we propose DRAFT, a new method capable of jointly learning depth, optical flow, and scene flow by combining synthetic data with geometric self-supervision. Building upon the RAFT architecture, we learn optical flow as an intermediate task to bootstrap depth and scene flow learning via triangulation. Our algorithm also leverages temporal and geometric consistency losses across tasks to improve multi-task learning. Our DRAFT architecture simultaneously establishes a new state of the art in all three tasks in the self-supervised monocular setting on the standard KITTI benchmark.
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2022.3145057