Multi-View Supervision for Single-View Reconstruction via Differentiable Ray Consistency

We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view. We do so by reformulating view consistency using a differentiable ray consistency (DRC) te...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on pattern analysis and machine intelligence 2022-12, Vol.44 (12), p.8754-8765
Main Authors: Tulsiani, Shubham, Zhou, Tinghui, Efros, Alexei A., Malik, Jitendra
Format: Article
Language:English
Subjects:
3D reconstruction
Cameras
Color
Color imagery
Consistency
Empirical analysis
Image reconstruction
Learning
multi-view supervision
ray consistency
Semantics
Shape
Solid modeling
Three-dimensional displays
Training data
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view. We do so by reformulating view consistency using a differentiable ray consistency (DRC) term. We show that this formulation can be incorporated in a learning framework to leverage different types of multi-view observations e.g., foreground masks, depth, color images, semantics etc. as supervision for learning single-view 3D prediction. We present empirical analysis of our technique in a controlled setting. We also show that this approach allows us to improve over existing techniques for single-view reconstruction of objects from the PASCAL VOC dataset.
ISSN:0162-8828
1939-3539
2160-9292
DOI:10.1109/TPAMI.2019.2898859