Stereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstruction

Dual pixels contain disparity cues arising from the defocus blur. This disparity information is useful for many vision tasks ranging from autonomous driving to 3D creative realism. However, directly estimating disparity from dual pixels is less accurate. This work hypothesizes that distilling high-p...

Full description

Saved in:
Bibliographic Details
Main Authors: Garg, Aryan, Mallampali, Raghav, Joshi, Akshat, Govindarajan, Shrisudhan, Mitra, Kaushik
Format: Conference Proceeding
Language:English
Subjects:
Dataset
Disparity Estimation
Dual Pixels
Knowledge Distillation
Knowledge engineering
Light Field
Light fields
Photography
Reconstruction algorithms
Self-Supervision
Source coding
Three-dimensional displays
Transformers
Vision Transformers
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Dual pixels contain disparity cues arising from the defocus blur. This disparity information is useful for many vision tasks ranging from autonomous driving to 3D creative realism. However, directly estimating disparity from dual pixels is less accurate. This work hypothesizes that distilling high-precision dark stereo knowledge, implicitly or explicitly, to efficient dual-pixel student networks enables faithful reconstructions. This dark knowledge distillation should also alleviate stereo-synchronization setup and calibration costs while dramatically increasing parameter and inference time efficiency. We collect the first and largest 3-view dual-pixel video dataset, dpMV, to validate our explicit dark knowledge distillation hypothesis. We show that these methods outperform purely monocular solutions, especially in challenging foreground-background separation regions using faithful guidance from dual pixels. Finally, we demonstrate an unconventional use case unlocked by dpMV and implicit dark knowledge distillation from an ensemble of teachers for Light Field (LF) video reconstruction. Our LF video reconstruction method is the fastest and most temporally consistent to date. It remains competitive in reconstruction fidelity while offering many other essential properties like high parameter efficiency, implicit disocclusion handling, zero-shot cross-dataset transfer, geometrically consistent inference on higher spatial-angular resolutions, and adaptive baseline control. All source code is available at the repository https://github.com/Aryan-Garg.
ISSN:2472-7636
DOI:10.1109/ICCP61108.2024.10644854