Stereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstruction

• Main Authors: Garg, Aryan, Mallampali, Raghav, Joshi, Akshat, Govindarajan, Shrisudhan, Mitra, Kaushik
• Format: Conference Proceeding
• Language: English
• Subjects: Dataset; Disparity Estimation; Dual Pixels; Knowledge Distillation; Knowledge engineering; Light Field; Light fields; Photography; Reconstruction algorithms; Self-Supervision; Source coding; Three-dimensional displays; Transformers; Vision Transformers
• ISSN: 2472-7636
• DOI: 10.1109/ICCP61108.2024.10644854

Dual pixels contain disparity cues arising from the defocus blur. This disparity information is useful for many vision tasks ranging from autonomous driving to 3D creative realism. However, directly estimating disparity from dual pixels is less accurate. This work hypothesizes that distilling high-precision dark stereo knowledge, implicitly or explicitly, to efficient dual-pixel student networks enables faithful reconstructions. This dark knowledge distillation should also alleviate stereo-synchronization setup and calibration costs while dramatically increasing parameter and inference time efficiency. We collect the first and largest 3-view dual-pixel video dataset, dpMV, to validate our explicit dark knowledge distillation hypothesis. We show that these methods outperform purely monocular solutions, especially in challenging foreground-background separation regions using faithful guidance from dual pixels. Finally, we demonstrate an unconventional use case unlocked by dpMV and implicit dark knowledge distillation from an ensemble of teachers for Light Field (LF) video reconstruction. Our LF video reconstruction method is the fastest and most temporally consistent to date. It remains competitive in reconstruction fidelity while offering many other essential properties like high parameter efficiency, implicit disocclusion handling, zero-shot cross-dataset transfer, geometrically consistent inference on higher spatial-angular resolutions, and adaptive baseline control. All source code is available at the repository https://github.com/Aryan-Garg.