Ray Deformation Networks for Novel View Synthesis of Refractive Objects

• Main Authors: Deng, Weijian, Campbell, Dylan, Sun, Chunyi, Kanitkar, Shubham, Shaffer, Matthew, Gould, Stephen
• Format: Conference Proceeding
• Language: English
• Subjects: 3D computer vision; Algorithms; Computational modeling; Computational photography; Computer vision; Deformable models; Deformation; Estimation; Geometry; image and video synthesis; Refractive index
• ISSN: 2642-9381
• DOI: 10.1109/WACV57701.2024.00309

Neural Radiance Fields (NeRF) have demonstrated exceptional capabilities in creating photorealistic novel views using volume rendering on a radiance field. However, the intrinsic assumption of straight light rays within NeRF becomes a limitation when dealing with transparent or translucent objects that exhibit refraction, and therefore have curved light paths. This hampers the ability of these approaches to accurately model the appearance of refractive objects, resulting in suboptimal novel view synthesis and geometry estimates. To address this issue, we propose an innovative solution using deformable networks to learn a tailored deformation field for refractive objects. Our approach predicts position and direction offsets, allowing NeRF to model the curved light paths caused by refraction and therefore the complex and highly view-dependent appearances of refractive objects. We also introduce a regularization strategy that encourages piece-wise linear light paths, since most physical systems can be approximated with a piece-wise constant index of refraction. By seamlessly integrating our deformation networks into the NeRF framework, our method significantly improves rendering refractive objects from novel views.