Refractive Pose Refinement: Generalising the Geometric Relation between Camera and Refractive Interface

• Published in: International journal of computer vision 2023-06, Vol.131 (6), p.1448-1476
• Main Authors: Hu, Xiao, Lauze, François, Pedersen, Kim Steenstrup
• Format: Article
• Language: English
• Subjects: Artificial Intelligence; Computer Imaging; Computer Science; Image Processing and Computer Vision; Pattern Recognition; Pattern Recognition and Graphics; Special Issue on Traditional Computer Vision in the Age of Deep Learning; Vision
• ISSN: 0920-5691; 1573-1405
• DOI: 10.1007/s11263-023-01763-4

In this paper, we investigate absolute and relative pose estimation under refraction, which are essential problems for refractive structure from motion. To cope with refraction effects, we first formulate geometric constraints for establishing iterative algorithms to optimize absolute and relative pose. By classifying two scenarios according to the geometric relationship between the camera and refractive interface, we derive the corresponding solutions to solve the optimization problems efficiently. In the scenario where the geometry between the camera and refractive interface is fixed (e.g., underwater imaging), we also show that the refractive epipolar constraint for relative pose can be established as a summation of the classical essential matrix and two correction terms caused by refraction by using the virtual camera transformation. Thanks to its succinct form, the resulting refractive epipolar constraint can be efficiently optimized. We evaluate our proposed algorithms on synthetic data showing superior accuracy and computational efficiency compared to state-of-the-art (SOTA) methods. We further demonstrate the application of the proposed algorithms in refractive structure from motion on real data. Our datasets (Hu et al., RefractiveSfM, https://github.com/diku-dk/RefractiveSfM , 2022) and code (Hu et al., DIKU Refractive Scenes Dataset 2022, Data, 2022) are publicly available.