Learning stratified 3D reconstruction

• Published in: Science China. Information sciences 2018-02, Vol.61 (2), p.220-235, Article 023101
• Main Authors: Dong, Qiulei, Shu, Mao, Cui, Hainan, Xu, Huarong, Hu, Zhanyi
• Format: Article
• Language: English
• Subjects: Computer Science; Computer vision; Deep learning; Geometry; Image reconstruction; Information Systems and Communication Service; Neural networks; Photogrammetry; Position Paper; Robustness; 学习方法; 计算机视觉; 3D; RANSAC; 空间视觉; 摄影测量学; 几何学; 孤立点
• ISSN: 1674-733X; 1869-1919
• DOI: 10.1007/s11432-017-9234-7

Stratified 3 D reconstruction, or a layer-by-layer 3 D reconstruction upgraded from projective to affine, then to the final metric reconstruction, is a well-known 3 D reconstruction method in computer vision. It is also a key supporting technology for various well-known applications, such as streetview, smart3 D, oblique photogrammetry. Generally speaking, the existing computer vision methods in the literature can be roughly classified into either the geometry-based approaches for spatial vision or the learning-based approaches for object vision. Although deep learning has demonstrated tremendous success in object vision in recent years,learning 3 D scene reconstruction from multiple images is still rare, even not existent, except for those on depth learning from single images. This study is to explore the feasibility of learning the stratified 3 D reconstruction from putative point correspondences across images, and to assess whether it could also be as robust to matching outliers as the traditional geometry-based methods do. In this study, a special parsimonious neural network is designed for the learning. Our results show that it is indeed possible to learn a stratified 3 D reconstruction from noisy image point correspondences, and the learnt reconstruction results appear satisfactory although they are still not on a par with the state-of-the-arts in the structurefrom-motion community due to largely its lack of an explicit robust outlier detector such as random sample consensus（RANSAC）. To the best of our knowledge, our study is the first attempt in the literature to learn3 D scene reconstruction from multiple images. Our results also show that how to implicitly or explicitly integrate an outlier detector in learning methods is a key problem to solve in order to learn comparable3 D scene structures to those by the current geometry-based state-of-the-arts. Otherwise any significant advancement of learning 3 D structures from multiple images seems difficult, if not impossible. Besides, we even speculate that deep learning might be, in nature, not suitable for learning 3 D structure from multiple images, or more generally, for solving spatial vision problems.