Surface normal and Gaussian weight constraints for indoor depth structure completion

• Published in: Pattern recognition 2023-06, Vol.138, p.109362, Article 109362
• Main Authors: Ren, Dongran, Yang, Meng, Wu, Jiangfan, Zheng, Nanning
• Format: Article
• Language: English
• Subjects: Convolution neural network; Depth structure completion; Gaussian weight; Kernel least-square method; Markov random field; Surface normal
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2023.109362

•An effective depth structure completion method is developed to jointly infer missing contents in flat regions and depth structures.•The texture-copy artifacts induced by RGB textures are well suppressed by a kernel least-square module.•The proposed method considerably outperforms recent baselines and exhibits good generalization ability. Raw depth maps captured by depth sensors generally contain missing contents due to glossy, transparent, and sparsity problems. Recent methods well completed flat regions of raw depth maps; however, ignored the accuracy of depth structures. In this paper, an effective depth structure completion method is developed to infer missing depth structures. First, a raw depth map is divided into flat regions and depth structures based on a structure prediction network. Second, two local features including surface normals and Gaussian weights are extracted from a reference RGB image to impose constraints on flat regions and depth structures, separately. Third, a kernel least-square module is adopted to handle the texture-copy artifacts problem. Finally, an iterative optimization model is developed by embedding the two constraints into a Markov random field. The cost function of the model comprises three terms, which limit data fidelity between completed depth map and raw depth map, smoothness of flat regions, and accuracy of depth structures, respectively. The proposed method is evaluated on four indoor datasets including Matterport3D, RealSense, ScanNet, and NYUv2, and compared with eight recent baselines. Quantitative results demonstrate that RMSE and MAE of completed depth maps are considerably reduced by 22.0% and 45.3%, respectively. Visual results show the superiority in completing depth structures and suppressing texture-copy artifacts. Generalization test verify the effectiveness on unseen datasets.