NIS-SLAM: Neural Implicit Semantic RGB-D SLAM for 3D Consistent Scene Understanding

In recent years, the paradigm of neural implicit representations has gained substantial attention in the field of Simultaneous Localization and Mapping (SLAM). However, a notable gap exists in the existing approaches when it comes to scene understanding. In this paper, we introduce NIS-SLAM, an effi...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on visualization and computer graphics 2024-11, Vol.30 (11), p.7129-7139
Main Authors: Zhai, Hongjia, Huang, Gan, Hu, Qirui, Li, Guanglin, Bao, Hujun, Zhang, Guofeng
Format: Article
Language:English
Subjects:
Cameras
Image reconstruction
Implicit representation
Neural dense SLAM
Rendering (computer graphics)
Scene understanding
Semantic segmentation
Semantics
Simultaneous localization and mapping
Surface reconstruction
Three-dimensional displays
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In recent years, the paradigm of neural implicit representations has gained substantial attention in the field of Simultaneous Localization and Mapping (SLAM). However, a notable gap exists in the existing approaches when it comes to scene understanding. In this paper, we introduce NIS-SLAM, an efficient neural implicit semantic RGB-D SLAM system, that leverages a pre-trained 2D segmentation network to learn consistent semantic representations. Specifically, for high-fidelity surface reconstruction and spatial consistent scene understanding, we combine high-frequency multi-resolution tetrahedron-based features and low-frequency positional encoding as the implicit scene representations. Besides, to address the inconsistency of 2D segmentation results from multiple views, we propose a fusion strategy that integrates the semantic probabilities from previous non-keyframes into keyframes to achieve consistent semantic learning. Furthermore, we implement a confidence-based pixel sampling and progressive optimization weight function for robust camera tracking. Extensive experimental results on various datasets show the better or more competitive performance of our system when compared to other existing neural dense implicit RGB-D SLAM approaches. Finally, we also show that our approach can be used in augmented reality applications. Project page: https://zju3dv.github.io/nis_slam.
ISSN:1077-2626
1941-0506
1941-0506
DOI:10.1109/TVCG.2024.3456201