Accurate and Efficient 3D Panoptic Mapping Using Diverse Information Modalities and Multidimensional Data Association

3D Panoptic perception is essential for the understanding of real-world environment and plays an increasingly important role in the field of robotics. However, most existing methods heavily rely on image panoptic segmentation networks to acquire panoptic information of the environment, which is time...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on circuits and systems for video technology 2024-06, Vol.34 (6), p.4489-4502
Main Authors: Ying, Zhongmou, Yuan, Xianfeng, Song, Boyi, Song, Yong, Zhou, Fengyu, Sheng, Weihua
Format: Article
Language:English
Subjects:
3D perception
Algorithms
Image acquisition
Image reconstruction
Image segmentation
Mapping
mobile robot
Multidimensional data
Multiple target tracking
Object detection
Object recognition
panoptic inference
Panoptic mapping
Real-time systems
Robotics
semantic mapping
Semantic segmentation
Semantics
Three-dimensional displays
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:3D Panoptic perception is essential for the understanding of real-world environment and plays an increasingly important role in the field of robotics. However, most existing methods heavily rely on image panoptic segmentation networks to acquire panoptic information of the environment, which is time-consuming and susceptible to interference. In this paper, we propose a novel and efficient panoptic mapping method based on multi-source information. Specifically, to improve the real-time performance of the system, we first apply lightweight object detection and semantic segmentation to extract 2D semantic and instance information from images. Second, a panoptic inference algorithm is designed that fully utilizes multi-source information, including geometry-based and learning-based information, to simultaneously reason about background and foreground objects in the environment. Finally, we take advantage of the scalability of the framework by introducing a multi-object tracking algorithm into the framework, thus providing the temporal information among consecutive frames to the data association module. Based on two popular datasets, extensive comparison experiments are conducted to illustrate the effectiveness of the proposed method. Experimental results show that compared with state-of-the-art panoptic mapping methods, the proposed method achieves superior performance in accuracy, real-timeness and stability. Furthermore, we also evaluate our method in real-world scenarios and CPU-only device to demonstrate the feasibility of its practical deployment.
ISSN:1051-8215
1558-2205
DOI:10.1109/TCSVT.2023.3339683