Loading…

Mask4D: End-to-End Mask-Based 4D Panoptic Segmentation for LiDAR Sequences

Scene understanding is crucial for autonomous systems to reliably navigate in the real world. Panoptic segmentation of 3D LiDAR scans allows us to semantically describe a vehicle's environment by predicting semantic classes for each 3D point and to identify individual instances through differen...

Full description

Saved in:
Bibliographic Details
Published in:IEEE robotics and automation letters 2023-11, Vol.8 (11), p.7487-7494
Main Authors: Marcuzzi, Rodrigo, Nunes, Lucas, Wiesmann, Louis, Marks, Elias, Behley, Jens, Stachniss, Cyrill
Format: Article
Language:English
Subjects:
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Scene understanding is crucial for autonomous systems to reliably navigate in the real world. Panoptic segmentation of 3D LiDAR scans allows us to semantically describe a vehicle's environment by predicting semantic classes for each 3D point and to identify individual instances through different instance IDs. To describe the dynamics of the surroundings, 4D panoptic segmentation further extends this information with temporarily consistent instance IDs to identify the different instances in the scans consistently over whole sequences. Previous approaches for 4D panoptic segmentation rely on post-processing steps and are often not end-to-end trainable. In this paper, we propose a novel approach that can be trained end-to-end and directly predicts a set of non-overlapping masks along with their semantic classes and instance IDs that are consistent over time without any post-processing like clustering or associations between predictions. We extend a mask-based 3D panoptic segmentation model to 4D by reusing queries that decoded instances in previous scans. This way, each query decodes the same instance over time, carries its ID and the tracking is performed implicitly. This enables us to jointly optimize segmentation and tracking and directly supervise for 4D panoptic segmentation.
ISSN:2377-3766
2377-3766
DOI:10.1109/LRA.2023.3320020