Divide and Conquer: Improving Multi-Camera 3D Perception With 2D Semantic-Depth Priors and Input-Dependent Queries

• Published in: IEEE transactions on image processing 2024, Vol.33, p.897-909
• Main Authors: Song, Qi, Hu, Qingyong, Zhang, Chi, Chen, Yongquan, Huang, Rui
• Format: Article
• Language: English
• Subjects: Cameras; Estimation; Image segmentation; Learning; Object recognition; Queries; Semantics; Space perception; Synergistic effect; Transformers
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2024.3352808

3D perception tasks, such as 3D object detection and Bird's-Eye-View (BEV) segmentation using multi-camera images, have drawn significant attention recently. Despite the fact that accurately estimating both semantic and 3D scene layouts are crucial for this task, existing techniques often neglect the synergistic effects of semantic and depth cues, leading to the occurrence of classification and position estimation errors. Additionally, the input-independent nature of initial queries also limits the learning capacity of Transformer-based models. To tackle these challenges, we propose an input-aware Transformer framework that leverages Semantics and Depth as priors (named SDTR). Our approach involves the use of an S-D Encoder that explicitly models semantic and depth priors, thereby disentangling the learning process of object categorization and position estimation. Moreover, we introduce a Prior-guided Query Builder that incorporates the semantic prior into the initial queries of the Transformer, resulting in more effective input-aware queries. Extensive experiments on the nuScenes and Lyft benchmarks demonstrate the state-of-the-art performance of our method in both 3D object detection and BEV segmentation tasks.