CCANet: Cross-Modality Comprehensive Feature Aggregation Network for Indoor Scene Semantic Segmentation

• Published in: IEEE transactions on cognitive and developmental systems 2024-09, p.1-13
• Main Authors: Zihao, Zhang, Yale, Yang, Huifang, Hou, Fanman, Meng, Fan, Zhang, Kangzhan, Xie, Chunsheng, Zhuang
• Format: Article
• Language: English
• Subjects: Accuracy; Bidirectional Rectification; Cross-Modality Fusion; Data mining; Feature extraction; Image color analysis; RGB-D; Semantic segmentation; Semantics; Transformers; Vision Transformers
• ISSN: 2379-8920; 2379-8939
• DOI: 10.1109/TCDS.2024.3455356

The semantic segmentation of indoor scenes based on RGB and Depth information has been a persistent and enduring research topic. However, how to fully utilize the complementarity of multimodal features and achieve efficient fusion remains a challenging research topic. To address this challenge, we proposed an innovative cross-modal comprehensive feature aggregation network (CCANet) to achieve high-precision semantic segmentation of indoor scenes. In this method, we first propose a bidirectional cross-modality feature rectification module (BCFR) to complement each other and remove noise in both channel and spatial correlations. After that, the adaptive criss-cross attention fusion module (CAF) is designed to realize multi-stage deep multi-modal feature fusion. Finally, a multi-supervision strategy is applied to accurately learn additional details of the target, guiding the gradual refinement of segmentation maps. By conducting thorough experiments on two openly accessible datasets of indoor scenes, the results demonstrate that CCANet exhibits outstanding performance and robustness in aggregating RGB and depth features.