Integrating Spatial Details With Long-Range Contexts for Semantic Segmentation of Very High-Resolution Remote-Sensing Images

• Published in: IEEE geoscience and remote sensing letters 2023, Vol.20, p.1-5
• Main Authors: Long, Jiang, Li, Mengmeng, Wang, Xiaoqin
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Auxiliary supervise; Blurring; Buildings; CLCFormer; Coders; Convolution; Convolutional neural networks; convolutional neural networks (CNNs); Datasets; Feature extraction; High resolution; Image processing; Image resolution; Image segmentation; Learning; Methods; Modules; Neural networks; Optimization; Remote sensing; Resolution; Semantic segmentation; Semantics; Spatial discrimination learning; Tiles; transformer; Transformers; very high-resolution (VHR) images
• ISSN: 1545-598X; 1558-0571
• DOI: 10.1109/LGRS.2023.3262586

This letter presents a cross-learning network (i.e., CLCFormer) integrating fine-grained spatial details within long-range global contexts based upon convolutional neural networks (CNNs) and transformer, for semantic segmentation of very high-resolution (VHR) remote-sensing images. More specifically, CLCFormer comprises two parallel encoders, derived from the CNN and transformer, and a CNN decoder. The encoders are backboned on SwinV2 and EfficientNet-B3, from which the extracted semantic features are aggregated at multiple levels using a bilateral feature fusion module (BiFFM). First, we used attention gate (ATG) modules to enhance feature representation, improving segmentation results for objects with various shapes and sizes. Second, we used an attention residual (ATR) module to refine spatial features's learning, alleviating boundary blurring of occluded objects. Finally, we developed a new strategy, called auxiliary supervise strategy (ASS), for model optimization to further improve segmentation performance. Our method was tested on the WHU, Inria, and Potsdam datasets, and compared with CNN-based and transformer-based methods. Results showed that our method achieved state-of-the-art performance on the WHU building dataset (92.31% IoU), Inria building dataset (83.71% IoU), and Potsdam dataset (80.27% MIoU). We concluded that CLCFormer is a flexible, robust, and effective method for the semantic segmentation of VHR images. The codes of the proposed model are available at https://github.com/long123524/CLCFormer .