Hybrid CNN and Transformer Network for Semantic Segmentation of UAV Remote Sensing Images

• Published in: IEEE journal on miniaturization for air and space systems 2024-03, Vol.5 (1), p.33-41
• Main Authors: Zhou, Xuanyu, Zhou, Lifan, Gong, Shengrong, Zhang, Haizhen, Zhong, Shan, Xia, Yu, Huang, Yizhou
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Autonomous aerial vehicles; Coders; Complexity; Convolutional neural networks; Feature extraction; Image segmentation; Modules; Remote sensing; Semantic segmentation; Semantics; Swin transformer; Technical services; Transformers; unmanned aerial vehicle (UAV); Unmanned aerial vehicles
• ISSN: 2576-3164; 2576-3164
• DOI: 10.1109/JMASS.2023.3332948

Semantic segmentation of unmanned aerial vehicle (UAV) remote sensing images is a recent research hotspot, offering technical support for diverse types of UAV remote sensing missions. However, unlike general scene images, UAV remote sensing images present inherent challenges. These challenges include the complexity of backgrounds, substantial variations in target scales, and dense arrangements of small targets, which severely hinder the accuracy of semantic segmentation. To address these issues, we propose a convolutional neural network (CNN) and transformer hybrid network for semantic segmentation of UAV remote sensing images. The proposed network follows an encoder-decoder architecture that merges a transformer-based encoder with a CNN-based decoder. First, we incorporate the Swin transformer as the encoder to address the limitations of CNN in global modeling, mitigating the interference caused by complex background information. Second, to effectively handle the significant changes in target scales, we design the multiscale feature integration module (MFIM) that enhances the multiscale feature representation capability of the network. Finally, the semantic feature fusion module (SFFM) is designed to filter the redundant noise during the feature fusion process, which improves the recognition of small targets and edges. Experimental results demonstrate that the proposed method outperforms other popular methods on the UAVid and Aeroscapes datasets.