Semantic Segmentation of Remote Sensing Imagery Based on Multiscale Deformable CNN and DenseCRF

• Published in: Remote sensing (Basel, Switzerland) Switzerland), 2023-03, Vol.15 (5), p.1229
• Main Authors: Cheng, Xiang, Lei, Hong
• Format: Article
• Language: English
• Subjects: Algorithms; Analysis; Artificial neural networks; Classification; Clustering; Computer applications; conditional random field (CRF); Conditional random fields; Context; Contours; convolutional neural network (CNN); Deep learning; Deformation; Digital imaging; Edge detection; Equipment and supplies; Feature extraction; Formability; Image processing; Image segmentation; Iterative methods; Neural networks; Pixels; Receptive field; Remote sensing; Resource consumption; Semantic segmentation; semantic segmentation of remote sensing imagery; Semantics; Spatial data
• ISSN: 2072-4292; 2072-4292
• DOI: 10.3390/rs15051229

The semantic segmentation of remote sensing images is a significant research direction in digital image processing. The complex background environment, irregular size and shape of objects, and similar appearance of different categories of remote sensing images have brought great challenges to remote sensing image segmentation tasks. Traditional convolutional-neural-network-based models often ignore spatial information in the feature extraction stage and pay less attention to global context information. However, spatial context information is important in complex remote sensing images, which means that the segmentation effect of traditional models needs to be improved. In addition, neural networks with a superior segmentation performance often suffer from the problem of high computational resource consumption. To address the above issues, this paper proposes a combination model of a modified multiscale deformable convolutional neural network (mmsDCNN) and dense conditional random field (DenseCRF). Firstly, we designed a lightweight multiscale deformable convolutional network (mmsDCNN) with a large receptive field to generate a preliminary prediction probability map at each pixel. The output of the mmsDCNN model is a coarse segmentation result map, which has the same size as the input image. In addition, the preliminary segmentation result map contains rich multiscale features. Then, the multi-level DenseCRF model based on the superpixel level and the pixel level is proposed, which can make full use of the context information of the image at different levels and further optimize the rough segmentation result of mmsDCNN. To be specific, we converted the pixel-level preliminary probability map into a superpixel-level predicted probability map according to the simple linear iterative clustering (SILC) algorithm and defined the potential function of the DenseCRF model based on this. Furthermore, we added the pixel-level potential function constraint term to the superpixel-based Gaussian potential function to obtain a combined Gaussian potential function, which enabled our model to consider the features of various scales and prevent poor superpixel segmentation results from affecting the final result. To restore the contour of the object more clearly, we utilized the Sketch token edge detection algorithm to extract the edge contour features of the image and fused them into the potential function of the DenseCRF model. Finally, extensive experiments on the Potsdam