Pix2pix Conditional Generative Adversarial Network with MLP Loss Function for Cloud Removal in a Cropland Time Series

• Published in: Remote sensing (Basel, Switzerland) Switzerland), 2022-01, Vol.14 (1), p.144
• Main Authors: Christovam, Luiz E., Shimabukuro, Milton H., Galo, Maria de Lourdes B. T., Honkavaara, Eija
• Format: Article
• Language: English
• Subjects: Agricultural land; Agriculture; Algorithms; cGAN; cloud removal; Clouds; Comparative analysis; Crops; custom loss function; Decision making; Generative adversarial networks; Image processing; Image segmentation; image-to-image; Medical research; Multilayers; Neural networks; Objective function; Optimization; Performance assessment; Pixels; Remote sensing; SAR to optical image translation; Satellite imagery; Semantic segmentation; Semantics; Spectral sensitivity; Statistical analysis; synthetic images; Time series
• ISSN: 2072-4292; 2072-4292
• DOI: 10.3390/rs14010144

Clouds are one of the major limitations to crop monitoring using optical satellite images. Despite all efforts to provide decision-makers with high-quality agricultural statistics, there is still a lack of techniques to optimally process satellite image time series in the presence of clouds. In this regard, in this article it was proposed to add a Multi-Layer Perceptron loss function to the pix2pix conditional Generative Adversarial Network (cGAN) objective function. The aim was to enforce the generative model to learn how to deliver synthetic pixels whose values were proxies for the spectral response improving further crop type mapping. Furthermore, it was evaluated the generalization capacity of the generative models in producing pixels with plausible values for images not used in the training. To assess the performance of the proposed approach it was compared real images with synthetic images generated with the proposed approach as well as with the original pix2pix cGAN. The comparative analysis was performed through visual analysis, pixel values analysis, semantic segmentation and similarity metrics. In general, the proposed approach provided slightly better synthetic pixels than the original pix2pix cGAN, removing more noise than the original pix2pix algorithm as well as providing better crop type semantic segmentation; the semantic segmentation of the synthetic image generated with the proposed approach achieved an F1-score of 44.2%, while the real image achieved 44.7%. Regarding the generalization, the models trained utilizing different regions of the same image provided better pixels than models trained using other images in the time series. Besides this, the experiments also showed that the models trained using a pair of images selected every three months along the time series also provided acceptable results on images that do not have cloud-free areas.