Missing data reconstruction in VHR images based on progressive structure prediction and texture generation

• Published in: ISPRS journal of photogrammetry and remote sensing 2021-01, Vol.171, p.266-277
• Main Authors: Xu, Hanwen, Tang, Xinming, Ai, Bo, Gao, Xiaoming, Yang, Fanlin, Wen, Zhen
• Format: Article
• Language: English
• Subjects: Deep learning; Missing data reconstruction; Progressive structure prediction; Texture generation; VHR images
• ISSN: 0924-2716; 1872-8235
• DOI: 10.1016/j.isprsjprs.2020.11.020

Very high resolution (VHR) satellite and aerial images often suffer from scene occlusion caused by redundant objects. The task of removing these redundant objects can be solved by missing data reconstruction technology. However, when dealing with VHR images with large-scale missing regions, existing spatial-based methods often destroy the structural information of ground objects. To alleviate this problem, this paper proposes a novel missing data reconstruction method based on deep learning. The reconstruction process is divided into two parts: structure prediction and texture generation. First, a progressive edge generation network (PEGN) is designed to predict the edges of objects in missing regions in a progressive manner. Then, the edge map predicted by PEGN is input to a texture generation network (TGN) as structural information to produce the reconstruction results. This is a spatial-based method that can produce realistic and reasonable results without any need for auxiliary spectral or temporal data. Experiments demonstrate that our model can better restore the structure of ground objects in VHR images than other spatial-based methods and outperform them in SSIM and PSNR indices. In addition, our model also has a strong generalization capability by introducing Poisson blending and histogram matching.