Sentinel-2 Sharpening Using a Single Unsupervised Convolutional Neural Network With MTF-Based Degradation Model

• Published in: IEEE journal of selected topics in applied earth observations and remote sensing 2021, Vol.14, p.6882-6896
• Main Authors: Nguyen, Han V., Ulfarsson, Magnus O., Sveinsson, Johannes R., Mura, Mauro Dalla
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Computer Science; Constellations; Convolutional neural networks (CNNs); Deep learning; Degradation; Electronics packaging; Evaluation; Handicrafts; Hyperspectral imaging; Image fusion; Image Processing; Image quality; Machine Learning; Methods; Modulation transfer function; MTF-based degradation; Neural and Evolutionary Computing; Neural networks; Optical communication; Pansharpening; Resolution; Sentinel-2 image sharpening; Sharpening; Spatial resolution; super-resolution; Training; unsupervised deep learning
• ISSN: 1939-1404; 2151-1535
• DOI: 10.1109/JSTARS.2021.3092286

The Sentinel-2 (S2) constellation provides multispectral images at 10 m, 20 m, and 60 m resolution bands. Obtaining all bands at 10 m resolution would benefit many applications. Recently, many model-based and deep learning (DL)-based sharpening methods have been proposed. However, the downside of those methods is that the DL-based methods need to be trained separately for the 20 m and the 60 m bands in a supervised manner at reduced resolution, while the model-based methods heavily depend on the hand-crafted image priors. To break the gap, this article proposes a novel unsupervised DL-based S2 sharpening method using a single convolutional neural network (CNN) to sharpen the 20 and 60 m bands at the same time at full resolution. The proposed method replaces the hand-crafted image prior by the deep image prior (DIP) provided by a CNN structure whose parameters are easily optimized using a DL optimizer. We also incorporate the modulation transfer function-based degradation model as a network layer, and add all bands to both network input and output. This setting improves the DIP and exploits the advantage of multitask learning since all S2 bands are highly correlated. Extensive experiments with real S2 data show that our proposed method outperforms competitive methods for reduced-resolution evaluation and yields very high quality sharpened image for full-resolution evaluation.