Multitemporal and Multispectral Data Fusion for Super-Resolution of Sentinel-2 Images

• Published in: IEEE transactions on geoscience and remote sensing 2023, Vol.61, p.1-19
• Main Authors: Tarasiewicz, Tomasz, Nalepa, Jakub, Farrugia, Reuben A., Valentino, Gianluca, Chen, Mang, Briffa, Johann A., Kawulok, Michal
• Format: Article
• Language: English
• Subjects: Band spectra; Data integration; Deep learning; Image enhancement; Image reconstruction; Image resolution; information fusion; multi-image super-resolution (MISR); multispectral images (MSIs); Satellite images; Sentinel-2 (S-2); Spatial data; Spatial discrimination; Spatial resolution; Spectral analysis; Spectral bands; super-resolution (SR); Superresolution; Training
• ISSN: 0196-2892; 1558-0644
• DOI: 10.1109/TGRS.2023.3311622

Multispectral Sentinel-2 (S-2) images are a valuable source of Earth observation data; however, spatial resolution of their spectral bands limited to 10-, 20-, and 60-m ground sampling distance (GSD) remains insufficient in many cases. This problem can be addressed with super-resolution (SR), aimed at reconstructing a high-resolution (HR) image from a low-resolution (LR) observation. For S-2, spectral information fusion allows for enhancing the 20- and 60-m bands to the 10-m resolution. Also, there were attempts to combine multitemporal stacks of individual S-2 bands; however, these two approaches have not been combined so far. In this article, we introduce DeepSent-a new deep network for super-resolving multitemporal series of multispectral S-2 images. It is underpinned with information fusion performed simultaneously in the spectral and temporal dimensions to generate an enlarged multispectral image (MSI). In our extensive experimental study, we demonstrate that our solution outperforms other state-of-the-art techniques that realize either multitemporal or multispectral data fusion. Furthermore, we show that the advantage of DeepSent results from how these two fusion types are combined in a single architecture, which is superior to performing such fusion in a sequential manner. Importantly, we have applied our method to super-resolve real-world S-2 images, enhancing the spatial resolution of all the spectral bands to 3.3-m nominal GSD, and we compare the outcome with very HR WorldView-2 images. We have made our implementation publicly available, and we expect it will increase the possibilities of exploiting super-resolved S-2 images in real-life applications.