Effective Pan-Sharpening With Transformer and Invertible Neural Network

In remote sensing imaging systems, pan-sharpening is an important technique to obtain high-resolution multispectral images from a high-resolution panchromatic image and its corresponding low-resolution multispectral image. Due to the powerful learning capability of convolution neural networks (CNNs)...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on geoscience and remote sensing 2022, Vol.60, p.1-15
Main Authors: Zhou, Man, Fu, Xueyang, Huang, Jie, Zhao, Feng, Liu, Aiping, Wang, Rujing
Format: Article
Language:English
Subjects:
Ablation
Algorithms
Artificial neural networks
Computer architecture
Convolution
Customization
Feature extraction
High resolution
Image resolution
Invertible neural network
Machine learning
Modules
Neural networks
pan-sharpening
Remote sensing
Resolution
Sharpening
Spatial resolution
Task analysis
transformer
Transformers
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In remote sensing imaging systems, pan-sharpening is an important technique to obtain high-resolution multispectral images from a high-resolution panchromatic image and its corresponding low-resolution multispectral image. Due to the powerful learning capability of convolution neural networks (CNNs), CNN-based methods have dominated this field. However, due to the limitation of the convolution operator, long-range spatial features are often not accurately obtained, thus limiting the overall performance. To this end, we propose a novel and effective method by exploiting a customized transformer architecture and information-lossless invertible neural module for long-range dependencies modeling and effective feature fusion in this article. Specifically, the customized transformer formulates the panchromatic (PAN) and multispectral (MS) features as queries and keys to encourage joint feature learning across two modalities, while the designed invertible neural module enables effective feature fusion to generate the expected pan-sharpened results. To the best of our knowledge, this is the first attempt to introduce a transformer and a invertible neural network into the pan-sharpening field. Extensive experiments over different kinds of satellite datasets demonstrate that our method outperforms state-of-the-art algorithms both visually and quantitatively with fewer parameters and flops. Furthermore, the ablation experiments also prove the effectiveness of the proposed customized long-range transformer and effective invertible neural feature fusion module for pan-sharpening.
ISSN:0196-2892
1558-0644
DOI:10.1109/TGRS.2021.3137967