Matrix Factorization Informed Interpretable Deep Network for Unregistered Hyperspectral and Multispectral Images Fusion

• Main Authors: Zhang, Tongzhen, Qu, Jiahui, Li, Yunsong, Du, Qian, Dong, Wenqian
• Format: Conference Proceeding
• Language: English
• Subjects: Accuracy; Computational modeling; Deep learning; Fusion; Geoscience and remote sensing; Hy-perspectral; Image fusion; Iterative methods; Matrices; Matrix factorization; Unregistered
• ISSN: 2153-7003
• DOI: 10.1109/IGARSS53475.2024.10640991

Considering the existing issues in unregistered hyper-spectral images (HSI) and multispectral images (MSI) fusion methods: i) the designed registration modules introduce a significant computational burden, and registration errors accumulate in fusion errors; ii) the methods lack model guidance, resulting in poor interpretability of the network. In this paper, we propose a matrix factorization informed interpretable deep network to address the challenges of unregistered HSI and MSI fusion (IUFNet). In particular, we derive an extended matrix factorization model for unregistered fusion (EUMF), which substitutes the abundance matrix of HSI containing low-resolution and distorted spatial information by the high-resolution abundance matrix of MSI. This substitution ingeniously eliminates the dependence of fusion performance on registration accuracy. Subsequently, IUFNet is designed to unfold the iterative results obtained by proximal gradient descent into the deep learning network, where each operation has a clear physical meaning. Overall, this network achieves the fusion of unregistered HSI and MSI and exhibits inter-pretability. Experimental results on the widely used Paiva Center dataset demonstrate the effectiveness and superiority of the proposed method.