Extending function mixture network for improved spectral super-resolution

• Published in: Computer vision and image understanding 2023-12, Vol.237, p.103834, Article 103834
• Main Authors: Hussain, Sadia, Lall, Brejesh
• Format: Article
• Language: English
• Subjects: Coding; Compression; Enhancement; Filtering; Image and video processing; Restoration; Super-resolution
• ISSN: 1077-3142; 1090-235X
• DOI: 10.1016/j.cviu.2023.103834

Hyperspectral Image reconstruction from RGB images is a low-cost and convenient alternative to acquiring hyperspectral images directly. The challenge in estimating the spectral response function and using it for generating the hyperspectral image data is addressed effectively by the use of convolutional neural networks for the task. The accuracy of the reconstruction techniques is improving and effective architectures are being adapted for the purpose. Specifically, in a recent work the authors propose the use of Function Mixture Network, to model the mapping from RGB to hyperspectral. This technique benefits from the use of an adaptive spatial receptive field, however, given that the goal is spectral superresolution additional use of adaptive spectral field is prudent. Accordingly, in this paper we propose an improved Function Mixture Network based reconstruction technique by additionally incorporating a variable spectral receptive field. The proposed updates have resulted in significant gains in performance as is demonstrated by the experiments conducted over multiple datasets. An additional update to the architecture is the use of non-convolutional branches in the function mixture network. The experiments of three standard datasets, clearly demonstrate the superiority of the proposed technique for hyperspectral image reconstruction based on RGB data. •While Pixel-wise receptive field is an effective approach, one cannot ignore correlation that exists across spectral bands. Spectral Receptive field is dynamically adapted (via the use of mixing function) to correctly represent the amount of correlation between pixels along the spectral domain. In this study we explore the effect of adaptive spectral receptive field on the quality of hyperspectral image reconstruction, within a dataset and across different datasets.•Spectral composition is not necessarily a linear operation. To effectively model such non-linearity, we propose the use of Rank order statistics (ROS) filter branches along with convolution branches in the FMB blocks. Such strategy results in clear performance gains as is evidenced by the experimental results obtained.