Hyperspectral imaging from a raw mosaic image with end-to-end learning

• Published in: Optics express 2020-01, Vol.28 (1), p.314-324
• Main Authors: Fu, Hao, Bian, Liheng, Cao, Xianbin, Zhang, Jun
• Format: Article
• Language: English
• ISSN: 1094-4087; 1094-4087
• DOI: 10.1364/OE.372746

Hyperspectral imaging provides rich spatial-spectral-temporal information with wide applications. However, most of the existing hyperspectral imaging systems require light splitting/filtering devices for spectral modulation, making the system complex and expensive, and sacrifice spatial or temporal resolution. In this paper, we report an end-to-end deep learning method to reconstruct hyperspectral images directly from a raw mosaic image. It saves the separate demosaicing process required by other methods, which reconstructs the full-resolution RGB data from the raw mosaic image. This reduces computational complexity and accumulative error. Three different networks were designed based on the state-of-the-art models in literature, including the residual network, the multiscale network and the parallel-multiscale network. They were trained and tested on public hyperspectral image datasets. Benefiting from the parallel propagation and information fusion of different-resolution feature maps, the parallel-multiscale network performs best among the three networks, with the average peak signal-to-noise ratio achieving 46.83dB. The reported method can be directly integrated to boost an RGB camera for hyperspectral imaging.