GAN meets chemometrics: Segmenting spectral images with pixel2pixel image translation with conditional generative adversarial networks

• Published in: Chemometrics and intelligent laboratory systems 2021-08, Vol.215, p.104362, Article 104362
• Main Authors: Mishra, Puneet, Herrmann, Ittai
• Format: Article
• Language: English
• Subjects: Artificial intelligence; Hyperspectral imagery; Image context; Semantic segmentation
• ISSN: 0169-7439; 1873-3239
• DOI: 10.1016/j.chemolab.2021.104362

In analytical chemistry, spectral imaging of complex analytical systems is commonly performed. A major task in spectral imaging analysis is to extract signals related to important analytes present in the imaged scene. Hence, the first task of spectral image analysis is to perform an image segmentation to extract the relevant signals to analyze. However, in the chemometric domain, the traditional image segmentation methods are limited either to threshold-based or pixel-wise classification, therefore, no approache uses the contextual information present in the imaging scene. This study presents for the first time a pixel2pixel (p2p) image translation using conditional generative adversarial networks (cGAN) for the segmentation of spectral images. The p2p cGAN trains two neural models simultaneously where one model (generator) learns to segment and the other model learns to detect if the segmentation performed by the generator model is correct. During the process of generation and detection, the model automatically learns to segment the spectral images accurately. As an application of the p2p cGAN, a case of segmenting visible and near-infrared spectral images of plants was presented. Furthermore, as a comparison, threshold-based and pixel-wise image classification based on partial least-square discriminant analysis were also presented. The results showed that the p2p cGAN based image translation performed the best segmentation task with an intersection over union score of 0.95 ​± ​0.04. The advanced new DL based image processing approaches can complement spectral image processing. •Generative adversarial networks (GAN) are proposed as a tool for spectral image segmentation.•An example of segmenting close range spectral images of plant is demonstrated.•The performance was compared with threshold and pixel-based segmentation.•The pixel2pixel GAN based image translation outperformed classical chemometric approach.