Multicolor image classification using the multimodal information bottleneck network (MMIB-Net) for detecting diabetic retinopathy

• Published in: Optics express 2021-07, Vol.29 (14), p.22732-22748
• Main Authors: Song, Jingqi, Zheng, Yuanjie, Wang, Jing, Zakir Ullah, Muhammad, Jiao, Wanzhen
• Format: Article
• Language: English
• Subjects: Deep Learning; Diabetic Retinopathy - diagnosis; Diagnostic Imaging - classification; Fundus Oculi; Humans; Retina - diagnostic imaging; Retrospective Studies
• ISSN: 1094-4087; 1094-4087
• DOI: 10.1364/OE.430508

Multicolor (MC) imaging is an imaging modality that records confocal scanning laser ophthalmoscope (cSLO) fundus images, which can be used for the diabetic retinopathy (DR) detection. By utilizing this imaging technique, multiple modal images can be obtained in a single case. Additional symptomatic features can be obtained if these images are considered during the diagnosis of DR. However, few studies have been carried out to classify MC Images using deep learning methods, let alone using multi modal features for analysis. In this work, we propose a novel model which uses the multimodal information bottleneck network (MMIB-Net) to classify the MC Images for the detection of DR. Our model can extract the features of multiple modalities simultaneously while finding concise feature representations of each modality using the information bottleneck theory. MC Images classification can be achieved by picking up the combined representations and features of all modalities. In our experiments, it is shown that the proposed method can achieve an accurate classification of MC Images. Comparative experiments also demonstrate that the use of multimodality and information bottleneck improves the performance of MC Images classification. To the best of our knowledge, this is the first report of DR identification utilizing the multimodal information bottleneck convolutional neural network in MC Images.