Multimodal image encoding pre-training for diabetic retinopathy grading

• Published in: Computers in biology and medicine 2022-04, Vol.143, p.105302-105302, Article 105302
• Main Authors: Hervella, Álvaro S., Rouco, José, Novo, Jorge, Ortega, Marcos
• Format: Article
• Language: English
• Subjects: Annotations; Artificial neural networks; Automation; Clinical medicine; Computer-aided diagnosis; Datasets; Deep learning; Diabetes; Diabetes mellitus; Diabetic retinopathy; Domains; Experimentation; Eye fundus; Image classification; Machine learning; Medical diagnosis; Medical imaging; Neural networks; Ophthalmology; Retina; Retinal images; Retinopathy; Self-supervised learning; Supervision; Training; Transfer learning
• ISSN: 0010-4825; 1879-0534
• DOI: 10.1016/j.compbiomed.2022.105302

Diabetic retinopathy is an increasingly prevalent eye disorder that can lead to severe vision impairment. The severity grading of the disease using retinal images is key to provide an adequate treatment. However, in order to learn the diverse patterns and complex relations that are required for the grading, deep neural networks require very large annotated datasets that are not always available. This has been typically addressed by reusing networks that were pre-trained for natural image classification, hence relying on additional annotated data from a different domain. In contrast, we propose a novel pre-training approach that takes advantage of unlabeled multimodal visual data commonly available in ophthalmology. The use of multimodal visual data for pre-training purposes has been previously explored by training a network in the prediction of one image modality from another. However, that approach does not ensure a broad understanding of the retinal images, given that the network may exclusively focus on the similarities between modalities while ignoring the differences. Thus, we propose a novel self-supervised pre-training that explicitly teaches the networks to learn the common characteristics between modalities as well as the characteristics that are exclusive to the input modality. This provides a complete comprehension of the input domain and facilitates the training of downstream tasks that require a broad understanding of the retinal images, such as the grading of diabetic retinopathy. To validate and analyze the proposed approach, we performed an exhaustive experimentation on different public datasets. The transfer learning performance for the grading of diabetic retinopathy is evaluated under different settings while also comparing against previous state-of-the-art pre-training approaches. Additionally, a comparison against relevant state-of-the-art works for the detection and grading of diabetic retinopathy is also provided. The results show a satisfactory performance of the proposed approach, which outperforms previous pre-training alternatives in the grading of diabetic retinopathy. •Multimodal image encoding allows learning from unlabeled multimodal visual data.•Neural network disentangles common and exclusive features of the input image modality.•Improved transfer learning performance for linear classification and full fine-tuning.•Self-supervised approach improves the grading and detection of diabetic retinopathy.•Competitive results on sm