Two-Stream CNN with Loose Pair Training for Multi-modal AMD Categorization

This paper studies automated categorization of age-related macular degeneration (AMD) given a multi-modal input, which consists of a color fundus image and an optical coherence tomography (OCT) image from a specific eye. Previous work uses a traditional method, comprised of feature extraction and cl...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2019-07
Main Authors: Wang, Weisen, Xu, Zhiyan, Yu, Weihong, Zhao, Jianchun, Yang, Jingyuan, He, Feng, Yang, Zhikun, Chen, Di, Ding, Dayong, Chen, Youxin, Li, Xirong
Format: Article
Language:English
Subjects:
Artificial neural networks
Classification
Eye (anatomy)
Feature extraction
Macular degeneration
Mapping
Optical Coherence Tomography
Training
Viability
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper studies automated categorization of age-related macular degeneration (AMD) given a multi-modal input, which consists of a color fundus image and an optical coherence tomography (OCT) image from a specific eye. Previous work uses a traditional method, comprised of feature extraction and classifier training that cannot be optimized jointly. By contrast, we propose a two-stream convolutional neural network (CNN) that is end-to-end. The CNN's fusion layer is tailored to the need of fusing information from the fundus and OCT streams. For generating more multi-modal training instances, we introduce Loose Pair training, where a fundus image and an OCT image are paired based on class labels rather than eyes. Moreover, for a visual interpretation of how the individual modalities make contributions, we extend the class activation mapping technique to the multi-modal scenario. Experiments on a real-world dataset collected from an outpatient clinic justify the viability of our proposal for multi-modal AMD categorization.
ISSN:2331-8422