Multi-modal neuroimaging feature fusion for diagnosis of Alzheimer’s disease

• Published in: Journal of neuroscience methods 2020-07, Vol.341, p.108795-108795, Article 108795
• Main Authors: Zhang, Tao, Shi, Mingyang
• Format: Article
• Language: English
• Subjects: Alzheimer’s disease; Attention model; Classification; Deep learning; Multi-modal fusion
• ISSN: 0165-0270; 1872-678X
• DOI: 10.1016/j.jneumeth.2020.108795

•Multi-modal classification can achieve better performance by fusing different information.•Expanding the early and the late fusion into a hierarchical fusion to effectively exploit low-level and high-level features.•The attention complementary strategy is introduced to extract the synergy between multi-modal images.•The attention strategy is introduced in the feature extraction task to suppresses the irrelevant information.•Experiments on ADNI show the effectiveness of proposed method and its superiority. Compared with single-modal neuroimages classification of AD, multi-modal classification can achieve better performance by fusing different information. Exploring synergy among various multi-modal neuroimages is contributed to identifying the pathological process of neurological disorders. However, it is still problematic to effectively exploit multi-modal information since the lack of an effective fusion method. In this paper, we propose a deep multi-modal fusion network based on the attention mechanism, which can selectively extract features from MRI and PET branches and suppress irrelevant information. In the attention model, the fusion ratio of each modality is assigned automatically according to the importance of the data. A hierarchical fusion method is adopted to ensure the effectiveness of Multi-modal Fusion. Evaluating the model on the ADNI dataset, the experimental results show that it outperforms the state-of-the-art methods. In particular, the final classification results of the NC/AD, SMCI/PMCI and Four-Class are 95.21 %, 89.79 %, and 86.15 %, respectively. : Different from the early fusion and the late fusion, the hierarchical fusion method contributes to learning the synergy between the multi-modal data. Compared with some other prominent algorithms, the attention model enables our network to focus on the regions of interest and effectively fuse the multi-modal data. Benefit from the hierarchical structure with attention model, the proposed network is capable of exploiting low-level and high-level features extracted from the multi-modal data and improving the accuracy of AD diagnosis. Results show its promising performance.