Self-supervised Multi-scale Multi-modal Graph Pool Transformer for Sellar Region Tumor Diagnosis

• Published in: IEEE journal of biomedical and health informatics 2024-11, Vol.PP, p.1-13
• Main Authors: Lei, Baiying, Cai, Gege, Zhu, Yun, Wang, Tianfu, Dong, Lei, Zhao, Cheng, Hu, Xinzhi, Zhu, Huijun, Lu, Lin, Feng, Feng, Feng, Ming, Wang, Renzhi
• Format: Article
• Language: English
• Subjects: Biological system modeling; Computational modeling; Data models; Deep learning; Hybrid loss; Imaging; Magnetic resonance imaging; Medical diagnostic imaging; Multi-scale multi-modal graph pool Transformer; Self-supervised learning; Sellar region tumor diagnosis; Solid modeling; Three-dimensional displays; Transformers
• ISSN: 2168-2194; 2168-2208; 2168-2208
• DOI: 10.1109/JBHI.2024.3496700

The sellar region tumor is a brain tumor that only exists in the brain sellar, which affects the central nervous system. The early diagnosis of the sellar region tumor subtypes helps clinicians better understand the best treatment and recovery of pa-tients. Magnetic resonance imaging (MRI) has proven to be an effective tool for the early detection of sellar region tumors. However, the existing sellar region tumor diagnosis still remains challenging due to the small amount of dataset and data imbalance. To overcome these challenges, we propose a novel self-supervised multi-scale multi-modal graph pool Transformer (MMGPT) network that can enhance the multi-modal fusion of small and imbalanced MRI data of sellar region tumors. MMGPT can strengthen feature interaction between multi-modal images, which makes our model more robust. A contrastive learning equipped auto-encoder (CAE) via self-supervised learning (SSL) is adopted to learn more detailed information between different samples. The proposed CAE transfers the pre-trained knowledge to the downstream tasks. Finally, a hybrid loss is equipped to relieve the performance degradation caused by data imbalance. The experimental results show that the proposed method outperforms state-of-the-art methods and obtains higher accuracy and AUC in the classification of sellar region tumors.