Systematic comparison of deep-learning based fusion strategies for multi-modal ultrasound in diagnosis of liver cancer

• Published in: Neurocomputing (Amsterdam) 2024-10, Vol.603, p.128257, Article 128257
• Main Authors: Li, Ming-De, Li, Wei, Lin, Man-Xia, Lin, Xin-Xin, Hu, Hang-Tong, Wang, Ying-Chen, Ruan, Si-Min, Huang, Ze-Rong, Lu, Rui-Fang, Li, Lv, Kuang, Ming, Lu, Ming-De, Chen, Li-Da, Wang, Wei, Huang, Qing-hua
• Format: Article
• Language: English
• Subjects: Classification; Fusion strategy; Hybrid-fusion; Liver cancer; Multi-modality; Ultrasound
• ISSN: 0925-2312
• DOI: 10.1016/j.neucom.2024.128257

For the diagnosis of liver cancer, conventional brightness mode (B-mode) can only provide morphological information. Multi-modal ultrasound, including shear-wave elastography (SWE) and contrast enhanced ultrasound (CEUS), can provide comprehensive diagnostic information on tumor microenvironment and tissue perfusion. The challenge is to effectively explore the multi-modal features of ultrasound. Besides, there are many fusion strategies currently available, but there is a lack of systematic comparative research on the various fusion strategies. In this study, we designed 'Lesions Pairing' to construct the dataset, addressing the challenge of small sample sizes in multi-modal learning. We then compared the effectiveness of different strategies and proposed hybrid-fusion strategies based on the combination of conventional layer-level fusion (i.e. early-fusion, mid-fusion and late-fusion), which can efficiently extract intra-/inter- modal information. Specifically, we first systematically compared different deep-learning-based fusion strategies for multi-modal ultrasound in the diagnosis of liver cancer. Secondly, based on the comparison results of a multimodal framework that integrates B-mode, SWE, CEUS ultrasound data, and clinical data simultaneously, we propose a hybrid-fusion strategies for the diagnosis of hepatocellular carcinoma and intrahepatic cholangiocarcinoma. The experimental results showed that the area under the curve of the early-late fusion strategy combined with clinical data was 0.9854, which was superior to other single mode and other fusion strategies, increasing by 13.8–25.88 % and 2.22 %-9.79 %, respectively.