Cross-Modality Interaction Network for Medical Image Fusion

Multi-modal medical image fusion maximizes the complementary information from diverse modality images by integrating source images. The fused medical image could offer enhanced richness and improved accuracy compared to the source images. Unfortunately, the existing deep learning-based medical image...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on consumer electronics 2024, p.1-1
Main Authors: Song, Wenhao, Zeng, Xiangqin, Abdelmoniem, Ahmed M., Zhang, Housheng, Gao, Mingliang
Format: Article
Language:English
Subjects:
AI-generated content
Biomedical imaging
Computational modeling
Convolutional neural networks
Deep learning
Feature extraction
Image fusion
Magnetic resonance imaging
Medical image fusion
Recursive neural network
Transformer
Transformers
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Multi-modal medical image fusion maximizes the complementary information from diverse modality images by integrating source images. The fused medical image could offer enhanced richness and improved accuracy compared to the source images. Unfortunately, the existing deep learning-based medical image fusion methods generally rely on convolutional operations, which may not effectively capture global information such as spatial relationships or shape features within and across image modalities. To address this problem, we propose a unified AI-Generated Content (AIGC)-based medical image fusion, termed Cross-Modal Interactive Network (CMINet). The CMINet integrates a recursive transformer with an interactive Convolutional Neural Network. Specifically, the recursive transformer is designed to capture extended spatial and temporal dependencies within modalities, while the interactive CNN aims to extract and merge local features across modalities. Benefiting from cross-modality interaction learning, the proposed method can generate fused images with rich structural and functional information. Additionally, the architecture of the recursive network is structured to reduce parameter count, which could be beneficial for deployment on resource-constrained devices. Comprehensive experiments on multi-model medical images (MRI and CT, MRI and PET, and MRI and SPECT) demonstrate that the proposed method outperforms the state-of-the-art fusion methods subjectively and objectively.
ISSN:0098-3063
1558-4127
DOI:10.1109/TCE.2024.3412879