An Improved Hybrid Network With a Transformer Module for Medical Image Fusion

Medical image fusion technology is an essential component of computer-aided diagnosis, which aims to extract useful cross-modality cues from raw signals to generate high-quality fused images. Many advanced methods focus on designing fusion rules, but there is still room for improvement in cross-moda...

Full description

Saved in:
Bibliographic Details
Published in:IEEE journal of biomedical and health informatics 2023-07, Vol.27 (7), p.1-12
Main Authors: Liu, Yanyu, Zang, Yongsheng, Zhou, Dongming, Cao, Jinde, Nie, Rencan, Hou, Ruichao, Ding, Zhaisheng, Mei, Jiatian
Format: Article
Language:English
Subjects:
Coders
Computer vision
Datasets
Diagnosis, Computer-Assisted
Discrete wavelet transforms
Electric Power Supplies
Encoders-Decoders
Feature extraction
Humans
Image fusion
Image Processing, Computer-Assisted
Image quality
Image reconstruction
Information retrieval
Information Storage and Retrieval
Medical diagnosis
Medical diagnostic imaging
Medical imaging
Modules
self-adaptive weight fusion
self-reconstruction
Signal quality
transformer
Transformers
Transforms
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Medical image fusion technology is an essential component of computer-aided diagnosis, which aims to extract useful cross-modality cues from raw signals to generate high-quality fused images. Many advanced methods focus on designing fusion rules, but there is still room for improvement in cross-modal information extraction. To this end, we propose a novel encoder-decoder architecture with three technical novelties. First, we divide the medical images into two attributes, namely pixel intensity distribution attributes and texture attributes, and thus design two self-reconstruction tasks to mine as many specific features as possible. Second, we propose a hybrid network combining a CNN and a transformer module to model both long-range and short-range dependencies. Moreover, we construct a self-adaptive weight fusion rule that automatically measures salient features. Extensive experiments on a public medical image dataset and other multimodal datasets show that the proposed method achieves satisfactory performance.
ISSN:2168-2194
2168-2208
2168-2208
DOI:10.1109/JBHI.2023.3264819