MURF: Mutually Reinforcing Multi-Modal Image Registration and Fusion

• Published in: IEEE transactions on pattern analysis and machine intelligence 2023-10, Vol.45 (10), p.12148-12166
• Main Authors: Xu, Han, Yuan, Jiteng, Ma, Jiayi
• Format: Article
• Language: English
• Subjects: Computer vision; contrastive learning; Deformation; Design methodology; Image enhancement; Image fusion; Image registration; Information retrieval; Magnetic resonance imaging; Measurement; Medical imaging; Modal data; Modules; Multi-modal images; Network architecture; Positron emission; Registration; Task analysis
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2023.3283682

Existing image fusion methods are typically limited to aligned source images and have to "tolerate" parallaxes when images are unaligned. Simultaneously, the large variances between different modalities pose a significant challenge for multi-modal image registration. This study proposes a novel method called MURF, where for the first time, image registration and fusion are mutually reinforced rather than being treated as separate issues. MURF leverages three modules: shared information extraction module (SIEM), multi-scale coarse registration module (MCRM), and fine registration and fusion module (F2M). The registration is carried out in a coarse-to-fine manner. During coarse registration, SIEM first transforms multi-modal images into mono-modal shared information to eliminate the modal variances. Then, MCRM progressively corrects the global rigid parallaxes. Subsequently, fine registration to repair local non-rigid offsets and image fusion are uniformly implemented in F2M. The fused image provides feedback to improve registration accuracy, and the improved registration result further improves the fusion result. For image fusion, rather than solely preserving the original source information in existing methods, we attempt to incorporate texture enhancement into image fusion. We test on four types of multi-modal data (RGB-IR, RGB-NIR, PET-MRI, and CT-MRI). Extensive registration and fusion results validate the superiority and universality of MURF.