A review of multimodal image matching: Methods and applications

• Published in: Information fusion 2021-09, Vol.73, p.22-71
• Main Authors: Jiang, Xingyu, Ma, Jiayi, Xiao, Guobao, Shao, Zhenfeng, Guo, Xiaojie
• Format: Article
• Language: English
• Subjects: Computer vision; Deep learning; Matching; Medical; Multimodal; Registration; Remote sensing
• ISSN: 1566-2535; 1872-6305
• DOI: 10.1016/j.inffus.2021.02.012

Multimodal image matching, which refers to identifying and then corresponding the same or similar structure/content from two or more images that are of significant modalities or nonlinear appearance difference, is a fundamental and critical problem in a wide range of applications, including medical, remote sensing and computer vision. An increasing number and diversity of methods have been proposed over the past decades, particularly in this deep learning era, due to the challenges in eliminating modality variance and geometrical deformation that intrinsically exist in multimodal image matching. However, a comprehensive review and analysis of traditional and recent trainable methods and their applications in different research fields are lacking. To this end and in this survey, we first introduce two general frameworks, saying area- and feature-based, in terms of their core components, taxonomy, and procedure details. Second, we provide a comprehensive review of multimodal image matching methods from handcrafted to deep methods for each research field according to their imaging nature, including medical, remote sensing and computer vision. Extensive experimental comparisons of interest point detection, description and matching, and image registration are performed on various datasets containing common types of multimodal image pairs that we collected and annotated. Finally, we briefly introduce and analyze several typical applications to reveal the significance of multimodal image matching and provide insightful discussions and conclusions to these multimodal image matching approaches, and simultaneously deliver their future trends for researchers and engineers in related research areas to achieve further breakthroughs. •We introduce two general matching frameworks and their extensions to multimodal case.•The multimodal matching methods and applications are comprehensively reviewed.•A multimodal benchmark containing 18 common types of data is constructed.•Extensive experiments on feature matching and image registration are performed.•Insightful discussions and prospects for future work are delivered.