DGLT-Fusion: A decoupled global–local infrared and visible image fusion transformer

• Published in: Infrared physics & technology 2023-01, Vol.128, p.104522, Article 104522
• Main Authors: Yang, Xin, Huo, Hongtao, Wang, Renhua, Li, Chang, Liu, Xiaowen, Li, Jing
• Format: Article
• Language: English
• Subjects: Convolution neural networks; Image fusion; Infrared image; Transformer; Visible image
• ISSN: 1350-4495; 1879-0275
• DOI: 10.1016/j.infrared.2022.104522

Convolution Neural Networks (CNN) and generative adversarial networks (GAN) based approaches have achieved substantial performance in image fusion field. However, these methods focus on extracting local features and pay little attention to learning global dependencies. In recent years, given the competitive long-term dependency modeling capability, the Transformer based fusion method has made impressive achievement, but this method simultaneously processes long-term correspondences and short-term features, which might result in deficiently global–local information interaction. Towards this end, we propose a decoupled global–local infrared and visible image fusion Transformer (DGLT-Fusion). The DGLT-Fusion decouples global–local information learning into Transformer module and CNN module. The long-term dependencies are modeled by a series of Transformer blocks (global-decoupled Transformer blocks), while the short-term features are extracted by local-decoupled convolution blocks. In addition, we design Transformer dense connection to reserve more information. These two modules are interweavingly stacked that enables our network retain texture and detailed information more integrally. Furthermore, the comparative experiment results show that DGLT-Fusion achieves better performance than state-of-the-art approaches. •A decoupled global–local infrared and visible image fusion Transformer (DGLT-Fusion) is proposed. The DGLT-Fusion decouples global–local information learning into Transformer and CNN modules. These two modules are interweavingly stacked which enables our network have better global–local information interaction.•The proposed method designs dense connection within global-decoupled Transformer module, so that long-term dependency information loss caused by network complexity can be avoid.•DGLT-Fusion is evaluated with eight fusion approaches both qualitatively and quantitatively, and the experimental results demonstrate DGLT-Fusion reach better performance.