Multi-scale unsupervised network for infrared and visible image fusion based on joint attention mechanism

• Published in: Infrared physics & technology 2022-09, Vol.125, p.104242, Article 104242
• Main Authors: Xu, Dongdong, Zhang, Ning, Zhang, Yuxi, Li, Zheng, Zhao, Zhikang, Wang, Yongcheng
• Format: Article
• Language: English
• Subjects: Attention mechanism; Deep learning; Image fusion; Infrared and visible images; Perceptual loss
• ISSN: 1350-4495; 1879-0275
• DOI: 10.1016/j.infrared.2022.104242

•A convolutional fusion model with joint channel and spatial attention was built.•Different perceptive fields were adopted in multi-branch structure to get features.•The particular perceptual loss with adjusted image was designed for image matching.•4 traditional and 9 deep learning-based methods were used to demonstrate the model.•Images with high fidelity and salient feature were friendly to human visual system. Infrared and visible image fusion can synthesize complementary features of salient objects and texture details which are important for all-weather detection and other tasks. Nowadays, the deep learning based unsupervised fusion solutions are preferred and have obtained good results since the reference images for fusion tasks are not available. In the existing methods, some prominent features are missing in the fused images and the visual vitality needs to be improved. From this thought, attention mechanism is introduced to the fusion network. Especially, channel dimension and spatial dimension attention are jointed to supplement each other for feature extraction. Multiple attention branches emphasize on multi-scale features to complete the encoding. Skip connections are added to learn residual information. The multi-layer perceptual loss, the structure similarity loss and the content loss together construct the strong constraints for training. Comparative experiments with subjective and objective evaluations on 4 traditional and 9 deep learning based methods demonstrate the advantages of the proposed model.