STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection

• Published in: IEEE transactions on instrumentation and measurement 2021, Vol.70, p.1-13
• Main Authors: Ma, Jiayi, Tang, Linfeng, Xu, Meilong, Zhang, Hao, Xiao, Guobao
• Format: Article
• Language: English
• Subjects: Algorithms; Computer vision; Data integration; Data mining; Deep learning; Feature extraction; Generative adversarial networks; Image fusion; Image processing; Image quality; Image reconstruction; infrared image; Infrared imagery; mask; Object detection; salient target detection; Target detection; Target masking; Target recognition; Texture; Transforms
• ISSN: 0018-9456; 1557-9662
• DOI: 10.1109/TIM.2021.3075747

In this article, we propose an infrared and visible image fusion network based on the salient target detection, termed STDFusionNet, which can preserve the thermal targets in infrared images and the texture structures in visible images. First, a salient target mask is dedicated to annotating regions of the infrared image that humans or machines pay more attention to, so as to provide spatial guidance for the integration of different information. Second, we combine this salient target mask to design a specific loss function to guide the extraction and reconstruction of features. Specifically, the feature extraction network can selectively extract salient target features from infrared images and background texture features from visible images, while the feature reconstruction network can effectively fuse these features and reconstruct the desired results. It is worth noting that the salient target mask is only required in the training phase, which enables the proposed STDFusionNet to be an end-to-end model. In other words, our STDFusionNet can fulfill salient target detection and key information fusion in an implicit manner. Extensive qualitative and quantitative experiments demonstrate the superiority of our fusion algorithm over the current state of the arts, where our algorithm is much faster and the fusion results look like high-quality visible images with clear highlighted infrared targets. Moreover, the experimental results on the public datasets reveal that our algorithm can improve the entropy (EN), mutual information (MI), visual information fidelity (VIF), and spatial frequency (SF) metrics with about 1.25%, 22.65%, 4.3%, and 0.89% gains, respectively. Our code is publicly available at https://github.com/jiayi-ma/STDFusionNet .