RGB-T Saliency Detection Based on Multiscale Modal Reasoning Interaction

How to explore the interaction between RGB and thermal infrared modalities is critical to the success of RGB-T salient object detection (SOD). Most existing methods collect and convey information by collecting and conveying information from both modalities without exploring the implicit relationship...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on instrumentation and measurement 2024, Vol.73, p.1-15
Main Authors: Wu, Yunhe, Jia, Tong, Chang, Xingya, Wang, Hao, Chen, Dongyue
Format: Article
Language:English
Subjects:
Cognition
Edge enhancement
Feature extraction
graph-based inference networks
Image edge detection
Lighting
multimodal fusion
multiscale feature
Object detection
RGB-T salient object detection (SOD)
Saliency detection
Task analysis
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:How to explore the interaction between RGB and thermal infrared modalities is critical to the success of RGB-T salient object detection (SOD). Most existing methods collect and convey information by collecting and conveying information from both modalities without exploring the implicit relationship between the two modalities. To address this problem, we innovatively propose a multiscale modal inference interaction network that fully realizes the advantageous expression and complementary fusion of the two modalities by inferring the implicit expression between them. Specifically, we design a multiscale attention feature (MSAF) extraction module to adaptively balance the relationship between shallow image details and deep image semantic features, thus enhancing the feature representation of the network at different scales. Then, we design a cross-modal interaction module based on the graph model (CMGM) to utilize the inference capability of graph models to explore complementary information in visible and infrared images and fuse them effectively. In addition, we propose the edge-enhanced saliency map generation module (EEM) to improve the accuracy of salient object boundaries and optimize the generation of saliency maps. Extensive experiments show that our method achieves the best performance on all three publicly available datasets compared to state-of-the-art RGB-T saliency detection methods, proving its superiority for saliency detection tasks.
ISSN:0018-9456
1557-9662
DOI:10.1109/TIM.2024.3419115