SiamTFA: Siamese Triple-Stream Feature Aggregation Network for Efficient RGBT Tracking

• Published in: IEEE transactions on intelligent transportation systems 2024-12, p.1-14
• Main Authors: Zhang, Jianming, Qin, Yu, Fan, Shimeng, Xiao, Zhu, Zhang, Jin
• Format: Article
• Language: English
• Subjects: Computational modeling; Correlation; Feature extraction; Fuses; lightweight attention; multi-modal feature fusion; Object tracking; Real-time systems; RGBT tracking; Semantics; siamese network; Streams; Target tracking; Transformers; triple-stream network
• ISSN: 1524-9050; 1558-0016
• DOI: 10.1109/TITS.2024.3512551

RGBT tracking is a task that utilizes images from visible (RGB) and thermal infrared (TIR) modalities to continuously locate a target, which plays an important role in various fields including intelligent transportation systems. Most existing RGBT trackers do not achieve high precision and real-time tracking speed simultaneously. To address this challenge, we propose an innovative RGBT tracker, the Siamese Triple-stream Feature Aggregation Network (SiamTFA). Firstly, a triple-stream backbone is presented to implement multi-modal feature extraction and fusion, which contains two parallel Swin Transformer feature extraction streams, and one feature fusion stream composed of joint-complementary feature aggregation (JCFA) modules. Secondly, our proposed JCFA module utilizes a joint-complementary attention to guide the aggregation of multi-modal features. Specifically, the joint attention can focus on spatial location information and semantic information of the target by combining the features of two modalities. Considering the complementarity between RGB and TIR modalities, the complementary attention is introduced to enhance the information of beneficial modality and suppress the information of ineffective modality. Thirdly, in order to reduce the computational complexity of the joint-complementary attention, we propose a depthwise shared attention structure, which utilizes depthwise convolution and shared features to achieve lightweight attention. Finally, we conduct extensive experiments on four official RGBT test datasets and the experimental results demonstrate that our proposed tracker outperforms some state-of-the-art trackers and the tracking speed reaches 37 frames per second (FPS). The code is available at https://github.com/zjjqinyu/SiamTFA.