Representation Selective Coupling via Token Sparsification for Multi-Spectral Object Re-Identification

To tackle the challenge of single-spectral object re-identification in complex and dynamic lighting scenarios, multi-spectral object re-identification, which integrates visible light and infrared information, is gradually taking the lead. Nevertheless, the significant heterogeneity across spectra ca...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on circuits and systems for video technology 2024-11, p.1-1
Main Authors: Yu, Zhi, Huang, Zhiyong, Hou, Mingyang, Pei, Jiaming, Yan, Yan, Liu, Yushi, Sun, Daming
Format: Article
Language:English
Subjects:
Attention-Fourier token sparsification
Benchmark testing
Circuits and systems
Couplings
Data mining
Feature extraction
Image reconstruction
Information unification constraint
Multi-spectral object re-identification
Object recognition
Representation selective coupling
Sun
Surveillance
Transformers
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To tackle the challenge of single-spectral object re-identification in complex and dynamic lighting scenarios, multi-spectral object re-identification, which integrates visible light and infrared information, is gradually taking the lead. Nevertheless, the significant heterogeneity across spectra causes formidable obstacles for this task. Most existing approaches alleviate inter-spectral disparities by amalgamating representations from different spectra, ignoring the selection of spectrum-specific crucial information. To address this issue, we propose a novel Representation Selective Coupling Network (RSCNet) for multi-spectral object re-identification. Specifically, we design an Attention-Fourier Token Sparsification (AFTS) module to adaptively sparse and join tokens from multi-spectral images in the attention domain and Fourier domain. This not only preserves spectrum-specific crucial information but also reduces inter-spectral gaps by selective coupling of multi-spectral representation. Meanwhile, to further align multi-spectral information and guide the model to learn more discriminative representation, we propose an Information Unification Constraint (IUC) learning strategy. Both feature-level information constraint and distribution-level information constraint are simultaneously deployed in IUC. Finally, we conduct extensive experiments on three multi-spectral object re-identification benchmarks, and the experimental results verify the effectiveness of our proposed method.
ISSN:1051-8215
DOI:10.1109/TCSVT.2024.3509817