Noise-Tolerant Learning for Audio-Visual Action Recognition

• Published in: IEEE transactions on multimedia 2024, Vol.26, p.7761-7774
• Main Authors: Han, Haochen, Zheng, Qinghua, Luo, Minnan, Miao, Kaiyao, Tian, Feng, Chen, Yan
• Format: Article
• Language: English
• Subjects: Action recognition; Activity recognition; Annotations; Audio data; audio-visual learning; Correlation; Correspondence; Datasets; Interference; Kinetic theory; Labels; Learning; Modal data; Noise measurement; noisy correspondence; noisy labels; Robustness; Task analysis; Training; Visual tasks; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2024.3371220

Recently, video recognition is emerging with the help of multi-modal learning, which focuses on integrating distinct modalities to improve the performance or robustness of the model. Although various multi-modal learning methods have been proposed and offer remarkable recognition results, almost all of these methods rely on high-quality manual annotations and assume that modalities among multi-modal data provide semantically relevant information. Unfortunately, the widely used video datasets are usually coarse-annotated or collected from the Internet. Thus, it inevitably contains a portion of noisy labels and noisy correspondence. To address this challenge, we use the audio-visual action recognition task as a proxy and propose a noise-tolerant learning framework to find anti-interference model parameters against both noisy labels and noisy correspondence. Specifically, our method consists of two phases that aim to rectify noise by the inherent correlation between modalities. First, a noise-tolerant contrastive training phase is performed to make the model immune to the possible noisy-labeled data. Despite the benefits brought by contrastive training, it would overfit the noisy correspondence and thus provide false supervision. To alleviate the influence of noisy correspondence, we propose a cross-modal noise estimation component to adjust the consistency between different modalities. As the noisy correspondence existed at the instance level, we further propose a category-level contrastive loss to reduce its interference. Second, in the hybrid-supervised training phase, we calculate the distance metric among features to obtain corrected labels, which are used as complementary supervision to guide the training. Furthermore, due to the lack of suitable datasets, we establish a benchmark of real-world noisy correspondence in audio-visual data by relabeling the Kinetics dataset. Extensive experiments on a wide range of noisy levels demonstrate that our method significantly improves the robustness of the action recognition model and surpasses the baselines by a clear margin.