Contrastive Learning Based Modality-Invariant Feature Acquisition for Robust Multimodal Emotion Recognition With Missing Modalities

Multimodal emotion recognition (MER) aims to understand the way that humans express their emotions by exploring complementary information across modalities. However, it is hard to guarantee that full-modality data is always available in real-world scenarios. To deal with missing modalities, research...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on affective computing 2024-10, Vol.15 (4), p.1856-1873
Main Authors: Liu, Rui, Zuo, Haolin, Lian, Zheng, Schuller, Bjorn W., Li, Haizhou
Format: Article
Language:English
Subjects:
Benchmark testing
Contrastive learning
Emotion recognition
Emotions
Feature extraction
Image reconstruction
Imagination
invariant feature
Invariants
Learning
missing modality imagination
modality gap
Modules
multimodal emotion recognition
Performance evaluation
Robustness
Self-supervised learning
Training
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Multimodal emotion recognition (MER) aims to understand the way that humans express their emotions by exploring complementary information across modalities. However, it is hard to guarantee that full-modality data is always available in real-world scenarios. To deal with missing modalities, researchers focused on meaningful joint multimodal representation learning during cross-modal missing modality imagination. However, the cross-modal imagination mechanism is highly susceptible to errors due to the "modality gap" issue, which affects the imagination accuracy, thus, the final recognition performance. To this end, we introduce the concept of a modality-invariant feature into the missing modality imagination network, which contains two key modules: 1) a novel contrastive learning-based module to extract modality-invariant features under full modalities and 2) a robust imagination module based on imagined invariant features to reconstruct missing information under missing conditions. Finally, we incorporate imagined and available modalities for emotion recognition. Experimental results on benchmark datasets demonstrate that our proposed method outperforms existing state-of-the-art strategies. Compared with our previous work, our extended version is more effective on multimodal emotion recognition with missing modalities.
ISSN:1949-3045
1949-3045
DOI:10.1109/TAFFC.2024.3378570