A Novel Decoupled Prototype Completion Network for Incomplete Multimodal Emotion Recognition

Reconstructing missing modality based on available modalities is widely used to address inevitable modality-missing for Multimodal Emotion Recognition (MER). However, due to explicit distribution gap across heterogeneous modalities, they fail to guarantee the consistency between the reconstructed da...

Full description

Saved in:
Bibliographic Details
Main Authors: Hu, Zhangfeng, Zheng, Wenming, Zong, Yuan, Wei, Mengting, Jiang, Xingxun, Shi, Mengxin
Format: Conference Proceeding
Language:English
Subjects:
Emotion recognition
Incomplete multimodal
Knowledge distillation
Multimodal emotion recognition
Prototype completion
Prototypes
Weight measurement
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reconstructing missing modality based on available modalities is widely used to address inevitable modality-missing for Multimodal Emotion Recognition (MER). However, due to explicit distribution gap across heterogeneous modalities, they fail to guarantee the consistency between the reconstructed data and the ground truth. To mitigate this problem, we propose a novel method to restore the missing modality using its weighted prototypes rather than other modalities. Specifically, prototypes of different classes of missing modality are used to encapsulate its representative knowledge. Then sample-to-prototype affinity measuring class similarity is used as weights to combine these prototypes for reconstruction, thereby effectively restoring the distribution-consistent modality. Furthermore, to improve the efficacy of prototype-based completion under seriously missing, we devise an adaptive knowledge distillation from the strong modality to the weaker ones. This reinforces the representation ability of weak modality features. Extensive experiments on CMU-MOSI and IEMOCAP datasets demonstrate the superiority of our method.
ISSN:1945-788X
DOI:10.1109/ICME57554.2024.10687673