Connecting Cross-Modal Representations for Compact and Robust Multimodal Sentiment Analysis With Sentiment Word Substitution Error

• Published in: IEEE transactions on affective computing 2024-11, p.1-13
• Main Authors: Sun, Qiyuan, Zuo, Haolin, Liu, Rui, Li, Haizhou
• Format: Article
• Language: English
• Subjects: Accuracy; Acoustics; Affective computing; Contrastive learning; Correlation; Cross-Modal representations; Fuses; multimodal sentiment analysis; Sentiment analysis; sentiment word substitution error; Sun; Training; Visualization
• ISSN: 1949-3045; 1949-3045
• DOI: 10.1109/TAFFC.2024.3490694

Multimodal Sentiment Analysis (MSA) seeks to fuse textual, acoustic, and visual information to predict a speaker's sentiment states effectively. However, in real-world scenarios, the text modality received by MSA systems is often obtained through automatic speech recognition (ASR) models. Unfortunately, ASR may erroneously recognize sentiment words as phonetically similar neutral alternatives, leading to sentiment degradation in text and impacting MSA accuracy. Recent attempts aim to first identify the sentiment word substitution (SWS) error in ASR results and then refine the corrupted word embeddings using multimodal information for final multimodal fusion. However, such a method includes a burdensome and ambiguous detection operation and ignores the inherent correlations and heterogeneity among different modalities. To address these issues, we propose a more compact system, termed ARF-MSA consisting of three key components to achieving robust MSA with SWS errors: 1) Alignment : we establish connections between the "text-acoustic' and "text-visual" representations to effectively map the "text-acoustic-visual" data into a unified sentiment space by leveraging their multimodal correlation knowledge; 2) Refinement : we perform fine-grained comparisons between the text modality and the other two modalities in the unified sentiment space, enabling refinement of the sentiment expression within the text modality more concisely; 3) Fusion : Finally, we hierarchically fuse the dominant and non-dominant representation from three heterogeneity modalities to obtain the multimodal feature for MSA. We conduct extensive experiments on the real-world datasets and the results demonstrate the effectiveness of our model. Codes are available at: https://github.com/ARFMSA/ARF-MSA .