SKEAFN: Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis

• Published in: Information fusion 2023-12, Vol.100, p.101958, Article 101958
• Main Authors: Zhu, Chuanbo, Chen, Min, Zhang, Sheng, Sun, Chao, Liang, Han, Liu, Yifan, Chen, Jincai
• Format: Article
• Language: English
• Subjects: External knowledge; Multi-head attention; Multi-view learning; Multimodal sentiment analysis; Multiple feature fusion
• ISSN: 1566-2535; 1872-6305
• DOI: 10.1016/j.inffus.2023.101958

Multimodal sentiment analysis is an active research field that aims to recognize the user’s sentiment information from multimodal data. The primary challenge in this field is to develop a high-quality fusion framework that effectively addresses the heterogeneity among different modalities. However, prior research has primarily concentrated on intermodal interactions while neglecting the semantic sentiment information conveyed by words in the text modality. In this paper, we propose the Sentiment Knowledge Enhanced Attention Fusion Network (SKEAFN), a novel end-to-end fusion network that enhances multimodal fusion by incorporating additional sentiment knowledge representations from an external knowledge base. Firstly, we construct an external knowledge enhancement module to acquire additional representations for the text modality. Then, we design a text-guided interaction module that facilitates the interaction between text and the visual/acoustic modality. Finally, we propose a feature-wised attention fusion module that achieves multimodal fusion by dynamically adjusting the weights of the additional and each modality’s representations. We evaluate our method on three challenging multimodal sentiment analysis datasets: CMU-MOSI, CMU-MOSEI, and Twitter2019. The experiment results demonstrate that our model significantly outperforms the state-of-the-art models. The source code is publicly available at https://github.com/doubibobo/SKEAFN. •Propose a sentiment knowledge-enhanced attention fusion network (SKEAFN).•Build an additional knowledge graph to leverage explicit sentiment information.•Use a multi-head attention mechanism to model the interactions among modalities.•Develop feature-wised attention to adjust the contributions of multiple modalities.