Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism

• Published in: Electronics (Basel) 2024-06, Vol.13 (11), p.2069
• Main Authors: Li, Hongchan, Lu, Yantong, Zhu, Haodong
• Format: Article
• Language: English
• Subjects: Ablation; Classification; Computational linguistics; Data mining; Deep learning; Dictionaries; Emotions; Esports; Image classification; Language processing; Machine learning; Natural language interfaces; Neural networks; Product reviews; Research methodology; Semantics; Sentiment analysis; Social networks
• ISSN: 2079-9292; 2079-9292
• DOI: 10.3390/electronics13112069

Research on uni-modal sentiment analysis has achieved great success, but emotions in real life are mostly multi-modal; there are not only texts but also images, audio, video, and other forms. The various modes play a role in mutual promotion. If the connection between various modalities can be mined, the accuracy of sentiment analysis will be further improved. To this end, this paper introduces a cross-attention-based multi-modal fusion model for images and text, namely, MCAM. First, we use the ALBert pre-training model to extract text features for text; then, we use BiLSTM to extract text context features; then, we use DenseNet121 to extract image features for images; and then, we use CBAM to extract specific areas related to emotion in images. Finally, we utilize multi-modal cross-attention to fuse the extracted features from the text and image, and we classify the output to determine the emotional polarity. In the experimental comparative analysis of MVSA and TumEmo public datasets, the model in this article is better than the baseline model, with accuracy and F1 scores reaching 86.5% and 75.3% and 85.5% and 76.7%, respectively. In addition, we also conducted ablation experiments, which confirmed that sentiment analysis with multi-modal fusion is better than single-modal sentiment analysis.