Cultural-Aware AI Model for Emotion Recognition

• Main Authors: Baradaran, Mehrdad, Zohari, Payam, Mahyar, Abtin, Motamednia, Hossein, Rahmati, Dara, Gorgin, Saeid
• Format: Conference Proceeding
• Language: English
• Subjects: Annotations; Deep learning; DeepFeature Extraction; Emotion recognition; Image recognition; Image Representation; Soft sensors; Speech recognition; Text Representation; Visualization
• ISSN: 2166-6784
• DOI: 10.1109/MVIP62238.2024.10491176

Emotion AI is a research domain that aims to understand human emotions from visual or textual data. However, existing methods often ignore the influence of cultural diversity on emotional interpretation. In this paper, we propose a multimodal deep learning model that integrates cultural awareness into emotion recognition. Our model uses images as the primary data source and comments from individuals across different regions as the secondary data source. Our results show that our model achieves robust performance across various scenarios. Our contribution is to introduce a novel fusion approach that bridges cultural gaps and fosters a more nuanced understanding of emotions. Due to the best of our knowledge, few works are using this approach, for Emotion AI, combining different types of data sources and models. We evaluate our model on the ArtELingo dataset, which contains image-comment pairs with Chinese, Arabic, and English annotations. The experimental results in the evaluation phase demonstrate an impressive 80% recognition accuracy for the model that merges image-text features.