Multimodal Speech Emotion Recognition Using Modality-specific Self-Supervised Frameworks

Emotion recognition is a topic of significant interest in assistive robotics due to the need to equip robots with the ability to comprehend human behavior, facilitating their effective interaction in our society. Consequently, efficient and dependable emotion recognition systems supporting optimal h...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-12
Main Authors: Rutherford Agbeshi Patamia, Santos, Paulo E, Acheampong, Kingsley Nketia, Ekong, Favour, Sarpong, Kwabena, She Kun
Format: Article
Language:English
Subjects:
Data points
Deep learning
Emotion recognition
Emotions
Machine learning
Motion capture
Representations
Robotics
Self-supervised learning
Speech recognition
System effectiveness
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Emotion recognition is a topic of significant interest in assistive robotics due to the need to equip robots with the ability to comprehend human behavior, facilitating their effective interaction in our society. Consequently, efficient and dependable emotion recognition systems supporting optimal human-machine communication are required. Multi-modality (including speech, audio, text, images, and videos) is typically exploited in emotion recognition tasks. Much relevant research is based on merging multiple data modalities and training deep learning models utilizing low-level data representations. However, most existing emotion databases are not large (or complex) enough to allow machine learning approaches to learn detailed representations. This paper explores modalityspecific pre-trained transformer frameworks for self-supervised learning of speech and text representations for data-efficient emotion recognition while achieving state-of-the-art performance in recognizing emotions. This model applies feature-level fusion using nonverbal cue data points from motion capture to provide multimodal speech emotion recognition. The model was trained using the publicly available IEMOCAP dataset, achieving an overall accuracy of 77.58% for four emotions, outperforming state-of-the-art approaches
ISSN:2331-8422