Beyond superficial emotion recognition: Modality-adaptive emotion recognition system

• Published in: Expert systems with applications 2024-01, Vol.235, p.121097, Article 121097
• Main Authors: Kang, Dohee, Kim, Daeha, Kang, Donghyun, Kim, Taein, Lee, Bowon, Kim, Deokhwan, Song, Byung Cheol
• Format: Article
• Language: English
• Subjects: Emotion recognition; Multi-modal fusion; Real-time application
• ISSN: 0957-4174
• DOI: 10.1016/j.eswa.2023.121097

With the rapid development of deep learning, emotion recognition from facial expressions has also greatly improved, but there are still limitations in terms of reliability when applied to the real world. In other words, facial expressions and the corresponding true emotions may be inconsistent. So, although audio or bio signals are additionally used to improve the reliability of emotion recognition, reasonable estimation performance or real-time operation of emotion recognition is not guaranteed yet. This paper addresses both of the above challenging issues. Each sensor input and feature extraction process is totally and asynchronously processed through a parallel processing library. And then, we predict the comprehensive state of the subject’s internal/external emotions through modality-adaptive fusion, which considers the influence of the features of each modality. We verified the performance of the proposed system through a real-time pilot test, and achieved accuracy of up to 33% higher than emotion recognition using only external signals, i.e., video and audio. •Real-time multimodal emotion recognition with visual, audio, and biological signals.•The mismatch between the inner and external expressions was considered first time.•An adaptive fusion method for adaptively analyzing a subject’s internal/external signals.•Our system can capture true emotions of subjects in a real-world environment.