Deep Auto-Encoders With Sequential Learning for Multimodal Dimensional Emotion Recognition

• Published in: IEEE transactions on multimedia 2022-01, Vol.24, p.1313-1324
• Main Authors: Nguyen, Dung, Nguyen, Duc Thanh, Zeng, Rui, Nguyen, Thanh Thi, N. Tran, Son, Nguyen, Thin, Sridharan, Sridha, Fookes, Clinton
• Format: Article
• Language: English
• Subjects: Affective computing; Artificial neural networks; Auto-encoder; Coders; Computer architecture; Convolution; dimensional emotion recognition; Emotion recognition; Emotions; Feature extraction; Long short term memory; Machine learning; multimodal emotion recognition; Short term; Streaming media; Streams; Two dimensional displays; Visual signals; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2021.3063612

Multimodal dimensional emotion recognition has drawn a great attention from the affective computing community and numerous schemes have been extensively investigated, making a significant progress in this area. However, several questions still remain unanswered for most of existing approaches including: (i) how to simultaneously learn compact yet representative features from multimodal data, (ii) how to effectively capture complementary features from multimodal streams, and (iii) how to perform all the tasks in an end-to-end manner. To address these challenges, in this paper, we propose a novel deep neural network architecture consisting of a two-stream auto-encoder and a long short term memory for effectively integrating visual and audio signal streams for emotion recognition. To validate the robustness of our proposed architecture, we carry out extensive experiments on the multimodal emotion in the wild dataset: RECOLA. Experimental results show that the proposed method achieves state-of-the-art recognition performance.