Audio-Visual Gated-Sequenced Neural Networks for Affect Recognition

The interest in automatic emotion recognition and the larger field of Affective Computing has recently gained momentum. The current emergence of large, video-based affect datasets offering rich multi-modal inputs facilitates the development of deep learning-based models for automatic affect analysis...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on affective computing 2023-07, Vol.14 (3), p.2193-2208
Main Authors: Aspandi, Decky, Sukno, Federico, Schuller, Bjorn W., Binefa, Xavier
Format: Article
Language:English
Subjects:
Ablation
Accuracy
Adaptation models
Affective computing
Arousal
Computational modeling
Datasets
deep learning
Emotion recognition
Emotions
Estimates
Feature extraction
Generators
Hidden Markov models
Machine learning
multi-modal fusion
Neural networks
sequence modelling
Visualization
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The interest in automatic emotion recognition and the larger field of Affective Computing has recently gained momentum. The current emergence of large, video-based affect datasets offering rich multi-modal inputs facilitates the development of deep learning-based models for automatic affect analysis that currently holds the state of the art. However, recent approaches to process these modalities cannot fully exploit them due to the use of oversimplified fusion schemes. Furthermore, the efficient use of temporal information inherent to these huge data are also largely unexplored hindering their potential progress. In this work, we propose a multi-modal, sequence-based neural network with gating mechanisms for Valence and Arousal based affect recognition. Our model consists of three major networks: Firstly, a latent-feature generator that extracts compact representations from both modalities that have been artificially degraded to add robustness. Secondly, a multi-task discriminator that estimates both input identity and a first step emotion quadrant estimation. Thirdly, a sequence-based predictor with attention and gating mechanisms that effectively merges both modalities and uses this information through sequence modelling. In our experiments on the SEMAINE and SEWA affect datasets, we observe the impact of both proposed methods with progressive increase in accuracy. We further show in our ablation studies how the internal attention weight and gating coefficient impact our models' estimates quality. Finally, we demonstrate state of the art accuracy through comparisons with current alternatives on both datasets.
ISSN:1949-3045
1949-3045
DOI:10.1109/TAFFC.2022.3156026