Recognition of facial expressions based on CNN features

• Published in: Multimedia tools and applications 2020-05, Vol.79 (19-20), p.13987-14007
• Main Authors: González-Lozoya, Sonia M., de la Calleja, Jorge, Pellegrin, Luis, Escalante, Hugo Jair, Medina, Ma. Auxilio, Benitez-Ruiz, Antonio
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial neural networks; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deep learning; Face recognition; Feature extraction; Feature recognition; Machine learning; Methods; Multimedia; Multimedia Information Systems; Neural networks; Special Purpose and Application-Based Systems
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-020-08681-4

Facial expressions are a natural way to communicate emotional states and intentions. In recent years, automatic facial expression recognition (FER) has been studied due to its practical importance in many human-behavior analysis tasks such as interviews, autonomous-driving, medical treatment, among others. In this paper we propose a method for facial expression recognition based on features extracted with convolutional neural networks (CNN), taking advantage of a pre-trained model in similar tasks. Unlike other approaches, the proposed FER method learns from mixed instances taken from different databases with the goal of improving generalization, a major issue in machine learning. Experimental results show that the FER method is able to recognize the six universal expressions with an accuracy above 92% considering five of the widely used databases. In addition, we have extended our method to deal with micro-expressions recognition (MER). In this regard, we propose three strategies to create a temporal-aggregated feature vector: mean, standard deviation and early fusion. In this case, the best result is 78.80% accuracy. Furthermore, we present a prototype system that implements the two proposed methods for FER and MER as a tool that allows to analyze videos.