Improvement of emotion recognition from facial images using deep learning and early stopping cross validation

• Published in: Multimedia tools and applications 2022-09, Vol.81 (21), p.29887-29917
• Main Authors: Bentoumi, Mohamed, Daoud, Mohamed, Benaouali, Mohamed, Taleb Ahmed, Abdelmalik
• Format: Article
• Language: English
• Subjects: Accuracy; Artificial neural networks; Classifiers; Computer Communication Networks; Computer Science; Correlation coefficients; Data Structures and Information Theory; Deep learning; Emotion recognition; Emotions; Engineering Sciences; Feature extraction; Likelihood ratio; Machine learning; Multilayer perceptrons; Multimedia Information Systems; Object recognition; Special Purpose and Application-Based Systems
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-12058-0

Inthis paper, we present a new approach for emotion recognition from facial images. The proposed method is based on the association of a pretrained convolutional neural network (CNN) model (VGG16, ResNet50) with a multilayer perceptron (MLP) classifier. The pretrained CNN model is used as a feature extractor. For this purpose, we adapt the original architecture by adding a global average pooling layer (GAP) without any fine tuning of the network parameters. In order to avoid overfitting for the MLP classifier, we introduce the early stopping criterion. It is proved that the aforementioned elements contribute in improving the performance of our approach in terms of generalization ability. The procedure for emotion recognition from facial images is applied on the CK+ (extended Cohen-Kanad), JAFFE (Japanese Female Facial Expression) and KDEF (Karolinska Directed Emotional Faces) databases. The k-fold cross validation procedure is used for accuracy estimation. The experimental results show the effectiveness of our facial emotion recognition (FER) approach compared to the existing methods yielding to recognition rates of 100 % , 96.40 % and 98.78 % for the CK+, JAFFE and KDEF databases, respectively. On the other hand, further improvement of our recognition performance is obtained for images from the JAFFE database by performing a data augmentation during the training phase. This allows to achieve an accuracy of 100 % for this database. Four other metrics, namely the F1-score, positive and negative likelihood ratios and Mathews correlation coefficient, confirm as well the classification results obtained from accuracy.