Identity-Aware Facial Expression Recognition Via Deep Metric Learning Based on Synthesized Images

• Published in: IEEE transactions on multimedia 2022, Vol.24, p.3327-3339
• Main Authors: Huang, Wei, Zhang, Siyuan, Zhang, Peng, Zha, Yufei, Fang, Yuming, Zhang, Yanning
• Format: Article
• Language: English
• Subjects: Datasets; Deep learning; Face recognition; facial expression recognition; Feature extraction; Generative adversarial networks; Image recognition; Image synthesis; Learning; Measurement; metric learning; Neural networks; Object recognition; Performance evaluation; person-dependent; Synthesis; Task analysis; Training
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2021.3096068

Person-dependent facial expression recognition has received considerable research attention in recent years. Unfortunately, different identities can adversely influence recognition accuracy, and the recognition task becomes challenging. Other adverse factors, including limited training data and improper measures of facial expressions, can further contribute to the above dilemma. To solve these problems, a novel identity-aware method is proposed in this study. Furthermore, this study also represents the first attempt to fulfill the challenging person-dependent facial expression recognition task based on deep metric learning and facial image synthesis techniques. Technically, a StarGAN is incorporated to synthesize facial images depicting different but complete basic emotions for each identity to augment the training data. Then, a deep-convolutional-neural-network-based network is employed to automatically extract latent features from both real facial images and all synthesized facial images. Next, a Mahalanobis metric network trained based on extracted latent features outputs a learned metric that measures facial expression differences between images, and the recognition task can thus be realized. Extensive experiments based on several well-known publicly available datasets are carried out in this study for performance evaluations. Person-dependent datasets, including CK+, Oulu (all 6 subdatasets), MMI, ISAFE, ISED, etc., are all incorporated. After comparing the new method with several popular or state-of-the-art facial expression recognition methods, its superiority in person-dependent facial expression recognition can be proposed from a statistical point of view.