A new context-based feature for classification of emotions in photographs

• Published in: Multimedia tools and applications 2021-04, Vol.80 (10), p.15589-15618
• Main Authors: Krishnani, Divya, Shivakumara, Palaiahnakote, Lu, Tong, Pal, Umapada, Lopresti, Daniel, Kumar, Govindaraju Hemantha
• Format: Article
• Language: English
• Subjects: Benchmarks; Bullying; Classification; Comparative studies; Computer Communication Networks; Computer Science; Context; Data Structures and Information Theory; Datasets; Digital media; Emotions; Feature extraction; Image classification; Internet; Multimedia Information Systems; Special Purpose and Application-Based Systems
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-020-10404-8

A high volume of images is shared on the public Internet each day. Many of these are photographs of people with facial expressions and actions displaying various emotions. In this work, we examine the problem of classifying broad categories of emotions based on such images, including Bullying, Mildly Aggressive, Very Aggressive, Unhappy, Disdain and Happy. This work proposes the Context-based Features for Classification of Emotions in Photographs (CFCEP). The proposed method first detects faces as a foreground component, and other information (non-face) as background components to extract context features. Next, for each foreground and background component, we explore the Hanman transform to study local variations in the components. The proposed method combines the Hanman transform (H) values of foreground and background components according to their merits, which results in two feature vectors. The two feature vectors are fused by deriving weights to generate one feature vector. Furthermore, the feature vector is fed to a CNN classifier for classification of images of different emotions uploaded on social media and public internet. Experimental results on our dataset of different emotion classes and the benchmark dataset show that the proposed method is effective in terms of average classification rate. It reports 91.7% for our 10-class dataset, 92.3% for 5 classes of standard dataset and 81.4% for FERPlus dataset. In addition, a comparative study with existing methods on the benchmark dataset of 5-classes, standard dataset of facial expression (FERPlus) and another dataset of 10-classes show that the proposed method is best in terms of scalability and robustness.