Hierarchical Group-Level Emotion Recognition

• Published in: IEEE transactions on multimedia 2021, Vol.23, p.3892-3906
• Main Authors: Fujii, Katsuya, Sugimura, Daisuke, Hamamoto, Takayuki
• Format: Article
• Language: English
• Subjects: Classification; Emotion recognition; Emotions; Exploitation; Face recognition; Faces; facial expression; Facial features; Feature extraction; Group-level emotion recognition; hierarchical classification; Image classification; Image enhancement; Labels; scene features; Semantics; Visualization
• ISSN: 1520-9210; 1941-0077
• DOI: 10.1109/TMM.2020.3033125

Group-level emotion recognition is a technique for estimating the emotion of a group of people. In this paper, we propose a novel method for group-level emotion recognition. Our method lies in the two-fold contributions: (1) recognition of group-level emotion using a hierarchical classification approach; (2) incorporation of novel features to contribute to the description of the group-level emotion. We consider that the use of facial expressions of people will only be effective in differentiating images labeled as "Positive" because those labeled as "Neutral" or "Negative" are likely to include similar facial expressions. Therefore, we first perform binary classification based on facial expression recognition to distinguish "Positive" labels that include discriminative facial expressions (e.g., smile) from the others. We evaluate outcomes that are not classified as "Positive" during the first classification by exploiting scene features that describe what type of events (e.g., demonstration or funeral) are shown in the image. The other novelty of our method lies in two-fold. The first is the exploitation of visual attention for the first classification. It allows us to estimate which faces are the main subjects in the target image, thereby suppressing the influences of faces in the background that contribute less to group-level emotion. The second is the exploitation of object-wise semantic information (labels) for the second classification. This allows a more detailed description of the scene context in the image and enables performance enhancement in the second classification. We demonstrate the effectiveness of our method through experiments using public datasets.