Feature fusion of multi-granularity and multi-scale for facial expression recognition

Although great progress has been made in facial expression recognition, it still faces challenges such as occlusion and pose changes in real-world scenario. To address this issue, we propose a simple yet effective multi-granularity and multi-scale feature fusion network (MM-Net) to achieve robust ex...

Full description

Saved in:
Bibliographic Details
Published in:The Visual computer 2024-03, Vol.40 (3), p.2035-2047
Main Authors: Xia, Haiying, Lu, Lidan, Song, Shuxiang
Format: Article
Language:English
Subjects:
Artificial Intelligence
Computer Graphics
Computer Science
Datasets
Emotions
Face recognition
Feature extraction
Image Processing and Computer Vision
Methods
Occlusion
Original Article
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Although great progress has been made in facial expression recognition, it still faces challenges such as occlusion and pose changes in real-world scenario. To address this issue, we propose a simple yet effective multi-granularity and multi-scale feature fusion network (MM-Net) to achieve robust expression recognition without either manually extracting local patches or designing complex sub-networks. Specifically, we use a puzzle generator to divide the image into local regions of different granularity, which are then randomly shuffled and reorganized to form a new input image. By feeding the facial puzzles in order from fine-grained to coarse-grained, the network progressively mines the local fine-grained information, the coarse-grained information, and the global information. Besides, considering the subtle inter-class variation characteristic of different expressions, we use the multi-scale feature fusion strategy in the shallow feature extraction module to obtain global features with detailed information for capturing the subtle differences in facial expression images. Extensive experimental results on three in-the-wild FER benchmarks demonstrate the superiority of the proposed MM-Net compared to state-of-the-art methods.
ISSN:0178-2789
1432-2315
DOI:10.1007/s00371-023-02900-3