Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns

• Published in: Neurocomputing (Amsterdam) 2016-01, Vol.175, p.564-578
• Main Authors: Huang, Xiaohua, Zhao, Guoying, Hong, Xiaopeng, Zheng, Wenming, Pietikäinen, Matti
• Format: Article
• Language: English
• Subjects: Discriminative; LBP-TOP; Micro-expression; Vector quantization
• ISSN: 0925-2312; 1872-8286
• DOI: 10.1016/j.neucom.2015.10.096

Spontaneous facial micro-expression analysis has become an active task for recognizing suppressed and involuntary facial expressions shown on the face of humans. Recently, Local Binary Pattern from Three Orthogonal Planes (LBP-TOP) has been employed for micro-expression analysis. However, LBP-TOP suffers from two critical problems, causing a decrease in the performance of micro-expression analysis. It generally extracts appearance and motion features from the sign-based difference between two pixels but not yet considers other useful information. As well, LBP-TOP commonly uses classical pattern types which may be not optimal for local structure in some applications. This paper proposes SpatioTemporal Completed Local Quantization Patterns (STCLQP) for facial micro-expression analysis. Firstly, STCLQP extracts three interesting information containing sign, magnitude and orientation components. Secondly, an efficient vector quantization and codebook selection are developed for each component in appearance and temporal domains to learn compact and discriminative codebooks for generalizing classical pattern types. Finally, based on discriminative codebooks, spatiotemporal features of sign, magnitude and orientation components are extracted and fused. Experiments are conducted on three publicly available facial micro-expression databases. Some interesting findings about the neighboring patterns and the component analysis are concluded. Comparing with the state of the art, experimental results demonstrate that STCLQP achieves a substantial improvement for analyzing facial micro-expressions. •We propose spatiotemporal completed local quantized pattern for micro-expression analysis.•We propose to use three useful information, including the sign-based, magnitude-based and orientation-based difference of pixels for LBP.•We propose to use an efficient vector quantization and discriminative codebook selection to make LBP-TOP more discriminative and compact.•We evaluate the framework on three publicly available facial micro-expression databases.•We evaluate the influence of parameters, different components and codebook selection to spatiotemporal completed local quantized pattern.