Dependency-Aware Attention Control for Image Set-Based Face Recognition

• Published in: IEEE transactions on information forensics and security 2020, Vol.15, p.1501-1512
• Main Authors: Liu, Xiaofeng, Guo, Zhenhua, You, Jane, Vijaya Kumar, B. V. K.
• Format: Article
• Language: English
• Subjects: actor-critic; attention control; Deep reinforcement learning; Dependence; Face; Face recognition; Facial recognition technology; Image quality; Learning; Markov processes; Measurement; Object recognition; Quality assessment; set-to-set; Task analysis; Testing; Training; Videos
• ISSN: 1556-6013; 1556-6021
• DOI: 10.1109/TIFS.2019.2938418

This paper considers the problem of image set-based face verification and identification. Unlike traditional single sample (an image or a video) setting, this situation assumes the availability of a set of heterogeneous collection of orderless images and videos. The samples can be taken at different check points, different identity documents etc . The importance of each image is usually considered either equal or based on a quality assessment of that image independent of other images and/or videos in that image set. How to model the relationship of orderless images within a set remains a challenge. We address this problem by formulating it as a Markov Decision Process (MDP) in a latent space. Specifically, we first propose a dependency-aware attention control (DAC) network, which uses actor-critic reinforcement learning for attention decision of each image to exploit the correlations among the unordered images. An off-policy experience replay is introduced to speed up the learning process. Moreover, the DAC is combined with a temporal model for videos using divide and conquer strategies. We also introduce a pose-guided representation (PGR) scheme that can further boost the performance at extreme poses. We propose a parameter-free PGR without the need for training as well as a novel metric learning-based PGR for pose alignment without the need for pose detection in testing stage. Extensive evaluations on IJB-A/B/C, YTF, Celebrity-1000 datasets demonstrate that our method outperforms many state-of-art approaches on the set-based as well as video-based face recognition databases.