A joint cascaded framework for simultaneous eye detection and eye state estimation

• Published in: Pattern recognition 2017-07, Vol.67, p.23-31
• Main Authors: Gou, Chao, Wu, Yue, Wang, Kang, Wang, Kunfeng, Wang, Fei-Yue, Ji, Qiang
• Format: Article
• Language: English
• Subjects: Cascade regression framework; Eye detection; Eye state estimation; Learning-by-synthesis
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2017.01.023

•An effective cascade regression method for simultaneous eye detection and eye state estimation is proposed.•Based on a cascade regression framework, it iteratively estimates the location of the eye and the probability of the eye being occluded by eyelid.•The regression models are learned from combination of generated synthetic photorealistic and real eye images.•Experimental results on benchmark database show that it outperforms other state-of-the-art methods both on eye detection and eye state estimation. And it achieves real time applications. Eye detection and eye state (close/open) estimation are important for a wide range of applications, including iris recognition, visual interaction and driver fatigue detection. Current work typically performs eye detection first, followed by eye state estimation by a separate classifier. Such an approach fails to capture the interactions between eye location and its state. In this paper, we propose a method for simultaneous eye detection and eye state estimation. Based on a cascade regression framework, our method iteratively estimates the location of the eye and the probability of the eye being occluded by eyelid. At each iteration of cascaded regression, image features from the eye center as well as contextual image features from eyelid and eye corners are jointly used to estimate the eye position and openness probability. Using the eye openness probability, the most likely eye state can be estimated. Since it requires large number of facial images with labeled eye related landmarks, we propose to combine the real and synthetic images for training. It further improves the performance by utilizing this learning-by-synthesis method. Evaluations of our method on benchmark databases such as BioID and Gi4E database as well as on real world driving videos demonstrate its superior performance comparing to state-of-the-art methods for both eye detection and eye state estimation.