Real-time lip reading system for isolated Korean word recognition

• Published in: Pattern recognition 2011-03, Vol.44 (3), p.559-571
• Main Authors: Shin, Jongju, Lee, Jin, Kim, Daijin
• Format: Article
• Language: English
• Subjects: Applied sciences; Artificial intelligence; Audio-visual ASR; Automatic speech recognition; Classifiers; Computer science; control theory; systems; Connectionism. Neural networks; Detectors; Exact sciences and technology; Information, signal and communications theory; Lip reading; Mathematical models; Mouth; Neural networks; Pattern recognition; Recognition; Signal and communications theory; Signal processing; Signal representation. Spectral analysis; Signal, noise; Speech processing; Telecommunications and information theory; Tracking; Two-stage lip tracking; Word classifier
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2010.09.011

This paper proposes a real-time lip reading system (consisting of a lip detector, lip tracker, lip activation detector, and word classifier), which can recognize isolated Korean words. Lip detection is performed in several stages: face detection, eye detection, mouth detection, mouth end-point detection, and active appearance model (AAM) fitting. Lip tracking is then undertaken via a novel two-stage lip tracking method, where the model-based Lucas–Kanade feature tracker is used to track the outer lip, and then a fast block matching algorithm is used to track the inner lip. Lip activation detection is undertaken through a neural network classifier, the input for which being a combination of the lip motion energy function and the first dominant shape feature. In the last step, input words are defined and recognized by three different classifiers: HMM, ANN, and K-NN. We combine the proposed lip reading system with an audio-only automatic speech recognition (ASR) system to improve the word recognition performance in the noisy environments. We then demonstrate the potential applicability of the combined system for use within hands free in-vehicle navigation devices. Results from experiments undertaken on 30 isolated Korean words using the K-NN classifier at a speed of 15 fps demonstrate that the proposed lip reading system achieves a 92.67% word correct rate (WCR) for person-dependent tests, and a 46.50% WCR for person-independent tests. Also, the combined audio-visual ASR system increases the WCR from 0% to 60% in a noisy environment.