Decoding lip language using triboelectric sensors with deep learning

• Published in: Nature communications 2022-03, Vol.13 (1), p.1401-1401, Article 1401
• Main Authors: Lu, Yijia, Tian, Han, Cheng, Jia, Zhu, Fei, Liu, Bin, Wei, Shanshan, Ji, Linhong, Wang, Zhong Lin
• Format: Article
• Language: English
• Subjects: 639/301/1005; 639/766/1130; Communication; Deep Learning; Directional control; Electric contacts; Electrical properties; Flexible components; Humanities and Social Sciences; Humans; Language; Language translation; Laryngology; Lip; Lips; multidisciplinary; Neural networks; Prototypes; Recurrent neural networks; Science; Science (multidisciplinary); Self concept; Sensors; Speech; Voice; Voice communication
• ISSN: 2041-1723; 2041-1723
• DOI: 10.1038/s41467-022-29083-0

Lip language is an effective method of voice-off communication in daily life for people with vocal cord lesions and laryngeal and lingual injuries without occupying the hands. Collection and interpretation of lip language is challenging. Here, we propose the concept of a novel lip-language decoding system with self-powered, low-cost, contact and flexible triboelectric sensors and a well-trained dilated recurrent neural network model based on prototype learning. The structural principle and electrical properties of the flexible sensors are measured and analysed. Lip motions for selected vowels, words, phrases, silent speech and voice speech are collected and compared. The prototype learning model reaches a test accuracy of 94.5% in training 20 classes with 100 samples each. The applications, such as identity recognition to unlock a gate, directional control of a toy car and lip-motion to speech conversion, work well and demonstrate great feasibility and potential. Our work presents a promising way to help people lacking a voice live a convenient life with barrier-free communication and boost their happiness, enriches the diversity of lip-language translation systems and will have potential value in many applications. Lip-language decoding systems are a promising technology to help people lacking a voice live a convenient life with barrier-free communication. Here, authors propose a concept of such system integrating self-powered triboelectric sensors and a well-trained dilated RNN model based on prototype learning.