AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove

• Published in: Nature communications 2021-09, Vol.12 (1), p.5378-5378, Article 5378
• Main Authors: Wen, Feng, Zhang, Zixuan, He, Tianyiyi, Lee, Chengkuo
• Format: Article
• Language: English
• Subjects: 639/166; 639/166/987; Artificial intelligence; Communication; Communication Aids for Disabled; Communications systems; Computer applications; Deafness - rehabilitation; Deep Learning; Dextrous hands; Gestures; Gloves; Humanities and Social Sciences; Humans; Language; multidisciplinary; Recognition; Recombination; Science; Science (multidisciplinary); Segmentation; Sentences; Sign Language; Virtual Reality; Wearable Electronic Devices; Words (language)
• ISSN: 2041-1723; 2041-1723
• DOI: 10.1038/s41467-021-25637-w

Sign language recognition, especially the sentence recognition, is of great significance for lowering the communication barrier between the hearing/speech impaired and the non-signers. The general glove solutions, which are employed to detect motions of our dexterous hands, only achieve recognizing discrete single gestures (i.e., numbers, letters, or words) instead of sentences, far from satisfying the meet of the signers’ daily communication. Here, we propose an artificial intelligence enabled sign language recognition and communication system comprising sensing gloves, deep learning block, and virtual reality interface. Non-segmentation and segmentation assisted deep learning model achieves the recognition of 50 words and 20 sentences. Significantly, the segmentation approach splits entire sentence signals into word units. Then the deep learning model recognizes all word elements and reversely reconstructs and recognizes sentences. Furthermore, new/never-seen sentences created by new-order word elements recombination can be recognized with an average correct rate of 86.67%. Finally, the sign language recognition results are projected into virtual space and translated into text and audio, allowing the remote and bidirectional communication between signers and non-signers. Though wearable gloves are widely used for gesture associated applications (e.g. sign language recognition), sentence identification of sign language remains a challenge. Here, the authors report AI-enabled recognition system helps barrier-free communication between signers and non-signers.