Decoding silent speech commands from articulatory movements through soft magnetic skin and machine learning

• Published in: Materials horizons 2023-11, Vol.1 (12), p.567-562
• Main Authors: Dong, Penghao, Li, Yizong, Chen, Si, Grafstein, Justin T, Khan, Irfaan, Yao, Shanshan
• Format: Article
• Language: English
• Subjects: Accuracy; Communication; Data acquisition; Humans; Machine learning; Magnetic Phenomena; Movement; Sensors; Smartphones; Speech; Speech Perception; Speech recognition; Voice communication; Voice recognition
• ISSN: 2051-6347; 2051-6355
• DOI: 10.1039/d3mh01062g

Silent speech interfaces have been pursued to restore spoken communication for individuals with voice disorders and to facilitate intuitive communications when acoustic-based speech communication is unreliable, inappropriate, or undesired. However, the current methodology for silent speech faces several challenges, including bulkiness, obtrusiveness, low accuracy, limited portability, and susceptibility to interferences. In this work, we present a wireless, unobtrusive, and robust silent speech interface for tracking and decoding speech-relevant movements of the temporomandibular joint. Our solution employs a single soft magnetic skin placed behind the ear for wireless and socially acceptable silent speech recognition. The developed system alleviates several concerns associated with existing interfaces based on face-worn sensors, including a large number of sensors, highly visible interfaces on the face, and obtrusive interconnections between sensors and data acquisition components. With machine learning-based signal processing techniques, good speech recognition accuracy is achieved (93.2% accuracy for phonemes, and 87.3% for a list of words from the same viseme groups). Moreover, the reported silent speech interface demonstrates robustness against noises from both ambient environments and users' daily motions. Finally, its potential in assistive technology and human-machine interactions is illustrated through two demonstrations - silent speech enabled smartphone assistants and silent speech enabled drone control. This article introduces a wireless, unobtrusive, and robust silent speech interface based on soft magnetic skin and machine learning. The magnetic skin precisely decodes articulatory movements at the temporomandibular joint for speech recognition.