Utilizing an Emotional Robot Capable of Lip-Syncing in Robot-Assisted Speech Therapy Sessions for Children with Language Disorders

• Published in: International journal of social robotics 2023, Vol.15 (2), p.165-183
• Main Authors: Esfandbod, Alireza, Rokhi, Zeynab, Meghdari, Ali F., Taheri, Alireza, Alemi, Minoo, Karimi, Mahdieh
• Format: Article
• Language: English
• Subjects: Algorithms; Consonants (speech); Control; Disorders; Engineering; Face recognition; Language disorders; Mechatronics; Morphing; Robotics; Robots; Service robots; Speech recognition; Speech therapy; Statistical analysis; Therapists; Therapy; Vowels
• ISSN: 1875-4791; 1875-4805
• DOI: 10.1007/s12369-022-00946-2

This study scrutinizes the impacts of utilizing a socially assistive robot, the RASA robot, during speech therapy sessions for children with language disorders. Two capabilities were developed for the robotic platform to enhance children-robot interactions during speech therapy interventions: facial expression communication (containing recognition and expression) and lip-syncing. Facial expression recognition was conducted by training several well-known CNN architectures on one of the most extensive facial expressions databases, the AffectNet database, and then modifying them using the transfer learning strategy performed on the CK+ dataset. The robot’s lip-syncing capability was designed in two steps. The first step was concerned with designing precise schemes of the articulatory elements needed during the pronunciation of the Persian phonemes (i.e., consonants and vowels). The second step included developing an algorithm to pronounce words by disassembling them into their components (including consonants and vowels) and then morphing them into each other successively. To pursue the study’s primary goal, two comparable groups of children with language disorders were considered, the intervention and control groups. The intervention group attended therapy sessions in which the robot acted as the therapist’s assistant, while the control group only communicated with the human therapist. The study’s first purpose was to compare the children’s engagement while playing a mimic game with the affective robot and the therapist, conducted via video coding. The second objective was to assess the efficacy of the robot’s presence in the speech therapy sessions alongside the therapist, accomplished by administering the Persian Test of Language Development, Persian TOLD. According to the first scenario, playing with the affective robot is more engaging than playing with the therapist. Furthermore, the statistical analysis of the study’s results indicates that participating in robot-assisted speech therapy (RAST) sessions enhances children with language disorders’ achievements in comparison with taking part in conventional speech therapy interventions.