Fostering inclusivity through effective communication: Real-time sign language to speech conversion system for the deaf and hard-of-hearing community

• Published in: Multimedia tools and applications 2024-05, Vol.83 (15), p.45859-45880
• Main Authors: Kaur, Binwant, Chaudhary, Aastha, Bano, Shahina, Yashmita, Reddy, S.R.N., Anand, Rishika
• Format: Article
• Language: English
• Subjects: Communication; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Datasets; Deafness; Deep learning; Hearing; Model accuracy; Multimedia Information Systems; Real time; Sign language; Special Purpose and Application-Based Systems; Speech; Track 2: Medical Applications of Multimedia
• ISSN: 1573-7721; 1380-7501; 1573-7721
• DOI: 10.1007/s11042-023-17372-9

In a world that values inclusivity, effective communication remains a cornerstone of empowerment for the deaf and hard-of-hearing community, with sign language serving as a pivotal means of expression. Yet, the limited proficiency in sign language among the general population underscores the need for innovative solutions like sign language translators. This study introduces a cutting-edge real-time sign language to speech conversion system, harnessing the power of a pre-trained InceptionResNetV2 deep learning model. To enhance accuracy, a bespoke American sign language dataset is employed, capturing 21 critical hand keypoints and sign images through Python libraries. The training dataset comprises 7200 images, categorized into 24 alphabet classes (excluding 'J' and 'Z'). Model refinement occurs over 20 epochs, each with a batch size of 16, culminating in remarkable training and validation accuracies of 98.23% and 97.07%, respectively. This impressive real-time sign language to speech conversion system, synergizing deep learning with Jetson Nano technology, paves the way for robust communication accessibility. Future advancements are envisaged, including expanding the system to support complete sentence translation and embracing diverse sign languages. By doing so, a comprehensive suite of sign language communication solutions will be offered, fostering universal understanding and inclusivity.