Automated Indian sign language recognition system by fusing deep and handcrafted feature

• Published in: Multimedia tools and applications 2023-05, Vol.82 (11), p.16905-16927
• Main Authors: Das, Soumen, Biswas, Saroj Kr, Purkayastha, Biswajit
• Format: Article
• Language: English
• Subjects: Accuracy; Artificial neural networks; Automation; Computer Communication Networks; Computer Science; Data Structures and Information Theory; Multimedia Information Systems; Sign language; Special Purpose and Application-Based Systems; Viewing; Words (language)
• ISSN: 1380-7501; 1573-7721
• DOI: 10.1007/s11042-022-14084-4

The deaf community faces some major challenges due to the communication gap with the hearing community. The traditional approach of employing a Sign Language (SL) interpreter is not an efficient and cost-effective solution to this problem. Thus, an automated Sign Language Recognition System (SLRS) is needed to provide an efficient and reliable solution. Existing SLRS for dynamic SL recognition utilizes the CNN-LSTM architecture, which has accomplished satisfactory results. However, spatial features extracted through Convolutional Neural Network (CNN) are insufficient for recognizing SL word that consists of identical hand orientation and multiple viewing angles. Thus, this paper proposes an SLRS named Automated Indian Sign Language Recognition System for Emergency Words (AISLRSEW) for recognizing ISL words which are frequently used in an emergency situation. The proposed AISLRSEW uses a combination of CNN and local handcrafted features to resolve the issue of identifying SL words with identical hand orientation and multiple viewing angles, which improves the recognition accuracy. The performance of the proposed AISLRSEW is evaluated with two fold cross-validation method and compared with existing models. The proposed model has achieved an average accuracy of 94.42%, which is comparatively better than existing models.