Occlusion robust sign language recognition system for indian sign language using CNN and pose features

The Sign Language Recognition System (SLRS) is a cutting-edge technology that aims to enhance communication accessibility for the deaf community in India by replacing the traditional approach of using a human interpreter. However, the existing SLRS for Indian Sign Language (ISL) do not focus on some...

Full description

Saved in:
Bibliographic Details
Published in:Multimedia tools and applications 2024, Vol.83 (36), p.84141-84160
Main Authors: Das, Soumen, Biswas, Saroj Kr, Purkayastha, Biswajit
Format: Article
Language:English
Subjects:
Artificial neural networks
Classification
Computer Communication Networks
Computer Science
Data Structures and Information Theory
Datasets
Feature extraction
Multimedia Information Systems
Occlusion
Robustness
Sign language
Special Purpose and Application-Based Systems
Track 3: Biometrics and HCI
Viewing
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The Sign Language Recognition System (SLRS) is a cutting-edge technology that aims to enhance communication accessibility for the deaf community in India by replacing the traditional approach of using a human interpreter. However, the existing SLRS for Indian Sign Language (ISL) do not focus on some major problems including occlusion, similar hand gesture, multi viewing angle problem and inefficiency due to extracting features from a large sequence of frame that contains redundant and unnecessary information. Therefore, in this research paper an occlusion robust SLRS named Multi Featured Deep Network (MF-DNet) is proposed for recognizing ISL words. The suggested MF-DNet uses a histogram difference based keyframe selection technique to remove redundant frames. To resolve occlusion, similar hand gesture, and multi viewing angle problem the suggested MF-DNet incorporates pose features with Convolution Neural Network (CNN) features. For classification the proposed system uses Bi Directional Long Shor Term Memory (BiLSTM) network, which is compared with different classifier such as LSTM, ConvLSTM and stacked LSTM networks. The proposed SLRS achieved an average classification accuracy of 96.88% on the ISL dataset, 99.06% on the benchmark LSA64 dataset and 99.85% on the WLASL dataset. The results obtained from the MF-DNet is compared with some of the existing SLRS where the proposed method outperformed the existing methods.
ISSN:1573-7721
1380-7501
1573-7721
DOI:10.1007/s11042-024-19068-0