Automatic Recognition of Mexican Sign Language Using a Depth Camera and Recurrent Neural Networks

Automatic sign language recognition is a challenging task in machine learning and computer vision. Most works have focused on recognizing sign language using hand gestures only. However, body motion and facial gestures play an essential role in sign language interaction. Taking this into account, we...

Full description

Saved in:
Bibliographic Details
Published in:Applied sciences 2022-06, Vol.12 (11), p.5523
Main Authors: Mejía-Peréz, Kenneth, Córdova-Esparza, Diana-Margarita, Terven, Juan, Herrera-Navarro, Ana-Marcela, García-Ramírez, Teresa, Ramírez-Pedraza, Alfonso
Format: Article
Language:English
Subjects:
Ablation
Accuracy
Cameras
Classification
Communication
Computer vision
Datasets
Deafness
Discriminant analysis
Hearing loss
Language
Model accuracy
Neural networks
Recurrent neural networks
RGB-D camera
Semantics
Sensors
Sign language
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Automatic sign language recognition is a challenging task in machine learning and computer vision. Most works have focused on recognizing sign language using hand gestures only. However, body motion and facial gestures play an essential role in sign language interaction. Taking this into account, we introduce an automatic sign language recognition system based on multiple gestures, including hands, body, and face. We used a depth camera (OAK-D) to obtain the 3D coordinates of the motions and recurrent neural networks for classification. We compare multiple model architectures based on recurrent networks such as Long Short-Term Memories (LSTM) and Gated Recurrent Units (GRU) and develop a noise-robust approach. For this work, we collected a dataset of 3000 samples from 30 different signs of the Mexican Sign Language (MSL) containing features coordinates from the face, body, and hands in 3D spatial coordinates. After extensive evaluation and ablation studies, our best model obtained an accuracy of 97% on clean test data and 90% on highly noisy data.
ISSN:2076-3417
2076-3417
DOI:10.3390/app12115523