A Classification Model Utilizing Facial Landmark Tracking to Determine Sentence Types for American Sign Language Recognition

The deaf and hard of hearing community relies on American Sign Language (ASL) as their primary mode of communication, but communication with others who do not know ASL can be difficult, especially during emergencies where no interpreter is available. As an effort to alleviate this problem, research...

Full description

Saved in:
Bibliographic Details
Main Authors: Nguyen, Janice, Wang, Y. Curtis
Format: Conference Proceeding
Language:English
Subjects:
Analytical models
Assistive technologies
Biological system modeling
Computational modeling
Computer vision
Gesture recognition
Gestures
Hearing Loss
Humans
Recognition, Psychology
Reproducibility of Results
Shape
Sign Language
United States
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The deaf and hard of hearing community relies on American Sign Language (ASL) as their primary mode of communication, but communication with others who do not know ASL can be difficult, especially during emergencies where no interpreter is available. As an effort to alleviate this problem, research in computer vision based real time ASL interpreting models is ongoing. However, most of these models are hand shape (gesture) based and lack the integration of facial cues, which are crucial in ASL to convey tone and distinguish sentence types. Thus, the integration of facial cues in computer vision based ASL interpreting models has the potential to improve performance and reliability. In this paper, we introduce a simple, computationally efficient facial expression based classification model that can be used to improve ASL interpreting models. This model utilizes the relative angles of facial landmarks with principal component analysis and a Random Forest Classification tree model to classify frames taken from videos of ASL users signing a complete sentence. The model classifies the frames as statements or assertions. The model was able to achieve an accuracy of 86.5%.
ISSN:2694-0604
DOI:10.1109/EMBC40787.2023.10340217