Sign Language Motion Generation from Sign Characteristics

This paper proposes, analyzes, and evaluates a deep learning architecture based on transformers for generating sign language motion from sign phonemes (represented using HamNoSys: a notation system developed at the University of Hamburg). The sign phonemes provide information about sign characterist...

Full description

Saved in:
Bibliographic Details
Published in:Sensors (Basel, Switzerland) Switzerland), 2023-11, Vol.23 (23), p.9365
Main Authors: Gil-Martín, Manuel, Villa-Monedero, María, Pomirski, Andrzej, Sáez-Trigueros, Daniel, San-Segundo, Rubén
Format: Article
Language:English
Subjects:
Algorithms
Automation
Communication
Datasets
Deafness
Deep learning
HamNoSys
Hand
Humans
landmarks extraction
Motion
motion dataset
motion generation
Movement
Natural language
Sign Language
sign phonemes
Speech
Time series
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper proposes, analyzes, and evaluates a deep learning architecture based on transformers for generating sign language motion from sign phonemes (represented using HamNoSys: a notation system developed at the University of Hamburg). The sign phonemes provide information about sign characteristics like hand configuration, localization, or movements. The use of sign phonemes is crucial for generating sign motion with a high level of details (including finger extensions and flexions). The transformer-based approach also includes a stop detection module for predicting the end of the generation process. Both aspects, motion generation and stop detection, are evaluated in detail. For motion generation, the dynamic time warping distance is used to compute the similarity between two landmarks sequences (ground truth and generated). The stop detection module is evaluated considering detection accuracy and ROC (receiver operating characteristic) curves. The paper proposes and evaluates several strategies to obtain the system configuration with the best performance. These strategies include different padding strategies, interpolation approaches, and data augmentation techniques. The best configuration of a fully automatic system obtains an average DTW distance per frame of 0.1057 and an area under the ROC curve (AUC) higher than 0.94.
ISSN:1424-8220
1424-8220
DOI:10.3390/s23239365