Deep multimodal-based finger spelling recognition for Thai sign language: a new benchmark and model composition

• Published in: Machine vision and applications 2024-07, Vol.35 (4), p.76, Article 76
• Main Authors: Vijitkunsawat, Wuttichai, Racharak, Teeradaj, Le Nguyen, Minh
• Format: Article
• Language: English
• Subjects: Communications Engineering; Computer Science; Datasets; Deep learning; Fingers; Graphical representations; Hand (anatomy); Image Processing and Computer Vision; Joints (anatomy); Networks; Obstructions; Pattern Recognition; Recognition; Sign language; Transformers
• ISSN: 0932-8092; 1432-1769
• DOI: 10.1007/s00138-024-01557-9

Video-based sign language recognition is vital for improving communication for the deaf and hard of hearing. Creating and maintaining quality of Thai sign language video datasets is challenging due to a lack of resources. Tackling this issue, we rigorously investigate a design and development of deep learning-based system for Thai Finger Spelling recognition, assessing various models with a new dataset of 90 standard letters performed by 43 diverse signers. We investigate seven deep learning models with three distinct modalities for our analysis: video-only methods (including RGB-sequencing-based CNN-LSTM and VGG-LSTM), human body joint coordinate sequences (processed by LSTM, BiLSTM, GRU, and Transformer models), and skeleton analysis (using TGCN with graph-structured skeleton representation). A thorough assessment of these models is conducted across seven circumstances, encompassing single-hand postures, single-hand motions with one, two, and three strokes, as well as two-hand postures with both static and dynamic point-on-hand interactions. The research highlights that the TGCN model is the optimal lightweight model in all scenarios. In single-hand pose cases, a combination of the Transformer and TGCN models of two modalities delivers outstanding performance, excelling in four particular conditions: single-hand poses, single-hand poses requiring one, two, and three strokes. In contrast, two-hand poses with static or dynamic point-on-hand interactions present substantial challenges, as the data from joint coordinates is inadequate due to hand obstructions, stemming from insufficient coordinate sequence data and the lack of a detailed skeletal graph structure. The study recommends integrating RGB-sequencing with visual modality to enhance the accuracy of two-handed sign language gestures.