Sign language recognition and translation network based on multi-view data

• Published in: Applied intelligence (Dordrecht, Netherlands) Netherlands), 2022-10, Vol.52 (13), p.14624-14638
• Main Authors: Li, Ronghui, Meng, Lu
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial Intelligence; Artificial neural networks; Computer Science; Datasets; Feature extraction; Hearing disorders; Language translation; Machines; Manufacturing; Mechanical Engineering; Neural networks; Processes; Recognition; Recurrent neural networks; Sentences; Sign language; Special Issue on Multi-view Learning; Translation methods and strategies
• ISSN: 0924-669X; 1573-7497
• DOI: 10.1007/s10489-022-03407-5

Sign language recognition and translation can address the communication problem between hearing-impaired and general population, and can break the sign language boundariesy between different countries and different languages. Traditional sign language recognition and translation algorithms use Convolutional Neural Networks (CNNs) to extract spatial features and Recurrent Neural Networks (RNNs) to extract temporal features. However, these methods cannot model the complex spatiotemporal features of sign language. Moreover, RNN and its variant algorithms find it difficult to learn long-term dependencies. This paper proposes a novel and effective network based on Transformer and Graph Convolutional Network (GCN), which can be divided into three parts: a multi-view spatiotemporal embedding network (MSTEN), a continuous sign language recognition network (CSLRN), and a sign language translation network (SLTN). MSTEN can extract the spatiotemporal features of RGB data and skeleton data. CSLRN can recognize sign language glosses and obtain intermediate features from multi-view input sign data. SLTN can translate intermediate features into spoken sentences. The entire network was designed as end-to-end. Our method was tested on three public sign language datasets (SLR-100, RWTH, and CSL-daily) and the results demonstrated that our method achieved excellent performance on these datasets.