Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors

• Published in: Expert systems with applications 2014-11, Vol.41 (16), p.7259-7271
• Main Authors: Moreira Almeida, Sílvia Grasiella, Guimarães, Frederico Gadelha, Arturo Ramírez, Jaime
• Format: Article
• Language: English
• Subjects: Applied sciences; Artificial intelligence; Brazil; Brazilian Sign Language Recognition; Computer science; control theory; systems; Data processing. List processing. Character string processing; Exact sciences and technology; Expert systems; Feature extraction; Feature recognition; Mathematics; Memory organisation. Data processing; Nonparametric inference; Pattern recognition. Digital image processing. Computational geometry; Probability and statistics; RGB-D sensors; Sciences and techniques of general use; Sensors; Software; Speech and sound recognition and synthesis. Linguistics; Speech recognition; Statistics; Structural members; Support vector machines
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2014.05.024

•We present a methodology for feature extraction in Brazilian Sign Language (BSL).•We investigate relation between extracted features and structural elements in BSL.•We obtain seven vision-based features from the RGB-D images.•We employ SVM’s to classify signs based on these features and linguistic elements. In contrast to speech recognition, whose speech features have been extensively explored in the research literature, feature extraction in Sign Language Recognition (SLR) is still a very challenging problem. In this paper we present a methodology for feature extraction in Brazilian Sign Language (BSL, or LIBRAS in Portuguese) that explores the phonological structure of the language and relies on RGB-D sensor for obtaining intensity, position and depth data. From the RGB-D images we obtain seven vision-based features. Each feature is related to one, two or three structural elements in BSL. We investigate this relation between extracted features and structural elements based on shape, movement and position of the hands. Finally we employ Support Vector Machines (SVM) to classify signs based on these features and linguistic elements. The experiments show that the attributes of these elements can be successfully recognized in terms of the features obtained from the RGB-D images, with accuracy results individually above 80% on average. The proposed feature extraction methodology and the decomposition of the signs into their phonological structure is a promising method to help expert systems designed for SLR.