SFINGE 3D: A novel benchmark for online detection and recognition of heterogeneous hand gestures from 3D fingers’ trajectories

• Published in: Computers & graphics 2020-10, Vol.91, p.232-242
• Main Authors: Caputo, Ariel, Giachetti, Andrea, Giannini, Franca, Lupinetti, Katia, Monti, Marina, Pegoraro, Marco, Ranieri, Andrea
• Format: Article
• Language: English
• Subjects: Computers and graphics; Formatting; Guidelines
• ISSN: 0097-8493; 1873-7684
• DOI: 10.1016/j.cag.2020.07.014

•SFINGE 3D, novel benchmark for evaluating online gesture detection and recognition.•13 gestures dictionary, recognition task for detection and recognition with 72 gesture sequences.•Different approaches to test the benchmark: visual rendering and convolutional nerual networks.•Different approaches to test the benchmark: geometry-based and dissimilarity-based classifiers. [Display omitted] In recent years gesture recognition has become an increasingly interesting topic for both research and industry. While interaction with a device through a gestural interface is a promising idea in several applications especially in the industrial field, some of the issues related to the task are still considered a challenge. In the scientific literature, a relevant amount of work has been recently presented on the problem of detecting and classifying gestures from 3D hands’ joints trajectories that can be captured by cheap devices installed on head-mounted displays and desktop computers. The methods proposed so far can achieve very good results on benchmarks requiring the offline supervised classification of segmented gestures of a particular kind but are not usually tested on the more realistic task of finding gestures execution within a continuous hand tracking session. In this paper, we present a novel benchmark, SFINGE 3D, aimed at evaluating online gesture detection and recognition. The dataset is composed of a dictionary of 13 segmented gestures used as a training set and 72 trajectories each containing 3–5 of the 13 gestures, performed in continuous tracking, padded with random hand movements acting as noise. The presented dataset, captured with a head-mounted Leap Motion device, is particularly suitable to evaluate gesture detection methods in a realistic use-case scenario, as it allows the analysis of online detection performance on heterogeneous gestures, characterized by static hand pose, global hand motions, and finger articulation. We exploited SFINGE 3D to compare two different approaches for the online detection and classification, one based on visual rendering and Convolutional Neural Networks and the other based on geometry-based handcrafted features and dissimilarity-based classifiers. We discuss the results, analyzing strengths and weaknesses of the methods, and deriving useful hints for their improvement.