Gamification of RF Data Acquisition for Classification of Natural Human Gestures

• Main Authors: Kurtoglu, Emre, DeHaan, Kenneth, Pezzarossi, Caroline Kobek, Griffin, Darrin J., Crawford, Chris, Gurbuz, Sevgi Z.
• Format: Conference Proceeding
• Language: English
• Subjects: American Sign Language; Data acquisition; deep neural networks; Human computer interaction; Micro-Doppler spectrogram; multi-modal; Predictive models; Radar; Radio frequency; RF sensors; Sign language; Transforms
• ISSN: 2375-5318
• DOI: 10.1109/RadarConf2458775.2024.10548148

In recent years, there have been significant developments in radio frequency (RF) sensor technology used in human-computer interaction (HCI) applications, specifically in areas like gesture recognition and more broadly, human activity recognition. Although extensive research has been conducted on these subjects, most experiments involve controlled settings where participants are instructed on how to perform specific movements. However, when such experiments are conducted on sign language recognition they lack capturing dialectal and background-related diversities. In this work, we explore the differences in RF datasets acquired under controlled experimental settings and in free form environments where users were not constrained by the experimental instructions and limitations. We show that directed (i.e., controlled) data acquisition approaches result in over-optimistic performances which do not perform well on naturally acquired data samples in a real-world use case. We evaluate different approaches on generating synthetic samples from directed dataset, but show that such methods do not offer much benefit over collecting natural data. Therefore, we propose an interactive data acquisition paradigm through gamification. We show that the proposed approach enables the recognition of American Sign Language (ASL) in real-world settings by achieving 69% accuracy on 29 words.