KD-Net: Continuous-Keystroke-Dynamics-Based Human Identification from RGB-D Image Sequences

• Published in: Sensors (Basel, Switzerland) Switzerland), 2023-10, Vol.23 (20), p.8370
• Main Authors: Dai, Xinxin, Zhao, Ran, Hu, Pengpeng, Munteanu, Adrian
• Format: Article
• Language: English
• Subjects: Analysis; Biometric identification; Biometrics; Datasets; human identification; image sequences; Keyboards; keystroke dynamics; Methods; Neural networks; RGB-D images; Sensors; Software
• ISSN: 1424-8220; 1424-8220
• DOI: 10.3390/s23208370

Keystroke dynamics is a soft biometric based on the assumption that humans always type in uniquely characteristic manners. Previous works mainly focused on analyzing the key press or release events. Unlike these methods, we explored a novel visual modality of keystroke dynamics for human identification using a single RGB-D sensor. In order to verify this idea, we created a dataset dubbed KD-MultiModal, which contains 243.2 K frames of RGB images and depth images, obtained by recording a video of hand typing with a single RGB-D sensor. The dataset comprises RGB-D image sequences of 20 subjects (10 males and 10 females) typing sentences, and each subject typed around 20 sentences. In the task, only the hand and keyboard region contributed to the person identification, so we also propose methods of extracting Regions of Interest (RoIs) for each type of data. Unlike the data of the key press or release, our dataset not only captures the velocity of pressing and releasing different keys and the typing style of specific keys or combinations of keys, but also contains rich information on the hand shape and posture. To verify the validity of our proposed data, we adopted deep neural networks to learn distinguishing features from different data representations, including RGB-KD-Net, D-KD-Net, and RGBD-KD-Net. Simultaneously, the sequence of point clouds also can be obtained from depth images given the intrinsic parameters of the RGB-D sensor, so we also studied the performance of human identification based on the point clouds. Extensive experimental results showed that our idea works and the performance of the proposed method based on RGB-D images is the best, which achieved 99.44% accuracy based on the unseen real-world data. To inspire more researchers and facilitate relevant studies, the proposed dataset will be publicly accessible together with the publication of this paper.