Semi-automation of gesture annotation by machine learning and human collaboration

Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be...

Full description

Saved in:
Bibliographic Details
Published in:Language resources and evaluation 2022-09, Vol.56 (3), p.673-700
Main Authors: Ienaga, Naoto, Cravotta, Alice, Terayama, Kei, Scotney, Bryan W., Saito, Hideo, Busà, M. Grazia
Format: Article
Language:English
Subjects:
Annotations
Computational Linguistics
Computer Science
Cooperative learning
Gestures
Language and Literature
Linguistics
Machine learning
Multimodality
Original Paper
Pose estimation
Social Sciences
User satisfaction
Video communication
Video data
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Gesture and multimodal communication researchers typically annotate video data manually, even though this can be a very time-consuming task. In the present work, a method to detect gestures is proposed as a fundamental step towards a semi-automatic gesture annotation tool. The proposed method can be applied to RGB videos and requires annotations of part of a video as input. The technique deploys a pose estimation method and active learning. In the experiment, it is shown that if about 27% of the video is annotated, the remaining parts of the video can be annotated automatically with an F-score of at least 0.85. Users can run this tool with a small number of annotations first. If the predicted annotations for the remainder of the video are not satisfactory, users can add further annotations and run the tool again. The code has been released so that other researchers and practitioners can use the results of this research. This tool has been confirmed to work in conjunction with ELAN.
ISSN:1574-020X
1574-0218
DOI:10.1007/s10579-022-09586-4