Self-supervised representation learning for robust fine-grained human hand action recognition in industrial assembly lines

• Published in: Machine vision and applications 2025-01, Vol.36 (1), p.19, Article 19
• Main Authors: Sturm, Fabian, Trat, Martin, Sathiyababu, Rahul, Allipilli, Harshitha, Menz, Benjamin, Hergenroether, Elke, Siegel, Melanie
• Format: Article
• Language: English
• Subjects: Activity recognition; Assembly lines; Corporate learning; Deep learning; Hand (anatomy); Labels; Machine learning; Moving object recognition; Regression models; Representations; Robustness; Self-supervised learning
• ISSN: 0932-8092; 1432-1769
• DOI: 10.1007/s00138-024-01638-9

Humans are still indispensable on industrial assembly lines, but in the event of an error, they need support from intelligent systems. In addition to the objects to be observed, it is equally important to understand the fine-grained hand movements of a human to be able to track the entire process. However, these deep-learning-based hand action recognition methods are very label intensive, which cannot be offered by all industrial companies due to the associated costs. This work therefore presents a self-supervised learning approach for industrial assembly processes that allows a spatio-temporal transformer architecture to be pre-trained on a variety of information from real-world video footage of daily life. Subsequently, this deep learning model is adapted to the industrial assembly task at hand using only a few labels. Well-known real-world datasets best suited for representation learning of such hand actions in a regression tasks are outlined and to what extent they optimize the subsequent supervised trained classification task. This subsequent fine-tuning is supplemented by concept drift detection, which makes the resulting productively employed models more robust against concept drift and future changing assembly movements.