Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?

• Published in: arXiv.org 2024-07
• Main Authors: Leonardi, Rosario, Furnari, Antonino, Ragusa, Francesco, Farinella, Giovanni Maria
• Format: Article
• Language: English
• Subjects: Adaptation; Annotations; Benchmarks; Environment models; Synthetic data; Three dimensional models
• ISSN: 2331-8422

In this study, we investigate the effectiveness of synthetic data in enhancing egocentric hand-object interaction detection. Via extensive experiments and comparative analyses on three egocentric datasets, VISOR, EgoHOS, and ENIGMA-51, our findings reveal how to exploit synthetic data for the HOI detection task when real labeled data are scarce or unavailable. Specifically, by leveraging only 10% of real labeled data, we achieve improvements in Overall AP compared to baselines trained exclusively on real data of: +5.67% on EPIC-KITCHENS VISOR, +8.24% on EgoHOS, and +11.69% on ENIGMA-51. Our analysis is supported by a novel data generation pipeline and the newly introduced HOI-Synth benchmark which augments existing datasets with synthetic images of hand-object interactions automatically labeled with hand-object contact states, bounding boxes, and pixel-wise segmentation masks. Data, code, and data generation tools to support future research are released at: https://fpv-iplab.github.io/HOI-Synth/.