UnrealFall: Overcoming Data Scarcity through Generative Models

• Main Authors: Mulero-Perez, David, Benavent-Lledo, Manuel, Ortiz-Perez, David, Garcia-Rodriguez, Jose
• Format: Conference Proceeding
• Language: English
• Subjects: 3D scenes reconstruction; Action video dataset; Analytical models; Data models; Digital twins; Fall detection; Neural networks; Synthetic data generation; Three-dimensional displays; Training; Video sequences
• ISSN: 2161-4407
• DOI: 10.1109/IJCNN60899.2024.10651116

Humans perform a variety of actions, some of which are infrequent but crucial for data collection. Synthetic generation techniques are highly effective in these situations, enhancing the data for such rare actions. In response to this need, we present UnrealFall, a robust framework developed within Unreal Engine 5, designed for the generation of human action video data in hyper-realistic virtual scenes. It addresses the scarcity and limited diversity in existing datasets for actions like falls by leveraging synthetic motion generation through text-guided generative models, Gaussian Splatting technology, and MetaHumans. The usefulness of the framework is demonstrated by its capability to produce a synthetic video dataset featuring elderly individuals falling in various settings. The value of the dataset is demonstrated by its successful use in training a VideoMAE model, in conjunction with the UCF101 and various fall-specific datasets. This versatility in generating data across a spectrum of actions and environments positions our framework as a valuable tool for broader applications such as digital twin creation and dataset augmentation. The code and data are available for research at project website, darkviid.github.io/UnrealFall/.