Topic modelling for routine discovery from egocentric photo-streams

• Published in: Pattern recognition 2020-08, Vol.104, p.107330, Article 107330
• Main Authors: Talavera, Estefania, Wuerich, Carolin, Petkov, Nicolai, Radeva, Petia
• Format: Article
• Language: English
• Subjects: Behaviour analysis; Egocentric vision; Lifestyle; Routine; Topic modelling
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2020.107330

•We introduce a novel automatic unsupervised pipeline for the identification and characterization of Routine-related days from egocentric photo-streams.•We prove that topic modelling is a powerful tool for discovery of patterns when addressing Bag-of-Words representation of photo-streams.•We prove that using Dynamic-Time-Warping and Distance-based clustering is a robust technique to detect the cluster of routine days where the method is tolerant to small temporal differences in the daily events.•We present and new egocentric dataset composed of a total of 100.000 images, from 104 days. Developing tools to understand and visualize lifestyle is of high interest when addressing the improvement of habits and well-being of people. Routine, defined as the usual things that a person does daily, helps describe the individuals’ lifestyle. With this paper, we are the first ones to address the development of novel tools for automatic discovery of routine days of an individual from his/her egocentric images. In the proposed model, sequences of images are firstly characterized by semantic labels detected by pre-trained CNNs. Then, these features are organized in temporal-semantic documents to later be embedded into a topic models space. Finally, Dynamic-Time-Warping and Spectral-Clustering methods are used for final day routine/non-routine discrimination. Moreover, we introduce a new EgoRoutine-dataset, a collection of 104 egocentric days with more than 100.000 images recorded by 7 users. Results show that routine can be discovered and behavioural patterns can be observed.