Clustering Egocentric Images in Passive Dietary Monitoring with Self-Supervised Learning

• Published in: arXiv.org 2022-08
• Main Authors: Peng, Jiachuan, Shi, Peilun, Qiu, Jianing, Ju, Xinwei, Lo, Frank P -W, Gu, Xiao, Jia, Wenyan, Baranowski, Tom, Steiner-Asiedu, Matilda, Anderson, Alex K, McCrory, Megan A, Sazonov, Edward, Sun, Mingui, Frost, Gary, Lo, Benny
• Format: Article
• Language: English
• Subjects: Annotations; Cameras; Clustering; Datasets; Food intake; Households; Monitoring; Self-supervised learning; Supervised learning; Wearable technology
• ISSN: 2331-8422

In our recent dietary assessment field studies on passive dietary monitoring in Ghana, we have collected over 250k in-the-wild images. The dataset is an ongoing effort to facilitate accurate measurement of individual food and nutrient intake in low and middle income countries with passive monitoring camera technologies. The current dataset involves 20 households (74 subjects) from both the rural and urban regions of Ghana, and two different types of wearable cameras were used in the studies. Once initiated, wearable cameras continuously capture subjects' activities, which yield massive amounts of data to be cleaned and annotated before analysis is conducted. To ease the data post-processing and annotation tasks, we propose a novel self-supervised learning framework to cluster the large volume of egocentric images into separate events. Each event consists of a sequence of temporally continuous and contextually similar images. By clustering images into separate events, annotators and dietitians can examine and analyze the data more efficiently and facilitate the subsequent dietary assessment processes. Validated on a held-out test set with ground truth labels, the proposed framework outperforms baselines in terms of clustering quality and classification accuracy.