Limits of speech in connected homes: Experimental comparison of self-reporting tools for human activity recognition

• Published in: International journal of human-computer studies 2025-01, Vol.195, p.103404, Article 103404
• Main Authors: Levasseur, Guillaume, Tang, Kejia, Bersini, Hugues
• Format: Article
• Language: English
• Subjects: Activity recognition; Machine learning; Self-reporting; Smart homes; Voice assistants; Water consumption
• ISSN: 1071-5819
• DOI: 10.1016/j.ijhcs.2024.103404

Data annotation for human activity recognition is a well-known challenge for researchers. In particular, annotation in daily life settings relies on self-reporting tools with unknown accuracy. Speech is a promising interface for activity labeling. In this work, we compare the accuracy of two commercially available tools for annotation: voice diaries and connected buttons. We retrofit the water meters of thirty homes in the USA for infrastructure-mediated sensing. Participants are split into equal groups and receive one of the self-reporting tools. The balanced accuracy metric is transferred from the field of machine learning to the evaluation of the annotation performance. Our results show that connected buttons perform significantly better than the voice diary, with 92% median accuracy and 65% median reporting rate. Using questionnaire answers, we highlight that annotation performance is impacted by habit formation and sentiments toward the annotation tool. The use case for data annotation is to disaggregate water meter data into human activities beyond the point of use. We show that it is feasible with a machine-learning model and the corrected annotations. Finally, we formulate recommendations for the design of studies and intelligent environments around the key ideas of proportionality and immediacy. •Self-reporting of human activities enables to train machine learning algorithms.•Speech is not the best interface to report chores in daily life settings.•Internet-connected buttons prove better with 92% accuracy and 65% reporting rate.•Human activities can be inferred from smart water meter data.