Emotion Recognition in the Real World: Passively Collecting and Estimating Emotions from Natural Speech Data of Individuals with Bipolar Disorder

• Published in: IEEE transactions on affective computing 2024-05, p.1-14
• Main Authors: Provost, Emily Mower, Sperry, Sarah H, Tavernor, James, Anderau, Steve, Yocum, Anastasia, McInnis, Melvin G
• Format: Article
• Language: English
• Subjects: Bipolar disorder; Cryptography; Depression; Diagnosis or assessment; Emotion recognition; Feature extraction; Modeling human emotion; Mood; Mood or core affect; Pipelines; Speech recognition
• ISSN: 1949-3045; 1949-3045
• DOI: 10.1109/TAFFC.2024.3407683

Emotions provide critical information regarding a person's health and well-being. Therefore, the ability to track emotion and patterns in emotion over time could provide new opportunities in measuring health longitudinally. This is of particular importance for individuals with bipolar disorder (BD), where emotion dysregulation is a hallmark symptom of increasing mood severity. However, measuring emotions typically requires self-assessment, a willful action outside of one's daily routine. In this paper, we describe a novel approach for collecting real-world natural speech data from daily life and measuring emotions from these data. The approach combines a novel data collection pipeline and validated robust emotion recognition models. We describe a deployment of this pipeline that included parallel clinical and self-report measures of mood and selfreported measures of emotion. Finally, we present approaches to estimate clinical and self-reported mood measures using a combination of passive and self-reported emotion measures. The results demonstrate that both passive and self-reported measures of emotion contribute to our ability to accurately estimate mood symptom severity for individuals with BD.