Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging

• Published in: JMIR mental health 2019-04, Vol.6 (4), p.e10140-e10140
• Main Authors: Cho, Youngjun, Julier, Simon J, Bianchi-Berthouze, Nadia
• Format: Article
• Language: English
• Subjects: Blood; Cameras; Electrocardiography; Machine learning; Medical equipment; Nervous system; Nose; Original Paper; Physiology; Sensors; Smartphones; Stress
• ISSN: 2368-7959; 2368-7959
• DOI: 10.2196/10140

A smartphone is a promising tool for daily cardiovascular measurement and mental stress monitoring. A smartphone camera-based photoplethysmography (PPG) and a low-cost thermal camera can be used to create cheap, convenient, and mobile monitoring systems. However, to ensure reliable monitoring results, a person must remain still for several minutes while a measurement is being taken. This is cumbersome and makes its use in real-life situations impractical. We proposed a system that combines PPG and thermography with the aim of improving cardiovascular signal quality and detecting stress responses quickly. Using a smartphone camera with a low-cost thermal camera added on, we built a novel system that continuously and reliably measures 2 different types of cardiovascular events: (1) blood volume pulse and (2) vasoconstriction/dilation-induced temperature changes of the nose tip. 17 participants, involved in stress-inducing mental workload tasks, measured their physiological responses to stressors over a short time period (20 seconds) immediately after each task. Participants reported their perceived stress levels on a 10-cm visual analog scale. For the instant stress inference task, we built novel low-level feature sets representing cardiovascular variability. We then used the automatic feature learning capability of artificial neural networks to improve the mapping between the extracted features and the self-reported ratings. We compared our proposed method with existing hand-engineered features-based machine learning methods. First, we found that the measured PPG signals presented high quality cardiac cyclic information (mean pSQI: 0.755; SD 0.068). We also found that the measured thermal changes of the nose tip presented high-quality breathing cyclic information and filtering helped extract vasoconstriction/dilation-induced patterns with fewer respiratory effects (mean pSQI: from 0.714 to 0.157). Second, we found low correlations between the self-reported stress scores and the existing metrics of the cardiovascular signals (ie, heart rate variability and thermal directionality) from short measurements, suggesting they were not very dependent upon one another. Third, we tested the performance of the instant perceived stress inference method. The proposed method achieved significantly higher accuracies than existing precrafted features-based methods. In addition, the 17-fold leave-one-subject-out cross-validation results showed that combining both modalities