Automatic Assessment Method and Device for Depression Symptom Severity Based on Emotional Facial Expression and Pupil-wave

• Published in: IEEE transactions on instrumentation and measurement 2024-01, Vol.73, p.1-1
• Main Authors: Li, Mi, Lu, Zeying, Cao, Qishuang, Gao, Junlong, Hu, Bin
• Format: Article
• Language: English
• Subjects: Artificial neural networks; Brain modeling; Convolutional neural networks; Data mining; Datasets; Deep learning; Depression; Effectiveness; Emotional factors; Feature extraction; HAMD (hamilton depression scale); Health services; Machine learning; Medical research; Mental depression; Mental disorders; Neural networks; Patients with depression; PHQ-9 (patient health questionnaire-9); Physiology; Predictive models; Pupils; Questionnaires; Symptom severity; Videos; Visualization
• ISSN: 0018-9456; 1557-9662
• DOI: 10.1109/TIM.2024.3415778

Depression is a serious mental disorder, significantly burdens individuals, families, and society. For clinical psychiatrists, assessing the severity of depression is a crucial tool in selecting treatment approaches and evaluating their effectiveness. Although many studies from machine learning on the automatic evaluation of self-rating scales (such as beck depression inventory-II and patient health questionnaire-8), the research on the machine learning-based automatic evaluation method of the medical clinical assessment scale (hamilton depression scale) has yet to be focused on. In this study, an end-to-end automatic evaluation device for hamilton depression scale and patient health questionnaire-9 scores was developed. In addition, we constructed a dataset consisting of emotional facial expression videos signals and emotional pupil-wave signals from 65 patients with depression. The dataset has hamilton depression scale and patient health questionnaire-9 score labels, encompassing two emotional states: sadness and happiness. We built a 3-dimension convolutional neural networks + long short-term memory model framework and a multi-scale 1-dimension convolutional neural network to learn and extract features from emotional facial expression videos and emotional pupil-wave automatically. The results showed that compared with the previous evaluation methods for depression levels, the evaluation precision of hamilton depression scale and patient health questionnaire-9 has been improved significantly. The results also showed that, in both hamilton depression scale and patient health questionnaire-9 evaluations, the evaluation precision of emotional facial expression videos was superior to emotional pupil-wave, and hamilton depression scale is better than patient health questionnaire-9. These studies indicated that both emotional facial expressions and emotional pupil-wave can better represent depressive mood in patients with depression, especially emotional facial expressions, and the predictive precision of the medical scale is significantly better than the self-rating scale. This automated assessment method and device can assist doctors in diagnosing depressive symptoms more effectively and serve as an evaluation tool for treatment efficacy.