Depth-Camera-Based Under-Blanket Sleep Posture Classification Using Anatomical Landmark-Guided Deep Learning Model

• Published in: International journal of environmental research and public health 2022-10, Vol.19 (20), p.13491
• Main Authors: Tam, Andy Yiu-Chau, Zha, Li-Wen, So, Bryan Pak-Hei, Lai, Derek Ka-Hei, Mao, Ye-Jiao, Lim, Hyo-Jung, Wong, Duo Wai-Chi, Cheung, James Chung-Wai
• Format: Article
• Language: English
• Subjects: Accuracy; Cameras; Classification; Data collection; Deep Learning; Experiments; Fetuses; Humans; Neural networks; Physiology; Pose estimation; Posture; Sensors; Sleep; Sleep deprivation; Sleep disorders; Sleep Wake Disorders; Support vector machines
• ISSN: 1660-4601; 1661-7827; 1660-4601
• DOI: 10.3390/ijerph192013491

Emerging sleep health technologies will have an impact on monitoring patients with sleep disorders. This study proposes a new deep learning model architecture that improves the under-blanket sleep posture classification accuracy by leveraging the anatomical landmark feature through an attention strategy. The system used an integrated visible light and depth camera. Deep learning models (ResNet-34, EfficientNet B4, and ECA-Net50) were trained using depth images. We compared the models with and without an anatomical landmark coordinate input generated with an open-source pose estimation model using visible image data. We recruited 120 participants to perform seven major sleep postures, namely, the supine posture, prone postures with the head turned left and right, left- and right-sided log postures, and left- and right-sided fetal postures under four blanket conditions, including no blanket, thin, medium, and thick. A data augmentation technique was applied to the blanket conditions. The data were sliced at an 8:2 training-to-testing ratio. The results showed that ECA-Net50 produced the best classification results. Incorporating the anatomical landmark features increased the F1 score of ECA-Net50 from 87.4% to 92.2%. Our findings also suggested that the classification performances of deep learning models guided with features of anatomical landmarks were less affected by the interference of blanket conditions.