A Two-Stream Deep-Learning Network for Heart Rate Estimation From Facial Image Sequence

• Published in: IEEE sensors journal 2024-01, Vol.24 (24), p.42343-42351
• Main Authors: Lie, Wen-Nung, Le, Dao Q., Huang, Po-Han, Fu, Guan-Hao, Nguyen Thi Quynh, Anh, Nguyen Quang Nhu, Quynh
• Format: Article
• Language: English
• Subjects: Computational modeling; Convolution; Convolutional neural networks; Data augmentation; Deep learning; Deep-learning (DL); Estimation; facial image sequence; Feature extraction; Heart rate; heart rate (HR) estimation; Modulation; Monitoring; remote PPG; Spatiotemporal data; Streams; Training; two-stream network
• ISSN: 1530-437X; 1558-1748
• DOI: 10.1109/JSEN.2024.3483629

This article presents a deep-learning-based two-stream network to estimate remote Photoplethysmogram (rPPG) signal and hence derive the heart rate (HR) from an RGB facial video. Our proposed network employs temporal modulation blocks (TMBs) to efficiently extract temporal dependencies and spatial attention blocks on a mean frame to learn spatial features. Our TMBs are composed of two subblocks that can simultaneously learn overall and channelwise spatiotemporal features, which are pivotal for the task. Data augmentation (DA) in training and multiple redundant estimations for noise removal in testing were also designed to make the training more effective and the inference more robust. Experimental results show that the proposed temporal shift-channelwise spatio-temporal network (TS-CST Net) has reached competitive and even superior performances among the state-of-the-art (SOTA) methods on four popular datasets, showcasing our network's learning capability.