DIR-BHRNet: A Lightweight Network for Real-Time Vision-Based Multiperson Pose Estimation on Smartphones

• Published in: IEEE transactions on industrial informatics 2024-11, Vol.20 (11), p.12533-12541
• Main Authors: Lan, Gongjin, Wu, Yu, Hao, Qi
• Format: Article
• Language: English
• Subjects: Computational efficiency; Computer architecture; Convolution; Deep learning; Feature extraction; human pose estimation (HPE); multiperson pose estimation (MPPE); Pose estimation; real time; Real-time systems; Smart phones; smartphones
• ISSN: 1551-3203; 1941-0050
• DOI: 10.1109/TII.2024.3421511

Human pose estimation (HPE), particularly multiperson pose estimation (MPPE), has been applied in many domains, such as human-machine systems. However, the current MPPE methods generally run on powerful GPU systems and take a lot of computational costs. Real-time MPPE on mobile devices with low-performance computing is a challenging task. In this article, we propose a lightweight neural network, DIR-BHRNet, for real-time MPPE on smartphones. In DIR-BHRNet, we design a novel lightweight convolutional module, dense inverted residual (DIR), to improve accuracy by adding a depthwise convolution and a shortcut connection into the well-known inverted residual, and a novel efficient neural network structure, balanced HRNet (BHRNet), to reduce computational costs by reconfiguring the proper number of convolutional blocks on each branch. We evaluate DIR-BHRNet on the well-known COCO and CrowdPose datasets. The results show that DIR-BHRNet outperforms the state-of-the-art methods in terms of accuracy with a real-time computational cost. Finally, we implement the DIR-BHRNet on the current mainstream Android smartphones, which perform more than 10 FPS. The free-used executable file (Android 10), source code, and a video description of this work are publicly available on the page 1 to facilitate the development of real-time MPPE on smartphones.