Efficient hand segmentation for rehabilitation tasks using a convolution neural network with attention

• Published in: Expert systems with applications 2023-12, Vol.234, p.121046, Article 121046
• Main Authors: Dutta, H Pallab Jyoti, Bhuyan, M.K., Neog, Debanga Raj, MacDorman, Karl Fredric, Laskar, Rabul Hussain
• Format: Article
• Language: English
• Subjects: Channel attention; Efficient attention mechanism; Encoder–decoder; Hand rehabilitation; Hand segmentation; Spatial attention
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2023.121046

We designed an interface to support hand rehabilitation tasks to restore hand function and relieve discomfort. The interface requires accurate hand segmentation, which is impeded by background clutter, occlusion, and variations in illumination. To overcome these challenges, we propose a novel encoder–decoder that segments the hand by encoding spatial and channel correlations using two attention blocks. This approach requires much less computation than benchmark self-attention mechanisms. Moreover, a novel loss function optimizes the model to resolve class imbalance, ensure boundary smoothness, and retain the hand’s shape. The quantitative and qualitative results show the model’s ability to segment the hands. It performed exceptionally well for images with different hand poses and orientations, the presence of a human face, background clutter, specularity, and variations in illumination. The model attained an F1-score of 97.3% for the Ouhands and 99.3% for the HGR dataset, higher than baseline models, with faster inference times. Furthermore, the model could generalize hand segmentation to multiple hands and unseen environments. Its segmentation precision enabled the development of the hand rehabilitation interface, which guided users to perform hand exercises. For five weeks, patients steadily improved hand function while using the interface. •We propose a novel hand segmentation architecture that uses attention mechanism.•The two proposed attention blocks emphasize the hand’s location and pixel’s class.•A composite loss function addresses class imbalance and hand’s shape related issues.•An interface designed to help patients with hand injuries via rehabilitation tasks.•Our method achieves state-of-the-art performance on benchmark hand datasets.