Grasping Time and Pose Selection for Robotic Prosthetic Hand Control Using Deep Learning Based Object Detection

• Published in: International journal of control, automation, and systems 2022, Automation, and Systems, 20(10), , pp.3410-3417
• Main Authors: Park, Hae-June, An, Bo-Hyeon, Joo, Su-Bin, Kwon, Oh-Won, Kim, Min Young, Seo, Joonho
• Format: Article
• Language: English
• Subjects: Algorithms; Control; Deep learning; Engineering; Grasping (robotics); Image acquisition; Machine learning; Mechatronics; Object recognition; Occlusion; Prostheses; Regular Papers; Robot control; Robotics; Three dimensional models; 제어계측공학
• ISSN: 1598-6446; 2005-4092
• DOI: 10.1007/s12555-021-0449-6

This paper presents an algorithm to control a robotic prosthetic hand by applying deep learning (DL) to select a grasping pose and a grasping time from 2D images and 3D point clouds. This algorithm consists of four steps: 1) Acquisition of 2D images and 3D point clouds of objects; 2) Object recognition in the 2D images; 3) Grasping pose selection; 4) Choice of a grasping time and control of the prosthetic hand. The grasping pose selection is necessary when the algorithm detects many objects in the same frame, and must decide which pose of the prosthetic hand should use. The pose was chosen considering the object that was to the prosthesis. The grasping time was determined by the operating point when approaching the selected target after selecting the grasping pose; this choice uses an empirically-determined distance threshold. The proposed method achieved 89% accuracy of the grasping the intended object. The failures occurred because of slight inaccuracy in object localization, occlusion of target objects, and the inability of DL object detection. Work to solve these shortcomings is ongoing. This algorithm will help to improve the convenience of the user of a prosthetic hand.