Imitation System of Humanoid Robots and Its Applications

• Published in: IEEE open journal of circuits and systems 2023, Vol.4, p.15-24
• Main Authors: Zhan, Ze-Feng, Huang, Han-Pang
• Format: Article
• Language: English
• Subjects: 3D pose estimation; Human engineering; human-robot interaction; Humanoid; Humanoid robot; Humanoid robots; imitation system; Legged locomotion; MediaPipe; motion database; Motion planning; Pose estimation; Robot dynamics; Robot kinematics; Robots; Skeleton; Three dimensional models; Three-dimensional displays; Trajectories; Trajectory; Vision systems
• ISSN: 2644-1225; 2644-1225
• DOI: 10.1109/OJCAS.2022.3231097

In this paper, we propose an imitation system that imitates human motions in videos to plan robot actions that are similar to human motions, with the aim of the complicated whole-body action planning of humanoid robots. Additionally, we created an interaction system that will enable basic human-robot interaction for our humanoid robot. To obtain the 3D coordinates of the key points on the human body, we used the 3D pose estimation model. The key points were then transformed into various trajectory files needed by the robot to complete the motion, using the mapping method proposed in this research, which refers to the control strategy and stability of the robot. In addition, we proposed some post-processing methods to post-process the trajectories. In the interaction system, we created a speech and vision system so that the robot could detect human gestures or postures and converse with people. It also has a music rhythm recognition system developed by seniors that enables the robot to dance to the beats of the song. Finally, through this system, we completed several human-robot interaction scenarios, which proved the convenience, and effectiveness of motion planning with an imitation system, and the completeness of the interaction system.