Learning Humanoid Robot Running Motions with Symmetry Incentive through Proximal Policy Optimization

• Published in: Journal of intelligent & robotic systems 2021-07, Vol.102 (3), Article 54
• Main Authors: Melo, Luckeciano C., Melo, Dicksiano C., Maximo, Marcos R. O. A.
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial Intelligence; Computer simulation; Control; Controllers; Electrical Engineering; Engineering; Humanoid; Machine learning; Mechanical Engineering; Mechatronics; Neural networks; Optimization; Regular Paper; Reliability analysis; Robotics; Robots; Running; Sensors; Soccer; State vectors; Symmetry; Topical collection on Robotica 2019
• ISSN: 0921-0296; 1573-0409
• DOI: 10.1007/s10846-021-01355-9

This article contributes with a methodology based on deep reinforcement learning to develop running skills in a humanoid robot with no prior knowledge. Specifically, the algorithm used for learning is the Proximal Policy Optimization (PPO). The chosen application domain is the RoboCup 3D Soccer Simulation (Soccer 3D), a competition where teams composed by 11 autonomous agents each compete in simulated soccer matches. In our approach, the state vector used as the neural network’s input consists of raw sensor measurements or quantities which could be obtained through sensor fusion, while the actions are the joint positions, which are sent to joint controllers. Our running behavior outperforms the state-of-the-art in terms of sprint speed by approximately 50%. We present results regarding the training procedure and also evaluate the controllers in terms of speed, reliability, and human similarity. Since the running policies with top speed display asymmetric motions, we also investigate a technique to encourage symmetry in the sagittal plane. Finally, we discuss key factors that lead us to surpass previous results in the literature and share some ideas for future research.