MARLadona - Towards Cooperative Team Play Using Multi-Agent Reinforcement Learning

Robot soccer, in its full complexity, poses an unsolved research challenge. Current solutions heavily rely on engineered heuristic strategies, which lack robustness and adaptability. Deep reinforcement learning has gained significant traction in various complex robotics tasks such as locomotion, man...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2024-10
Main Authors: Li, Zichong, Bjelonic, Filip, Klemm, Victor, Hutter, Marco
Format: Article
Language:English
Subjects:
Deep learning
Heuristic methods
Locomotion
Multiagent systems
Robot dynamics
Robot learning
Robotics
Soccer
Task complexity
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Robot soccer, in its full complexity, poses an unsolved research challenge. Current solutions heavily rely on engineered heuristic strategies, which lack robustness and adaptability. Deep reinforcement learning has gained significant traction in various complex robotics tasks such as locomotion, manipulation, and competitive games (e.g., AlphaZero, OpenAI Five), making it a promising solution to the robot soccer problem. This paper introduces MARLadona. A decentralized multi-agent reinforcement learning (MARL) training pipeline capable of producing agents with sophisticated team play behavior, bridging the shortcomings of heuristic methods. Further, we created an open-source multi-agent soccer environment based on Isaac Gym. Utilizing our MARL framework and a modified a global entity encoder as our core architecture, our approach achieves a 66.8% win rate against HELIOS agent, which employs a state-of-the-art heuristic strategy. Furthermore, we provided an in-depth analysis of the policy behavior and interpreted the agent's intention using the critic network.
ISSN:2331-8422