Multi-agent cooperation policy gradient method based on enhanced exploration for cooperative tasks

• Published in: International journal of machine learning and cybernetics 2024-04, Vol.15 (4), p.1431-1452
• Main Authors: Zhao, Li-yang, Chang, Tian-qing, Zhang, Lei, Zhang, Xin-lu, Wang, Jiang-feng
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial Intelligence; Coding; Complex Systems; Computational Intelligence; Control; Cooperation; Decision making; Deep learning; Distillation; Efficiency; Engineering; Learning; Machine learning; Mechatronics; Methods; Multiagent systems; Optimization; Original Article; Pattern Recognition; Robotics; Systems Biology
• ISSN: 1868-8071; 1868-808X
• DOI: 10.1007/s13042-023-01976-6

Multi-agent cooperation and coordination are often essential for task fulfillment. Multi-agent deep reinforcement learning (MADRL) can effectively learn the solutions to problems, but its application is still primarily restricted by the exploration–exploitation trade-off. Therefore, the focus of MADRL research is placed on how to effectively explore the environment and collect good experience with rich information to strengthen cooperative behaviors and optimize policy learning. To address this problem, we propose a novel multi-agent cooperation policy gradient method called multi-agent proximal policy optimization based on self-imitation learning and random network distillation (MAPPOSR). MAPPOSR consists of two policy gradient-based additional components, namely (1) random network distillation (RND) exploration bonus component that produces intrinsic rewards and encourages agents to access new states and actions, thereby helping them explore better trajectories and avoiding the algorithm prematurely converging or getting stuck in local optima; and (2) self-imitation learning (SIL) policy update component that stores and reuses high-return trajectory samples generated by agents themselves, thereby strengthening their cooperation and boosting learning efficiency. The experimental results show that in addition to effectively solving the hard-exploration problem, the proposed method significantly outperforms other SOTA MADRL algorithms in learning efficiency as well as in escaping local optima. Moreover, the effect of different function inputs on algorithm performance is investigated in the centralized training and decentralized execution (CTDE) framework, based on which a joint-observation coding method based on individual is developed. By encouraging the agent to focus more on the local observation information of other agents related to it and abandon global state information provided by the environment, the developed coding method can remove the effects of excessive value function input dimensions and redundant feature information on algorithm performance.