Autonomous underwater vehicle path planning based on actor-multi-critic reinforcement learning

• Published in: Proceedings of the Institution of Mechanical Engineers. Part I, Journal of systems and control engineering Journal of systems and control engineering, 2021-11, Vol.235 (10), p.1787-1796
• Main Authors: Wang, Zhuo, Zhang, Shiwei, Feng, Xiaoning, Sui, Yancheng
• Format: Article
• Language: English
• Subjects: Adaptability; Algorithms; Autonomous underwater vehicles; Back propagation; Back propagation networks; Machine learning; Mechanical engineering; Moving obstacles; Neural networks; Obstacle avoidance; Path planning; Underwater exploration
• ISSN: 0959-6518; 2041-3041
• DOI: 10.1177/0959651820937085

The environmental adaptability of autonomous underwater vehicles is always a problem for its path planning. Although reinforcement learning can improve the environmental adaptability, the slow convergence of reinforcement learning is caused by multi-behavior coupling, so it is difficult for autonomous underwater vehicle to avoid moving obstacles. This article proposes a multi-behavior critic reinforcement learning algorithm applied to autonomous underwater vehicle path planning to overcome problems associated with oscillating amplitudes and low learning efficiency in the early stages of training which are common in traditional actor–critic algorithms. Behavior critic reinforcement learning assesses the actions of the actor from perspectives such as energy saving and security, combining these aspects into a whole evaluation of the actor. In this article, the policy gradient method is selected as the actor part, and the value function method is selected as the critic part. The strategy gradient and the value function methods for actor and critic, respectively, are approximated by a backpropagation neural network, the parameters of which are updated using the gradient descent method. The simulation results show that the method has the ability of optimizing learning in the environment and can improve learning efficiency, which meets the needs of real time and adaptability for autonomous underwater vehicle dynamic obstacle avoidance.