A Reinforcement Learning Approach to Find Optimal Propulsion Strategy for Microrobots Swimming at Low Reynolds Number

• Published in: Robotics and autonomous systems 2024-05, Vol.175, p.104659, Article 104659
• Main Authors: Jebellat, Iman, Jebellat, Ehsan, Amiri-Margavi, Alireza, Vahidi-Moghaddam, Amin, Nejat Pishkenari, Hossein
• Format: Article
• Language: English
• Subjects: Basic Coding; machine learning; propulsion strategy; reinforcement learning; smart microswimmer; states and actions
• ISSN: 0921-8890; 1872-793X
• DOI: 10.1016/j.robot.2024.104659

•The smart microswimmers learn and find the optimal propulsion strategy without previous knowledge and without the need to try all propulsion cycles.•Q-Learning along with the Basic Coding is utilized to train microrobots.•Basic Coding is a powerful and computationally inexpensive computer method for defining states and actions in different Reinforcement Learning problems.•Parametric study shows the importance of selecting appropriate learning parameters to avoid suboptimal solutions or high computational costs. The development of artificial microscopic robots, like synthetic microswimmers, is one of the state-of-the-art research topics due to their promising biomedical applications. The movement of microswimmers is affected by stringent constraints because of the low Reynolds number of the surrounding environment. Researchers have been working on enhancing microrobots’ propulsion tactics and figuring out new approaches to find optimal propulsion strategies. In this research, we employ a Reinforcement Learning (RL) algorithm, specifically Q-Learning, to train linear-shaped microrobots, comprised of spheres and rods, by introducing an innovative and pioneer coding approach, termed “Basic Coding”, which is utilized to specify states and actions within the RL framework. Basic Coding is a powerful general method that can be employed for different agents in any discrete RL environment. We show how to apply Basic Coding to various microrobots with different geometrical configurations, like a triangular one. Our smart microswimmers, with different numbers of spheres, acquire the knowledge of the optimal propulsion cycle to accomplish large net mechanical displacement without relying on any pre-existing locomotion expertise. The three-sphere linear microrobot recovers the cycle Najafi and Golestanian suggested. The N-sphere microrobots with higher degrees of freedom can find the optimal propulsion cycle within a reasonable number of learning steps and low computational cost utilizing our RL and Basic Coding approach, while the learning step number significantly increases using other methods like Brute-force search. For example, we show this number for the 5-sphere microrobot is 1.19E03 and 8.97E10 steps using our methodology and Brute-force, respectively. Furthermore, our intelligent microrobots can successfully and adaptively find new optimal strategies in indeterministic environments in the presence of uncertainty. Moreover, the effects of learning parameters on