Reinforcement Learning Based Optimal Control of Linear Singularly Perturbed Systems

• Published in: IEEE transactions on circuits and systems. II, Express briefs Express briefs, 2022-03, Vol.69 (3), p.1362-1366
• Main Authors: Zhao, Jianguo, Yang, Chunyu, Gao, Weinan
• Format: Article
• Language: English
• Subjects: Algorithms; Control systems design; Convergence; Cost function; Eigenvalues and eigenfunctions; Machine learning; Mathematical model; model-free control; Optimal control; policy iteration; Reinforcement learning; Riccati equation; Robustness (mathematics); Singular perturbation; Singularly perturbed systems; Symmetric matrices
• ISSN: 1549-7747; 1558-3791
• DOI: 10.1109/TCSII.2021.3105652

This brief studies the optimal control problem of linear singularly perturbed systems (SPSs) via reinforcement learning (RL). We first present an offline model-based algorithm on the basis of Kleinman algorithm to solve the underlying algebraic Riccati equation with singular perturbation parameter and its convergence is proven. This revised version of Kleinman algorithm is the key to develop the subsequent learning algorithm. Then, by means of two time-scale characteristics of SPSs, we present a novel model-free learning algorithm relating to the online measurement of state and input to design the optimal controller. Compared with the existing learning approaches, the obtained RL algorithm is free of ill-conditioned numerical issues caused by the co-existence of fast and slow modes in SPSs and thus is more robust with respect to the computation and measurement errors. Finally, we verify the effectiveness of the developed results by a simulation example.