Linear Quadratic Control Using Model-Free Reinforcement Learning

• Published in: IEEE transactions on automatic control 2023-02, Vol.68 (2), p.737-752
• Main Authors: Yaghmaie, Farnaz Adib, Gustafsson, Fredrik, Ljung, Lennart
• Format: Article
• Language: English
• Subjects: Adaptation models; Adaptive control; Algorithms; Benchmarks; Cost analysis; Costs; Dynamical systems; Exact solutions; Heuristic algorithms; Iterative algorithms; Iterative methods; Linear quadratic (LQ) control; Machine learning; Noise measurement; Optimal control; Process control; reinforcement learning (RL); Stability analysis; State variable
• ISSN: 0018-9286; 1558-2523; 1558-2523
• DOI: 10.1109/TAC.2022.3145632

In this article, we consider linear quadratic (LQ) control problem with process and measurement noises. We analyze the LQ problem in terms of the average cost and the structure of the value function. We assume that the dynamics of the linear system is unknown and only noisy measurements of the state variable are available. Using noisy measurements of the state variable, we propose two model-free iterative algorithms to solve the LQ problem. The proposed algorithms are variants of policy iteration routine where the policy is greedy with respect to the average of all previous iterations. We rigorously analyze the properties of the proposed algorithms, including stability of the generated controllers and convergence. We analyze the effect of measurement noise on the performance of the proposed algorithms, the classical off-policy, and the classical Q-learning routines. We also investigate a model-building approach, inspired by adaptive control, where a model of the dynamical system is estimated and the optimal control problem is solved assuming that the estimated model is the true model. We use a benchmark to evaluate and compare our proposed algorithms with the classical off-policy, the classical Q-learning, and the policy gradient. We show that our model-building approach performs nearly identical to the analytical solution and our proposed policy iteration-based algorithms outperform the classical off-policy and the classical Q-learning algorithms on this benchmark but do not outperform the model-building approach.