Output‐feedback Q‐learning for discrete‐time linear H∞ tracking control: A Stackelberg game approach

• Published in: International journal of robust and nonlinear control 2022-08, Vol.32 (12), p.6805-6828
• Main Authors: Ren, Yunxiao, Wang, Qishao, Duan, Zhisheng
• Format: Article
• Language: English
• Subjects: Algorithms; Feedback; Game theory; Games; H-infinity control; H∞$$ {H}_{\infty } $$ tracking control; Machine learning; Optimal control; Optimization; output feedback control; reinforcement learning; Riccati equation; Stackelberg game; System dynamics; Tracking control
• ISSN: 1049-8923; 1099-1239
• DOI: 10.1002/rnc.6169

In this article, an output‐feedback Q‐learning algorithm is proposed for the discrete‐time linear system to deal with the H∞$$ {H}_{\infty } $$ tracking control problem. The problem is formulated as a zero‐sum game in the Stackelberg game framework with a discount factor to make the value function bounded. According to the principle of optimality, the game algebraic Riccati equation (GARE) is derived and solved by the Q‐learning algorithm to get the optimal solution of the Stackelberg game without requiring the knowledge of system dynamics and state. It is proved that the solution of the algorithm converges to the optimal control input and the worst‐case disturbance with excitation noises during training, and the Stackelberg strategy can achieve a lower L2$$ {L}_2 $$ disturbance attenuation level than the Nash one. Moreover, the impacts of the discount factor on the stability of the closed‐loop system and solvability of the GARE are analyzed to provide some criteria for the choice of the discount factor. Simulation examples are provided to validate the effectiveness of the algorithm.