Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems

This paper proposes an integral Q-learning for continuous-time (CT) linear time-invariant (LTI) systems, which solves a linear quadratic regulation (LQR) problem in real time for a given system and a value function, without knowledge about the system dynamics A and B. Here, Q-learning is referred to...

Full description

Saved in:
Bibliographic Details
Published in:Automatica (Oxford) 2012-11, Vol.48 (11), p.2850-2859
Main Authors: Lee, Jae Young, Park, Jin Bae, Choi, Yoon Ho
Format: Article
Language:English
Subjects:
Adaptative systems
Adaptive control
Algorithms
Applied sciences
Computer science
control theory
systems
Control system synthesis
Control theory. Systems
Dynamical systems
Dynamics
Exact sciences and technology
Exploration
Integrals
LQR
Optimal control
Optimization
Optimization under uncertainties
Policies
Policy iteration
Polyimide resins
Q-learning
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper proposes an integral Q-learning for continuous-time (CT) linear time-invariant (LTI) systems, which solves a linear quadratic regulation (LQR) problem in real time for a given system and a value function, without knowledge about the system dynamics A and B. Here, Q-learning is referred to as a family of reinforcement learning methods which find the optimal policy by interaction with an uncertain environment. In the evolution of the algorithm, we first develop an explorized policy iteration (PI) method which is able to deal with known exploration signals. Then, the integral Q-learning algorithm for CT LTI systems is derived based on this PI and the variants of Q-functions derived from the singular perturbation of the control input. The proposed Q-learning scheme evaluates the current value function and the improved control policy at the same time, and are proven stable and convergent to the LQ optimal solution, provided that the initial policy is stabilizing. For the proposed algorithms, practical online implementation methods are investigated in terms of persistency of excitation (PE) and explorations. Finally, simulation results are provided for the better comparison and verification of the performance.
ISSN:0005-1098
1873-2836
DOI:10.1016/j.automatica.2012.06.008