Hamiltonian-Driven Adaptive Dynamic Programming With Approximation Errors

In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton-Jacobi-Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, "...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on cybernetics 2022-12, Vol.52 (12), p.13762-13773
Main Authors: Yang, Yongliang, Modares, Hamidreza, Vamvoudakis, Kyriakos G., He, Wei, Xu, Cheng-Zhong, Wunsch, Donald C.
Format: Article
Language:English
Subjects:
Algorithms
Approximation
Approximation error
Costs
Dynamic programming
Errors
Hamilton-Jacobi-Bellman (HJB) equation
Hamiltonian-driven framework
inexact adaptive dynamic programming (ADP)
Iterative algorithms
Iterative methods
Mathematical analysis
Nonlinear systems
Optimal control
Stability analysis
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this article, we consider an iterative adaptive dynamic programming (ADP) algorithm within the Hamiltonian-driven framework to solve the Hamilton-Jacobi-Bellman (HJB) equation for the infinite-horizon optimal control problem in continuous time for nonlinear systems. First, a novel function, "min-Hamiltonian," is defined to capture the fundamental properties of the classical Hamiltonian. It is shown that both the HJB equation and the policy iteration (PI) algorithm can be formulated in terms of the min-Hamiltonian within the Hamiltonian-driven framework. Moreover, we develop an iterative ADP algorithm that takes into consideration the approximation errors during the policy evaluation step. We then derive a sufficient condition on the iterative value gradient to guarantee closed-loop stability of the equilibrium point as well as convergence to the optimal value. A model-free extension based on an off-policy reinforcement learning (RL) technique is also provided. Finally, numerical results illustrate the efficacy of the proposed framework.
ISSN:2168-2267
2168-2275
DOI:10.1109/TCYB.2021.3108034