The Expected Total Cost Criterion for Markov Decision Processes under Constraints: A Convex Analytic Approach

This paper deals with discrete-time Markov decision processes (MDPs) under constraints where all the objectives have the same form of expected total cost over the infinite time horizon. The existence of an optimal control policy is discussed by using the convex analytic approach. We work under the a...

Full description

Saved in:
Bibliographic Details
Published in:Advances in applied probability 2012-09, Vol.44 (3), p.774-793
Main Authors: Dufour, François, Horiguchi, M., Piunovskiy, A. B.
Format: Article
Language:English
Subjects:
60J10
90C40
90C90
constraint
Control theory
Cost estimates
Cost functions
Decision making models
Determinism
expected total cost criterion
General Applied Probability
Linear programming
Marketing
Markov analysis
Markov decision process
Markov processes
Mathematical analysis
Mathematical functions
Mathematical models
Mathematics
Occupation
occupation measure
Optimal control
Optimal solutions
Optimization
Optimization and Control
Optimization techniques
Policies
Probability
Studies
Tenants
Topology
Total costs
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper deals with discrete-time Markov decision processes (MDPs) under constraints where all the objectives have the same form of expected total cost over the infinite time horizon. The existence of an optimal control policy is discussed by using the convex analytic approach. We work under the assumptions that the state and action spaces are general Borel spaces, and that the model is nonnegative, semicontinuous, and there exists an admissible solution with finite cost for the associated linear program. It is worth noting that, in contrast to the classical results in the literature, our hypotheses do not require the MDP to be transient or absorbing. Our first result ensures the existence of an optimal solution to the linear program given by an occupation measure of the process generated by a randomized stationary policy. Moreover, it is shown that this randomized stationary policy provides an optimal solution to this Markov control problem. As a consequence, these results imply that the set of randomized stationary policies is a sufficient set for this optimal control problem. Finally, our last main result states that all optimal solutions of the linear program coincide on a special set with an optimal occupation measure generated by a randomized stationary policy. Several examples are presented to illustrate some theoretical issues and the possible applications of the results developed in the paper.
ISSN:0001-8678
1475-6064
DOI:10.1239/aap/1346955264