Transition Based Discount Factor for Model Free Algorithms in Reinforcement Learning

Reinforcement Learning (RL) enables an agent to learn control policies for achieving its long-term goals. One key parameter of RL algorithms is a discount factor that scales down future cost in the state’s current value estimate. This study introduces and analyses a transition-based discount factor...

Full description

Saved in:
Bibliographic Details
Published in:Symmetry (Basel) 2021-07, Vol.13 (7), p.1197
Main Authors: Sharma, Abhinav, Gupta, Ruchir, Lakshmanan, K., Gupta, Atul
Format: Article
Language:English
Subjects:
Algorithms
Approximation
Convergence
Costs
Discounts
Machine learning
model free algorithms
Q-learning
reinforcement learning
SARSA
transition dependent discount factor
Wireless networks
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reinforcement Learning (RL) enables an agent to learn control policies for achieving its long-term goals. One key parameter of RL algorithms is a discount factor that scales down future cost in the state’s current value estimate. This study introduces and analyses a transition-based discount factor in two model-free reinforcement learning algorithms: Q-learning and SARSA, and shows their convergence using the theory of stochastic approximation for finite state and action spaces. This causes an asymmetric discounting, favouring some transitions over others, which allows (1) faster convergence than constant discount factor variant of these algorithms, which is demonstrated by experiments on the Taxi domain and MountainCar environments; (2) provides better control over the RL agents to learn risk-averse or risk-taking policy, as demonstrated in a Cliff Walking experiment.
ISSN:2073-8994
2073-8994
DOI:10.3390/sym13071197