Exploration and exploitation balance management in fuzzy reinforcement learning

This paper offers a fuzzy balance management scheme between exploration and exploitation, which can be implemented in any critic-only fuzzy reinforcement learning method. The paper, however, focuses on a newly developed continuous reinforcement learning method, called fuzzy Sarsa learning (FSL) due...

Full description

Saved in:
Bibliographic Details
Published in:Fuzzy sets and systems 2010-02, Vol.161 (4), p.578-595
Main Authors: Derhami, Vali, Majd, Vahid Johari, Nili Ahmadabadi, Majid
Format: Article
Language:English
Subjects:
Applied sciences
Circuit properties
Computer science
control theory
systems
Decision analysis
Digital circuits
Electric, optical and optoelectronic circuits
Electronic circuits
Electronics
Exact sciences and technology
Exploitation
Exploration
Fuzzy control
Information, signal and communications theory
Mathematical analysis
Mathematical methods
Mathematics
Miscellaneous
Real functions
Reinforcement learning
Sciences and techniques of general use
Telecommunications and information theory
Theoretical computing
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper offers a fuzzy balance management scheme between exploration and exploitation, which can be implemented in any critic-only fuzzy reinforcement learning method. The paper, however, focuses on a newly developed continuous reinforcement learning method, called fuzzy Sarsa learning (FSL) due to its advantages. Establishing balance greatly depends on the accuracy of action value function approximation. At first, the overfitting problem in approximating action value function in continuous reinforcement learning algorithms is discussed, and a new adaptive learning rate is proposed to prevent this problem. By relating the learning rate to the inverse of “fuzzy visit value” of the current state, the training data set is forced to have uniform effect on the weight parameters of the approximator and hence overfitting is resolved. Then, a fuzzy balancer is introduced to balance exploration vs. exploitation by generating a suitable temperature factor for the Softmax formula. Finally, an enhanced FSL (EFSL) is offered by integrating the proposed adaptive learning rate and the fuzzy balancer into FSL. Simulation results show that EFSL eliminates overfitting, well manages balance, and outperforms FSL in terms of learning speed and action quality.
ISSN:0165-0114
1872-6801
DOI:10.1016/j.fss.2009.05.003