Accelerating Model-Free Reinforcement Learning With Imperfect Model Knowledge in Dynamic Spectrum Access

Current studies that Our records indicate that Hao-Hsuan Chang is a Graduate Student Member of the IEEE. Please verify. Our records indicate that Jonathan D. Ashdown is a Member of the IEEE. Please verify. apply reinforcement learning (RL) to dynamic spectrum access (DSA) problems in wireless commun...

Full description

Saved in:
Bibliographic Details
Published in:IEEE internet of things journal 2020-08, Vol.7 (8), p.7517-7528
Main Authors: Li, Lianjun, Liu, Lingjia, Bai, Jianan, Chang, Hao-Hsuan, Chen, Hao, Ashdown, Jonathan D., Zhang, Jianzhong, Yi, Yang
Format: Article
Language:English
Subjects:
Acceleration
Algorithms
Complexity
Complexity theory
Computational modeling
Dynamic spectrum access (DSA)
imperfect model
Internet of Things
Learning
Learning (artificial intelligence)
reinforcement learning (RL)
Sensors
training acceleration
Wireless communication
Wireless communication systems
Wireless communications
wireless communications systems
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Current studies that Our records indicate that Hao-Hsuan Chang is a Graduate Student Member of the IEEE. Please verify. Our records indicate that Jonathan D. Ashdown is a Member of the IEEE. Please verify. apply reinforcement learning (RL) to dynamic spectrum access (DSA) problems in wireless communications systems mainly focus on model-free RL (MFRL). However, in practice, MFRL requires a large number of samples to achieve good performance making it impractical in real-time applications such as DSA. Combining model-free and model-based RL can potentially reduce the sample complexity while achieving a similar level of performance as MFRL as long as the learned model is accurate enough. However, in a complex environment, the learned model is never perfect. In this article, we combine model-free and model-based RL, and introduce an algorithm that can work with an imperfectly learned model to accelerate the MFRL. Results show our algorithm achieves higher sample efficiency than the standard MFRL algorithm and the Dyna algorithm (a standard algorithm integrating model-based RL and MFRL) with much lower computation complexity than the Dyna algorithm. For the extreme case where the learned model is highly inaccurate, the Dyna algorithm performs even worse than the MFRL algorithm while our algorithm can still outperform the MFRL algorithm.
ISSN:2327-4662
2327-4662
DOI:10.1109/JIOT.2020.2988268