Basis Adaptation for Sparse Nonlinear Reinforcement Learning

• Published in: Proceedings of the ... AAAI Conference on Artificial Intelligence 2013-06, Vol.27 (1), p.654-660
• Main Authors: Mahadevan, Sridhar, Giguere, Stephen, Jacek, Nicholas
• Format: Article
• Language: English
• ISSN: 2159-5399; 2374-3468
• DOI: 10.1609/aaai.v27i1.8665

This paper presents a new approach to representation discovery in reinforcement learning (RL) using basis adaptation. We introduce a general framework for basis adaptation as {\em nonlinear separable least-squares value function approximation} based on finding Frechet gradients of an error function using variable projection functionals. We then present a scalable proximal gradient-based approach for basis adaptation using the recently proposed mirror-descent framework for RL. Unlike traditional temporal-difference (TD) methods for RL, mirror descent based RL methods undertake proximal gradient updates of weights in a dual space, which is linked together with the primal space using a Legendre transform involving the gradient of a strongly convex function. Mirror descent RL can be viewed as a proximal TD algorithm using Bregman divergence as the distance generating function. We present a new class of regularized proximal-gradient based TD methods, which combine feature selection through sparse L1 regularization and basis adaptation. Experimental results are provided to illustrate and validate the approach.