A Geometric Perspective on Optimal Representations for Reinforcement Learning

We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks. Our formulation considers adapting...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2019-06
Main Authors: Bellemare, Marc G, Dabney, Will, Dadashi, Robert, Adrien Ali Taiga, Castro, Pablo Samuel, Nicolas Le Roux, Schuurmans, Dale, Lattimore, Tor, Lyle, Clare
Format: Article
Language:English
Subjects:
Approximation
Decision making models
Machine learning
Mathematical analysis
Pareto optimum
Representations
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks. Our formulation considers adapting the representation to minimize the (linear) approximation of the value function of all stationary policies for a given environment. We show that this optimization reduces to making accurate predictions regarding a special class of value functions which we call adversarial value functions (AVFs). We demonstrate that using value functions as auxiliary tasks corresponds to an expected-error relaxation of our formulation, with AVFs a natural candidate, and identify a close relationship with proto-value functions (Mahadevan, 2005). We highlight characteristics of AVFs and their usefulness as auxiliary tasks in a series of experiments on the four-room domain.
ISSN:2331-8422