Safe Exploration for Identifying Linear Systems via Robust Optimization

Safely exploring an unknown dynamical system is critical to the deployment of reinforcement learning (RL) in physical systems where failures may have catastrophic consequences. In scenarios where one knows little about the dynamics, diverse transition data covering relevant regions of state-action s...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2017-11
Main Authors: Lu, Tyler, Zinkevich, Martin, Boutilier, Craig, Binz, Roy, Schuurmans, Dale
Format: Article
Language:English
Subjects:
Confidence intervals
Cooling
Data centers
Error correction
Exploration
Linear systems
Mathematical models
Model accuracy
Optimization
Parameter estimation
Parameter identification
Process variables
Random noise
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Safely exploring an unknown dynamical system is critical to the deployment of reinforcement learning (RL) in physical systems where failures may have catastrophic consequences. In scenarios where one knows little about the dynamics, diverse transition data covering relevant regions of state-action space is needed to apply either model-based or model-free RL. Motivated by the cooling of Google's data centers, we study how one can safely identify the parameters of a system model with a desired accuracy and confidence level. In particular, we focus on learning an unknown linear system with Gaussian noise assuming only that, initially, a nominal safe action is known. Define safety as satisfying specific linear constraints on the state space (e.g., requirements on process variable) that must hold over the span of an entire trajectory, and given a Probably Approximately Correct (PAC) style bound on the estimation error of model parameters, we show how to compute safe regions of action space by gradually growing a ball around the nominal safe action. One can apply any exploration strategy where actions are chosen from such safe regions. Experiments on a stylized model of data center cooling dynamics show how computing proper safe regions can increase the sample efficiency of safe exploration.
ISSN:2331-8422