Off-Policy Risk Assessment in Contextual Bandits

Even when unable to run experiments, practitioners can evaluate prospective policies, using previously logged data. However, while the bandits literature has adopted a diverse set of objectives, most research on off-policy evaluation to date focuses on the expected reward. In this paper, we introduc...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2021-06
Main Authors: Huang, Audrey, Liu Leqi, Lipton, Zachary C, Azizzadenesheli, Kamyar
Format: Article
Language:English
Subjects:
Algorithms
Confidence
Distribution functions
Error analysis
Estimates
Estimators
Evaluation
Experimentation
Importance sampling
Policies
Risk assessment
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Even when unable to run experiments, practitioners can evaluate prospective policies, using previously logged data. However, while the bandits literature has adopted a diverse set of objectives, most research on off-policy evaluation to date focuses on the expected reward. In this paper, we introduce Lipschitz risk functionals, a broad class of objectives that subsumes conditional value-at-risk (CVaR), variance, mean-variance, many distorted risks, and CPT risks, among others. We propose Off-Policy Risk Assessment (OPRA), a framework that first estimates a target policy's CDF and then generates plugin estimates for any collection of Lipschitz risks, providing finite sample guarantees that hold simultaneously over the entire class. We instantiate OPRA with both importance sampling and doubly robust estimators. Our primary theoretical contributions are (i) the first uniform concentration inequalities for both CDF estimators in contextual bandits and (ii) error bounds on our Lipschitz risk estimates, which all converge at a rate of \(O(1/\sqrt{n})\).
ISSN:2331-8422