Robust Reinforcement Learning for Decision Making Under Uncertainty in Electricity Markets

Reinforcement learning (RL) is a powerful tool for market agents solving decision-making problems in electricity markets. Vanilla RL enables agents to learn optimal policies in dynamic and uncertain market environments via trial and error. However, uncertainties in state transitions are often treate...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on power systems 2024-11, p.1-14
Main Authors: Qiu, Dawei, Wang, Jianhong, Ruan, Guangchun, Zhang, Qianzhi, Strbac, Goran
Format: Article
Language:English
Subjects:
Decision making
decision making under uncertainty
Electricity
Electricity markets
Electricity supply industry
Heuristic algorithms
Power system dynamics
Pricing
robust reinforcement learning
Robustness
Stochastic processes
Training
Uncertainty
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Reinforcement learning (RL) is a powerful tool for market agents solving decision-making problems in electricity markets. Vanilla RL enables agents to learn optimal policies in dynamic and uncertain market environments via trial and error. However, uncertainties in state transitions are often treated as exogenous state features with statistical errors. This approach can result in policies that are sensitive to perturbations of these uncertainties, potentially leading to performance degradation. This sensitivity is particularly critical in electricity markets, where the penetration of renewable energy and demand variability are increasing. To address this issue, this paper proposes a robust adversarial RL algorithm aimed at learning a robust optimal policy that accounts for market uncertainties in state transitions to systematically mitigate sensitivity to perturbations in uncertain environments. Specifically, we leverage the uncertainty set regularizer technique to define uncertainty sets within the parametric space of state transitions. Furthermore, we introduce a novel adversarial approach to generate unknown uncertainty sets using the value function as a basis. We finally conduct a comprehensive assessment of the robust adversarial RL algorithm across three electricity market applications: strategic bidding, retail pricing, and peer-to-peer energy trading, demonstrating significant improvements in robustness performance against various uncertainties.
ISSN:0885-8950
1558-0679
DOI:10.1109/TPWRS.2024.3502639