Surrogate model enabled deep reinforcement learning for hybrid energy community operation

• Published in: Applied energy 2021-05, Vol.289, p.116722, Article 116722
• Main Authors: Wang, Xiaodi, Liu, Youbo, Zhao, Junbo, Liu, Chang, Liu, Junyong, Yan, Jinyue
• Format: Article
• Language: English
• Subjects: Community market; Deep reinforcement learning; Optimization; P2P transaction; Transactive control
• ISSN: 0306-2619; 1872-9118; 1872-9118
• DOI: 10.1016/j.apenergy.2021.116722

•A hybrid community P2P market for multi-energy systems is presented.•A market surrogate model is designed to estimate community P2P transaction state.•The market surrogate model is integrated with DDPG for retail price selection.•The P2P market enables safe delivery service of P2P energy flow with dynamic prices. Local peer-to-peer (P2P) transactions in a community are becoming a trend for energy integration and management. The introduction of P2P trading scheme requires comprehensive consideration on various aspects, such as peer privacy, computational efficiency, network security and operational economics. This paper provides a novel hybrid community P2P market framework for multi-energy systems, where a data-driven market surrogate model-enabled deep reinforcement learning (DRL) method is proposed to facilitate P2P transaction within technical constraints of the community delivery networks. Specifically, to achieve privacy protection, a market surrogate model based on deep belief network (DBN) is developed to characterize P2P transaction behaviors of peers in the community without disclosing their private data. Since the energy inputs and outputs of peers are highly correlated with real time signals of retail energy prices, the data-driven market surrogate model is further integrated into the DRL-enabled optimization model of a community agent (CA) for on-line retail energy price generation. Particularly, by integrating network constraints into DRL reward function, the P2P transaction scheme among community peers under specific retail energy price is guaranteed to proceed within a feasible region of community networks. Numerical results indicate that the proposed market framework can achieve 7.6% energy cost saving for community peers over none P2P transaction scheme while increase 284.4$ economic benefits for CA in one day over other comparison algorithms. This study provides an effective prototype to supplement existing P2P markets.