Energy-Efficient Mode Selection and Resource Allocation for D2D-Enabled Heterogeneous Networks: A Deep Reinforcement Learning Approach

• Published in: IEEE transactions on wireless communications 2021-02, Vol.20 (2), p.1175-1187
• Main Authors: Zhang, Tao, Zhu, Kun, Wang, Junhua
• Format: Article
• Language: English
• Subjects: Algorithms; Base stations; Cellular communication; deep deterministic policy gradient; Deep learning; device-to-device (D2D) communication; Device-to-device communication; Energy conversion efficiency; Energy efficiency; Heterogeneous networks; Machine learning; Markov processes; Modal choice; mode selection; Networks; Optimization; Power control; Resource allocation; Resource management; Wireless communication
• ISSN: 1536-1276; 1558-2248
• DOI: 10.1109/TWC.2020.3031436

Improving energy efficiency has shown increasing importance in designing future cellular system. In this work, we consider the issue of energy efficiency in D2D-enabled heterogeneous cellular networks. Specifically, communication mode selection and resource allocation are jointly considered with the aim to maximize the energy efficiency in the long term. And an Markov decision process (MDP) problem is formulated, where each user can switch between traditional cellular mode and D2D mode dynamically. We employ deep deterministic policy gradient (DDPG), a model-free deep reinforcement learning algorithm, to solve the MDP problem in continuous state and action space. The architecture of proposed method consists of one actor network and one critic network. The actor network uses deterministic policy gradient scheme to generate deterministic actions for agent directly, and the critic network employs value function based Q networks to evaluate the performance of the actor network. Simulation results show the convergence property of proposed algorithm and the effectiveness in improving the energy efficiency in a D2D-enabled heterogeneous network.