Proactive Caching With Distributed Deep Reinforcement Learning in 6G Cloud-Edge Collaboration Computing

• Published in: IEEE transactions on parallel and distributed systems 2024-08, Vol.35 (8), p.1387-1399
• Main Authors: Wu, Changmao, Xu, Zhengwei, He, Xiaoming, Lou, Qi, Xia, Yuanyuan, Huang, Shuman
• Format: Article
• Language: English
• Subjects: 6G mobile communication; Algorithms; Caching; Collaboration; Convolution; Cooperation; Costs; Decision making; Deep learning; deep reinforcement learning; distributed edge computing; Generative adversarial networks; Machine learning; multi-agent learning architecture; Multiagent systems; Multimedia; Multiple objective analysis; Network latency; Optimization; Predictive models; proactive caching; Servers; Traffic congestion; Training
• ISSN: 1045-9219; 1558-2183
• DOI: 10.1109/TPDS.2024.3406027

Proactive caching in 6G cloud-edge collaboration scenarios, intelligently and periodically updating the cached contents, can either alleviate the traffic congestion of backhaul link and edge cooperative link or bring multimedia services to mobile users. To further improve the network performance of 6G cloud-edge, we consider the issue of multi-objective joint optimization, i.e., maximizing edge hit ratio while minimizing content access latency and traffic cost. To solve this complex problem, we focus on the distributed deep reinforcement learning (DRL)-based method for proactive caching, including content prediction and content decision-making. Specifically, since the prior information of user requests is seldom available practically in the current time period, a novel method named temporal convolution sequence network (TCSN) based on the temporal convolution network (TCN) and attention model is used to improve the accuracy of content prediction. Furthermore, according to the value of content prediction, the distributional deep Q network (DDQN) seeks to build a distribution model on returns to optimize the policy of content decision-making. The generative adversarial network (GAN) is adapted in a distributed fashion, emphasizing learning the data distribution and generating compelling data across multiple nodes. In addition, the prioritized experience replay (PER) is helpful to learn from the most effective sample. So we propose a multivariate fusion algorithm called PG-DDQN. Finally, faced with such a complex scenario, a distributed learning architecture, i.e., multi-agent learning architecture is efficiently used to learn DRL-based methods in a manner of centralized training and distributed inference. The experiments prove that our proposal achieves satisfactory performance in terms of edge hit ratio, traffic cost and content access latency.