On-Policy Versus Off-Policy Reinforcement Learning for Multi-Domain SFC Embedding in SDN/NFV-Enabled Networks

• Published in: IEEE access 2024, Vol.12, p.123049-123070
• Main Authors: Zhao, Donghao, Shi, Wei, Lu, Yu, Li, Xi, Liu, Yicen
• Format: Article
• Language: English
• Subjects: Business; Cloud computing; Computational modeling; Computer architecture; Heuristic algorithms; hidden Markov model (HMM); Hidden Markov models; Markov decision process (MDP); Network function virtualization; network function virtualization (NFV); Reinforcement learning; reinforcement learning (RL); Resource management; Service function chain (SFC); software defined network (SDN); Software defined networking
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2024.3430865

In the software defined network (SDN)/network function virtualization (NFV)-enabled networks, service function chains (SFCs) should typically be allocated to deploy these services, which not only entails meeting the service's Quality of Service (QoS) requirements, but also considering the infrastructure's limitations. Although this issue has received much attention in the literature, the dynamics, intricacy, complexity and unpredictability of the issue provide several difficulties for researchers and engineers. The traditional methods (e.g., exact, heuristic, meta-heuristic, and game, etc.) are subjected to the complexity of multi-domain cloud network scenarios with dynamic network states, high-speed computational requirements, and enormous service requests. Recent studies have shown that reinforcement learning (RL) is a promising way to deal with the limitations of the traditional methods. On-policy and off-policy are two key categories in the field of RL models, and they both have promising advantages in deal with dynamic resource allocation problems. This paper contains two innovative points at two levels. Firstly, in order to deal with SFC embedding problem in dynamic multi-domain networks, a mixed Markov model combining Markov decision process (MDP) and hidden Markov model (HMM) is constructed, and the corresponding RL model-solving algorithms are proposed. Secondly, in order to distinguish the appropriate model in a given network scenario, the on-policy RL based multiple domain SFC embedding algorithm is compared with the off-policy one. The obtained simulation results show that the proposed RL algorithms can outperform the current baselines in terms of delay, load balancing and response time. Furthermore, we also point out that the off-policy based algorithm is more suitable for small-scale dynamic network scenarios, while the on-policy based algorithm is more suitable for medium to large-scale network scenarios with high convergence requirements.