Deep Reinforcement Learning-Based Network Slicing for Beyond 5G

Joint modulation-coding scheme (MCS) adaptation and resource block (RB) allocation is an effective approach to guarantee different quality of service (QoS) requirements of all UEs under dynamic network environments. In this paper, we consider a fifth generation (5 G) cellular network with time-varying wireless channels, in which the BS serves multiple user equipments (UEs) under limited available RBs. We aim to minimize the total RB consumption subject to the rigorous constraints of each UE's QoS requirement. To attain this objective, this paper puts forth an online learning technique, referred to as integrated Deep Reinforcement learning and stable Matching (DeepRM), in the sense that the MCS adaptation and RB allocation decisions are conducted without acquiring the real-time channel quality indicator (CQI) feedback. DeepRM is a closed-loop framework, where the output of deep reinforcement learning (DRL) is imported into the stable matching to guide optimal RB allocation whilst the output of stable matching is fed into the DRL framework to assist efficient MCS decision-making. Specifically, in DeepRM, we first develop a powerful DRL algorithm, termed as Action-and-Reward Branching Deep Q-network (ARBDQ), by incorporating the action branch architecture into conventional DRL and modifying the traditional deep neural network training mechanism, to perform judicious MCS decisions on different links in parallel. Then, a new many-to-one stable matching algorithm, called adaptive deferred acceptance, is exploited to dynamically adjust the RB quota of each UE in a computationally efficient fashion. Simulation results demonstrate that compared with ACO-HM, OLLA-ADA, and ARBDQ-Random algorithms, DeepRM induces much less RB consumption while guaranteeing the QoS requirements of all UEs in various network scenarios. Furthermore, under miscellaneous QoS requirement, number of UEs, and CQI reporting period setups, DeepRM is more robust than other baselines.

In 5G networks, the deployment of network slices enabled by Software-Defined Networking (SDN) is becoming a critical component for delivering tailored services to meet diverse application needs. However, this introduces challenges in network management, particularly in efficiently allocating resources to ensure that each network slice meets its specific Quality of Service (QoS) and availability requirements. Simultaneously, it must optimize overall network performance and network operator’s profit, which is linked to the Quality of Experience (QoE) of the end-users. Existing works offer either an availability-based solution or a QoE-aware solution to this problem, but not both. This paper addresses the end-to-end network slice resource allocation problem by simultaneously considering QoS and availability requirements in slice placement, while employing a QoE-aware strategy for resource allocation. We propose a framework that optimizes the network operator profit i.e. the highest QoE with the least resource usage, and can be flexibly configured to model realistic scenarios. Arbitrary network slice requirements can be defined using slice-specific QoS/QoE mapping, resource requirements, end-to-end latency and availability. For solving the formulated problem a Mixed Integer Nonlinear Programming (MINLP) formulation and efficient heuristic methods are proposed. Our solution accounts for the non-linear QoS/QoE relationship, utilizes redundantly placed Service Function Chains (SFCs) to increase availability, and supports the sharing of Virtual Network Functions (VNFs) among SFCs to optimize resource usage. Through extensive simulations on realistic network topologies and slice requests, we demonstrate the framework’s effectiveness in offering flexible and efficient network slice placement and resource allocation, utilizing a baseline heuristic from related studies. The results indicate that while the exact method delivers an optimal solution, heuristic approaches are suitable for time-sensitive tasks, such as dynamic slice configuration.

End-to-end network slicing in unmanned aerial vehicle (UAV) networks has attracted significant attention for delivering differentiated services tailored to the quality of service (QoS) requirements of user terminals (UTs). In a sliced UAV network, the dual mobility of both UTs and UAVs, combined with the three-layer association of UT-UAV-network slice (NS), makes handoff a critical and challenging problem for maintaining the communication performance of UTs. This paper proposes a sequential handoff process where a central controller makes handoff decisions for each UT sequentially according to the UT queue. This process is modeled as a Markov decision process (MDP). In the MDP, the state space includes NS available resources, UTs' QoS requirements, and UT queue information. Additionally, the reward function accounts for UTs' QoS and handoff overhead. The double deep Q-network (DDQN) technique is then employed to design a handoff algorithm. This algorithm achieves the optimization goal of ensuring UTs' QoS requirements while avoiding frequent handoffs by maximizing the total reward of the MDP. Additionally, a state-dependent exploration technique with adaptive intensity adjustment, called noisy network (NoisyNet), is utilized to drive the DDQN for effective exploration, thereby enhancing the performance of the handoff algorithm. Finally, the runtime complexity of the proposed NoisyNet-DDQN-based sequential handoff (NoisyNet-DDQN-SH) algorithm is analyzed. The simulation results show that the NoisyNet-DDQN-SH algorithm converges well. Moreover, it exhibits strong robustness. Additionally, the NoisyNet technique effectively enhances the algorithm's performance, making the NoisyNet-DDQN-SH algorithm superior to the vanilla DDQN (with an ϵ-greedy strategy), greedy, maximum signal-to-interference-plus-noise ratio (MaxSINR), and random algorithms.

In this chapter, we present a comprehensive end-to-end (E2E) network slicing framework for fifth-generation (5G) wireless and core networks and provide our vision on network slicing for beyond 5G (B5G) networks using artificial intelligence (AI) technologies. We first give an introduction on the 5G communication networks by specifying distinct 5G networking characteristics. As the key features of 5G pose challenges on quality-of-service (QoS)-oriented resource provisioning, a new concept of network slicing is presented, based on software-defined networking (SDN) and network function virtualization (NFV) technologies, to enhance the overall resource utilization while guaranteeing QoS isolation among services. We focus on developing specific network slicing solutions for both 5G wireless networks and core networks. In addition, the evolution of B5G networks is discussed in terms of a potential networking paradigm shift with new service key performance indicators (KPIs). To satisfy B5G service requirements, a framework of AI-assisted network slicing lifecycle is developed to automate the slice creation with reduced slice management complexity. A case study is presented to demonstrate the effectiveness of the proposed E2E network slicing solutions.Keywords5G communication networksNetwork slicingWireless networksRadio resource slicingCore networksComputing and transmission resource slicingBeyond 5G networksAI-assisted network slicingQoSE2E delay

The rapid development of smart grid has put forward higher requirements on communication technology. With the rapid development of power wireless communication network, the demand of power business becomes more and more complicated and diverse. However, the current single-standard network has poor coverage, poor power service carrying capacity, and weak transmission stability and security. To address the above problems, this paper explores the network slicing technology for intelligent power distribution services. Based on the analysis of existing network access selection methods, a network selection strategy for quality of service (QoS) assurance of power services is proposed for the characteristics of power distribution services. Combining its own QoS requirements and the characteristics of the network to be accessed, the power service can realize the access selection of the optimal adaptation network. In order to better meet various vertical industry applications, network slicing is required in both the core network and the radio access network, i.e., end-to-end network slicing is realized, and network slicing can be customized to tailor network functions and reasonably allocate network resources according to service scenario requirements. After receiving a request for network slicing, operators need to map the virtual network functions required for network slicing to the underlying physical network, a process that involves the optimal allocation of physical resources.

Effective zero-touch management and orchestration (ZSM&O) is crucial for scaling network slicing, particularly transitioning toward Beyond 5G (B5G) and 6G networks. This paper empirically validates the network slicing framework developed under the European Union Horizon 2020 MonB5G project. Building on three years of academia-industry collaboration, MonB5G introduces a flexible slicing model featuring umbrella slices that orchestrate modular, specialized slices across multi-domain environments to address next-generation service demands. For the first time, we evaluate its practicality in a 5G cloud-native testbed through a virtual reality (VR) streaming use case, supported by solutions such as federated learning-based CPU forecasting, anomaly detection, and deep reinforcement learning (DRL) for radio access network (RAN) optimization. The paper offers insights from technically demanding experimental tests and highlights challenges and development paths for managing next-generation mobile networks.

Artificial intelligence (AI)-driven zero-touch massive network slicing is envisioned to be a disruptive technology in beyond 5G (B5G)/6G, where tenancy would be extended to the final consumer in the form of advanced digital use-cases. In this paper, we propose a novel model-free deep reinforcement learning (DRL) framework, called collaborative statistical Actor-Critic (CS-AC) that enables a scalable and farsighted slice performance management in a 6G-like RAN scenario that is built upon mobile edge computing (MEC) and massive multiple-input multiple-output (mMIMO). In this intent, the proposed CS-AC targets the optimization of the latency cost under a long-term statistical service-level agreement (SLA). In particular, we consider the Q-th delay percentile SLA metric and enforce some slice-specific preset constraints on it. Moreover, to implement distributed learners, we propose a developed variant of soft Actor-Critic (SAC) with less hyperparameter sensitivity. Finally, we present numerical results to showcase the gain of the adopted approach on our built OpenAI-based network slicing environment and verify the performance in terms of latency, SLA Q-th percentile, and time efficiency. To the best of our knowledge, this is the first work that studies the feasibility of an AI-driven approach for massive network slicing under statistical SLA.

The communication industry is rapidly advancing towards 5G and beyond 5G (B5G) wireless technologies in order to fulfill the ever-growing needs for higher data rates and improved quality-of-service (QoS). Emerging applications require wireless connectivity with tremendously increased data rates, substantially reduced latency, and growing support for a large number of devices. These requirements pose new challenges that can no longer be efficiently addressed by conventional approaches. Artificial intelligence (AI) is considered as one of the most promising solutions to improve the performance and robustness of 5G and B5G systems, fueled by the massive amount of data generated in 5G and B5G networks and the availability of powerful data processing fabrics. As a consequence, a plethora of research on AI-based communication technologies has emerged recently, promising higher data rates and improved QoS with affordable implementation overhead. In this overview paper, we summarize the state-of-the-art of AI-based 5G and B5G techniques on the algorithm, implementation, and optimization levels. We shed light on the advantages and limitations of AI-based solutions, and we provide a summary of emerging techniques and open research problems.

To bring 5G systems and networks to life in large-scale commercial applications, academia community has started the research beyond 5G (B5G), in which network slicing (NS) is proposed as a new paradigm for building service-tailored B5G networks. In each network slice, to precisely control the service quality and cost, deploying the service-required virtual network functions (VNFs) by utilizing the linkage between the characteristics of this slicing task and the characteristics of different servers in the B5G network is essential. Therefore, aiming at gaining the ability of learning and adapting new tasks quickly and cost effectively, we view the NFV deployment problem as a meta relational learning process that explores the meta mapping relation between service-tailored slicing tasks and the B5G physical network and propose a service-tailored VNF deployment framework, abbreviated as StailNet. Instead of training a one-strategy-fits-all deployment model, we focus on “learning” how to train a deployment model and propose to learn the features of servers and slicing tasks from the perspective of knowledge graph-based representation learning, then locate the initial meta mapping relation by extracting meta information in the task-agnostic meta space and exploring the service-tailored meta mapping relation in the task space for each task, so that we can quickly obtain the solution by a few gradients on the initial meta mapping relation. To highlight the performances of StailNet, we do comprehensive simulations. Simulation results demonstrate that our StailNet outperforms the selected representative algorithms in the literature.

Fifth-generation (5G) networks are already available in major urban areas and are expected to bring a major transformation to citizens’ lives. 5G services, such as enhanced mobile broadband (eMBB), ultra-reliable low latency communications (URLLC), and massive machine-type communications (mMTC), require a network infrastructure capable of supporting stringent requirements in terms of latency and bandwidth demands; as such, it must be highly dynamic and flexible. Network slicing is a key enabler technology that can provide dynamic and flexible characteristics to 5G network architecture. A network slice (NS) can be defined as a partition of network and IT resources, that is, network links and nodes capacity dedicated to a specific set of service demands. As a result, different NSs can coexist over the same physical infrastructure network and can be used to dynamically and flexibly deploy the aforementioned 5G services. However, to efficiently implement NSs with different requirements, communication service providers (CSPs) that own the physical infrastructure network must adopt sophisticated techniques for admission control and resource allocation of NSs. In this paper, we present a novel framework for admission control and resource allocation of 5G NSs in metro-core networks. Specifically, our framework is based on a deep reinforcement learning (DRL) algorithm called Advantage Actor Critic (A2C), which performs admission control, i.e. it is capable of learning which slice to admit based on the availability of the physical network resources. Then, given the diversity of requirements for each 5G service, we propose different resource allocation algorithms based on integer linear programming (ILP) and heuristics to treat each service accordingly. Results show that our proposed framework can increase the number of admitted NSs with respect to the case in which the admission control is disabled by improving the resource allocation performance.

The deployment of reliable maritime communication systems integrated with terrestrial networks is challenging due to the large maritime regions, the difficulty of deploying conventional base stations at sea, and the heterogeneity of proprietary equipment. In this paper, we propose an AI-based network slicing framework for Open Radio Access Network (O-RAN) integrated aerial-terrestrial maritime networks that incorporates non-tethered and tethered unmanned aerial vehicles (UAVs) and marine buoys. The network provides ubiquitous connectivity while addressing diverse maritime user requirements. Specifically, we leverage network slicing to accommodate the needs of two distinct slices: the maritime infotainment slice, which demands high data rates, and the maritime emergency communication slice, which requires high-reliability and low-latency. Moreover, we adopt virtualization principles to enable flexible deployment approaches for virtualized network functions (VNFs), that can be dynamically scaled and/or migrated across virtualized network nodes. Then, we design a network slicing framework based on a Deep Reinforcement Learning (DRL) that takes into account the characteristics of the maritime environment, and present two algorithms using Advantage Actor-Critic (A2C) and Proximal Policy Optimization (PPO). Our findings highlight the importance of the integration of aerial and terrestrial networks with network slicing to enhance the energy efficiency of maritime communications while meeting diverse Quality of Service (QoS) requirements.

To achieve the service-oriented features of the 5G, network slicing aims to create logical virtual networks where multiple services are provided on a common physical infrastructure. The performance of network slicing depends on the intelligent management of multi-dimensional resources, which are exactly what multi-access edge computing (MEC) provides. This paper proposes joint optimization of communication, computing and caching (3C) resources in multi-access edge network slicing. The optimization objective of the two-level resource allocation problem is to maximize the utility obtained by mobile virtual network operators while ensuring the quality of service (QoS). The deep reinforcement learning (DRL) approach is employed which enables the resource allocation scheme to intelligently adapt to the dynamic environment. Specifically, we propose a novel DRL approach named twin-actor deep deterministic policy gradient (twin-actor DDPG). Since the action space is continuous, the DDPG is adopted where the actor generates the deterministic policy while the critic evaluates the policy and guides the actor to obtain the optimal policy. A novel twin-actor structure is put forward to replace the actor of the DDPG, thus the slice-level action and user-level action can be generated respectively. The convergence and effectiveness of the proposed DRL based algorithm is are verified by numerical simulation.

Network slicing has been proposed as an innovative concept to partition a physical network into different configurable slices, thereby allowing the network to meet the diverse requirements of expected services in the 5G era. Existing network-slicing schemes focus almost exclusively on the partitioning of the core network or the radio access network. However, to achieve greater flexibility and better resource utilization in service provisioning, the potential of network slicing in the 5G optical mobile fronthaul (MFH) network should be jointly exploited. To address this challenge, we propose a flexible hierarchical edge cloud architecture to enable 5G optical MFH network slicing. Additionally, a corresponding integrated network resource management scheme is proposed to support various quality-of-service (QoS) requirements and diverse resource requests of the network slices. Simulation results indicate that the proposed scheme is able to jointly allocate the bandwidth resources to network slices and realize cloud-computing offloading to meet various QoS requirements, and therefore reduce the fronthaul bandwidth burden.

The 5th generation of mobile communications is expected to provide a network suitable for any service requirements using a technology termed network slicing. In network slicing, network resources are divided and are allocated to slices for services. Radio access network slicing is essential for end-to-end network slicing. The state of the radio access network changes from moment to moment and automatic control of network slicing is necessary to respond to service requirements in real time. Automatic control is applied using deep reinforcement learning that can learn optimal control by repeating trial and error. In this paper, we propose a method to allocate radio resources that satisfy the service requirements regardless of the number of slices using deep reinforcement learning. The proposed method flexibly controls multiple slices by observing the state of one slice and calling a model that controls one slice multiple times. The proposed method was evaluated by simulation using a scenario in which the number of slices changes with time. The results of the evaluation show that variations in slice numbers achieve almost completely satisfying of slice requirements without being affected by other slices.

Network slicing is an architectural solution that enables the future 5G network to offer a high data traffic capacity and efficient network connectivity. Moreover, software defined network (SDN) and network functions virtualization (NFV) empower this architecture to visualize the physical network resources. The network slicing identified as a multiple logical network, where each network slice dedicates as an end-to-end network and works independently with other slices on a common physical network resources. Most user devices have more than one smart wireless interfaces to connect to different radio access technologies (RATs) such as WiFi and LTE, thereby network operators utilize this facility to offload mobile data traffic. Therefore, it is important to enable a network slicing to manage different RATs on the same logical network as a way to mitigate the spectrum scarcity problem and enables a slice to control its users' mobility across different access networks. In this paper, we propose a mobility management architecture based network slicing where each slice manages its users across heterogeneous radio access technologies such as WiFi, LTE and 5G networks. In this architecture, each slice has a different mobility demands and these demands are governed by a network slice configuration and service characteristics. Therefore, our mobility management architecture follows a modular approach where each slice has individual module to handle the mobility demands and enforce the slice policy for mobility management. The advantages of applying our proposed architecture include: i) Sharing network resources between different network slices; ii) creating logical platform to unify different RATs resources and allowing all slices to share them; iii) satisfying slice mobility demands.