A Collaborative Statistical Actor-Critic Learning Approach for 6G Network Slicing Control

In this paper, we address the difficulty of solving large-scale multi-dimensional knapsack instances (MKP), presenting a novel deep reinforcement learning (DRL) framework. In this DRL framework, we train different agents compatible with a discrete action space for sequential decision-making while still satisfying any resource constraint of the MKP. This novel framework incorporates the decision variable values in the 2D DRL where the agent is responsible for assigning a value of 1 or 0 to each of the variables. To the best of our knowledge, this is the first DRL model of its kind in which a 2D environment is formulated, and an element of the DRL solution matrix represents an item of the MKP. Our framework is configured to solve MKP instances of different dimensions and distributions. We propose a K-means approach to obtain an initial feasible solution that is used to train the DRL agent. We train four different agents in our framework and present the results comparing each of them with the CPLEX commercial solver. The results show that our agents can learn and generalize over instances with different sizes and distributions. Our DRL framework shows that it can solve medium-sized instances at least 45 times faster in CPU solution time and at least 10 times faster for large instances, with a maximum solution gap of 0.28% compared to the performance of CPLEX. Furthermore, at least 95% of the items are predicted in line with the CPLEX solution. Computations with DRL also provide a better optimality gap with respect to state-of-the-art approaches.

This paper presents a novel model-free deep reinforcement learning (DRL) framework for optimal hydrogen production and delivery operations in a green hydrogen hub (GHH), where produced hydrogen is delivered to customer locations via a fleet of portable tanks. The problem formulation takes into account (1) realistic nonlinear physical constraints and characteristics of the GHH components, such as the proton exchange membrane (PEM) electrolyzer and storage system, and (2) hydrogen delivery challenges, including hydrogen demand and traffic flow uncertainty. The proposed framework adopts 1) a safe proximal policy optimization algorithm that embeds an interior-point log-barrier in the actor loss, which ensures hard operational constraint satisfaction during learning over high-dimensional continuous states and actions, and 2) a deep neural network-based forecasting algorithm that actively monitors traffic flows and travel times between the GHH and consumer locations and estimates hydrogen demand and renewable power generation. Moreover, the DRL method can track electrolyzer degradation by using the predicted remaining useful life of the PEM unit, thereby introducing more realistic conditions that account for the impact of the PEM unit's practical lifetime on GHH operation. The proposed framework effectively schedules the GHH and manages portable hydrogen tank distribution under uncertainties. Comparisons with model-based information-gap decision theory optimization and state-of-the-art DRL methods validate the effectiveness of the proposed framework across various conditions.

With the advent of 5G era, network slicing has received a great deal of attention as a means to support a variety of wireless services in a flexible manner. Network slicing is a technique to divide a single physical resource network into multiple slices supporting independent services. In beyond 5G (B5G) systems, the main goal of network slicing is to assign the physical resource blocks (RBs) such that the quality of service (QoS) requirements of eMBB, URLLC, and mMTC services are satisfied. Since the goal of each service category is dearly distinct and the computational burden caused by the increased number of time slots is huge, it is in general very difficult to assign RB properly. In this paper, we propose a deep reinforcement learning (DRL)-based network slicing technique to find out the resource allocation policy maximizing the long-term throughput while satisfying the QoS requirements in the B5G systems. Key ingredient of the proposed technique is to reduce the action space by eliminating undesirable actions that cannot satisfy the QoS requirements. Numerical results demonstrate that the proposed technique is effective in maximizing the long-term throughput and handling the coexistence of use cases in the B5G environments.

Two-timescale hierarchical DRL for resource allocation in UAV-assisted 6G edge networks

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

In this chapter, we present a comprehensive end-to-end (E2E) network slicing framework for fifth-generation (5G) wireless and core networks and provide our vision on network slicing for beyond 5G (B5G) networks using artificial intelligence (AI) technologies. We first give an introduction on the 5G communication networks by specifying distinct 5G networking characteristics. As the key features of 5G pose challenges on quality-of-service (QoS)-oriented resource provisioning, a new concept of network slicing is presented, based on software-defined networking (SDN) and network function virtualization (NFV) technologies, to enhance the overall resource utilization while guaranteeing QoS isolation among services. We focus on developing specific network slicing solutions for both 5G wireless networks and core networks. In addition, the evolution of B5G networks is discussed in terms of a potential networking paradigm shift with new service key performance indicators (KPIs). To satisfy B5G service requirements, a framework of AI-assisted network slicing lifecycle is developed to automate the slice creation with reduced slice management complexity. A case study is presented to demonstrate the effectiveness of the proposed E2E network slicing solutions.Keywords5G communication networksNetwork slicingWireless networksRadio resource slicingCore networksComputing and transmission resource slicingBeyond 5G networksAI-assisted network slicingQoSE2E delay

Data offloading in mobile edge computing (MEC) allows the low power IoT devices in the edge to optionally offload power-consuming computation tasks to MEC servers. In this paper, we consider a novel backscatter-aided hybrid data offloading scheme to further reduce the power consumption in data transmission. In particular, each device has a dual-mode radio that can offload data via either the conventional active RF communications or the passive backscatter communications with extreme low power consumption. The flexibility in the radio mode switching makes it more complicated to design the optimal offloading strategy, especially in a dynamic network with time-varying workload and energy supply at each device. Hence, we propose the deep reinforcement learning (DRL) framework to handle huge state space under uncertain network state information. By a simple quantization scheme, we design the learning policy in the Double Deep Q-Network (DDQN) framework, which is shown to have better stability and convergence properties. The numerical results demonstrate that the proposed DRL approach can learn and converge to the maximal energy efficiency compared with other baseline approaches.

Effective ways to optimise real-time pump scheduling to maximise energy efficiency are being sought to meet the challenges in the energy market. However, the considerable number of evaluations of popular optimisation methods based on metaheuristics cause significant delays for real-time pump scheduling, and the simplification of traditional deterministic methods may introduce bias towards the optimal solutions. To address these limitations, an exploration-enhanced deep reinforcement learning (DRL) framework is proposed to address real-time pump scheduling problems in water distribution systems. The experimental results indicate that E-PPO can learn suboptimal scheduling policies for various demand distributions and can control the application time to 0.42 s by transferring the online computation-intensive optimisation task offline. Furthermore, a form of penalty of the tank level was found that can reduce energy costs by up to 11.14% without sacrificing the water level in the long term. Following the DRL framework, the proposed method makes it possible to schedule pumps in a more agile way as a timely response to changing water demand while still controlling the energy cost and level of tanks.

In coupled power distribution and transportation (CPT) system, a joint scheduling framework for mobile energy storage systems (MESSs) and Volt/VAR control (VVC) ensures reliable power distribution grid operations while supporting electric vehicle loads at electric vehicle charging stations (EVCSs). However, conventional model-based optimization methods for MESS scheduling and VVC may yield suboptimal solutions and greater computation times because of MESS operation and VVC in uncertain environment of CPT systems. To resolve this issue, this study proposes a model-free deep reinforcement learning (DRL) framework. In this framework, smart inverters of MESSs and solar photovoltaic (PV) systems cooperate to minimize the real power loss and mitigate the violations of both MESSs’ state of charge (SOC) and voltage in the power distribution network, while MESSs travel via the transportation network to satisfy EV loads at EVCSs. A MESS routing algorithm based on Dijkstra’s algorithm is developed to determine the optimal destinations of the MESSs. In addition, two safety modules are developed to ensure that neither SOC nor voltage violations occur by adjusting real and/or reactive power of MESSs and PV systems during the training process. The developed MESS routing algorithm and safety modules are integrated into the proposed DRL framework, wherein the DRL agent performs the desired MESS scheduling and VVC through safe exploration during the training procedure. The proposed approach is tested in coupled IEEE 33-bus power distribution and 15-node transportation systems and coupled IEEE 57-bus power distribution and 42-node transportation systems. Numerical examples demonstrate the advantages of the proposed approach in terms of training convergence, real power loss, and SOC/voltage violation.

With the rapid development of smart grid and renewable energy, demand response mechanism has become a key means to balance microgrid supply and demand and improve energy efficiency. As the traditional power demand response algorithm is difficult to deal with the uncertainty of power demand and the adverse effects caused by customers, this paper proposes a power demand response algorithm based on deep reinforcement learning. Firstly, we study and design a price -based layered electricity energy demand response model. Then, the dynamic optimal pricing decision of power trading market is described as Markov decision process, and the learning mechanism under the framework of deep reinforcement learning is expounded. Finally, a solution algorithm based on deep reinforcement learning is designed. The simulation results show that the proposed algorithm can adjust the electricity price adaptively according to the load demand, which can reduce the cost of users by about 17% and reduce the peak load demand.

Vascular interventional surgery has the advantages of small trauma, rapid postoperative recovery, and a high safety factor. It is widely used in the treatment of various cardiovascular diseases. In order to reduce the X-ray radiation exposure to doctors, the existing vascular interventional surgery robot is mainly operated by human teleoperation. Currently, how to achieve autonomously intelligent guidewire feeding has become an attractive issue for researchers. This paper proposes an autonomous wire feeding method in interventional surgery based on the deep reinforcement learning framework SAC (Soft Actor-Critic). We adopt the target detection algorithm YOLOv5s to obtain the guidewire tip, which is used for state update of the SAC agent. The guidewire feeding experiments on the vessel phantom verified that the deep reinforcement learning SAC method could well complete the guidewire feeding task in interventional surgery. Further research will utilize the combination of guidewire and guide catheter to solve more complex surgical tasks. The experiment results show that guidewire end detection accuracy can reach at least 92.9% and the guidewire feeding accuracy is less than 5mm.

To alleviate communication pressure and terminal resource constraints in mobile edge computing (MEC) networks, this paper proposes a resource allocation optimization method for MEC systems that integrates data compression technology and non-orthogonal multiple access technology. This method considers practical constraints such as terminal device battery capacity and computational resource limitations. By jointly optimizing computational resource allocation, task offloading strategies, and data compression ratios, it constructs an optimization model aimed at minimizing the total task processing latency. Addressing the challenges stemming from the non-convex nature of the problem and the dynamic variations in channel conditions and task requirements, this paper proposes a softmax deep double deterministic policy gradient algorithm, where softmax operator function mitigates both overestimation and underestimation biases inherent in traditional reinforcement learning frameworks, enhancing convergence performance. Utilizing a deep reinforcement learning framework, the algorithm achieves joint decision-making optimization for computational resources, task offloading, and compression ratios, thereby minimizing the total task processing latency while satisfying transmit power and computational resource constraints. Simulation results demonstrate that the proposed scheme exhibits significant advantages over benchmark algorithms in terms of convergence speed and task processing latency.

Effective zero-touch management and orchestration (ZSM&O) is crucial for scaling network slicing, particularly transitioning toward Beyond 5G (B5G) and 6G networks. This paper empirically validates the network slicing framework developed under the European Union Horizon 2020 MonB5G project. Building on three years of academia-industry collaboration, MonB5G introduces a flexible slicing model featuring umbrella slices that orchestrate modular, specialized slices across multi-domain environments to address next-generation service demands. For the first time, we evaluate its practicality in a 5G cloud-native testbed through a virtual reality (VR) streaming use case, supported by solutions such as federated learning-based CPU forecasting, anomaly detection, and deep reinforcement learning (DRL) for radio access network (RAN) optimization. The paper offers insights from technically demanding experimental tests and highlights challenges and development paths for managing next-generation mobile networks.

The integration of mobile edge computing (MEC) and network slicing can provide low-latency and customized services. In such integrated wireless networks, we propose a pricing-driven joint MEC server orchestration and network slice deployment scheme (PD-JSOSD), jointly solving the pricing, MEC server orchestration and network slicing deployment issues. We divide the system into an infrastructure provider layer (IPL), network planning layer (NPL) and resource allocation layer (RAL), and a three-stage Stackelberg game is proposed to describe their relationships. To obtain the Stackelberg equalization, we propose a three-layer deep reinforcement learning (DRL) algorithm. Specifically, the dueling double deep Q-network (D3QN) is used in the IPL, and the DRL with branching dueling Q-network (BDQ) is used in the NPL and the RAL to cope with the large-scale discrete action spaces. Moreover, we propose an innovative illegal action modification algorithm to improve the convergence of the BDQ. Simulations verify the convergence of the three-layer DRL and the superiority of modified-BDQ in dealing with large-scale action spaces, where modified-BDQ can improve the convergence by 21.9% and 28.3%. Furthermore, compared with the benchmark algorithms, JSOSD in the NPL and the RAL can improve system utility by up to 52.1%, proving the superiority of the server orchestration and slice deployment scheme.

We present a hierarchical deep reinforcement learning (DRL) framework with prominent sampling efficiency and sim-to-real transfer ability for fast and safe navigation: the low-level DRL policy enables the robot to move towards the target position and keep a safe distance to obstacles simultaneously; the high-level DRL policy is supplemented to further enhance the navigation safety. We select a waypoint located on the path from the robot to the ultimate goal as the sub-goal to reduce the state space and avoid sparse reward. Moreover, the path is generated based on either a local or a global map, which can significantly improve the sampling efficiency, safety, and generalization ability of the proposed DRL framework. Additionally, a target-directed representation for the action space can be derived based on the sub-goal to improve the motion efficiency and reduce the action space. In order to demonstrate the eminent sampling efficiency, motion performance, obstacle avoidance, and generalization ability of the proposed framework, we implement sufficient comparisons with the non-learning navigation methods and DRL-based baselines, with videos, data, code, and other supplemental material shown on our website<xref ref-type="fn" rid="fn1">¹</xref><fn id="fn1"><label>¹</label> https://github.com/zw199502/RL <inline-formula><tex-math notation="LaTeX">$\_$</tex-math></inline-formula> navigation </fn>.