5G Network Slicing: Methods to Support Blockchain and Reinforcement Learning

Introduction Reinforcement learning formalizes the concept of learning from interactions.1 Broadly, reinforcement learning focuses on a setting in which an agent (decision maker) sequentially interacts with an environment that is partially unknown to them. At each stage, the agent takes an action and receives a reward. The objective of the agent is to maximize rewards accumulated in the long run. There are many situations in health care where decisions are made sequentially for which reinforcement learning approaches could prove useful for decision making. Throughout this article, we consider treatment prescription as an archetypical example to connect reinforcement learning concepts to a health care setting. In this setting, the care provider, the prescribed treatment, and the patients can be viewed as the agent, the action, and the environment, respectively, as depicted in Figure 1.Figure 1: Sequential treatment of AKI or CKD complications modeled as a reinforcement learning problem.Background In this section, with the objective of making reinforcement learning literature more accessible to a clinical audience, we briefly introduce related fundamental concepts and approaches. We refer the interested reader to Sutton and Barto1 for a comprehensive introduction to reinforcement learning. Markov Decision Processes Markov decision processes (MDPs) are a formalism of the sequential decision-making problem that has been central to the theoretical and practical advancements of reinforcement learning. In each stage of an MDP, the agent observes the state of the environment and takes an action, which, in turn, results in a change of the state. This change of state is assumed to be probabilistic with the next state being determined only by the preceding state, the chosen action, and the transition probability. The agent also receives a reward that is a function of the taken action, the preceding state, and the subsequent state. In an MDP, the objective of the agent is to maximize the return defined as the reward accumulated over a time horizon. In some applications, it is common to consider the horizon to be infinite, in which case the future rewards are discounted by a factor smaller than one. The selection of action by the agent on the basis of the observed state is known as the policy. More formally, a policy is a probabilistic mapping from states to each possible action. Because the policy and the reward are a function of the state, it is critical to estimate the utility of being in a certain state. More specifically, the value function is defined as the expected return starting from a given state under the chosen policy. Under this formalism, the objective of the agent is to find the optimal policy that maximizes the value function for all states. Reinforcement Learning Methods Action-value methods are a class of reinforcement learning methods in which the actions are chosen on the basis of the estimation of their long-term value. A prominent example of an action-value method is Q-learning in which the agent iteratively takes actions with the highest estimated values and updates the action-state value function on the basis of new observations. Policy gradient methods are another class of reinforcement learning methods that seek to optimize the policy directly instead of choosing actions on the basis of their respective estimated value. Such methods could be advantageous in health care applications that entail a large number of possible actions, e.g., when recommending a wide range of drug dosages or treatment options. Clinical Applications Reinforcement learning frameworks and methods are broadly applicable to clinical settings in which decisions are made sequentially. A prominent clinical application of reinforcement learning is for treatment recommendation, which has been studied across a variety of diseases and treatments including radiation and chemotherapy for cancer, brain stimulation for epilepsy, and treatment strategies for sepsis.2–5 In such treatment recommendation settings, a policy is commonly known as a dynamic treatment regime. There are various other clinical applications of reinforcement learning including diagnosis, medical imaging, and decision support tools (see refs. 2–5 and the references therein). Reinforcement Learning in Nephrology Although there have been recent applications of machine learning in nephrology,6,7 to the best of the authors' knowledge, the application of reinforcement learning to nephrology has been primarily limited to optimizing the erythropoietin dosage in hemodialysis patients.8,9 However, there are other settings where reinforcement learning has the potential to improve patient care in nephrology. For example, reinforcement learning methods can be adopted in the treatment of the complications of AKI or CKD (Figure 1). In this problem, the state models the conditions of the patient (e.g., vital signs, laboratory test results including urine and blood tests, and urine output measurements). The action refers to the treatment options (e.g., the dosage of medications such as sodium polystyrene sulfonate, and hemodialysis). The reward models the improvement in patient conditions. Similarly, reinforcement learning can help automate and optimize the dosage of immunosuppressive drugs in kidney transplants. Challenges and Opportunities Despite the success of reinforcement learning in several simplified clinical settings, their large-scale application to patient care faces several open challenges. The complexity of human biology complicates modeling clinical decision making as a reinforcement learning problem. The state space in such settings is often enormous, which could make a purely computational approach infeasible. Moreover, modeling all potential objectives a priori as a reward function may not be feasible. To overcome these challenges and realize the potential of reinforcement learning, clinical insight can play a pivotal role. More specifically, restricting the state space to only include highly relevant clinical variables could greatly reduce the computational complexity. Furthermore, using inverse reinforcement learning,2 relevant reward functions can be learned from retrospective studies assuming the optimality of clinical decisions. Another critical challenge is addressing moral and ethical concerns. It is imperative to ensure that reinforcement learning methods do not cause harm to the patient. To this end, there exists a need for a thorough validation of such methods before their use in patient care. Hence, there is a need to go beyond retrospective studies that have been used for the proof of concept of most existing reinforcement learning methods in health care applications.2,3 The lessons learned from the success of reinforcement learning in other application areas (e.g., self-driving cars) can help navigate the path to realizing its potential in health care. Accessible open-source simulation environments that enable researchers to compare various approaches are essential to the field of reinforcement learning. OpenAI Gym is currently the leading toolkit containing a wide range of simulated environments, e.g., surgical robotics.10 The development of high-quality and reliable simulation environments for nephrology and other health care applications can facilitate the development and validation of reinforcement learning methods beyond limited retrospective studies. The adoption of methods validated in such simulation environments in actual clinical settings will require clinicians' oversight. Similar to how self-driving cars require a human driver to ensure collision avoidance, clinicians' oversight is critical to ensure the safety of the patients, especially in the early stages of the adoption of reinforcement learning methods. The data from clinicians' decisions (e.g., overruling the automated treatment recommendation) can be used to improve the reliability of autonomous systems over time and reduce the burden of clinicians' oversight.

The widespread deployment of smart meters and communication technologies brings opportunities to improve the adaptability and flexibility of future manufacturing systems under growing complexity and uncertainty. As a result, data-driven approaches, especially Reinforcement Learning (RL) methods, are gaining wide attention in recent years. Although in the current literature, RL approaches show superiority over traditional methods in many applications, a comprehensive review and comparison of different RL methods and their use in discrete manufacturing system control under changing environments have not yet been established. This paper first provides a literature review of RL algorithms for decision-making in discrete manufacturing systems, and then systematically discusses the underlying mechanisms of four most commonly used RL algorithms. In addition, the performance of these RL algorithms is compared by solving a production control problem, which aims to maximize the profit under time-varying production costs. The comparison results show that the single-agent RL methods with discrete action space can provide satisfactory results in small-scale systems, while multi-agent RL algorithms are more suitable to solve problems in complex, large-scale systems to remit the curse of dimensionality. This research sheds light on the impacts of RL method selection on the search for high-quality solutions in manufacturing systems, which is envisioned to drive future research on RL-supported manufacturing systems.

Reinforcement learning (RL) has shown promising performance in autonomous driving applications in recent years. The early end-to-end RL method is usually unexplainable and fails to generate stable actions, while the hierarchical RL (HRL) method can tackle the above issues by dividing complex problems into multiple sub-tasks. Prior HRL works either select discrete driving behaviors with continuous control commands, or generate expected goals for the low-level controller. However, they typically have strong scenario dependence or fail to generate goals with good quality. To address the above challenges, we propose a Continuous-Lattice Hierarchical RL (Cola-HRL) method for autonomous driving tasks to make high-quality decisions in various scenarios. We utilize the continuous-lattice module to generate reasonable goals, ensuring temporal and spatial reachability. Then, we train and evaluate our method under different traffic scenarios based on real-world High Definition maps. Experimental results show our method can handle multiple scenarios. In addition, our method also demonstrates better performance and driving behaviors compared to existing RL methods.

Adaptive bitrate (ABR) streaming algorithms play an important role in ensuring a high Quality of Experience (QoE) for the consumer. However, a lot of ABR algorithms tend to be too ad hoc. In response, methods based on a Markov Decision Process (MDP) offer more intelligent models. In particular, Reinforcement Learning (RL) methods typically do so via QoE metrics. However, RL methods are plagued by high complexity and long convergence times due to their model-free nature. This paper proposes qMDP, which is an RL method with an MDP partially modeled by an M/D/1/K queue. Our study shows that qMDP results in higher QoE and faster convergence compared to a QoE-only model-free version.

Newly developed fifth- and sixth-generation (5G and 6G) networks are very secure, reliable, less latent, and versatile. These networks provide various services, like Massive Machine Type Communication (mMTC), Ultra-Reliable Low Latency Communication (URLLC), and Enhance Mobile Broadband (eMBB). Network slicing (NS) is important in both 5G and 6G networks as it enables the provision of the above services in a single physical network. This paper proposes an Optimal Hybrid Quantum-Classical Convolutional Neural Network for efficient 5G Network Slicing (HQCNN-5GNS). Initially, the input data are gathered from Unicauca IP flow version 2 and 5G network slicing dataset. The collected data are fed to the pre-processing phase using a Pseudo-linear Maximum Correntropy Kalman Filter (PMCKF) to remove duplicate data and replace the missing values from the dataset. The pre-processing information is provided to the Hybrid Quantum-Classical Convolutional Neural Network (HQCNN) to classify the network slicing into mMTC, eMBB, and URLLC. In general, HQCNN does not adopt any optimization methods to determine the optimal parameters to guarantee accurate categorization. Therefore, the Quantum Beetle Antennae Search Algorithm (QBASOA) is employed to optimize the weight parameter of HQCNN. The experimental outcomes show that the proposed HQCNN-5GNS attains 17.25%, 15.65%, and 19.55% higher accuracy than the existing approaches: Harris Hawks optimization depends on hybrid deep learning for effectual network slicing on 5G network (HHO-HDL-5GNS), Highly accurate with dependable wireless network slicing in the 5G network: a hybrid deep learning model (WNS-HDL-5GNS), Deep reinforcement learning-basis network slicing for beyond 5G (DRL-5GNS), respectively.

The reinforcement learning method for occupant behavior in building control: A review

In the wake of the vast population of smart device users worldwide, mobile health (mHealth) technologies are hopeful to generate positive and wide influence on people's health. They are able to provide flexible, affordable and portable health guides to devise users. Current online decision-making methods for mHealth assume that the users are completely heterogeneous. They share no information among users and learn a separate policy for each user. However, data for each user is very limited in size to support the separate online learning, leading to unstable policies that contain lots of variances. Besides, we find the truth that a user may be similar with some, but not all, users, and connected users tend to have similar behaviors. In this paper, we propose a network cohesion constrained (actor-critic) Reinforcement Learning (RL) method for mHealth. The goal is to explore how to share information among similar users to better convert the limited user information into sharper learned policies. To the best of our knowledge, this is the first online actor-critic RL for mHealth and first network cohesion constrained (actor-critic) RL method in all applications. The network cohesion is important to derive effective policies. We come up with a novel method to learn the network by using the warm start trajectory, which directly reflects the users' property. The optimization of our model is difficult and very different from the general supervised learning due to the indirect observation of values. As a contribution, we propose two algorithms for the proposed online RLs. Apart from mHealth, the proposed methods can be easily applied or adapted to other health-related tasks. Extensive experiment results on the HeartSteps dataset demonstrates that in a variety of parameter settings, the proposed two methods obtain obvious improvements over the state-of-the-art methods.

In any complex or large scale sequential decision making problem, there is a crucial need to use function approximation to represent the relevant functions such as the value function or the policy. The Dynamic Programming (DP) and Reinforcement Learning (RL) methods introduced in previous chapters make the implicit assumption that the value function can be perfectly represented (i.e. kept in memory), for example by using a look-up table (with a finite number of entries) assigning a value to all possible states (assumed to be finite) of the system. Those methods are called exact because they provide an exact computation of the optimal solution of the considered problem (or at least, enable the computations to converge to this optimal solution). However, such methods often apply to toy problems only, since in most interesting applications, the number of possible states is so large (and possibly infinite if we consider continuous spaces) that a perfect representation of the function at all states is impossible. It becomes necessary to approximate the function by using a moderate number of coefficients (which can be stored in a computer), and therefore extend the range of DP and RL to methods using such approximate representations. These approximate methods combine DP and RL methods with function approximation tools.

In the fifth-generation of mobile communications, network slicing is used to provide an optimal network for various services as a slice. In this paper, we propose a radio access network (RAN) slicing method that flexibly allocates RAN resources using deep reinforcement learning (DRL). In RANs, the number of slices controlled by a base station fluctuates in terms of user ingress and egress from the base station coverage area and service switching on the respective sets of user equipment. Therefore, when resource allocation depends on the number of slices, resources cannot be allocated when the number of slices changes. We consider a method that makes optimal-resource allocation independent of the number of slices. Resource allocation is optimized using DRL, which learns the best action for a state through trial and error. To achieve independence from the number of slices, we show a design for a model that manages resources on a one-slice-by-one-agent basis using Ape-X, which is a DRL method. In Ape-X, because agents can be employed in parallel, models that learn various environments can be generated through trial and error of multiple environments. In addition, we design a model that satisfies the slicing requirements without over-allocating resources. Based on this design, it is possible to optimally allocate resources independently of the number of slices by changing the number of agents. In the evaluation, we test multiple scenarios and show that the mean satisfaction of the slice requirements is approximately 97%.

In this chapter, we present a comprehensive end-to-end (E2E) network slicing framework for fifth-generation (5G) wireless and core networks and provide our vision on network slicing for beyond 5G (B5G) networks using artificial intelligence (AI) technologies. We first give an introduction on the 5G communication networks by specifying distinct 5G networking characteristics. As the key features of 5G pose challenges on quality-of-service (QoS)-oriented resource provisioning, a new concept of network slicing is presented, based on software-defined networking (SDN) and network function virtualization (NFV) technologies, to enhance the overall resource utilization while guaranteeing QoS isolation among services. We focus on developing specific network slicing solutions for both 5G wireless networks and core networks. In addition, the evolution of B5G networks is discussed in terms of a potential networking paradigm shift with new service key performance indicators (KPIs). To satisfy B5G service requirements, a framework of AI-assisted network slicing lifecycle is developed to automate the slice creation with reduced slice management complexity. A case study is presented to demonstrate the effectiveness of the proposed E2E network slicing solutions.Keywords5G communication networksNetwork slicingWireless networksRadio resource slicingCore networksComputing and transmission resource slicingBeyond 5G networksAI-assisted network slicingQoSE2E delay

Deep reinforcement learning based energy management for a hybrid electric vehicle

The field of autonomous driving has seen increasing proposed use of machine learning methodologies. However, there are still challenges in applying such methods since autonomous driving involves complex and dynamic interactions with the environment. Supervised learning algorithms such as imitation learning can work in environments represented in the training data set, however, it is impractical or cost-prohibitive to collect data for all possible environments. Reinforcement learning methods can train the agent through trial and error, but it may still fail in new environments. To overcome these shortcomings, we thus propose a meta reinforcement learning (MRL) method to improve the agent's generalization capabilities to new environments. The method is applied to automated lane-changing maneuvers at different traffic congestion levels. Specifically, we train the model at light to moderate traffic conditions under a reinforcement learning framework, and then test it at heavy traffic conditions that are never encountered during training. For performance evaluation, we use both collision rate and success rate of the lane-change maneuvers to quantify the safety and effectiveness of the proposed model. A pretrained model is established as a benchmark, which uses the same network structure and training tasks as our proposed model for fair comparison. Simulation results show that the proposed method achieves an overall success rate up to 20% higher than the benchmark model when it is generalized to the new environment of heavy traffic density. The collision rate is also reduced by up to 18% compared with the benchmark model. Furthermore, the proposed model shows more stable and efficient adaptation capabilities and can achieve 100% successful rate and 0% collision rate with only a few steps of gradient update adaptation.

Reinforcement Learning (RL) methods are very hopeful because they can learn useful behavior based on rewards from environment by trial and error. This paper tackles more difficult problems than the ones tackled by many ordinary RL methods: RL in POMDP (Partially Observable Markov Decision Process) environments with multiple rewards.

Aiming at Distributed Permutation Flow-shop Scheduling Problems (DPFSPs), this study took the minimization of the maximum completion time of the workpieces to be processed in all production tasks as the goal, and took the multi-agent Reinforcement Learning (RL) method as the main frame of the solution model, then, combining with the NASH equilibrium theory and the RL method, it proposed a NASH Q-Learning algorithm for Distributed Flow-shop Scheduling Problem (DFSP) based on Mean Field (MF). In the RL part, this study designed a two-layer online learning mode in which the sample collection and the training improvement proceed alternately, the outer layer collects samples, when the collected samples meet the requirement of batch size, it enters to the inner layer loop, which uses the Q-learning model-free batch processing mode to proceed, and adopts neural network to approximate the value function to adapt to large-scale problems. By comparing the Average Relative Percentage Deviation (ARPD) index of the benchmark test questions, the calculation results of the proposed algorithm outperformed other similar algorithms, which proved the feasibility and efficiency of the proposed algorithm.

This paper presents a novel approach for acquiring dynamic whole-body movements on humanoid robots focused on learning a control policy for the center of mass (CoM). In our approach, we combine both a model-based CoM controller and a model-free reinforcement learning (RL) method to acquire dynamic whole-body movements in humanoid robots. (i) To cope with high dimensionality, we use a model-based CoM controller as a basic controller that derives joint angular velocities from the desired CoM velocity. The balancing issue can also be considered in the controller. (ii) The RL method is used to acquire a controller that generates the desired CoM velocity based on the current state. To demonstrate the effectiveness of our approach, we apply it to a ball-punching task on a simulated humanoid robot model. The acquired whole-body punching movement was also demonstrated on Fujitsu's Hoap-2 humanoid robot.