Intelligent QoS-Aware Slice Resource Allocation With User Association Parameterization for Beyond 5G O-RAN-Based Architecture Using DRL

Efficient spectrum and resource allocation remains a fundamental challenge in fifth generation (5G) wireless networks due to heterogeneous traffic requirements, spectrum scarcity, and dynamic interference conditions. Conventional scheduling and optimization techniques struggle to meet the stringent quality of service (QoS) demands of enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (URLLC), and massive machine-type communication (mMTC). This paper presents an improved deep reinforcement learning (DRL)-based optimization framework for adaptive spectrum and resource allocation in 5G networks. The proposed model integrates Deep Q-Network (DQN), Double DQN (DDQN), and DDQN with Prioritized Experience Replay (PER) to enhance convergence speed, stability, and scalability. A realistic simulation environment combining ray-traced channel data and packet-level traffic modeling was developed to evaluate system performance. Results demonstrate that the proposed DRL framework achieves up to 7% throughput improvement over proportional fair scheduling, reduces average latency by 10–12%, and lowers URLLC violation rates by approximately 15% under high-interference and heterogeneous traffic scenarios. These findings confirm the feasibility of DRL as a practical and scalable solution for real-time 5G resource management.

In this paper, a puncturing-based resource slicing technique is proposed to schedule the Ultra-Reliable Low Latency Communication (URLLC) transmission on top of the pre-scheduled enhanced Mobile Broadband (eMBB) traffic in a Reconfigurable Intelligent Surface (RIS)-assisted down-link 6G system. The objective is to minimize the eMBB rate loss while satisfying the Quality of Service (QoS) of both eMBB and URLLC services. We use an alternating optimization to solve the slot-based eMBB allocation problem and a Deep Reinforcement Learning (DRL)-based Dueling Deep Q Networks (D-DQN) algorithm to solve the mini-slot-based URLLC allocation problem. The performance of the proposed solution is compared with other state-of-the-art techniques, including the Deep Q Network (DQN) with and without using the RIS.

To decrease the impact of Partially Observable MDP on deep reinforcement learning performance of intersection signal control, A deep reinforcement learning is proposed in this paper with utilizing the real-time GPS data as well as learning the control of the traffic lights in single intersection. We integrate deep reinforcement learning network (DRQN) with recurrent neural network (RNN) and apply deep network, experience pool and greedy strategy in deep reinforcement learning strategy. It solves the problem of overestimation of target Q value and insufficient long-term experience learning in the standard reinforcement learning of traffic signal control. The comparison of performance was made between the proposed method and standard Deep Q-Network (DQN) on the partial observation of traffic situations. The experimental results show that both DQN and DRQN methods can adjust their traffic signal timing control strategies according to the specific traffic conditions as well as calculating a lower average delay time of vehicle than that of fixed-time control. Besides, the simulation effect of DRQN learning method is better than that of DQN learning method in different probe vehicle proportion environment.

Network slicing is a promising technique for wireless service providers to support enhanced mobile broadband (eMBB) and ultra-reliable low-latency communication (URLLC) services in a shared radio access network (RAN) infrastructure. In this paper, we apply numerology, mini-slot based transmission, and punctured scheduling techniques to support eMBB and URLLC network slices. For efficient allocation of radio resources (e.g., physical resource blocks, transmit power) to the users, we formulate RAN slicing problem as a multi-timescale problem. To solve this problem and address the dynamics of the traffic, we propose a hierarchical deep learning framework. Specifically, in each long time slot, the service provider employs a deep reinforcement learning (DRL) algorithm to determine the slice configuration parameters. The eMBB and URLLC schedulers use their own attention-based deep neural network (DNN) algorithm to allocate radio resources to their corresponding users in each short and mini time slot, respectively. Simulation results show that the proposed framework can achieve a higher aggregate throughput and a higher service level agreement (SLA) satisfaction ratio compared to some other RAN slicing approaches, including the resource proportional placement algorithm, decomposition and relaxation based resource allocation algorithm, and distributed bandwidth optimization algorithm.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

In this paper, a resource allocation algorithm based on deep reinforcement learning (DRL) is proposed to solve the problem of co-existing enhanced mobile broadband (eMBB) slices and ultrareliable low latency communication (URLLC) slices based on Non-orthogonal Multiple Access (NOMA) in downlink network scenarios. Double Deep Q network (DDQN) is designed to output subcarrier and power allocation simultaneously. In addition, expert data is added in the training process to accelerate network convergence, and our goal is to optimize the spectral efficiency of the system. Simulation results show that compared with the baseline based on heuristic joint user pairing and power allocation algorithm [1] and Orthogonal Multiple Access (OMA), proposed algorithm can achieve higher spectral efficiency and ensure isolation between slices.KeywordsDeep reinforcement learningNetwork slicingResource allocation

To optimally cover users in millimeter-Wave (mmWave) networks, clustering is needed to identify the number and direction of beams. The mobility of users motivates the need for an online clustering scheme to maintain up-to-date beams towards those clusters. Furthermore, mobility of users leads to varying patterns of clusters (i.e., users move from the coverage of one beam to another), causing dynamic traffic load per beam. As such, efficient radio resource allocation and beam management is needed to address the dynamicity that arises from mobility of users and their traffic. In this paper, we consider the coexistence of Ultra-Reliable Low-Latency Communication (URLLC) and enhanced Mobile BroadBand (eMBB) users in 5G mmWave networks and propose a Quality-of-Service (QoS) aware clustering and resource allocation scheme. Specifically, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is used for online clustering of users and the selection of the number of beams. In addition, Long Short Term Memory (LSTM)-based Deep Reinforcement Learning (DRL) scheme is used for resource block allocation. The performance of the proposed scheme is compared to a baseline that uses K-means and priority-based proportional fairness for clustering and resource allocation, respectively. Our simulation results show that the proposed scheme outperforms the baseline algorithm in terms of latency, reliability, and rate of URLLC users as well as rate of eMBB users.

Low-Earth orbit (LEO) satellites have limited on-board resources, user terminals are unevenly distributed in the constantly changing coverage area, and the service requirements vary significantly. It is urgent to optimize resource allocation under the constraint of limited satellite spectrum resources and ensure the fairness of service admission control. Therefore, we propose an intelligent hierarchical admission control (IHAC) strategy based on deep reinforcement learning (DRL). This strategy combines the deep deterministic policy gradient (DDPG) and the deep Q network (DQN) intelligent algorithm to construct upper and lower hierarchical resource allocation and admission control frameworks. The upper controller considers the state features of each ground zone and satellite resources from a global perspective, and determines the beam resource allocation ratio of each ground zone. The lower controller formulates the admission control policy based on the decision of the upper controller and the detailed information of the users' services. At the same time, a designed reward and punishment mechanism is used to optimize the decisions of the upper and lower controllers. The fairness of users' services admissions in each ground zone is achieved as far as possible while ensuring the reasonable allocation of beam resources among zones. Finally, online decision-making and offline learning were combined, so that the controller could make full use of a large number of historical data to learn and generate intelligent strategies with stronger adaptive ability while interacting with the network environment in real time. A large number of simulation results show that IHAC has better performance in terms of a successful service admission rate, service drop rate, and fair resource allocation. Among them, the number of accepted services increased by 20.36% on average, the packet loss rate decreased by 17.56% on average, and the resource fairness increased by 17.16% on average.

Background. Ultra-Reliable Low Latency Communication (URLLC) as a service offered by fifth and sixth generation (5G/6G) wireless systems is a technological response to the needs of various mission-critical applications that require reliable data transmission with low latency. These applications also include Intelligent Transportation Systems services, which, among other things, provide connectivity and autonomous vehicle control. The combination of high reliability and low latency requirements in URLLC usage scenarios creates a security problem for URLLC data transmission that cannot be solved using conventional complex cryptographic methods based on a secret key. The article discusses in detail the approach to using physical layer security mechanisms (physical layer security - PLS) as a powerful alternative to classical cryptographic security methods for URLLC, and also proposes the application of the wire-tap channel concept in URLLC with an analysis of the efficiency that can be achieved for physical layer security. Objective. The aim of the article is to provide an overview of information security solutions in URLLC usage scenarios, as well as to propose a constructive method for information protection for reliable data transmission with low latency without the use of cryptographic mechanisms based on a secret key. Methods. Theoretical research in the field of the branch channel concept was used to create solutions that allow data protection with information-theoretic stability in URLLC usage scenarios for providing IoT, connected car and autonomous driving services. Results. The article examines in detail the data security issues in ultra-reliable low latency communication (URLLC) as a service offered by fifth and sixth generation (5G/6G) wireless systems. It is determined that URLLC is a technological response to the needs of various critical applications that require reliable signal transmission with low latency, and among these applications are Intelligent Transportation Systems services, which, among other things, provide connectivity and autonomous vehicle control. It is shown that the combination of high reliability and low latency requirements in URLLC scenarios creates a security problem for URLLC data transmission that cannot be solved using conventional complex cryptographic methods based on a secret key. The feasibility of using physical layer security mechanisms (PLS) as a powerful alternative to classical cryptographic security methods for URLLC is substantiated. The approach to applying the concept of a wire-tap channel in URLLC is considered in detail, as well as the results that can be achieved for physical layer security, and the influence of code parameters for probabilistic cryptographic transformations in accordance with the concept of a wire-tap channel on PLS URLLC. Estimates of the effectiveness of PLS URLLC for finite block length codes are provided. Conclusions. An effective way to ensure data security for ultra-reliable low-latency physical layer link (PLS URLLC) of fifth-generation 5G wireless systems in the field of connected cars and vehicles of 4-5 levels of automation can be approaches based on the concept of a tapped channel ("wire-tap channel").

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

Ultra reliable low latency communication (URLLC) in vehicular networks is crucial for safety-related vehicular applications. Mini-slot with a short packet that carries only a few symbols is used to reduce the transmission time interval and enable quick scheduling for URLLC that requires extremely low latency. However, a single air interface transmission of URLLC packets may fail due to the high mobility of vehicles. Leveraging multi-connectivity technologies, the real-time reliability of URLLC can be greatly enhanced without relying on packet retransmission. In this paper, we propose a multi-connectivity URLLC downlink transmission scheme for vehicular networks, where the URLLC packet is duplicated and transmitted over multiple independent wireless links to improve packet reliability. Specifically, we design a multi-agent cooperative deep reinforcement learning algorithm, called transformer associated proximal policy optimization (TAPPO), to achieve real-time robust power allocation for multi-connectivity URLLC with imperfect channel state information (CSI). The transformer neural network architecture is employed to share the information among multiple links serving the same URLLC user and choose appropriate transmit powers, enabling cooperation to ensure reliability while minimizing inter-cell interference and energy consumption. Extensive simulation results validate the effectiveness of multi-connectivity packet duplication for URLLC and proposed TAPPO for power allocation.

Fifth-generation (5G) networks are already available in major urban areas and are expected to bring a major transformation to citizens’ lives. 5G services, such as enhanced mobile broadband (eMBB), ultra-reliable low latency communications (URLLC), and massive machine-type communications (mMTC), require a network infrastructure capable of supporting stringent requirements in terms of latency and bandwidth demands; as such, it must be highly dynamic and flexible. Network slicing is a key enabler technology that can provide dynamic and flexible characteristics to 5G network architecture. A network slice (NS) can be defined as a partition of network and IT resources, that is, network links and nodes capacity dedicated to a specific set of service demands. As a result, different NSs can coexist over the same physical infrastructure network and can be used to dynamically and flexibly deploy the aforementioned 5G services. However, to efficiently implement NSs with different requirements, communication service providers (CSPs) that own the physical infrastructure network must adopt sophisticated techniques for admission control and resource allocation of NSs. In this paper, we present a novel framework for admission control and resource allocation of 5G NSs in metro-core networks. Specifically, our framework is based on a deep reinforcement learning (DRL) algorithm called Advantage Actor Critic (A2C), which performs admission control, i.e. it is capable of learning which slice to admit based on the availability of the physical network resources. Then, given the diversity of requirements for each 5G service, we propose different resource allocation algorithms based on integer linear programming (ILP) and heuristics to treat each service accordingly. Results show that our proposed framework can increase the number of admitted NSs with respect to the case in which the admission control is disabled by improving the resource allocation performance.

With the rapid development of AI technology, personalized learning path planning is becoming an important research field. Dual system teaching models have great influence on the learning effect of students. Therefore, how to optimize the learning process of students through the personalized learning path is a great challenge. A Deep Reinforcement Learning (DRL) algorithm for personalized learning path planning is proposed. Based on deep Q-networks and reinforcement learning feedback mechanisms, the algorithm achieves adaptive optimization. The experimental results show that the algorithm based on the DRL is superior to the traditional method in learning performance, learning efficiency and satisfaction. Compared with traditional path planning algorithms based on rules and collaborative filtering, the proposed algorithm improves learning performance by 20%. Moreover, it has obvious advantages in learning motivation and efficiency. This paper provides a new way of thinking to intelligent education system, especially dual-system teaching model, and it is of great practical significance.

Machine learning offers advanced tools for efficient management of radio resources in modern wireless networks. In this study, we leverage a multi-agent deep reinforcement learning (DRL) approach, specifically the Parameterized Deep Q-Network (DQN), to address the challenging problem of power allocation and user association in massive multiple-input multiple-output (M-MIMO) communication networks. Our approach tackles a multi-objective optimization problem aiming to maximize network utility while meeting stringent quality of service requirements in M-MIMO networks. To address the non-convex and nonlinear nature of this problem, we introduce a novel multi-agent DQN framework. This framework defines a large action space, state space, and reward functions, enabling us to learn a near-optimal policy. Simulation results demonstrate the superiority of our Parameterized Deep DQN (PD-DQN) approach when compared to traditional DQN and RL methods. Specifically, we show that our approach outperforms traditional DQN methods in terms of convergence speed and final performance. Additionally, our approach shows 72.2% and 108.5% improvement over DQN methods and the RL method, respectively, in handling large-scale multi-agent problems in M-MIMO networks.

The growing number of complex and heterogeneous Internet of Things (IoT) applications has imposed a high demand for scarce communications and computing resources. To meet this stringent requirement, it is desirable to develop large-scale highly adaptive online resource allocation strategies to streamline existing network operations. Deep reinforcement learning (DRL), which combines the merits of reinforcement learning and deep learning, is capable of addressing complex decision-making tasks, thus enabling efficient online resource allocation. In this article, we present a DRL-based resource allocation framework. We begin a discussion on DRL basics and review its several recent applications. Then, we develop two new DRL algorithms that facilitate unlocking the potential of DRL and offer viable solutions to many more complex resource allocation problems. The first one tackles an optimization problem exposed to mixed (discrete and continuous) action spaces and bound by a number of highly non-linear quality-of-service (QoS) constraints. The second one extends the single-agent DRL to a more challenging multi-agent DRL by introducing a novel semi-distributed architecture. Finally, we discuss the challenges and future visions of applying DRL to real-world IoT networks.