EMBB and URLLC Service Multiplexing Based on Deep Reinforcement Learning in 5G and Beyond

Study on deep reinforcement learning techniques for building energy consumption forecasting

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

Improving the interpretability of deep reinforcement learning in urban drainage system operation

Reinforcement and deep reinforcement learning for wireless Internet of Things: A survey

SummaryOne desired aspect of microservice architecture is the ability to self‐adapt its own architecture and behavior in response to changes in the operational environment. To achieve the desired high levels of self‐adaptability, this research implements distributed microservice architecture model running a swarm cluster, as informed by the Monitor, Analyze, Plan, and Execute over a shared Knowledge (MAPE‐K) model. The proposed architecture employs multiadaptation agents supported by a centralized controller, which can observe the environment and execute a suitable adaptation action. The adaptation planning is managed by a deep recurrent Q‐learning network (DRQN). It is argued that such integration between DRQN and Markov decision process (MDP) agents in a MAPE‐K model offers distributed microservice architecture with self‐adaptability and high levels of availability and scalability. Integrating DRQN into the adaptation process improves the effectiveness of the adaptation and reduces any adaptation risks, including resource overprovisioning and thrashing. The performance of DRQN is evaluated against deep Q‐learning and policy gradient algorithms, including (1) a deep Q‐learning network (DQN), (2) a dueling DQN (DDQN), (3) a policy gradient neural network, and (4) deep deterministic policy gradient. The DRQN implementation in this paper manages to outperform the aforementioned algorithms in terms of total reward, less adaptation time, lower error rates, plus faster convergence and training time. We strongly believe that DRQN is more suitable for driving the adaptation in distributed services‐oriented architecture and offers better performance than other dynamic decision‐making algorithms.

Spectrum sharing has been proposed to effectively utilize the limited licensed spectrum resource and provide extra capacity for mobile network operator (MNO) who requires more spectrum in 5G network. In spectrum sharing, the initialized licensed spectrum of each MN O is divided into a reserved and shared spectrum. The reserved spectrum is for the private use of an MNO, while the shared spectrum of all MNOs forms a spectrum pool which will be dynamically used by MNO who requires an additional spectrum. However, the spectrum pool management problem significantly affects the efficiency of spectrum sharing among multi-MNO. In this paper, we formulate this problem into a non-linear programming problem that aims to maximize the average data rate of UEs, which subscribe to an MNO sending the spectrum request event, of all events from all MNOs. We propose an event-driven Deep Reinforcement Learning based spectrum sharing mechanism (EDRL-SSM) that utilizes a spectrum pool manager to effectively manage the spectrum pool to achieve long-term optimization of data rate performance of UEs. The spectrum pool manager intelligently allocates spectrum pool resource for each incoming spectrum request event. The proposed EDRL-SSM fully utilizes a DRL technique, Deep Deterministic Policy Gradient (DDPG), to deal with stochastic spectrum request event arrivals in the spectrum pool manager. The simulation results show that the proposed EDRL-SSM can significantly improve the data rate performance of UEs as compared to greedy spectrum pool allocation and without spectrum sharing under both identical and different initialized allocation of licensed spectrum scenarios.

The high cost of space mission operations has motivated several space agencies to prioritize the development of autonomous spacecraft command and control technologies. Deep reinforcement learning (DRL) techniques present one promising domain for the creation of autonomous agents for complex, multifaceted operations problems. This work examines the feasibility of adapting DRL-driven policy generation algorithms to problems in spacecraft decision-making, including strategies for framing spacecraft decision-making problems such as Markov decision processes, avenues for dimensionality reduction, and simplification using expert domain knowledge, sensitivity to hyperparameters, and robustness in the face of mismodeled environmental dynamics. In addition, consideration is given to ensuring the safety of these approaches by hybridizing them with correct-by-construction control techniques in a novel adaptation of shielded deep reinforcement learning. These strategies are demonstrated against a prototypical low-fidelity stationkeeping scenario and a high-fidelity attitude mode management scenario involving flight heritage attitude control and momentum management algorithms. DRL techniques are found to compare favorably to other black-box optimization tools or heuristic solutions for these problems and to require similar network sizes and training durations as widely used testing datasets in the deep learning community.

Ground penetrating radars (GPRs) carried by mobile platforms, such as vehicles and drones, have been applied in various applications, for instance, subsurface utility detection, structural health inspection, and autonomous driving. However, existing GPR systems are not able to operate autonomously and adaptively due to several challenges, including the lack of intelligence, uncertain and dynamic nature of sensing environments, and huge state and action spaces. To overcome these challenges, in this article, we propose an autonomous cognitive GPR (AC-GPR) enabled by a deep reinforcement learning (DRL) approach. Specifically, the operation of the proposed AC-GPR is first formulated as a sequential decision process. A novel reward function is developed for the DRL model by defining and combining two different types of entropy-based rewards resulting from object detection and recognition, respectively. A deep Q-learning network (DQN) is developed to address the extreme curse of dimensionality in the state space and learn a policy directing the actions of the AC-GPR. The AC-GPR is evaluated using software called GprMax by combining DRL with GPR modeling and simulation. Results show that our proposed DRL-based AC-GPR outperforms other GPR systems using different approaches in terms of detection accuracy and operating time.

In the domain of continuous control, deep reinforcement learning (DRL) demonstrates promising results. However, the dependence of DRL on deep neural networks (DNNs) results in the demand for extensive data and increased computational cost. To address this issue, a novel hybrid actor-critic reinforcement learning (RL) framework is introduced. The proposed framework integrates the broad learning system (BLS) with DNN, aiming to merge the strengths of both distinct architectural paradigms. Specifically, the critic network employs BLS for rapid value estimation via ridge regression, while the actor network retains the DNN structure to optimize policy gradients. This hybrid design is generalizable and can enhance existing actor-critic algorithms. To demonstrate its versatility, the proposed framework is integrated into three widely used actor-critic algorithms-deep deterministic policy gradient (DDPG), soft actor-critic (SAC), and twin delayed DDPG (TD3), resulting in BLS-augmented variants. The experimental results reveal that all BLS-enhanced versions surpass their original counterparts in terms of training efficiency and accuracy. These improvements highlight the suitability of the proposed framework for real-time control scenarios, where computational efficiency and rapid adaptation are critical.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

In a complex and changeable stock market, it is very important to design a trading agent that can benefit investors. In this paper, we propose two stock trading decision-making methods. First, we propose a nested reinforcement learning (Nested RL) method based on three deep reinforcement learning models (the Advantage Actor Critic, Deep Deterministic Policy Gradient, and Soft Actor Critic models) that adopts an integration strategy by nesting reinforcement learning on the basic decision-maker. Thus, this strategy can dynamically select agents according to the current situation to generate trading decisions made under different market environments. Second, to inherit the advantages of three basic decision-makers, we consider confidence and propose a weight random selection with confidence (WRSC) strategy. In this way, investors can gain more profits by integrating the advantages of all agents. All the algorithms are validated for the U.S., Japanese and British stocks and evaluated by different performance indicators. The experimental results show that the annualized return, cumulative return, and Sharpe ratio values of our ensemble strategy are higher than those of the baselines, which indicates that our nested RL and WRSC methods can assist investors in their portfolio management with more profits under the same level of investment risk.

The design and implementation of a deep reinforcement learning and quantum finance theory-inspired portfolio investment management system

Unmanned Aerial Vehicle (UAV) has emerged as a promising technology to provide wireless signals from air to the ground users in specific scenarios such as earthquakes, tsunamis and other disasters. The performance of the UAV is degraded when the signals are blocked by obstacles in dense urban scenarios. To address this issue and enhance the signal quality available to the ground users, Reconfigurable Intelligent Surface (RIS) has emerged as a new technological paradigm. It offers an intelligent configuration for the signal propagation environment by redirecting the signals to the users. In this article, we solve a non-convex optimization problem of RIS-assisted UAV network by jointly optimizing the RIS phase shift and 3D trajectory of UAV to maximize the energy efficiency of a rotatory-wing UAV. The considered optimization problem is solved using Deep Reinforcement Learning (DRL) based techniques in an on-line fashion to reduce the computational complexity. We leverage Twin-delayed Deep Deterministic Policy Gradient (TD3) to solve the problem by considering the UAV trajectory as a set of continuous actions. For comparison, we also use the Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG) and Double Deep Q-Network (DDQN) for continuous and discrete optimization of the UAV trajectory, respectively. Extensive simulations show that the TD3 outperforms all the considered DRL techniques with the highest energy efficiency and throughput, and the lowest propulsion energy.

The following interdisciplinary article presents a memetic algorithm with applying deep reinforcement learning (DRL) for solving practically oriented dual resource constrained flexible job shop scheduling problems (DRC-FJSSP). From research projects in industry, we recognize the need to consider flexible machines, flexible human workers, worker capabilities, setup and processing operations, material arrival times, complex job paths with parallel tasks for bill of material (BOM) manufacturing, sequence-dependent setup times and (partially) automated tasks in human-machine-collaboration. In recent years, there has been extensive research on metaheuristics and DRL techniques but focused on simple scheduling environments. However, there are few approaches combining metaheuristics and DRL to generate schedules more reliably and efficiently. In this paper, we first formulate a DRC-FJSSP to map complex industry requirements beyond traditional job shop models. Then we propose a scheduling framework integrating a discrete event simulation (DES) for schedule evaluation, considering parallel computing and multicriteria optimization. Here, a memetic algorithm is enriched with DRL to improve sequencing and assignment decisions. Through numerical experiments with real-world production data, we confirm that the framework generates feasible schedules efficiently and reliably for a balanced optimization of makespan (MS) and total tardiness (TT). Utilizing DRL instead of random metaheuristic operations leads to better results in fewer algorithm iterations and outperforms traditional approaches in such complex environments.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.