Adversarial Sybil attacks against Deep RL based drone trajectory planning

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Multi-agent quantum-inspired deep reinforcement learning for real-time distributed generation control of 100% renewable energy systems

In this chapter, the regulation of an Unmanned Aerial Vehicle (UAV) communication network is investigated in the presence of dynamic changes in the UAV lineup and user distribution. We target an optimal UAV control policy which is capable of identifying the upcoming change in the UAV lineup (quit or join-in) or user distribution, and proactively relocating the UAVs ahead of the change rather than passively dispatching the UAVs after the change. Specifically, a deep reinforcement learning (DRL)-based UAV control framework is developed to maximize the accumulated user satisfaction (US) score for a given time horizon which is able to handle the change in both the UAV lineup and user distribution. The framework accommodates the changed dimension of the state-action space before and after the UAV lineup change by deliberate state transition design. In addition, to handle the continuous state and action space, deep deterministic policy gradient (DDPG) algorithm, which is an actor-critic based DRL method, is exploited. Furthermore, to promote learning exploration around the timing of the change, the original DDPG scheme is adapted into an asynchronous parallel computing (APC) structure which leads to better training performance in both the critic and actor networks. Finally, extensive simulations are conducted to validate the convergence of the proposed learning approach, and demonstrate its capability in jointly handling the dynamics in UAV lineup and user distribution as well as its superiority over a passive reaction method.KeywordsUnmanned aerial vehicle (UAV)UAV-based communication networksDeep reinforcement learningDynamic user spatial distributionUAV trajectory design

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

The modern network is equipped with many service functions to acquire high-quality service. The emergence of network function virtualization (NFV) provides a convenient way to accomplish the network services in the form of virtual network function (VNF) and also makes the scheduling solution of VNFs flexible. The VNFs can be deployed on commodity servers as software processes. Besides, multiple VNFs are chained in a specified order as a service function chain (SFC) to serve a given flow, increasing the scheduling difficulty to minimize the average flow completion time. In this paper, we study the problem of scheduling multiple SFCs with the constraint of different resource limitations in various commodity servers. This problem is typically formulated as an Integer Linear Programming (ILP) problem, which is NP-hard. To well tackle this problem, we propose a deep reinforcement learning (DRL) approach. It involves multi-step decision making, which can be naturally transformed into a DRL problem. We design specific reward and state representations for such a multi-resource scheduling problem. We also consider how to use DRL to handle online requests of SFCs. The experiment results demonstrate that the DRL approach can significantly reduce the average completion time of a set of SFC and achieves a cost saving of 39.94% against the benchmark method.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

With the increasing real-time demand in the internet era, especially dynamic requests for last-mile delivery, route planning is becoming more computationally expensive than ever before. Many supply chains (SCs) choose the joint distribution of multiple depots to reduce transportation costs and delivery times. However, providing real-time and high-quality solutions for such complex routing problems remains challenging. Current solution methods like mathematical programming and heuristics suffer from scalability issues and long computation times. In contrast, artificial intelligence, especially Deep Reinforcement Learning (DRL), provides a general-purpose framework for sequential decision-making that has produced good results for many challenging real-life problems. However, applying DRL to route multiple vehicles is nontrivial, as the joint distribution requires an effective method that facilitates collaboration and communication among all agents while they carry out the delivery mission. In this research, a collaborative Multi-Agent Deep Reinforcement Learning (MADRL) approach is proposed for routing multiple vehicles in the SC. The proposed MADRL model leverages the power of two frameworks, deep learning and reinforcement learning, to generate routing policies for all agents in real time. Experimental results show the ability of the proposed learning model to obtain fast and quality solutions for complex delivery problems. Furthermore, the generalization ability of MADRL is also validated by testing the well-trained model on different scale problems.

Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm in the field of artificial intelligence (AI), offering unprecedented capabilities in decision-making across diverse domains. This article explores the profound impact of DRL on enhancing the decision-making capabilities of AI systems, elucidating its underlying principles, applications, and implications.DRL represents a fusion of deep learning and reinforcement learning, enabling machines to learn complex behaviors and make decisions by interacting with their environment. The utilization of neural networks allows DRL algorithms to handle high-dimensional input spaces, making it well-suited for tasks that involve intricate decision-making processes.One of the key strengths of DRL lies in its ability to address problems with sparse and delayed rewards, common challenges in traditional reinforcement learning. Through a process of trial and error, DRL algorithms can learn optimal decision strategies by navigating through a vast decision space, adapting to dynamic environments, and maximizing cumulative rewards over time.The applications of DRL span various domains, including robotics, finance, healthcare, gaming, and autonomous systems. In robotics, DRL facilitates the development of intelligent agents capable of autonomously navigating complex environments, performing intricate tasks, and adapting to unforeseen circumstances. In finance, DRL is leveraged for portfolio optimization, algorithmic trading, and risk management, demonstrating its potential to revolutionize traditional financial strategies.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

The food industry faces significant challenges in managing operational costs due to its high energy intensity and rising energy prices. Industrial food-processing facilities, with substantial thermal capacities and large demands for cooling and heating, offer promising opportunities for demand response (DR) strategies. This study explores the application of deep reinforcement learning (RL) as an innovative, data-driven approach for DR in the food industry. By leveraging the adaptive, self-learning capabilities of RL, energy costs in the investigated plant are effectively decreased. The RL algorithm was compared with the well-established optimization method Mixed Integer Linear Programming (MILP), and both were benchmarked against a reference scenario without DR. The two optimization strategies demonstrate cost savings of 17.57% and 18.65% for RL and MILP, respectively. Although RL is slightly less efficient in cost reduction, it significantly outperforms in computational speed, being approximately 20 times faster. During operation, RL only needs 2ms per optimization compared to 19s for MILP, making it a promising optimization tool for edge computing. Moreover, while MILP’s computation time increases considerably with the number of binary variables, RL efficiently learns dynamic system behavior and scales to more complex systems without significant performance degradation. These results highlight that deep RL, when applied to DR, offers substantial cost savings and computational efficiency, with broad applicability to energy management in various applications.

Millimeter-wave (mmWave) communication is a key technology of fifth generation wireless systems to achieve the expected 1000x data rate. With large bandwidth at the mmWave band, the link capacity between users and base stations (BSs) can be much higher compared to sub-6 GHz wireless systems. Meanwhile, due to the high cost of infrastructure upgrade, it would be difficult for operators to drastically enhance the capacity of backhaul links between mmWave BSs and the core network. As a result, the data rate provided by backhaul may not be sufficient to support all mmWave links; hence, the backhaul connection becomes the new bottleneck. On the other hand, as mmWave channels are subject to random blockage, the data rates of mmWave users significantly vary over time. With limited backhaul capacity and highly dynamic data rates of users, how to allocate backhaul resource to each user remains a challenge for mmWave systems. In this article, we present a deep reinforcement learning (DRL) approach to address this challenge. By learning the blockage pattern, the system dynamics can be captured and predicted, resulting in efficient utilization of backhaul resource. We begin with a discussion on DRL and its application in wireless systems. We then investigate the problem of backhaul resource allocation and present the DRL-based solution. Finally, we discuss open problems for future research and conclude this article.

This paper investigates the covert communications via spectrum allocations in a cellular-enabled unmanned aerial vehicle (UAV) network consisting of a base station (BS), UAVs, ground users (GUs), and a warden, where warden attempts to detect the transmission from a target GU to a UAV receiver. We formulate the spectrum allocation as an optimization problem with the constraints of covertness performance requirement and the qualities of service (QoS) of cellular communications. This is a nonlinear and nonconvex problem, which is generally challenging to be solved. Thus, we propose a deep reinforcement learning (DRL) approach to solve it. Under such an approach, we first model the multi-agent DRL environment in such networks. Then we define the state, action, reward and interaction mechanism of the DRL environment. Finally, a DRL algorithm is presented for learning the optimal policy of spectrum allocation.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.