Autonomous Vehicle Driving Path Control with Deep Reinforcement Learning

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.

Irrigation decision-making using Reinforcement Learning (RL) performs well in changing environment, but easily falls into sub-optimal solutions with high-dimensional data. Deep Reinforcement Learning (DRL) has fused RL with Deep Learning (DL) and excels at learning adaptive and long-term irrigation strategies directly from high-dimensional environment data. This paper systematically reviews the applications of DRL in irrigation optimization, covering both pre-trained environments based on crop growth simulators and dynamic environments driven by real-time sensors. We discussed the strengths of classic DRL algorithms, including their ability to handle dynamic and non-linear environments, and reviewed their performance in irrigation multi-objective optimization and decision-making. In addition, we identified constraints in applying DRL in irrigation decision making, which include data scarcity, poor model interpretability, and difficulties in field deployment. It shows DRL can provide a powerful framework for adaptive irrigation, but is constrained by the gap between simulation and real-world complexity. To address these limitations, we discussed approaches in future work, such as developing multi-objective DRL algorithms. These approaches will improve DRL modeling outcomes and provide a technological foundation for smart agriculture and sustainable resource management. • Review the application of Deep Reinforcement Learning (DRL) in agricultural irrigation. • Analyze the performance of DRL algorithms in irrigation decision-making. • Compare DRL models based on different environment in irrigation optimization. • Discuss the further work to improve DRL performance in irrigation optimization.

In recent years, deep reinforcement learning has combined the advantages of reinforcement learning and deep learning, and has made great progress in decision-making tasks. However, the training of deep reinforcement learning requires frequent interactions between the agent and the environment and repeated experiments. Adversaries have chances to poison the sample data collected by the agent by attacking the experimental environment in the training process, thereby bringing security risks and serious consequences to the training process of reinforcement learning. This work is committed to addressing the security risks in the field of deep reinforcement learning. However, this work improves the algorithm from the perspective of sample data filtering, and improves the security performance of deep reinforcement learning algorithm. There are two contributions in this work: one is to defend against adversarial attacks against deep reinforcement learning through cluster analysis and sample value evaluation; the other is to propose a deep reinforcement learning algorithm based on sample value evaluation on the basis of deterministic strategy gradient algorithm. The algorithm uses the clustering method to classify the sample pool, and measures the contribution value and security risk of the sample to the model training through the sample value evaluation. The classic game experiments show that the proposed algorithm is safe and effective. It reduces the threat of the agent falling into the adversarial sample attack and improves the training performance of deep reinforcement learning.

In recent years, deep reinforcement learning (DRL) has made impressive achievements in many fields. However, existing DRL algorithms usually require a large amount of exploration to obtain a good action policy. In addition, in many complex situations, the reward function cannot be well designed to meet task requirements. These two problems will make it difficult for DRL to learn a good action policy within a relatively short period. The use of expert data can provide effective guidance and avoid unnecessary exploration. This study proposes a deep imitation reinforcement learning (DIRL) algorithm that uses a certain amount of expert demonstration data to speed up the training of DRL. In the proposed method, the learning agent imitates the expert's action policy by learning from demonstration data. After imitation learning, DRL is used to optimise the action policy in a self‐learning way. By experimental comparison on a video game called the Mario racing game, it is shown that the proposed DIRL algorithm with expert demonstration data can obtain much better performance than previous DRL algorithms without expert guidance.

Deep Reinforcement Learning (DRL) has emerged as a transformative paradigm in the field of artificial intelligence (AI), offering unprecedented capabilities in decision-making across diverse domains. This article explores the profound impact of DRL on enhancing the decision-making capabilities of AI systems, elucidating its underlying principles, applications, and implications.DRL represents a fusion of deep learning and reinforcement learning, enabling machines to learn complex behaviors and make decisions by interacting with their environment. The utilization of neural networks allows DRL algorithms to handle high-dimensional input spaces, making it well-suited for tasks that involve intricate decision-making processes.One of the key strengths of DRL lies in its ability to address problems with sparse and delayed rewards, common challenges in traditional reinforcement learning. Through a process of trial and error, DRL algorithms can learn optimal decision strategies by navigating through a vast decision space, adapting to dynamic environments, and maximizing cumulative rewards over time.The applications of DRL span various domains, including robotics, finance, healthcare, gaming, and autonomous systems. In robotics, DRL facilitates the development of intelligent agents capable of autonomously navigating complex environments, performing intricate tasks, and adapting to unforeseen circumstances. In finance, DRL is leveraged for portfolio optimization, algorithmic trading, and risk management, demonstrating its potential to revolutionize traditional financial strategies.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Reinforcement learning, as a branch of machine learning, has been gradually applied in the control field. However, in the practical application of the algorithm, the hyperparametric approach to network settings for deep reinforcement learning still follows the empirical attempts of traditional machine learning (supervised learning and unsupervised learning). This method ignores part of the information generated by agents exploring the environment contained in the updating of the reinforcement learning value function, which will affect the performance of the convergence and cumulative return of reinforcement learning. The reinforcement learning algorithm based on dynamic parameter adjustment is a new method for setting learning rate parameters of deep reinforcement learning. Based on the traditional method of setting parameters for reinforcement learning, this method analyzes the advantages of different learning rates at different stages of reinforcement learning and dynamically adjusts the learning rates in combination with the temporal-difference (TD) error values to achieve the advantages of different learning rates in different stages to improve the rationality of the algorithm in practical application. At the same time, by combining the Robbins–Monro approximation algorithm and deep reinforcement learning algorithm, it is proved that the algorithm of dynamic regulation learning rate can theoretically meet the convergence requirements of the intelligent control algorithm. In the experiment, the effect of this method is analyzed through the continuous control scenario in the standard experimental environment of ”Car-on-The-Hill” of reinforcement learning, and it is verified that the new method can achieve better results than the traditional reinforcement learning in practical application. According to the model characteristics of the deep reinforcement learning, a more suitable setting method for the learning rate of the deep reinforcement learning network proposed. At the same time, the feasibility of the method has been proved both in theory and in the application. Therefore, the method of setting the learning rate parameter is worthy of further development and research.

Energy efficient task scheduling based on deep reinforcement learning in cloud environment: A specialized review

The growing number of complex and heterogeneous Internet of Things (IoT) applications has imposed a high demand for scarce communications and computing resources. To meet this stringent requirement, it is desirable to develop large-scale highly adaptive online resource allocation strategies to streamline existing network operations. Deep reinforcement learning (DRL), which combines the merits of reinforcement learning and deep learning, is capable of addressing complex decision-making tasks, thus enabling efficient online resource allocation. In this article, we present a DRL-based resource allocation framework. We begin a discussion on DRL basics and review its several recent applications. Then, we develop two new DRL algorithms that facilitate unlocking the potential of DRL and offer viable solutions to many more complex resource allocation problems. The first one tackles an optimization problem exposed to mixed (discrete and continuous) action spaces and bound by a number of highly non-linear quality-of-service (QoS) constraints. The second one extends the single-agent DRL to a more challenging multi-agent DRL by introducing a novel semi-distributed architecture. Finally, we discuss the challenges and future visions of applying DRL to real-world IoT networks.

Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

Reinforcement Learning (RL) has become one of the most powerful methods in Artificial Intelligence (AI) which allows a system to learn as it is engaged in an interaction with its environment and to maximize its actions by performing a trial and error process. Although classical RL algorithms can give us a fundamental idea on RL, like the Q-Learning, its usefulness wanes in high-dimensional or continuous-state spaces. To overcome these shortcomings, Deep Reinforcement Learning (DRL) algorithms including Deep Q-Networks (DQN) build on neural networks to learn action-value functions, allowing horizontal scalability and more efficient learning. This paper reports on the experimental comparison of Q-Learning and DQN based on the environment CartPole-v1. Both of the algorithms were trained in the controlled environment and tested under various metrics that include cumulative rewards, success rate, convergence speed, safety (failure rate), and path efficiency. Findings indicate that Q-Learning moves towards moderate stability with slower convergence but DQN takes shorter learning time, is more reliable and performs better. Such results indicate the ground-breaking potential of the DRL in the progress of AI toward transcending classical RL restrictions. Future research topics encompass implementation of more challenging experimental settings, a test of other DRL algorithms (e.g., PPO, SAC), and generalization, scalability and safety in real-world environments.

Deep Reinforcement Learning (DRL) algorithm provides an effective and powerful way for the computer learning to make decisions according to what the computer sees, the most famous application of DRL is training the computer to play computer games, such as the computer Go and Atari Games. In this paper, we present that the DRL can used to improve the classical PID controller by training an adaptive PID controller with its parameters change according to the changing state, we name our new controller as DRPID, the deep reinforcement learning algorithm that we used to train the PID controller in this paper is deep deterministic policy gradient, which is very suitable to solve continue action control problem. We test our DRPID algorithm by training it to control an inverted pendulum in the OpenAI gym simulation environment, and we found DRPID controller works very well, outperforms the common PID controller with fixed parameters, by a great margin.

Using Deep Reinforcement Learning (DRL) algorithm to deal with autonomous driving tasks usually have unsatisfied performance due to lack of robustness and means to escape local optimum. In this article, we designs a Survival-Oriented Reinforcement Learning (SORL) model that tackle these problems by setting survival rather than maximize total reward as first priority. In SORL model, we model autonomous driving task as Constrained Markov Decision Process (CMDP) and introduce Negative-Avoidance Function to learn from previous failure. The SORL model greatly speed up the training process and improve the robustness of normal Deep Reinforcement Learning algorithm.