Transformable Gaussian Reward Function for Socially Aware Navigation Using Deep Reinforcement Learning.

In this paper, we address the difficulty of solving large-scale multi-dimensional knapsack instances (MKP), presenting a novel deep reinforcement learning (DRL) framework. In this DRL framework, we train different agents compatible with a discrete action space for sequential decision-making while still satisfying any resource constraint of the MKP. This novel framework incorporates the decision variable values in the 2D DRL where the agent is responsible for assigning a value of 1 or 0 to each of the variables. To the best of our knowledge, this is the first DRL model of its kind in which a 2D environment is formulated, and an element of the DRL solution matrix represents an item of the MKP. Our framework is configured to solve MKP instances of different dimensions and distributions. We propose a K-means approach to obtain an initial feasible solution that is used to train the DRL agent. We train four different agents in our framework and present the results comparing each of them with the CPLEX commercial solver. The results show that our agents can learn and generalize over instances with different sizes and distributions. Our DRL framework shows that it can solve medium-sized instances at least 45 times faster in CPU solution time and at least 10 times faster for large instances, with a maximum solution gap of 0.28% compared to the performance of CPLEX. Furthermore, at least 95% of the items are predicted in line with the CPLEX solution. Computations with DRL also provide a better optimality gap with respect to state-of-the-art approaches.

Deep reinforcement learning (DRL) is one of the emerging areas of machine learning which focuses on maximized rewards. DRL is a type of machine learning that combines reinforcement learning and deep learning. It uses a series of algorithms to enable an agent to learn how to make decisions in a complex environment. DRL is a subset of artificial intelligence that focuses on making decisions based on the environment and the rewards associated with each action.The goal of DRL is to maximize the long-term reward of an agent. In order to do this, the agent must use a combination of deep learning, reinforcement learning and other AI techniques to learn which actions will lead to the highest reward. DRL is used to solve a variety of problems, from playing video games to controlling robots. It is also used in autonomous driving and robotics, as well as for financial trading. DRL is a powerful tool for solving complex problems and has been used in a variety of research projects. DRL has the potential to revolutionize the way we interact with machines and the environment.

Manually designing an effective and efficient controller for complex mechanics, such as bio-inspired robots or underactuated mechanical system, typically are very difficult. It requires precise motion planning and dynamic control. Reinforcement learning or genetic algorithm based learning methods suffers from representing the high dimensional models. The combination of deep learning and reinforcement learning provide a feasible way to handle such difficulties. However, priori-less searching sometimes tends to be low efficient and usually finds the “mechanic” solution instead of the “natural” one. In this paper, the traditional nonlinear control concept is integrated into the deep reinforcement learning (DRL) framework. The whole process is implemented on the brachiation control problem of a two link bio-primate robot. Deep Deterministic Policy Gradient (DDPG) is used to search for the optimal control policy. The searching process is realized by interacting with the dynamic model instead of real robot. The energy based planning and control concept is adopted, which utilize the fact that when the shoulder joint angle is fixed, energy of the whole system keeps constant. By regulating the angle and energy, the robot can be restricted on a particular trajectory. The energy concept is encoded within the reward function and trained in the Gym environment. For varying targets point-to-point control, the network structure is also modified to accept the target coordinates. Effectiveness of the proposed methods are verified by simulation and experimental results.

Learning ability is the basic characteristic of human intelligence. The July 1, 2005 issue of Science published a list of 125 important questions in science. Among them, the question 94 “What are the limits of learning by machines?”. The annotation “Computers can already beat the world’s best chess players, and they have a wealth of information on the Web to draw on. But abstract reasoning is still beyond any machine”. In recent artificial intelligence has made great progresses. In 1997, the rise of the man-machine war, IBM Supercomputer Deep Blue defeated the chess master Garry Kasparov. On February 14, 2011, IBM’s Watson supercomputer won a practice round against Jeopardy champions Ken Jennings and Brad Rutter. In March 2016, Google DeepMind’s AlphaGo sealed a 4-1 victory over a South Korean Go grandmaster Lee Se-dol. This paper focuses on the machine learning methods of AlphaGo, including reinforcement learning, deep learning, deep reinforcement learning, analysis of the existing problems and the latest research progress. Deep reinforcement learning is the combination of deep learning and reinforcement learning, which can realize the learning algorithm from the perception to action. Simply said, this is the same as human behavior, input sensing information such as vision, and then, direct output action through the deep neural network. Deep reinforcement learning has the potential to learn a variety of skills for the robot to achieve full autonomy. Even though reinforcement learning is practiced successfully, but feature states need to manually set, for complex scene is a difficult thing, especially easy to cause the dimension disaster, and expression is not good. In 2010, Sascha Lange and Martin Riedmiller proposed deep auto-encoder neural networks in reinforcement learning to extract feature, which is used to control the visual correlation. In 2013, DeepMind proposed deep Q-network (DQN) in NIPS 2013, using convolution neural network to extract features, and then applied in reinforcement learning. They continue to improve and published an improved version of DQN on Nature in 2015, which has aroused widespread concern. In order to break through the limits of learning by machines, cognitive machine learning is proposed, which is the combination of machine learning and brain cognition, so that the machine intelligence is constantly evolving, and gradually reaches the human level of artificial intelligence. A cognitive model entitled Consciousness And Memory (CAM) is proposed by author, which consists of memory, consciousness, high-level cognitive functions, perception and motor. High-level cognitive functions of the brain include learning, language, thinking, decision making, emotion, and so on. Learning is a course to accept the stimulus through the nervous system and obtain new behavior, habits and accumulation experience. According to the current research progress of brain science and cognitive science, cognitive machine learning may be interested in learning emergence, procedural memory knowledge learning, learning evolution and so on. For intelligence, so-called evolution is refers to the learning of learning and the structure also follows the change. It is important to record the learning result by structure changing and improve the learning method.

Study on deep reinforcement learning techniques for building energy consumption forecasting

Energy management and power allocation policy is considered for energy harvesting (EH) communications. In this letter, we propose a joint optimization problem with the continuous EH time and transmit power to maximize the long-term throughput based on deep deterministic policy gradient (DDPG). However, the joint optimization problem leads to a large continuous action space. In order to reduce the dimension of action space, we present a deep reinforcement learning (DRL) framework by combining DDPG and convex program. The original problem is decomposed into two-layer optimization subproblems by using the primal decomposition method. The primary problem can be solved by DDPG with a low-dimensional action space. The lower-layer subproblem can be solved by using the existing convex toolbox. Numerical simulation results show that, compared with the existing energy management or power allocation policies for EH communications, the proposed DRL framework can achieve higher long-term throughput.

A multi-objective deep reinforcement learning framework

Dynamic movement primitives (DMPs) are a learning from demonstration (LfD) method that imitates movement primitives. Recently, reinforcement learning (RL) has been used to modify DMPs. In this regard, most methods have concentrated on improving DMPs with RL after DMPs are learned with supervised learning. In this way, there is no guarantee that the DMPs stay close to demonstrations while modified. To address this problem, in this study, a framework is proposed for learning DMPs with deep RL, which can be further used to modify DMPs. To do so, first, DMPs are modeled in the framework of deep RL. Then, after the reward function for learning is defined, they are learned with deep RL. The deep RL algorithm used in the present study is the twin-delayed deep deterministic policy gradient (TD3) algorithm. Our proposed algorithm is evaluated with various demonstrations, and a case study of a pick-and-place task is done. The results show that our method can learn all demonstrations with high accuracy.

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

Path planning refers to that the mobile robot can obtain the surrounding environment information and its own state information through the sensor carried by itself, which can avoid obstacles and move towards the target point. Deep reinforcement learning consists of two parts: reinforcement learning and deep learning, mainly used to deal with perception and decision-making problems, has become an important research branch in the field of artificial intelligence. This paper first introduces the basic knowledge of deep learning and reinforcement learning. Then, the research status of deep reinforcement learning algorithm based on value function and strategy gradient in path planning is described, and the application research of deep reinforcement learning in computer game, video game and autonomous navigation is described. Finally, I made a brief summary and outlook on the algorithms and applications of deep reinforcement learning.

With The advent of the Internet of Things (IoT) and its use cases there is a necessity for improved latency which has led to edgecomputing technologies. IoT applications need a cloud environment and appropriate scheduling based on the underlying requirements of a given workload. Due to the mobility nature of IoT devices and resource constraints and resource heterogeneity, IoT application tasks need more efficient scheduling which is a challenging problem. The existing conventional and deep learning scheduling techniques have limitations such as lack of adaptability, issues with synchronous nature and inability to deal with temporal patterns in the workloads. To address these issues, we proposed a learning-based framework known as the Deep Reinforcement Learning Framework (DRLF). This is designed in such a way that it exploits Deep Reinforcement Learning (DRL) with underlying mechanisms and enhanced deep network architecture based on Recurrent Neural Network (RNN). We also proposed an algorithm named Reinforcement Learning Dynamic Scheduling (RLbDS) which exploits different hyperparameters and DRL-based decision-making for efficient scheduling. Real-time traces of edge-cloud infrastructure are used for empirical study. We implemented our framework by defining new classes for CloudSim and iFogSim simulation frameworks. Our empirical study has revealed that RLbDS out performs many existing scheduling methods.

Non-terrestrial network (NTN) communication is included in the 3GPP standard because of its excellent features such as resistance to ground physical attacks and wide coverage. Since the available power and storage resources are limited on high throughput satellites (HTSs), optimizing the resource allocation can greatly improve the performance of the HTS based communication system. Notably, weather and other factors make the channel status between the satellite and the terrestrial base stations constantly change. In this paper, we first exploit the model-free feature of reinforcement learning to formulate the aforementioned dynamic and unpredictable channel conditions into the power allocation problem in HTS systems. Due to the complexity of environment state and power allocation, a deep neural network is introduced to replace the Q table, and a deep reinforcement learning framework is built. Furthermore, a power allocation algorithm based on a deep reinforcement learning framework is presented. Finally, the simulation results show that the proposed algorithm is superior to existing algorithms in terms of the long-term system throughput performance, and has better adaptability to dynamic channel environment, thereby improving network performance.

Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach

With the rapid development of smart grid and renewable energy, demand response mechanism has become a key means to balance microgrid supply and demand and improve energy efficiency. As the traditional power demand response algorithm is difficult to deal with the uncertainty of power demand and the adverse effects caused by customers, this paper proposes a power demand response algorithm based on deep reinforcement learning. Firstly, we study and design a price -based layered electricity energy demand response model. Then, the dynamic optimal pricing decision of power trading market is described as Markov decision process, and the learning mechanism under the framework of deep reinforcement learning is expounded. Finally, a solution algorithm based on deep reinforcement learning is designed. The simulation results show that the proposed algorithm can adjust the electricity price adaptively according to the load demand, which can reduce the cost of users by about 17% and reduce the peak load demand.

Reinforcement learning is a mathematical framework for agents to interact intelligently with their environment. Unlike supervised learning, where a system learns with the help of labeled data, reinforcement learning agents learn how to act by trial and error only receiving a reward signal from their environments. A field where reinforcement learning has been prominently successful is robotics [3]. However, real-world control problems are also particularly challenging because of the noise and high- dimensionality of input data (e.g., visual input). In recent years, in the field of supervised learning, deep neural networks have been successfully used to extract meaning from this kind of data. Building on these advances, deep reinforcement learning was used to solve complex problems like Atari games and Go. Mnih et al. [1] built a system with fixed hyper parameters able to learn to play 49 different Atari games only from raw pixel inputs. However, in order to apply the same methods to real-world control problems, deep reinforcement learning has to be able to deal with continuous action spaces. Discretizing continuous action spaces would scale poorly, since the number of discrete actions grows exponentially with the dimensionality of the action. Furthermore, having a parametrized policy can be advantageous because it can generalize in the action space. Therefore with this thesis we study state-of-the-art deep reinforcement learning algorithm, Deep Deterministic Policy Gradients. We provide a theoretical comparison to other popular methods, an evaluation of its performance, identify its limitations and investigate future directions of research. The remainder of the thesis is organized as follows. We start by introducing the field of interest, machine learning, focusing our attention of deep learning and reinforcement learning. We continue by describing in details the two main algorithms, core of this study, namely Deep Q-Network (DQN) and Deep Deterministic Policy Gradients (DDPG). We then provide implementatory details of DDPG and our test environment, followed by a description of benchmark test cases. Finally, we discuss the results of our evaluation, identifying limitations of the current approach and proposing future avenues of research.