Optimal Elevator Group Control via Deep Asynchronous Actor-Critic Learning.

In this paper, we propose a novel deep reinforcement learning (DRL) method for optimal path planning for mobile robots using dynamic programming (DP)-based data collection. The proposed method can overcome the slow learning process and improve training data quality inherently in DRL algorithms. The main idea of our approach is as follows. First, we mapped the dynamic programming method to typical optimal path planning problems for mobile robots, and created a new efficient DP-based method to find an exact, analytical, optimal solution for the path planning problem. Then, we used high-quality training data gathered using the DP method for DRL, which greatly improves training data quality and learning efficiency. Next, we established a two-stage reinforcement learning method where, prior to the DRL, we employed extreme learning machines (ELM) to initialize the parameters of actor and critic neural networks to a near-optimal solution in order to significantly improve the learning performance. Finally, we illustrated our method using some typical path planning tasks. The experimental results show that our DRL method can converge much easier and faster than other methods. The resulting action neural network is able to successfully guide robots from any start position in the environment to the goal position while following the optimal path and avoiding collision with obstacles.

The study examines the complex processes of intelligent management of energy resilience in modern healthcare facilities during critical situations, large-scale failures, and prolonged outages of the external centralized power supply. The aim is to develop a comprehensive mathematical model and a Safe Deep Reinforcement Learning (Safe DRL) method that ensures guaranteed compliance with the strict physical and operational constraints of hospital energy systems, even during the intensive training phase of a neural network agent. The objectives are: to formalize in detail the decision-making procedure in an energy system in detail by transitioning to the paradigm of Constrained Markov Decision Processes (CMDP); to develop an innovative mathematical model featuring the implementation of a specialized safety layer based on Lyapunov functions; and to ensure high resilience and autonomy of the system through the implementation of a decentralized Edge-Fog data processing architecture. The methods used include: the theory of Constrained Markov Decision Processes (CMDP), deep reinforcement learning methods based on the Actor-Critic architecture, the mathematical apparatus of Lyapunov stability theory for the analytical correction of actions, and methods of simulation modeling for complex dynamic energy systems. The following results were obtained. In the course of the study, a Safe DRL method was proposed and substantiated, which integrates a Lyapunov-based projection directly into the training loop for the immediate correction of the agent’s control actions. This makes it possible to ensure strict theoretical guarantees of maintaining the required State of Charge (SoC) of battery systems and to prevent critical violations of energy system parameters beyond established safety limits, which is crucial for patient life-support. The effectiveness of the proposed approach was confirmed by a series of numerical experiments in the specialized environment, HospitalEnergyEnv. Under a full blackout scenario, the agent demonstrated adaptability and high accuracy in resource management without any violation of the established physical limits during the entire process of autonomous operation. Conclusions. The scientific novelty of the obtained results lies in the following: the existing optimization model for building energy management systems (BEMS) has been improved by introducing an analytical safety projection mechanism, which minimizes the risks of emergency equipment shutdown during the adaptation of artificial intelligence algorithms; further development of decentralized control methods for critical infrastructure based on Edge-Fog computing has been achieved, which significantly increases system fault tolerance in the event of a loss of connection with the global network and ensures obtaining quasi-optimal solutions in high-dimensional problems. The practical value of this work lies in the potential to create highly reliable autonomous energy systems for critical infrastructure facilities.

Recently, the development of Low Earth Orbit (LEO) satellites and the advancement of the Mobile Edge Computing (MEC) paradigm have driven the emergence of the Satellite Mobile Edge Computing (Sat-MEC). Sat-MEC has been developed to support communication and task computation for Internet of Things (IoT) Mobile Devices (IMDs) in the absence of terrestrial networks. However, due to the heterogeneity of tasks and Sat-MEC servers, it is still a great challenge to efficiently schedule tasks in Sat-MEC servers. Here, we propose a scheduling algorithm based on the Deep Reinforcement Learning (DRL) method in the Sat-MEC architecture to minimize the average task processing time. We consider multiple factors, including the cooperation between LEO satellites, the concurrency and heterogeneity of tasks, the dynamics of LEO satellites, the heterogeneity of the computational capacity of Sat-MEC servers, and the heterogeneity of the initial queue for task computation. Further, we use the self-attention mechanism to act as a Q-network to extract high-dimensional dynamic information of tasks and Sat-MEC servers. In this work, we model the Sat-MEC environment simulation at the application level and propose a DRL-based task scheduling algorithm. The simulation results confirm the effectiveness of our proposed scheduling algorithm, which reduces the average task processing time by 22.1%, 30.6%, and 41.3%, compared to the genetic algorithm(GA), the greedy algorithm, and the random algorithm, respectively.

First-person shooter (FPS) video games play an important role in game artificial intelligence (AI). In this paper, we present an effective deep reinforcement learning (DRL) method to learn battles in ViZDoom. Our approach utilizes the actor-critic with Kronecker-factored trust region (ACKTR), a sample-efficient and computationally inexpensive DRL method. We train our ACKTR agents in two battle scenarios, and compare with the advantage actor-critic (A2C) baseline agent. The experimental results demonstrate that DRL methods successfully teach agents to battle in these scenarios. In addition, the ACKTR agents significantly outperform the A2C agents in terms of all the metrics by a significant margin.

Adaptive computation offloading and resource allocation strategy in a mobile edge computing environment

The hybrid flow shop scheduling problem (HFSP) has been extensively studied due to its wide applications in modern manufacturing systems. However, existing research primarily focuses on machine-centric scheduling while neglecting the critical role of human resources, whose varying skill levels impact processing efficiency and whose availability constraints complicate scheduling decisions. Therefore, this paper investigates the HFSP with human resource constraints (HFSP-HRC), which requires the joint optimization of job sequencing, machine allocation, and worker assignment to obtain efficient scheduling solutions. We propose a deep reinforcement learning (DRL) method integrated with a meta-path-based heterogeneous graph neural network (MP-HGNN) to solve this problem. It innovatively captures complex interdependencies among operations, machines, and workers through machine-centric and worker-centric meta-paths, overcoming limitations of existing methods that fail to model heterogeneous resource competition relationships. The integration enables high-order semantic feature fusion for enhanced scheduling robustness. Combined with the proximal policy optimization (PPO) algorithm, the framework realizes end-to-end learning from raw scheduling states to optimal decisions. Experiments across four different problem scales demonstrate that the proposed method significantly outperforms baseline heuristic rules and DRL method in makespan, with particularly outstanding performance on large-scale instances.

Automated driving (AD) is a new technology that aims to mitigate traffic accidents and enhance driving efficiency. This study presents a deep reinforcement learning (DRL) method for autonomous vehicles that can safely and efficiently handle highway overtaking scenarios. The first step is to create a highway traffic environment where the agent can be guided safely through surrounding vehicles. A hierarchical control framework is then provided to manage high-level driving decisions and low-level control commands, such as speed and acceleration. Next, a special DRL-based method called deep deterministic policy gradient (DDPG) is used to derive decision strategies for use on the highway. The performance of the DDPG algorithm is compared with that of the DQN and PPO algorithms, and the results are evaluated. The simulation results show that the DDPG algorithm can effectively and safely handle highway traffic tasks.

Deep reinforcement learning for energy trading and load scheduling in residential peer-to-peer energy trading market

The popularity of deep reinforcement learning (DRL) applications in economics has increased exponentially. DRL, through a wide range of capabilities from reinforcement learning (RL) to deep learning (DL), offers vast opportunities for handling sophisticated dynamic economics systems. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this paper, we initially consider a brief review of DL, RL, and deep RL methods in diverse applications in economics, providing an in-depth insight into the state-of-the-art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher efficiency as compared to the traditional algorithms while facing real economic problems in the presence of risk parameters and the ever-increasing uncertainties.

AUV position tracking and trajectory control based on fast-deployed deep reinforcement learning method

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

Due to its chaotic and high volatility character, as well as many other uncertainties from reality, predicting the price of stock market is always a challenging goal to achieve. Due to those characteristics mentioned above, it could be regarded as a classification problem, and then many methods using different machine learning tools could be well applied to solve the challenging problem. Within these methods, deep neural network is a popular and highly noticed one in recent years. This is mainly because of its unique advantages compared to the more conventional machine learning methods, the highly complex nonlienearity and deep nonlinear topologies, to appropriately describe the complex situations. Later, after adding advantages of reinforcement learning to enable the model of the advantages to improve feature dimensions, the deep reinforcement learning method is well proposed to improve the performance. Deep reinforcement learning is a method to combine the advantages of deep learning and the advantages of reinforcement learning, and this paper will discuss its characters and advantages, and finally talk about its limitations and future development.

This paper proposes a hierarchical framework integrating deep reinforcement learning (DRL) and rule-based methods for multi-lane autonomous driving. We define an instantaneous desired speed (IDS) to mimic the common motivation for higher speed in different traffic situations as an intermediate action. High-level DRL is utilized to generate IDS directly, while the low-level rule-based policies including car following (CF) models and three-stage lane changing (LC) models are governed by the common goal of IDS. Not only the coupling between CF and LC behaviors is captured by the hierarchy, but also utilizing the benefits from both DRL and rule-based methods like more interpretability and learning ability. Owing to the decomposition and combination with rule-based models, traffic flow operations can be taken into account for individually controlled automated vehicles (AVs) with an extension of traffic flow adaptive (TFA) strategy through exposed critical parameters. A comprehensive evaluation for the proposed framework is conducted from both the individual and system perspective, comparing with a pure DRL model and widely used rule-based model IDM with MOBIL. The simulation results prove the effectiveness of the proposed framework.

Reinforcement learning (RL), a promising framework for data-driven decision making in an uncertain environment, has successfully been applied in many real-world operation and control problems. However, the application of RL in a large-scale decentralized multi-agent environment remains a challenging problem due to the partial observability and limited communications between agents. In this paper, we develop a model-based kernel RL approach and a model-free deep RL approach for learning a decentralized, shared policy among homogeneous agents. By leveraging the strengths of both these methods, we further propose a novel deep ensemble multi-agent reinforcement learning (MARL) method that efficiently learns to arbitrate between the decisions of the local kernel-based RL model and the wider-reaching deep RL model. We validate the proposed deep ensemble method on a highly challenging real-world air traffic control problem, where the goal is to provide effective guidance to aircraft to avoid air traffic congestion, conflicting situations, and to improve arrival timeliness, by dynamically recommending adjustments of aircraft speeds in real-time. Extensive empirical results from an open-source air traffic management simulation model, developed by Eurocontrol and built on a real-world data set including thousands of aircrafts, demonstrate that our proposed deep ensemble MARL method significantly outperforms three state-of-the-art benchmark approaches.

Reinforcement learning (RL) has distinguished itself as a prominent learning method to augment the efficacy of autonomous systems. Recent advances in deep learning studies have complemented existing RL methods and led to a crucial breakthrough in the effort of applying RL to automation and robotics. Artificial agents based on deep RL can take selective and intelligent actions comparable with those of a human to maximize the feedback reward from the interactive environment. In this paper, we survey recent developments in the literature regarding deep RL methods for building human-level agents. As a result, prominent studies that involve modeling every aspect of a human-level agent will be examined. We also provide an overview of constructing a framework for prospective autonomous systems. Moreover, various toolkits and frameworks are suggested to facilitate the development of deep RL methods. Finally, we open a discussion that potentially raises a range of future research directions in deep RL.