Multi-agent deep reinforcement learning based distributed control architecture for interconnected multi-energy microgrid energy management and optimization

With the improvement in the intelligence level of UAVs and the development of cluster control technology, the intelligent decision-making method of UAV cluster confrontation will become the key technology of UAV combat in the future. The UAV cluster confrontation learning environment is of complex characteristics of high dimension, non-linearity, incomplete information, continuous action space, and so on. Recently, artificial intelligence technology represented by deep learning and reinforcement learning has made a great breakthrough. Deep reinforcement learning has shown a great ability to solve intelligent decision-making problems in complex environments. In this paper, enlightened by the multi-agent centralized training distributed execution framework and the idea of maximum policy entropy, we propose a deep reinforcement learning method of the multi-agent soft actor-critic (MASAC) with incomplete information. A game model for UAV cluster confrontation based on multi-agent deep reinforcement learning is established, and a continuous space multiple UAV combat environment is constructed. Simulation experiments are performed on the asymmetric confrontation of UAV clusters with red and blue teams. The experimental results show that MASAC outperforms the existing popular multi-agent deep reinforcement learning methods, making the game players converge to a higher return equilibrium point of the game. Moreover, the convergence of MASAC is investigated and analyzed extensively, and the results show that MASAC is of good convergence and stability, ensuring the practicability of MASAC in intelligent decision-making regarding UAV cluster confrontation.

In this paper, we propose a novel deep reinforcement learning (DRL) method for optimal path planning for mobile robots using dynamic programming (DP)-based data collection. The proposed method can overcome the slow learning process and improve training data quality inherently in DRL algorithms. The main idea of our approach is as follows. First, we mapped the dynamic programming method to typical optimal path planning problems for mobile robots, and created a new efficient DP-based method to find an exact, analytical, optimal solution for the path planning problem. Then, we used high-quality training data gathered using the DP method for DRL, which greatly improves training data quality and learning efficiency. Next, we established a two-stage reinforcement learning method where, prior to the DRL, we employed extreme learning machines (ELM) to initialize the parameters of actor and critic neural networks to a near-optimal solution in order to significantly improve the learning performance. Finally, we illustrated our method using some typical path planning tasks. The experimental results show that our DRL method can converge much easier and faster than other methods. The resulting action neural network is able to successfully guide robots from any start position in the environment to the goal position while following the optimal path and avoiding collision with obstacles.

In contemporary metropolitan environments, where growing traffic and environmental concerns necessitate creative solutions, efficient traffic flow management is essential. In this work, a multi-agent deep reinforcement learning (DRL) method for managing dynamic traffic flow in a smart city framework is presented. This model optimizes traffic flow and reduces congestion by dynamically adjusting signal timings depending on real-time traffic circumstances by utilizing many intelligent agents, each of which represents a traffic light at an intersection. Through interactions with their separate environments, agents cooperatively learn and modify their activities to create a traffic network that is globally efficient. The study assesses the model's effectiveness in terms of shorter travel times, shorter wait times, and increased fuel efficiency using simulation data. According to the results, the suggested DRL model performs noticeably better than conventional static and rule-based systems, offering smart cities that want to improve urban mobility and lessen their environmental effect a scalable, adaptable option. This study advances intelligent transportation systems by demonstrating how multi-agent DRL may be used to create effective and sustainable urban traffic control plans.

The study examines the complex processes of intelligent management of energy resilience in modern healthcare facilities during critical situations, large-scale failures, and prolonged outages of the external centralized power supply. The aim is to develop a comprehensive mathematical model and a Safe Deep Reinforcement Learning (Safe DRL) method that ensures guaranteed compliance with the strict physical and operational constraints of hospital energy systems, even during the intensive training phase of a neural network agent. The objectives are: to formalize in detail the decision-making procedure in an energy system in detail by transitioning to the paradigm of Constrained Markov Decision Processes (CMDP); to develop an innovative mathematical model featuring the implementation of a specialized safety layer based on Lyapunov functions; and to ensure high resilience and autonomy of the system through the implementation of a decentralized Edge-Fog data processing architecture. The methods used include: the theory of Constrained Markov Decision Processes (CMDP), deep reinforcement learning methods based on the Actor-Critic architecture, the mathematical apparatus of Lyapunov stability theory for the analytical correction of actions, and methods of simulation modeling for complex dynamic energy systems. The following results were obtained. In the course of the study, a Safe DRL method was proposed and substantiated, which integrates a Lyapunov-based projection directly into the training loop for the immediate correction of the agent’s control actions. This makes it possible to ensure strict theoretical guarantees of maintaining the required State of Charge (SoC) of battery systems and to prevent critical violations of energy system parameters beyond established safety limits, which is crucial for patient life-support. The effectiveness of the proposed approach was confirmed by a series of numerical experiments in the specialized environment, HospitalEnergyEnv. Under a full blackout scenario, the agent demonstrated adaptability and high accuracy in resource management without any violation of the established physical limits during the entire process of autonomous operation. Conclusions. The scientific novelty of the obtained results lies in the following: the existing optimization model for building energy management systems (BEMS) has been improved by introducing an analytical safety projection mechanism, which minimizes the risks of emergency equipment shutdown during the adaptation of artificial intelligence algorithms; further development of decentralized control methods for critical infrastructure based on Edge-Fog computing has been achieved, which significantly increases system fault tolerance in the event of a loss of connection with the global network and ensures obtaining quasi-optimal solutions in high-dimensional problems. The practical value of this work lies in the potential to create highly reliable autonomous energy systems for critical infrastructure facilities.

Recently, the development of Low Earth Orbit (LEO) satellites and the advancement of the Mobile Edge Computing (MEC) paradigm have driven the emergence of the Satellite Mobile Edge Computing (Sat-MEC). Sat-MEC has been developed to support communication and task computation for Internet of Things (IoT) Mobile Devices (IMDs) in the absence of terrestrial networks. However, due to the heterogeneity of tasks and Sat-MEC servers, it is still a great challenge to efficiently schedule tasks in Sat-MEC servers. Here, we propose a scheduling algorithm based on the Deep Reinforcement Learning (DRL) method in the Sat-MEC architecture to minimize the average task processing time. We consider multiple factors, including the cooperation between LEO satellites, the concurrency and heterogeneity of tasks, the dynamics of LEO satellites, the heterogeneity of the computational capacity of Sat-MEC servers, and the heterogeneity of the initial queue for task computation. Further, we use the self-attention mechanism to act as a Q-network to extract high-dimensional dynamic information of tasks and Sat-MEC servers. In this work, we model the Sat-MEC environment simulation at the application level and propose a DRL-based task scheduling algorithm. The simulation results confirm the effectiveness of our proposed scheduling algorithm, which reduces the average task processing time by 22.1%, 30.6%, and 41.3%, compared to the genetic algorithm(GA), the greedy algorithm, and the random algorithm, respectively.

Multienergy distribution network (MEDN) with high penetration photovoltaics (PVs) may suffer from sharp voltage fluctuations and increased network losses. Existing methods struggle to achieve voltage control due to challenges such as high interarea communication latency and difficulties in power flow modeling caused by low coverage of measurement devices. To address these issues, this paper proposes a multiagent deep reinforcement learning (MADRL) method to realize the collaborative optimization of controllable devices, including hybrid energy storage system (HESS) and PV inverters. Furthermore, under the framework of decentralized partially observable Markov decision processes (Dec‐POMDP), we integrate cross‐agent attention (CAA) and factored value networks to enhance perception capabilities and improve value function fitting. The proposed method explicitly assigns credit to agents and dynamically captures electrical coupling relationships between agents and buses. The improved IEEE 33‐bus and IEEE 141‐bus distribution systems were used as case studies to compare with mainstream MADRL. Experimental results demonstrate that after offline deployment, the agents achieve global voltage control based solely on limited local observations within each zone, without relying on a complete power flow model or interarea communication. The comparative experiments verify the effectiveness, robustness, and scalability of this method.

With the increasing real-time demand in the internet era, especially dynamic requests for last-mile delivery, route planning is becoming more computationally expensive than ever before. Many supply chains (SCs) choose the joint distribution of multiple depots to reduce transportation costs and delivery times. However, providing real-time and high-quality solutions for such complex routing problems remains challenging. Current solution methods like mathematical programming and heuristics suffer from scalability issues and long computation times. In contrast, artificial intelligence, especially Deep Reinforcement Learning (DRL), provides a general-purpose framework for sequential decision-making that has produced good results for many challenging real-life problems. However, applying DRL to route multiple vehicles is nontrivial, as the joint distribution requires an effective method that facilitates collaboration and communication among all agents while they carry out the delivery mission. In this research, a collaborative Multi-Agent Deep Reinforcement Learning (MADRL) approach is proposed for routing multiple vehicles in the SC. The proposed MADRL model leverages the power of two frameworks, deep learning and reinforcement learning, to generate routing policies for all agents in real time. Experimental results show the ability of the proposed learning model to obtain fast and quality solutions for complex delivery problems. Furthermore, the generalization ability of MADRL is also validated by testing the well-trained model on different scale problems.

In this article, a new deep reinforcement learning (RL) method, called asynchronous advantage actor-critic (A3C) method, is developed to solve the optimal control problem of elevator group control systems (EGCSs). The main contribution of this article is that the optimal control law of EGCSs is designed via a new deep RL method, such that the elevator system sends passengers to the desired destination floors as soon as possible. Deep convolutional and recurrent neural networks, which can update themselves during applications, are designed to dispatch elevators. Then, the structure of the A3C method is developed, and the training phase for the learning optimal law is discussed. Finally, simulation results illustrate that the developed method effectively reduces the average waiting time in a complex building environment. Comparisons with traditional algorithms further verify the effectiveness of the developed method.

Deep reinforcement learning for energy trading and load scheduling in residential peer-to-peer energy trading market

Task assignment is a fundamental research problem in mobile crowdsensing (MCS) since it directly determines an MCS system’s practicality and economic value. Due to the complex dynamics of tasks and workers, task assignment problems are usually NP-hard, and approximation-based methods are preferred to impractical optimal methods. In the literature, a graph neural network-based deep reinforcement learning (GDRL) method is proposed in Xu and Song (2022) to solve routing problems in MCS and shows high performance and time efficiency. However, GDRL, as a centralized method, has to cope with the limitation in scalability and the challenge of privacy protection. In this paper, we propose a multi-agent deep reinforcement learning-based method named CQDRL to solve a task assignment problem in a decentralized fashion. The CQDRL method not only inherits the merits of GDRL over traditional heuristic and metaheuristic methods but also exploits computation potentials in mobile devices and protects workers’ privacy with a decentralized decision-making scheme. Our extensive experiments show that the CQDRL method can achieve significantly better performance than other traditional methods and performs fairly close to the centralized GDRL method.

The hybrid flow shop scheduling problem (HFSP) has been extensively studied due to its wide applications in modern manufacturing systems. However, existing research primarily focuses on machine-centric scheduling while neglecting the critical role of human resources, whose varying skill levels impact processing efficiency and whose availability constraints complicate scheduling decisions. Therefore, this paper investigates the HFSP with human resource constraints (HFSP-HRC), which requires the joint optimization of job sequencing, machine allocation, and worker assignment to obtain efficient scheduling solutions. We propose a deep reinforcement learning (DRL) method integrated with a meta-path-based heterogeneous graph neural network (MP-HGNN) to solve this problem. It innovatively captures complex interdependencies among operations, machines, and workers through machine-centric and worker-centric meta-paths, overcoming limitations of existing methods that fail to model heterogeneous resource competition relationships. The integration enables high-order semantic feature fusion for enhanced scheduling robustness. Combined with the proximal policy optimization (PPO) algorithm, the framework realizes end-to-end learning from raw scheduling states to optimal decisions. Experiments across four different problem scales demonstrate that the proposed method significantly outperforms baseline heuristic rules and DRL method in makespan, with particularly outstanding performance on large-scale instances.

The ongoing quest for transportation electrification with the massive proliferation of EV charging stations (EVCSs) will deepen the interaction and require the further coordination of coupled power and transportation networks (PTN). The individually-owned EVCSs located in an urban transportation network (UTN) will compete using price signals to maximize their respective payoffs. In this paper, a multi-agent deep reinforcement learning (MA-DRL) method is proposed to model the pricing game in UTN and determine the optimal charging prices for a single EVCS. The EVCS charging demand is first analyzed using a modified user equilibrium traffic assignment problem (UE-TAP) with elastic traveling demands and different charging prices. The price competition problem is then formulated as a game with incomplete information in which the market environment is complex due to nonlinear traffic assignments. Thus, the MA-DRL approach is proposed to learn the charging pricing strategies of multiple EVCSs and approximate the Nash Equilibrium (NE) of the pricing game using the incomplete information. The proposed solution will determine the optimal pricing strategies for an EVCS in UTN. The case studies on a 24-node Sioux-Falls network, and the real-world Xi'an and Hangzhou cities are conducted to verify the effectiveness and scalability of the proposed approach.

First-person shooter (FPS) video games play an important role in game artificial intelligence (AI). In this paper, we present an effective deep reinforcement learning (DRL) method to learn battles in ViZDoom. Our approach utilizes the actor-critic with Kronecker-factored trust region (ACKTR), a sample-efficient and computationally inexpensive DRL method. We train our ACKTR agents in two battle scenarios, and compare with the advantage actor-critic (A2C) baseline agent. The experimental results demonstrate that DRL methods successfully teach agents to battle in these scenarios. In addition, the ACKTR agents significantly outperform the A2C agents in terms of all the metrics by a significant margin.

Adaptive computation offloading and resource allocation strategy in a mobile edge computing environment

The popularity of deep reinforcement learning (DRL) applications in economics has increased exponentially. DRL, through a wide range of capabilities from reinforcement learning (RL) to deep learning (DL), offers vast opportunities for handling sophisticated dynamic economics systems. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this paper, we initially consider a brief review of DL, RL, and deep RL methods in diverse applications in economics, providing an in-depth insight into the state-of-the-art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher efficiency as compared to the traditional algorithms while facing real economic problems in the presence of risk parameters and the ever-increasing uncertainties.