Online Multi-Agent Reinforcement Learning for Decentralized Inverter-Based Volt-VAR Control

Advancements in data-driven voltage control in active distribution networks: A Comprehensive review

This paper proposes a new methodology for optimal coordination of network reconfiguration (NR) and Volt/VAR control (VVC) in the active distribution network by considering renewable energy sources, switched capacitor banks, and electric vehicle charging stations (EVCSs). NR and VVC are essential tools to enhance the performance of radial distribution systems by minimising active power losses and improving the system’s security and reliability. NR will decrease active power losses as there is a power flow diversion due to changes in network topology. Whereas VVC decreases active power losses due to power factor improvement. Because of the reliability considerations, NR is performed only a few times a day. However, the VVC is performed much more frequently. Coordinating these two controls over a day is essential for getting the best results compared to independent and sequential control. An integrated approach for solving the coordination of NR and VVC problems over a day is proposed. Total power loss and voltage deviation minimizations are considered as objective functions, and they are solved by using the Teaching Learning Based Optimisation algorithm. The simulation results are performed on IEEE 14 bus and IEEE 33 bus test systems by considering the wind, solar PV generators, and EVCSs.

Volt-VAR control is critical to keeping distribution network voltages within allowable range, minimizing losses, and reducing wear and tear of voltage regulating devices. To deal with incomplete and inaccurate distribution network models, we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner. The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space, and solved by our proposed constrained soft actor-critic algorithm. Our proposed reinforcement learning algorithm achieves scalability, sample efficiency, and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework, the method of multiplier, a device-decoupled neural network structure, and an ordinal encoding scheme. Comprehensive numerical studies with the IEEE distribution test feeders show that our proposed algorithm outperforms the existing reinforcement learning algorithms and conventional optimization-based approaches on a large feeder.

The rapid advancement of generative models and optimization algorithms has opened new opportunities for intelligent garden design systems. However, existing approaches often lack the ability to integrate aesthetic, functional and ecological considerations simultaneously and fail to adapt dynamically to complex design constraints. To address these challenges, this study proposes a dynamic garden design decision-making framework that integrates StyleGAN, Convolutional Neural Networks (CNNs) and the Soft Actor-Critic (SAC) reinforcement learning algorithm. The model combines StyleGAN’s capability to generate diverse and high-quality garden layouts, CNN’s feature extraction for spatial and visual representation and SAC’s optimization for balancing multiple objectives, including ecological benefit, resource efficiency and aesthetic quality. Experimental results on the Places Dataset and AID Dataset demonstrate that the proposed model significantly outperforms baseline methods in image quality, design diversity and optimization effectiveness. Specifically, it achieves superior performance in generating realistic and context-aware garden layouts while maintaining flexibility to adapt to environmental constraints. This research provides an adaptive framework for garden design and offers a foundation for extending dynamic design systems to fields like urban planning and interior design, promoting multi-objective optimization in diverse contexts.

In this research, Advantage Actor-Critic(A2C) Reinforcement Learning algorithm is applied on high precision augmented navigation system with GPS/GLONASS dual frequency receiver and navigation grade INS. Despite the complementary navigation characteristics, GNSS and INS augmented system suffers from insufficient model error especially on altitude. It is caused by vertical divergence of INS and vertical error budget of GNSS such as ionospheric delay. The approach dealt in this paper is to train the system with neural network utilizing accurate post processed carrier phase differential GNSS position. The flight data for the research is obtained through 2.5-hour flight, including IMU measurements, raw GNSS range and ephemeris information, etc. In order to train the networks, a custom reinforcement learning environment was designed. The environment defined the interaction between agent and its surrounding, with state, action, and reward. Applied policy and value optimizing A2C algorithm for continuous action space is described. With proposed continuous action A2C, the network achieved higher score compared to the baseline. Increasing the number of trainable weights and biases resulted in enhancement of accuracy and stability.

Comparison of model-free and model-based methods for time optimal hit control of a badminton robot

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

P2P power trading based on reinforcement learning for nanogrid clusters

Volt-VAR control (VVC) is a critical application in active distribution network management system to reduce network losses and improve voltage profile. To remove dependency on inaccurate and incomplete network models and enhance resiliency against communication or controller failure, we propose consensus multi-agent deep reinforcement learning algorithm to solve the VVC problem, which determines the operation schedules for voltage regulators, on-load tap changers, and capacitors. The VVC problem is formulated as a networked multi-agent Markov decision process, which is solved using the maximum entropy reinforcement learning framework and a novel communication-efficient consensus strategy. The proposed algorithm allows individual agents to learn a group control policy using local rewards. Numerical studies on IEEE distribution test feeders show that our proposed algorithm matches the performance of single-agent reinforcement learning benchmark. In addition, the proposed algorithm is shown to be communication efficient and resilient.

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

Swimming microrobots are increasingly developed with complex materials and dynamic shapes and are expected to operate in complex environments in which the system dynamics are difficult to model and positional control of the microrobot is not straightforward to achieve. Deep reinforcement learning is a promising method of autonomously developing robust controllers for creating smart microrobots, which can adapt their behavior to operate in uncharacterized environments without the need to model the system dynamics. This article reports the development of a smart helical magnetic hydrogel microrobot that uses the soft actor critic reinforcement learning algorithm to autonomously derive a control policy which allows the microrobot to swim through an uncharacterized biomimetic fluidic environment under control of a time varying magnetic field generated from a three-axis array of electromagnets. The reinforcement learning agent learned successful control policies from both state vector input and raw images, and the control policies learned by the agent recapitulated the behavior of rationally designed controllers based on physical models of helical swimming microrobots. Deep reinforcement learning applied to microrobot control is likely to significantly expand the capabilities of the next generation of microrobots.

We investigate the problem of non-covariant behavior of policy gradient reinforcement learning algorithms. The policy gradient approach is amenable to analysis by information geometric methods. This leads us to propose a natural metric on controller parameterization that results from considering the manifold of probability distributions over paths induced by a stochastic controller. Investigation of this approach leads to a covariant gradient ascent rule. Interesting properties of this rule are discussed, including its relation with actor-critic style reinforcement learning algorithms. The algorithms discussed here are computationally quite efficient and on some interesting problems lead to dramatic performance improvement over noncovariant rules.

With advancements in computational power and Vehicle-to-Infrastructure (V2I) technologies, modern traffic signal control (TSC) systems are designed in a more intelligent way by leveraging Connected Vehicle (CV) data. While prior research primarily focuses on CV positions and speeds, route data—which can provide foresight into future traffic demand—remains largely overlooked. To fill this gap, we propose a distributed TSC model based on the Soft Actor-Critic (SAC) reinforcement learning algorithm. Through V2I communication, the model incorporates vehicle positions, route information, downstream capacity, and neighboring traffic pressure into its state representation. This enables the agent to anticipate traffic demand and mitigate issues caused by partial observability. Furthermore, by acquiring the real-time data of priority vehicles and assigning weighted priority, the proposed model can adapt to real-time priority control. Simulation results show that the proposed model significantly outperforms baseline approaches, reducing average queue length and vehicle waiting time by approximately 19.2% and 57.9%, respectively. The model also demonstrates strong generalizability under traffic disturbances. In priority scenarios, it cuts the waiting time for high-priority vehicles by around 75%, with minimal impact on overall traffic efficiency. These findings demonstrate the model’s effectiveness, adaptability, and potential for deployment in intelligent traffic management systems.

Recent advancement on reinforcement learning (RL) algorithms shows that effective learning of parametric action- selection policies can often be achieved through direct opti- mization of a performance lower bound subject to pre-defined policy behavioral constraints. Driven by this understanding, this paper seeks to develop new policy search techniques where RL is achieved through maximizing a performance lower bound ob- tained originally based on an Expectation-Maximization method. For reliable RL, our new learning techniques must also simul- taneously guarantee constrained policy behavioral changes mea- sured through KL divergence. Two separate approaches will be pursued to tackle our constrained policy optimization problems, resulting in two new RL algorithms. The first algorithm utilizes a conjugate gradient technique and a Bayesian learning method for approximate optimization. The second algorithm focuses on minimizing a loss function derived from solving the Lagrangian for constrained policy search. Both algorithms have been experi- mentally examined on several benchmark problems provided by OpenAI GYM. The experiment results clearly demonstrate that our algorithms can be highly effective in comparison to several well-known RL algorithms.

To find the path that minimizes the time to navigate between two given points in a fluid flow is known as Zermelo's problem. Here, we investigate it by using a Reinforcement Learning (RL) approach for the case of a vessel that has a slip velocity with fixed intensity, Vs, but variable direction and navigating in a 2D turbulent sea. We show that an Actor-Critic RL algorithm is able to find quasioptimal solutions for both time-independent and chaotically evolving flow configurations. For the frozen case, we also compared the results with strategies obtained analytically from continuous Optimal Navigation (ON) protocols. We show that for our application, ON solutions are unstable for the typical duration of the navigation process and are, therefore, not useful in practice. On the other hand, RL solutions are much more robust with respect to small changes in the initial conditions and to external noise, even when Vs is much smaller than the maximum flow velocity. Furthermore, we show how the RL approach is able to take advantage of the flow properties in order to reach the target, especially when the steering speed is small.