Optimal dispatch of unbalanced distribution networks with phase-changing soft open points based on safe reinforcement learning

In recent years, the penetration rate of intermittent distributed generation (DG) in active distribution networks has been steadily increasing, leading to prominent issues such as voltage violations and network congestion. Coordinated operation of soft open points (SOPs) and distributed energy storage systems (DESSs) enable flexible resource management in both spatial and temporal dimensions, allowing real-time voltage regulation and flow control. This enhances the controllability, sustainability, and economic efficiency of the distribution network, ultimately improving user satisfaction. The optimization of this coordinated configuration has become a central challenge in research. Taking the different characteristics of DESSs and SOPs into account, this paper proposes a coordinated configuration method by introducing local marginal price (LMP) under the configuration scheme. The framework is modeled as a three-level problem, including planning and operation levels. Initially, typical scenarios are generated to address the uncertainty of distributed generation operation. At the upper level, the DESS configuration is optimized to minimize annual operational costs, while at the middle level, SOPs are planned based on the annual comprehensive operational cost of the distribution network. At the lower level, the objective is set as social welfare maximization to reflect user satisfaction by incorporating LMP as a planning indicator. It is then transformed and solved as a mixed-integer second-order programming model with a hybrid optimization algorithm. The model is established with the modified IEEE 33-node distribution system as a case study to validate the feasibility of the proposed configuration method. The case study results demonstrate the effectiveness of the proposed approach in optimally configuring SOPs with DESSs to reduce overall annual operating costs and enhance the economic efficiency of the system.

Distributed energy storage planning in soft open point based active distribution networks incorporating network reconfiguration and DG reactive power capability

Self-Preserving Genetic Algorithms (SPGA) combine the evolutionary strategy of a genetic algorithm with safety assurance methods commonly implemented in safe reinforcement learning (SRL), a branch of reinforcement learning (RL) that accounts for safety in the exploration and decision-making process of the agent. Safe learning approaches are especially important in safety-critical environments, where failure to account for the safety of the controlled system could result in the loss of millions of dollars in hardware or bodily harm to people working nearby, as is true of many cyber-physical systems. While SRL is a viable approach to safe learning, there are many challenges that must be taken into consideration when training agents, such as sample efficiency, stability, and exploration---an issue that is easily addressed by the evolutionary strategy of a genetic algorithm. By combining GAs with the safety mechanisms used with SRL, SPGA offers a safe learning alternative that is able to explore large areas of the solution space, addressing SRL's challenge of exploration. This work implements SPGA with both action masking and run time assurance safety strategies to evolve safe controllers for three types of discrete action space environments applicable to cyber physical systems (control, routing, and operations) and under various safety conditions. Training and testing evaluation metrics are compared with results from SRL trained controllers to validate results. SPGA and SRL controllers are trained across 5 random seeds and evaluated on 500 episodes to calculate average wall time to train, average expected return, and percentage of safe action evaluation metrics. SPGA achieves comparable reward and safety performance results with significantly improved training efficiency (55x faster on average), demonstrating the effectiveness of this safe learning approach.

Summary In the process of waterflood development in oil and gas reservoirs, the injection/production ratio is one of the key indicators affecting both the development effectiveness and economic benefits. A reasonable injection/production ratio directly impacts the development parameters, such as reservoir pressure maintenance levels and water-cut rise rates, and it helps maintain high production capacity in the oil field, ultimately achieving a higher recovery factor. However, existing constrained-waterflood reservoir production optimization methods often face challenges such as time-consuming calculations and the risk of getting trapped in local optima when dealing with such nonlinear constraint problems, making it difficult to effectively achieve a balance between constraints and economic benefits. To address this issue, we propose an injection/production ratio–constrained production optimization method based on the worst-case soft actor-critic (WCSAC) algorithm. This method effectively handles the constraint issue by introducing a safety critic, ensuring the injection/production ratio constraint is met, and maximizing economic benefits throughout the entire life cycle. Specifically, this method models the injection/production ratio–constrained production optimization problem as a constrained Markov decision process (CMDP). On the one hand, it uses conditional value at risk (CVaR) as a safety metric to assess the safety risks of the policy and achieves a balance between reward and safety by adjusting adaptive safety weights. On the other hand, it uses the maximum entropy mechanism to enhance policy exploration and improve the efficiency of global optimization. This results in the training of a stable and reliable safe reinforcement learning (RL) agent to achieve injection/production ratio–constrained production optimization. The agent can adjust and determine the production optimization plan that satisfies the injection/production ratio constraint in real time based on the current reservoir development status, without the need for retraining, thus reducing computational time. To validate the performance of the proposed method, tests were conducted in two reservoir models. The analysis results indicate that, compared with other optimization methods, the proposed method can achieve a higher net present value (NPV) while satisfying the injection/production ratio constraints, demonstrating good model applicability.

The intelligent soft open point (SOP) has powerful power flow regulation capabilities in the distribution network. If applied to the distribution network, it can flexibly cope with the output uncertainty of unmanageable distributed energy sources. However, considering the investment, operation, and maintenance costs, as well as the assistance of reactive power equipment, the site selection and capacity determination of SOP have become an urgent problem to be solved. This article proposes the optimal configuration strategy of SOP in a flexible interconnected distribution network, taking into account the features of distributed generation and reactive power sources. Firstly, based on the unpredictability of DG output, this paper uses improved sensitivity analysis to determine the optimal SOP installation location. Subsequently, with the optimization objective of minimizing the annual cost of the distribution network, this paper considers the characteristics of DGs, CBs, and OLCTs and uses second-order cone programming to optimize and solve SOP capacity under constraints such as trends. Finally, in the enhanced IEEE 33-node distribution system model, the effects of different scenarios on node voltage, reactive power components, and SOP location and capacity are compared.

Dual-layer peer-to-peer energy trading method with multiple SOPs pricing in distribution networks

A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid

This study proposes an improved method of the Artificial Bee Colony (ABC) algorithm for the distribution network in scenarios where distributed generation sources and Soft Open Points (SOPs) are connected to optimize power control. Improvement is achieved by integrating the ABC algorithm with the Grenade Explosion Method and Cauchy to accelerate the ABC algorithm's speed. The objective function is considered to reduce power losses over a day. The proposed method was tested on the IEEE-33 bus test system under various scenarios: Case 1 with 3 DGs installed, Case 2 with 3 DGs and 1 SOP simultaneously installed in the distribution network, and Case 3 having the same configuration as Case 2 but operating for 24 hours. In addition to reducing power losses, the voltage at the nodes in the distribution grid was also improved, maintained above 0.95 pu and close to 1 pu. Case 3 showed that integrating a Wind Turbine (WT), two Photovoltaic (PV) generators, and one SOP during operation resulted in the lowest energy losses, smaller than a system with only one WT and two PVs, and significantly lower than the baseline system without any DGs and SOPs. Therefore, employing SOPs in a distribution network with integrated DGs can offer significant benefits in reducing energy losses.

As part of the smart grid concept, different strategies specialized for power flow control in distribution networks have been developed. One of the possible solutions to optimize the utilization of existing capacities and increase distributed generation penetration is the implementation of Soft Open Points. Soft Open Points are modular devices based on power electronic converters that enable closing loops in the network without negative consequences regarding fault current propagation. Introducing the concept of soft open points in the distribution network enables power flow control in a particular part of the network and voltage control in the soft open point connecting nodes. The control strategy proposed in this paper addresses the main obstacle for appropriate exploitation of soft open points, defining reference values for control variables in the case of large-scale distribution systems. Furthermore, the proposed strategy also deals with data unavailability problems, that is, soft open point control under communication interruption. The proposed control algorithm incorporates centralized optimal power flow calculations and an estimation algorithm based on a multivariate polynomial regression. The optimal power flow is used to calculate the control variables in normal operation modes. The procedure based on multivariate polynomial regression was used to estimate the reference values of the control variables when the optimal power flow results were unavailable. This feature makes the proposed algorithm applicable to communication interruptions when only limited data capture is available. The algorithm proposed in this study was implemented and tested on a test network considering different scenarios. Conclusions and simulation results make this algorithm applicable to an actual soft open point controller.

With the increasing amount of distributed generation (DG) integrated into electrical distribution networks, various operational problems, such as excessive power losses, over-voltage and thermal overloading issues become gradually remarkable. Innovative approaches for power flow and voltage controls are required to ensure the power quality, as well as to accommodate large DG penetrations. Using power electronic devices is one of the approaches. In this paper, a multi-objective optimization framework was proposed to improve the operation of a distribution network with distributed generation and a soft open point (SOP). An SOP is a distribution-level power electronic device with the capability of real-time and accurate active and reactive power flow control. A novel optimization method that integrates a Multi-Objective Particle Swarm Optimization (MOPSO) algorithm and a local search technique – the Taxi-cab method, was proposed to determine the optimal set-points of the SOP, where power loss reduction, feeder load balancing and voltage profile improvement were taken as objectives. The local search technique is integrated to fine tune the non-dominated solutions obtained by the global search technique, overcoming the drawback of MOPSO in local optima trapping. Therefore, the search capability of the integrated method is enhanced compared to the conventional MOPSO algorithm. The proposed methodology was applied to a 69-bus distribution network. Results demonstrated that the integrated method effectively solves the multi-objective optimization problem, and obtains better and more diverse solutions than the conventional MOPSO method. With the DG penetration increasing from 0 to 200%, on average, an SOP reduces power losses by 58.4%, reduces the load balance index by 68.3% and reduces the voltage profile index by 62.1%, all compared to the case without an SOP. Comparisons between SOP and network reconfiguration showed the outperformance of SOP in operation optimization.

Volt-VAR control is critical to keeping distribution network voltages within allowable range, minimizing losses, and reducing wear and tear of voltage regulating devices. To deal with incomplete and inaccurate distribution network models, we propose a safe off-policy deep reinforcement learning algorithm to solve Volt-VAR control problems in a model-free manner. The Volt-VAR control problem is formulated as a constrained Markov decision process with discrete action space, and solved by our proposed constrained soft actor-critic algorithm. Our proposed reinforcement learning algorithm achieves scalability, sample efficiency, and constraint satisfaction by synergistically combining the merits of the maximum-entropy framework, the method of multiplier, a device-decoupled neural network structure, and an ordinal encoding scheme. Comprehensive numerical studies with the IEEE distribution test feeders show that our proposed algorithm outperforms the existing reinforcement learning algorithms and conventional optimization-based approaches on a large feeder.

With frequent occurrences of extreme natural disasters such as typhoons in urban integrated energy systems (UIES), it is of great significance to cope with different kinds of disturbances. This paper proposes a load restoration method under typhoon weather for urban distribution and natural gas networks based on soft open points (SOP). Firstly, the typhoon wind speed model is introduced and the line fault rates of distribution networks under typhoons are calculated. Secondly, the gas turbine and electric‐driven compressor are considered as the coupling units of the integrated electric–gas energy system, and related models are constructed. The fault analysis method of the natural gas network is proposed considering the faults in the distribution network lead by typhoons. Thirdly, SOP installed in the distribution network with the V/f control mode is applied to restore electrical loads and provide voltage support for the loads on the fault side. After that, the loads of the gas network could also be restored because of the restoration of the coupling units. Optimal energy flow is applied to determine the output of the power and gas sources, coupling units and also the loads to be restored. Finally, the fault rate of each line under typhoon disaster is analysed and the correctness and effectiveness of the resilience improving method based on SOP are verified with test systems UIES E33‐G14 and UIES E123‐G48.

We propose a computationally efficient approach to safe reinforcement learning (RL) for frequency regulation in power systems with high levels of variable renewable energy resources. The approach draws on set-theoretic control techniques to craft a neural network-based control policy that is guaranteed to satisfy safety-critical state constraints, without needing to solve a model predictive control or projection problem in real time. By exploiting the properties of robust controlled-invariant polytopes, we construct a novel, closed-form "safety-filter" that enables end-to-end safe learning using any policy gradient-based RL algorithm. We then apply the safety filter in conjunction with the deep deterministic policy gradient (DDPG) algorithm to regulate frequency in a modified 9-bus power system, and show that the learned policy is more cost-effective than robust linear feedback control techniques while maintaining the same safety guarantee. We also show that the proposed paradigm outperforms DDPG augmented with constraint violation penalties.

Dispatchable region for distributed renewable energy generation in reconfigurable AC–DC distribution networks with soft open points

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.