A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid

Summary In the process of waterflood development in oil and gas reservoirs, the injection/production ratio is one of the key indicators affecting both the development effectiveness and economic benefits. A reasonable injection/production ratio directly impacts the development parameters, such as reservoir pressure maintenance levels and water-cut rise rates, and it helps maintain high production capacity in the oil field, ultimately achieving a higher recovery factor. However, existing constrained-waterflood reservoir production optimization methods often face challenges such as time-consuming calculations and the risk of getting trapped in local optima when dealing with such nonlinear constraint problems, making it difficult to effectively achieve a balance between constraints and economic benefits. To address this issue, we propose an injection/production ratio–constrained production optimization method based on the worst-case soft actor-critic (WCSAC) algorithm. This method effectively handles the constraint issue by introducing a safety critic, ensuring the injection/production ratio constraint is met, and maximizing economic benefits throughout the entire life cycle. Specifically, this method models the injection/production ratio–constrained production optimization problem as a constrained Markov decision process (CMDP). On the one hand, it uses conditional value at risk (CVaR) as a safety metric to assess the safety risks of the policy and achieves a balance between reward and safety by adjusting adaptive safety weights. On the other hand, it uses the maximum entropy mechanism to enhance policy exploration and improve the efficiency of global optimization. This results in the training of a stable and reliable safe reinforcement learning (RL) agent to achieve injection/production ratio–constrained production optimization. The agent can adjust and determine the production optimization plan that satisfies the injection/production ratio constraint in real time based on the current reservoir development status, without the need for retraining, thus reducing computational time. To validate the performance of the proposed method, tests were conducted in two reservoir models. The analysis results indicate that, compared with other optimization methods, the proposed method can achieve a higher net present value (NPV) while satisfying the injection/production ratio constraints, demonstrating good model applicability.

Electric vehicle parking lot is a kind of profit service facilities nearby commercial office buildings for electric vehicles to park and charge. A V2P model is built to describe electric vehicles charging in the parking lot based on V2G (vehicle to grid) technology. Under the conditions of real-time pricing, improved particle swarm optimization (PSO) algorithm is applied to study on charging strategies of electric vehicles. Real-time electrovalency data of PJM and the parameters of mainstream electric vehicles are used in simulation. Results of the simulations show that the proposed model is reasonable and the improved PSO algorithm is both efficient and effective.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

With the interest from car owners in going green being at an all-time high, electric vehicles (EVs) are flooding the automobile market. One of the primary concerns in owning an EV is the availability of charging infrastructure while away from home. There has been a renewed interest in managing and pricing the usage of shared commercial EV chargers, while maximizing the operator's profits. Towards this end, we propose a combined pricing-scheduling quadratic integer programming (QIP) model that iteratively prices and schedules EV charging. A pricing module is used to accept/reject charging requests and control the right number and types (arrival-departure times, charge demand etc.) of EVs to charge. The scheduling module ensures that the demand can be met subject to price-demand sensitivity and other scheduling constraints. Once the EVs to be accepted have been finalized and their permit prices determined, the scheduling module can be run every night once the day-to-day arrival and departure times of each EV is revealed to the operator.

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

When a large-scale random charging load is connected to the regional power grid, it can negatively affect the safe and stable operation of the power grid. Therefore, we need to study its charging load and response potential in advance so that electric vehicles can interact well with the grid after being connected to the regional grid. Firstly, after analyzing the influencing factors of regional electric vehicle ownership, an electric vehicle ownership prediction model based on the sparrow search algorithm-improved BP neural network (SSA-BPNN) is established. On this basis, an electric vehicle charging load prediction model is established based on the sparrow search algorithm-improved BP neural network and Monte Carlo algorithm (SSA-BPNN-MC). Secondly, the charging behavior of different types of electric vehicles is analyzed and modeled, and the data from a certain area are taken as an example for the prediction. Then, according to the load forecasting results, the potential of electric vehicles participating in demand response in the region in the future is deeply analyzed using the scenario analysis method. Finally, with the aim of resolving the problems of massive multi-source heterogeneous data processing and the management of electric vehicles participating in the regional power grid demand response, a basic framework of electric vehicles participating in the regional power grid demand response is developed, which provides effective support for promoting electric vehicles to participate in regional grid demand response.

Electric vehicle (EV) batteries have a proven flexibility potential which could serve as an alternative to conventional electricity storage solutions. EV batteries could support the balancing of supply and demand, and the integration of variable renewable energy into the electricity system. The flexibility potential from EVs, in distinction to conventional battery storage, depends on the vehicle user’s willingness and opportunity to make their vehicle available for flexibility. This rate of participation is often not considered in modelling studies, despite the impact EV flexibility could have on the electricity system. This work presents a modelling study of the Norwegian electricity system using the high spatial and temporal resolution electricity system model (highRES). Two different charging strategies for EVs, flexible charging and vehicle-to-grid (V2G), are optimised in conjunction with the design and dispatch of the electricity system. By varying the rate of participation in the charging strategies we demonstrate their benefits for a future net-zero electricity system. Our findings show EVs’ potential to eliminate the need for stationary battery storage with just 50% participation in V2G. In addition, we find that the EV flexibility contributes to reductions in the total system cost by almost 4% and 15% assuming 100% participation in flexible charging and V2G, respectively.

Optimal dispatch of unbalanced distribution networks with phase-changing soft open points based on safe reinforcement learning

The vigorous development of electric vehicles (EVs) is an important means of reducing carbon emissions and mitigating environmental problems such as the greenhouse effect. Battery swapping stations (BSSs) can both provide battery swapping services for large-scale EVs and charge batteries centrally. As the supply of fully charged batteries in the BSS shrinks, it becomes necessary to schedule the charging of the depleted batteries rapidly that users have swapped for fully-charged ones. The charging schedule for depleted batteries must be made without knowledge of future battery arrivals. In this context, this paper develops a mathematical model for online charging scheduling of BSSs, formulates the charging strategy as a two-dimensional rectangle packing problem, and quickly calculates the scheduling arrangement of batteries by partitioning the remaining available capacity of a BSS. Since there are limited battery types within the BSS which can provide battery replacement services, this paper supplements the proposed model with known battery types, which improves the utilization of the available capacity of BSSs. Finally, numerical results verify the effectiveness of the proposed model. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Electric vehicles (EVs) are becoming an alternative way to reduce carbon emissions in transportation systems. Herein, the optimal battery charging problem is the core problem when it comes to dispatching a huge number of EVs. Up to now, battery-swapping is widely used for EVs due to its simple, convenient way. Furthermore, a business model for the battery swapping stations (BSSs) is brought up, where EV users send their depleted batteries to the BSS and the BSS provides the users with a fully charged replacement battery from its warehouse, which only takes a few minutes. Since the maximum charging power of the BSS is limited by the capacity of the transformer connecting the BSS to the power grid, the BSS will adopt an optimal charging schedule that maximizes the charging benefit for large quantities of depleted batteries in the warehouse. However, the challenge is that the charging schedule for depleted batteries must be made without knowledge of future battery arrivals because the EV behaviors are difficult to predict. To address this problem, this paper developed an online charging scheduling algorithm, which formulates the charging strategy as a two-dimensional rectangle packing problem. The proposed method can provide battery replacement services in real-time and solve quickly without any information about incoming depleted EV batteries. The proposed model and method have been tested on the system with different numbers of batteries to show the effectiveness. Besides, the online two-dimensional rectangle packing problem can provide an online decision for BSSs.

Renewable energy and electric vehicle technology are the two pillars for achieving a sustainable future. Floating solar power plants use PV modules on water infrastructure to save the land and increase module efficiency. Furthermore, the reduction in evaporation saves water. Electric vehicles are one of the fastest-growing markets and the most successful technologies to combat the problem of energy and climate change. This research aims to construct a floating PV system on the lake of the Vellore Institute of Technology (VIT), to analyze electric vehicle performance and greenhouse gas (GHG) emissions when charged using the installed floating PV system. To address this, a 1.5 MWP floating PV system was simulated and analyzed using Helioscope software. When charged from the proposed floating PV plant, electric bikes, scooters, and cars saved CO2 emissions. When charged from a floating PV, E-bike, E-scooter, and E-car Net CO2 emissions became zero in 25.5, 12.1, and 7.7 months, respectively. After the aforementioned time periods, all three electric vehicle types were zero-emission vehicles. The required charge for all three types of vehicles (1,000,000 km) was analyzed using a floating PV system. E-bike, E-scooter, and E-car CO2 emission savings were −8,516,000 g/kWh, −328,000 g/kWh, and 525,600,000 g/kWh, respectively. All three types of electric vehicles can reduce CO2 emissions for nations that rely on renewable energy, but only electric cars save carbon emissions over fixed distances. Through this research, we finally conclude that electric cars reduce CO2 emissions the most compared to other electric vehicles.

With the development of intelligent distribution networks and the proposal of the carbon peaking and carbon neutrality goals, the importance of demand‐side management for the improvement of flexible power operation systems is becoming increasingly prominent. To solve the problems of the excessive peak–valley load differences, the insufficient utilization of demand‐side resources, and the unreasonable pricing of aggregators, this paper proposes a three‐stage economy optimization method for the aggregator based on EV(electric vehicle) user response volumes. To gain the advantages of different types of EVs while increasing the dispatchable capacity, the contract mode between the aggregator and EVs is divided into three categories: complete dispatching, rolling reward and punishment mechanism dispatching, and free dispatching. Based on the cloud model, considering loss aversion, the user's own time flexibility, and the impact of the tariff set by the aggregator on the user's decision‐making, we introduced indicator weights to obtain the improved cloud model. Based on the improved cloud model, a user response volume model is obtained. Then, the aggregator performs a three‐stage pricing optimization operation for EVs based on the bid‐winning peak shaving capacity. In the first stage, the attraction between EVs and charging stations is determined based on the law of gravity, and the charging and discharging reward electricity price is set. In the second stage, the dynamic electricity price of three types of EVs and the charging punishment electricity prices of the rolling reward and punishment mechanism dispatching type are determined. In the third stage, the electricity price of the rolling reward and punishment dispatching EVs is further optimized based on their lack of peak shaving capacity. The method proposed in the paper has been verified to be effective in increasing aggregator profits, increasing EV dispatch capacity, and reducing EV charging costs through the example analysis.

Capabilities of compressed air energy storage in the economic design of renewable off-grid system to supply electricity and heat costumers and smart charging-based electric vehicles

In recent years, Electric vehicles (EVs) are receiving significant attention as an environmental-sustainable and cost-effective substitute of vehicles with internal combustion engine, for the solution of the dependence from fossil fuels and for the saving of Green-House Gasses emission The present paper deals with an overview on different types of EVs charging stations and a comparison between the related European and American Standards. The work includes also a summary on possible types of Energy Storage Systems (ESSs), that are important for the integration of EVs fast charging stations of the last generation in smart grids. Finally a brief analysis on the possible electrical layout for the ESS integration in EVs charging system, proposed in literature, is reported.

Intelligent Transport System (ITS) intentions to attain traffic efficiency by diminishing traffic difficulties. It supplies information like traffic issues, real-time traveling information, parking availability, etc., in advance to the users who are connected with the smart cities that ensure travelers’ safety and comfort. This ITS technique should merge with Electric Vehicles (EVs) because nowadays, EVs have become familiar in the last decade owing to the requirement to cut greenhouse gas emissions and fossil fuels. However, traffic jams caused by EVs driven to the charging stations (CSs) can result in the complex charging scheduling of EVs. Therefore, an effective algorithm is developed for optimal charging scheduling using the proposed Grey Sail Fish Optimization (GSFO). The proposed charging scheduling algorithm integrates Grey Wolf Optimizer (GWO) and Sail Fish Optimization (SFO). For each EV, the demand when charging is computed. The path used by the EV to travel to the charging station is determined by computing the path decision factor. In comparison to existing techniques, the proposed GSFO-based charging algorithm schedules EVs to charging stations based on the fitness function, and the performance was improved with a traffic density of 26.11 km, a distance of 0.0278 kW, and a power of 2.3377. To be more specific, the proposed GSFO improved when many vehicles were considered.

The rapid expansion of electric vehicles (EVs) brings forth a range of possibilities and obstacles for the electrical grid. EVs are a hopeful way to cut down on greenhouse gas emissions and reliance on fossil fuels, but the way they charge can put a lot of stress on the grid, which can cause problems for both grid operators and EV users. The uncontrolled approach to EV charging could result in the development of new peak loads. This paper formulates a problem for the minimisation of EV charging cost while reducing the load variance using a dynamic charging strategy. This dynamic charging strategy forecasts the charging price of EV for each time stamp based on the available energy generation. The proposed formulation has also considered the seasonal variation in load profile. The model is solved using different optimisation algorithm and different scenarios have been formulated to validate the effectiveness of the optimisation problem which has been formulated to achieve the objectives of minimised charging cost and load variance. The proposed strategy has improved the charging cost by 67% from other strategies of EV charging while reduction in load variance comes out to be 69%. This proposed formulation will help in the higher adoption of EV's and will subsequently lead to sustainable growth.