A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning

The growing demand for low delay vehicular content has put tremendous strain on the backbone network. As a promising alternative, cooperative content caching among different cache nodes can reduce content access delay. However, heterogeneous cache nodes have different communication modes and limited caching capacities. In addition, the high mobility of vehicles renders the more complicated caching environment. Therefore, performing efficient cooperative caching becomes a key issue. In this paper, we propose a cross-tier cooperative caching architecture for all contents, which allows the distributed cache nodes to cooperate. Then, we devise the communication link and content caching model to facilitate timely content delivery. Aiming at minimizing transmission delay and cache cost, an optimization problem is formulated. Furthermore, we use a multi-agent deep reinforcement learning (MADRL) approach to model the decision-making process for caching among heterogeneous cache nodes, where each agent interacts with the environment collectively, receives observations yet a common reward, and learns its own optimal policy. Extensive simulations validate that the MADRL approach can enhance hit ratio while reducing transmission delay and cache cost.

In mixed-mode operation, arrivals and departures are allowed to land and depart on the same runway. An appropriate strategy from air traffic controllers for arrivals and departures sequencing would boost the runway through-put significantly. On the other hand, safety is still the most crucial feature in the operation. Therefore, to assist air traffic controllers to make decisions on departures and arrivals with efficient utilization of runway capacity and safe operations, this paper proposed a Multi-agent Deep Reinforcement Learning approach using Multi-agent Deep Deterministic Policy Gradient to train two agents simultaneously: departure agent, and arrival agent. The departure agent makes departure slotting decisions for departures while the arrival agent determines the time delay or spacing decision on the arrival stream. A data-driven simulation environment is developed using Singapore Changi Airport data to support the learning process. Besides, a random sampling technique is also introduced to reduce redundant samples and increase off-policy sample efficiency. Moreover, the impact of different reward functions on runway throughput is also investigated and two specific models, e.g., ‘arrival priority’ and ‘departure priority’, are selected for further analysis in this study. As the result, by comparing the trained models with the ad-hoc model, the proposed approach increases the runway throughput significantly, with the highest 12.8% additional departure releases and overall 5.3% additional departure releases in identical environments while safety separations are maintained.

Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management

The ongoing quest for transportation electrification with the massive proliferation of EV charging stations (EVCSs) will deepen the interaction and require the further coordination of coupled power and transportation networks (PTN). The individually-owned EVCSs located in an urban transportation network (UTN) will compete using price signals to maximize their respective payoffs. In this paper, a multi-agent deep reinforcement learning (MA-DRL) method is proposed to model the pricing game in UTN and determine the optimal charging prices for a single EVCS. The EVCS charging demand is first analyzed using a modified user equilibrium traffic assignment problem (UE-TAP) with elastic traveling demands and different charging prices. The price competition problem is then formulated as a game with incomplete information in which the market environment is complex due to nonlinear traffic assignments. Thus, the MA-DRL approach is proposed to learn the charging pricing strategies of multiple EVCSs and approximate the Nash Equilibrium (NE) of the pricing game using the incomplete information. The proposed solution will determine the optimal pricing strategies for an EVCS in UTN. The case studies on a 24-node Sioux-Falls network, and the real-world Xi'an and Hangzhou cities are conducted to verify the effectiveness and scalability of the proposed approach.

Toward social equity access and mobile charging stations for electric vehicles: A case study in Los Angeles

Electric vehicles (EVs) are a sustainable transportation solution with environmental benefits and energy efficiency. However, their popularity has raised challenges in locating appropriate charging stations, especially in cities with limited infrastructure and dynamic charging demands. To address this, we propose a multi-agent deep deterministic policy gradient (MADDPG) method for optimal EV charging station recommendations, considering real-time traffic conditions. Our approach aims to minimize total travel time in a stochastic environment for efficient smart transportation management. We adopt a centralized learning and decentralized execution strategy, treating each region of charging stations as an individual agent. Agents cooperate to recommend optimal charging stations based on various incentive functions and competitive contexts. The problem is modeled as a Markov game, suitable for analyzing multi-agent decisions in stochastic environments. Intelligent transportation systems provide us with traffic information, and each charging station feeds relevant data to the agents. Our MADDPG method is challenged with a substantial number of EV requests, enabling efficient handling of dynamic charging demands. Simulation experiments compare our method with DDPG and deterministic approaches, considering different distributions and EV numbers. The results highlight MADDPG’s superiority, emphasizing its value for sustainable urban mobility and efficient EV charging station scheduling.

Global warming and some dangerous climate changes are becoming more prevalent as the demand for modern transportation systems grows for economic development and cultural comfort. To tackle this global warming issue due to transportation every country pushing for Electric Vehicles (EVs), As the number of electric vehicles on the road rises, Charging EVs with a fossil fuel-based infrastructure alone is not cost-effective or efficient. As a result, a charging station based on renewable energy has enormous potential and control for electric vehicle charging. In the current scenario, a solar-powered electric vehicle charging station and a Battery Energy Storage System (BESS) are required. Additional grid assistance is recommended to ensure that the charging station has uninterrupted power without putting additional burden on the grid, For effective power management in the charging station between solar, BESS, grid, and EVs an efficient charging station design with Adaptive Neuro-Fuzzy Inference System (ANFIS) voltage-controlled MPPT, PID controller, Grid with Neural Network technique is designed and evaluated in MATLAB/Simulink.

Efficient operation of battery energy storage systems, electric-vehicle charging stations and renewable energy sources linked to distribution systems

This paper aims to tackle the complex challenge of channel assignment and joint power-energy allocation within a cellular-vehicle-to-everything (C-V2X) network, which is deployed to manage vehicular dynamics at an urban traffic intersection. The primary function of the C-V2X network is to facilitate the coordination of multiple vehicle platoons formed by closely spaced same-lane vehicles. This coordination involves two critical communication tasks including the timely update of platoon states to a roadside unit (RSU) and the reliable exchange of cooperative awareness messages (CAMs) among vehicles within the same platoon. The main objective of this paper is to minimise the average age of information (AoI) to ensure the timely update between vehicle platoons and the RSU, maximise the CAM delivery probability (CDP) to guarantee the successful exchange of CAMs among vehicles, and promote sustainable, green communication practices through the implementation of our optimal power-energy management strategies. Recognising the intricate and dynamic nature of this challenge, we adopt a multi-agent deep reinforcement learning (MADRL) approach based on the Markov decision process (MDP). Two innovative algorithms based on the multi-agent deep deterministic policy gradient (MADDPG) and twin delayed deep deterministic policy gradient (TD3) algorithms are proposed to address this optimisation problem effectively. Finally, comprehensive simulation results are presented, which demonstrate the remarkable performance of our proposed schemes, particularly in terms of energy efficiency when compared to existing research. Importantly, these gains in energy efficiency are achieved while maintaining competitive algorithm convergence speeds, low AoI levels, and high CDP, showcasing the practical viability of the developed methods.

PASCAL: PopulAtion-Specific Curriculum-based MADRL for collision-free flocking with large-scale fixed-wing UAV swarms

Optimal deadline scheduling for electric vehicle charging with energy storage and random supply

With the advantages of real-time data processing and flexible deployment, unmanned aerial vehicle (UAV)-assisted mobile edge computing systems are widely used in both civil and military fields. However, due to limited energy, it is usually difficult for UAVs to stay in the air for long periods and to perform computational tasks. In this paper, we propose a full-duplex air-to-air communication system (A2ACS) model combining mobile edge computing and wireless power transfer technologies, aiming to effectively reduce the computational latency and energy consumption of UAVs, while ensuring that the UAVs do not interrupt the mission or leave the work area due to insufficient energy. In this system, UAVs collect energy from external air-edge energy servers (AEESs) to power onboard batteries and offload computational tasks to AEESs to reduce latency. To optimize the system’s performance and balance the four objectives, including the system throughput, the number of low-power alarms of UAVs, the total energy received by UAVs and the energy consumption of AEESs, we develop a multi-objective optimization framework. Considering that AEESs require rapid decision-making in a dynamic environment, an algorithm based on multi-agent deep deterministic policy gradient (MADDPG) is proposed, to optimize the AEESs’ service location and to control the power of energy transfer. While training, the agents learn the optimal policy given the optimization weight conditions. Furthermore, we adopt the K-means algorithm to determine the association between AEESs and UAVs to ensure fairness. Simulated experiment results show that the proposed MODDPG (multi-objective DDPG) algorithm has better performance than the baseline algorithms, such as the genetic algorithm and other deep reinforcement learning algorithms.

The provision of electric vehicle (EV) charging at airports is a very new and evolving practice. EV charging technology and the costs associated with providing EV charging stations are still evolving and there are still many unknowns, conflicting information, and speculation about this new practice. Purchases of EVs are increasing throughout the United States, with the West Coast states accounting for a disproportionately high number. Research undertaken for this report has shown that at least 37 airports in the United States are providing EV charging stations, most commonly in short-term or long-term parking facilities, and usually with no fee for charging. The report presents a primer on EV charging and includes information to understand the policy approaches, infrastructure needs, and funding mechanisms that airports have used in EV hosting. This report (1) summarizes the state of the EV industry and motivations for airport sponsors to provide EV charging stations, (2) identifies current EV charging technologies, and (3) describes effective practices for installing EV charging stations at airports in terms of finances, planning, implementation, and operation of the charging stations. This report is intended to help airport management staff, specifically those environmental, parking, and facilities managers who may be interested in installed EV charging stations.

This paper presents a modern yearly based park-and-ride management scheme. The electric vehicles’ owners are encouraged to keep their cars away from the crowded areas in cities and use the public facilities such as bus, train, and metro. This action will help the owners to reach their work on time inside these crowded cities. Electric vehicle charging stations are designed to charge 1000 electric vehicles using the proposed park-and-ride control approach. A case study of Fuxin, China is considered. The electric vehicle charging stations demand is met using renewable energy sources, namely photovoltaic and battery energy storage systems. Meeting the load demand and minimizing the total life cycle cost are considered two objective functions to formulate a multiobjective approach. The optimal sizes of the photovoltaic and battery energy storage systems are obtained using a multiobjective genetic algorithm and ε-MOGA. The robustness and effectiveness of the proposed control methodology are verified by detailed analysis and comparison using MATLAB®.

Dynamic pricing, which aims to dynamically adjust the charging price in a timely fashion to unlock the flexibility of electric vehicle (EV) customers, has been extensively studied with the rapid development of charging technologies. Many existing works on dynamic pricing focus on maximizing the social welfare of charging service providers and EV customers. Cases of high-dimensional charging environments, which are often encountered with the rapid growth of EV market penetration, have been rarely considered to date. This article proposes a new dynamic pricing framework for EV charging stations that can offer multiple charging options to customers over a finite-time horizon. The charging price can be dynamically adjusted to maximize the quality of service (QoS) with a differentiated service requirement level (SRL) whenever the arrival rates and queuing system capacities of the charging systems are given at the end of a time period. The dynamic pricing problem is formulated as a finite-discrete horizon Markov decision process (MDP) with a mixed state space. A customized deep reinforcement learning (DRL) approach is employed to solve the examined EV dynamic pricing problem. The simulation results demonstrate the effectiveness of the proposed method.