Proximal policy optimization via enhanced exploration efficiency

Implementing action mask in proximal policy optimization (PPO) algorithm

Distributed multi-agent collaborative decision-making technology is the key to general artificial intelligence. This paper takes the self-developed Unity3D collaborative combat environment as the test scenario, setting a task that requires heterogeneous unmanned aerial vehicles (UAVs) to perform a distributed decision-making and complete cooperation task. Aiming at the problem of the traditional proximal policy optimization (PPO) algorithm’s poor performance in the field of complex multi-agent collaboration scenarios based on the distributed training framework Ray, the Critic network in the PPO algorithm is improved to learn a centralized value function, and the muti-agent proximal policy optimization (MAPPO) algorithm is proposed. At the same time, the inheritance training method based on course learning is adopted to improve the generalization performance of the algorithm. In the experiment, MAPPO can obtain the highest average accumulate reward compared with other algorithms and can complete the task goal with the fewest steps after convergence, which fully demonstrates that the MAPPO algorithm outperforms the state-of-the-art.

Currently, helicopter towing operations mainly rely on towing systems and multi-person coordinated control, resulting in low work efficiency and posing a threat to life safety. With the development of artificial intelligence and computer science, automating helicopter towing systems is key to achieving safe and efficient helicopter operations. This paper proposes a helicopter tracking control method based on the Proximal Policy Optimization (PPO) algorithm for the helicopter towing trajectory tracking problem. This paper first improves the PPO algorithm network structure and sets the hyperparameters of the algorithm. Then, a deep learning environment is constructed based on the Markov decision process, and the state space and action space are reasonably set. A reward function is designed based on lateral deviation, longitudinal deviation, and heading deviation. Finally, the PPO algorithm is used to train the trajectory tracking control strategy, and the optimal tracking control strategy is obtained and verified by simulation. The simulation results show that the helicopter tracking control strategy based on the PPO algorithm has a fast convergence speed, and the tracking effect verifies the feasibility and generalization of the algorithm.

This study applied deep reinforcement learning (DRL) with the Proximal Policy Optimization (PPO) algorithm within a two-dimensional computational fluid dynamics (CFD) model to achieve closed-loop control in microfluidics. The objective was to achieve the desired droplet size with minimal variability in a microfluidic capillary flow-focusing device. An artificial neural network was utilized to map sensing signals (flow pressure and droplet size) to control actions (continuous phase inlet pressure). To validate the numerical model, simulation results were compared with experimental data, which demonstrated a good agreement with errors below 11%. The PPO algorithm effectively controlled droplet size across various targets (50, 60, 70, and 80 μm) with different levels of precision. The optimized DRL + CFD framework successfully achieved droplet size control within a coefficient of variation (CV%) below 5% for all targets, outperforming the case without control. Furthermore, the adaptability of the PPO agent to external disturbances was extensively evaluated. By subjecting the system to sinusoidal mechanical vibrations with frequencies ranging from 10 Hz to 10 KHz and amplitudes between 50 and 500 Pa, the PPO algorithm demonstrated efficacy in handling disturbances within limits, highlighting its robustness. Overall, this study showcased the implementation of the DRL+CFD framework for designing and investigating novel control algorithms, advancing the field of droplet microfluidics control research.

To address the resource allocation problem in dynamic environments where multiple unmanned aerial vehicle base stations (UAV-BSs) provide efficient downlink services to ground users, this paper proposes a novel hierarchical decision-making mechanism based on the Proximal Policy Optimization (PPO) algorithm. The proposed method optimizes time-frequency resource allocation in the downlink, aiming to maximize the total user throughput over multiple time slots. By constructing channel and interference models, the complex multi-channel resource allocation problem is decomposed into a series of single-channel decision subproblems, significantly reducing the action space complexity. Specifically, the original exponential complexity O(NM) (where N is the number of users and M is the number of channels) is reduced to a linear complexity O(N), effectively alleviating the curse of dimensionality. Simulation results demonstrate that the proposed hierarchical architecture, integrated with the PPO algorithm, achieves superior performance in terms of total throughput, convergence speed, and stability compared to existing methods. This study provides new insights and technical support for efficient resource management in UAV-BS systems operating in complex and dynamic environments.

Reinforcement learning-based control with application to the once-through steam generator system

As the penetration rate of new energy continues to increase, the active voltage support capability of the power system is decreasing. The grid-forming static var generator (GFM-SVG) features the advantages of fast dynamic response, strong reactive power support, and high overload capacity, which play an important role in maintaining voltage stability. However, the parameters of the GFM-SVG are often unknown due to trade secret reasons. Meanwhile, the parameters may be changed during the long-term operation of the system, which brings challenges to the system stability analysis and control. Aiming at this problem, a parameter identification method based on trajectory sensitivity analysis and the proximal policy optimization (PPO) algorithm is proposed in this paper. Firstly, through trajectory sensitivity analysis, the key influential parameters on the output characteristics of the GFM-SVG can be selected, which can reduce the dimensionality of the identification parameters and improve the identification efficiency. Then, a parameter identification framework based on the PPO algorithm is constructed for GFM-SVGs, which utilizes its adaptive learning capability to achieve accurate identification of the key parameters of the system. Finally, the effectiveness of the proposed parameter identification method is verified through simulation examples. The simulation results show that the identification error of the parameters in the GFM-SVG is small. The proposed method can characterize the output response of the GFM-SVG under different operating conditions.

Industry 5.0 promotes the transformation of manufacturing toward flexibility, personalization, and sustainability. As a critical component of closed-loop manufacturing systems, disassembly operations urgently require more flexible and efficient human–robot collaboration models. To this end, this work, for the first time, proposes a multihuman–robot collaborative circular disassembly line balancing problem. By allowing workers to move between robotic workstations, the proposed system enhances operational flexibility. Furthermore, a multiworker mechanism is introduced to improve fault tolerance and system stability, overcoming the limitations of fixed worker positions in existing collaborative disassembly research. To solve this problem, we formulate a discrete-time mixed-integer programming model based on product AND/OR graphs, aiming to maximize disassembly profit. The model’s correctness is verified using CPLEX. Additionally, we develop a heterogeneous graph neural network-enhanced proximal policy optimization (PPO) algorithm. By integrating product and workstation information into a heterogeneous graph, the algorithm performs two-stage feature extraction and node embedding via graph neural networks. Based on these embeddings, the agent dynamically selects multiple actions per decision step to simulate the behavior of multiple workers moving simultaneously. Experimental results show that the proposed method outperforms traditional reinforcement learning algorithms such as PPO and deep Q-network algorithm in terms of disassembly profit. Moreover, it demonstrates strong generalization capability in cross-task transfer and scalability experiments involving different task graph sizes. The improved performance is achieved with acceptable computational time.

With the rapid development of renewable energy, power systems face increased flexibility and dispatch requirements, particularly in response to grid load fluctuations and energy reserve security. Pumped hydro storage, a mature energy storage technology, is widely used in power regulation and reserve capacity provision, but its dispatching still presents significant optimization challenges. Traditional dispatching methods struggle to simultaneously meet multi-objective optimization requirements. Therefore, this study proposes a dispatching optimization method based on Proximal Policy Optimization (PPO) combined with Multi-Objective Optimization (MOO) to maximize profit, ensure sufficient reserve capacity, and safeguard grid stability. We construct a multi-objective optimization model for pumped hydro storage and combine it with the Proximal Policy Optimization (PPO) algorithm for dynamic dispatch optimization. A multi-objective reward function is used to weightedly optimize profit, reserve capacity, and grid stability. Experimental results demonstrate that the PPO + MOO method significantly outperforms traditional methods (such as piecewise linear approximation, genetic algorithm, and particle swarm optimization) across multiple evaluation metrics. It effectively improves dispatching accuracy and system stability, particularly in complex power markets and under conditions of large load fluctuations. Furthermore, the PPO algorithm demonstrates excellent convergence and stability during training, ensuring continuous model optimization in dynamic environments. Although computational time is relatively long, optimization techniques such as weight pruning have significantly improved computational efficiency. Ultimately, this study provides a new solution for the scheduling optimization of pumped hydro energy storage systems and provides an important reference for scheduling problems in smart grids and renewable energy systems.

In this paper, we apply the Proximal Policy Optimization (PPO) algorithm in intelligent traffic signal control at a single intersection with eight lanes and four signal phases. The optimization goal is to minimize the average waiting time of vehicles so as to improve the traffic efficiency of the intersection. Extensive experiments are conducted in Simulation of Urban MObility (SUMO) to evaluate the performance of the proposed algorithm, and compare it with other classic algorithms including Deep Q-network (DQN), Advantage Actor Critic (A2C) and Fixed Time. Simulation results show that the proposed PPO algorithm outperforms the others under various traffic scenarios to different extent. The performance gain is significant under unbalanced traffic where one direction is saturated while the other is not, and becomes marginal when all the directions are saturated or unsaturated. PPO also demonstrates good portability and robustness over time-varying traffic patterns, while implies it could be a preferable option for implementation in real world intelligent traffic signal control systems.

The fluctuations of renewable energies and loads pose a great challenge to reactive power regulation. Considering the time-varying characteristics of new energies and loads, the multi-time reactive power regulation problem is constructed as a reinforcement learning problem. A constraint-objective partitioning and objective preconditioning approach is proposed to design the reward function, and a proximal policy optimization algorithm is used to solve the reinforcement learning problem and obtain the reactive power regulation policy. A case study is carried out with the improved IEEE39 system, and the results show that the proposed reward function can improve the convergence speed of the agent, and the reactive power regulation strategy solved based on reinforcement learning outperforms the traditional deterministic optimization algorithm in terms of decision effect and decision time.

With the gradual development of sports data analysis, data-driven game prediction has gradually become an important tool for improving the decision-making efficiency of teams and coaches. Basketball, as a team sport, is influenced by multiple factors, and traditional statistical analysis methods make it difficult to intuitively identify the factors that have the greatest impact on the game. This study proposes a reinforcement learning model based on the Proximal Policy Optimization (PPO) algorithm for predicting the winning rate of NBA games. By collecting career statistics of NBA players and combining them with the team's current season winning rate, a feature vector containing individual player characteristics and team winning rate is constructed. The model is trained using the PPO algorithm to identify the most critical features that affect the winning rate of games. This study not only provides new ideas for sports event prediction, but also provides solutions for future decision-making and dataization in big data environments.

Tuning of fuzzy controller with arbitrary triangular input fuzzy sets based on proximal policy optimization for time-delays system

Reinforcement learning based fractional fuzzy controller for photovoltaic systems

In the face of a complex three-dimensional environment, the computational complexity of the traditional path planning algorithms is extremely increased. But the performance of path planning results is relatively poor. Deep reinforcement learning (DRL) algorithms don’t rely on accurate environmental models, and its overall efficiency is much higher than traditional algorithms. Aiming at the problem of AUV path planning in a three-dimensional environment, proximal policy optimization algorithm (PPO) is selected. The PPO algorithm with improved network structure is used to train the model based on the established AUV model and gym environment. Through simulation experiments, the accuracy and effectiveness of the algorithm is verified.