Accelerating Proximal Policy Optimization on CPU-FPGA Heterogeneous Platforms

Reinforcement Learning (RL) is a major branch of AI that enables agents to learn optimal decision making via interaction with the environment. Proximal Policy Optimization (PPO) is the state-of-the-art policy optimization based RL algorithm which achieves superior overall performance on various benchmarks. A PPO agent iteratively optimizes its policy - a function which chooses optimal actions approximated by a DNN, with each iteration consisting of two computationally intensive phases: Sample Generation - where agents inference on its policy and interact with the environment to collect data, and Model Update - where the policy is trained using the collected data. In this paper, we develop the first high-throughput PPO accelerator on CPU-FPGA heterogeneous platform. Our unified systolic-array based design accelerates both the inference and the training of the deep neural network used in a RL algorithm, and is generalizable to various MLP and CNN models across a wide range of RL applications. We develop novel optimizations to simultaneously reduce data access and computation latencies, specifically: (a) optimal data flow mapping to systolic array, (b) novel memory-blocked data layout to enable streaming stall-free data access in both forward and backward propagations, and, (c) a systolic array compute sharing technique to mitigate load imbalance in the training of two networks. We evaluate our design on widely used robotics and gaming benchmarks, achieving 1.4×–26× and 1.3×–2.7× improvements in throughput, respectively, when compared with state-of-the-art CPU/CPU-GPU implementations.

Implementing action mask in proximal policy optimization (PPO) algorithm

The application of reinforcement learning algorithms to motion planning is a research hotspot in robotics in recent years. However, training reinforcement learning agents from scratch has low training efficiency and difficulty in convergence. In this paper, a robot motion planning method based on residual reinforcement learning is proposed. This method divides the agent's policy of motion planning into initial policy and residual policy. The initial policy is composed of a neural network motion planner responsible for guiding the training direction of residual policy. The residual policy is composed of Proximal Policy Optimization (PPO) algorithm in reinforcement learning. A motion planning experiment is carried out in a simulation environment, and the result shows that the method can successfully perform motion planning. The comparison experiment between PPO and the proposed algorithm demonstrates that the proposed algorithm has better motion planning performance.

Wireless Power Transfer (WPT) for Electric Vehicles (EVs) offers a promising solution for convenient and efficient charging. However, misalignments, sensor noise, and parameter variability can significantly degrade Power Transfer Efficiency (PTE). This study proposes a novel unified artificial intelligence (AI)-driven optimization and control framework that integrates Genetic Algorithm (GA)-based static optimization, Artificial Neural Network (ANN) surrogate modeling, and Reinforcement Learning (RL) dynamic control using the Proximal Policy Optimization (PPO) algorithm. This unified design bridges the gap between previous static-only optimization methods and dynamic adaptive controllers, enabling both peak efficiency and verified robustness within a single digital twin simulation environment. A high-fidelity MATLAB/Simulink model of the WPT system was developed and validated using an ANN surrogate model (Test MSE: 7.87×10−13). The GA-optimized configuration achieved a peak PTE of 85.47%, representing a 2.11 percentage-point improvement over the baseline. The RL controller, based on PPO, maintained a mean efficiency of approximately 80% under unseen trajectories, ±10% hardware parameter variations, and Gaussian sensor noise (σ=0.56%), demonstrating superior adaptability. Comparative analysis with state-of-the-art studies confirms that the proposed approach not only matches or exceeds the reported efficiency gains, but also uniquely integrates robustness validation and generalization testing. The results suggest that combining offline GA optimization with online RL adaptation provides a scalable, real-time control strategy for practical WPT deployments.

Proximal policy optimization via enhanced exploration efficiency

The rapid advancement of autonomous robots in domains such as healthcare, transportation, and disaster response necessitates robust ethical decision-making frameworks to mitigate risks associated with uncertain environments. This research delves deeply into the application of reinforcement learning (RL) for enabling "safe" decisions in robots facing multiple risky outcomes. By synthesizing over 80 peer- reviewed sources, including recent 2024-2025 surveys on ethical RL and safe RL benchmarks, it systematically addresses three pivotal research questions: (1) How should a robot choose between two risky paths? (2) Can RL augmented with reward shaping effectively reduce harm in unpredictable environments? (3) How do prominent RL algorithms compare in terms of safety efficacy in robotic applications? Key findings indicate that multi-objective RL (MORL) and constrained policy optimization variants, such as Proximal Policy Optimization (PPO) with Lagrangian constraints, outperform traditional RL in balancing ethical imperatives with task performance, achieving up to 40% reductions in violation rates in simulated benchmarks. Reward shaping, particularly potential-based and ethics-informed variants, accelerates ethical convergence by 3-6x while curbing reward hacking—a common pitfall where agents exploit reward loopholes. A novel student-led simulation in a custom gridworld environment demonstrates that shaped rewards enhance average returns by approximately 12% with near-zero harm incidents over 1,000 episodes. This expanded study proposes an integrated Ethical Safe RL (ESRL) framework, incorporating human-in-the-loop feedback and verifiable constraints, tailored for accessible implementation in educational settings. It underscores persistent challenges like distributional shifts in real-world deployment and calls for interdisciplinary ethics integration. By bridging theoretical surveys with empirical simulations, this work empowers high-school researchers to contribute to responsible AI robotics.

Automated source code refactoring, particularly extract method refactoring, is a crucial and frequently employed technique during software development. Despite its importance and frequent use by practitioners, current automated techniques face significant limitations such as the lack of automation. While machine learning-based approaches have shown promise in intelligent code refactoring, existing such approaches overlook code-specific sequence-level characteristics, including but not limited to compilability, syntactic correctness, and functional integrity. To address these challenges, we propose a novel reinforcement learning-based approach for fine-tuning and aligning code language models to perform automated, intelligent extract method refactoring on Java source code. Our approach fine-tunes sequence-to-sequence generative models and aligns them using the Proximal Policy Optimization (PPO) algorithm using code compilation and presence of the refactoring in the generated code as reward signals. Our experiments demonstrate that our approach significantly enhances the performance of language models in code refactoring. The supervised fine-tuned model, further aligned with PPO, surpasses traditional supervised fine-tuning by 11.96% and 16.45% in terms of BLEU and CodeBLEU scores, respectively. When subjected to a suite of 122 unit tests, the number of successful tests increased from 41 to 66 for the reinforcement learning aligned fine tuned Code-T5 model, highlighting the effectiveness of our approach in producing functionally correct refactorings. Our work paves the way for intelligent, automated code refactoring tools that can significantly reduce developers’ manual effort.

This paper proposes a reinforcement learning (RL) based path following strategy for underactuated airships with magnitude and rate saturation. The Markov decision process (MDP) model for the control problem is established. Then an error bounded line-of-sight (LOS) guidance law is investigated to restrain the state space. Subsequently, a proximal policy optimization (PPO) algorithm is employed to approximate the optimal action policy through trial and error. Since the optimal action policy is generated from the action space, the magnitude and rate saturation can be avoided. The simulation results, involving circular, general, broken-line, and anti-wind path following tasks, demonstrate that the proposed control scheme can transfer to new tasks without adaptation, and possesses satisfying real-time performance and robustness.

Obstacle avoidance during high-speed, low-altitude flight remains a significant challenge for unmanned aerial vehicles (UAVs), particularly in unfamiliar environments where prior maps and heavy onboard sensors are unavailable. To address this, we present an entropy-aware deep reinforcement learning framework that enables fixed-wing UAVs to navigate safely using only monocular onboard cameras. Our system features a lightweight, single-frame depth estimation module optimized for real-time execution on edge computing platforms, followed by a reinforcement learning controller equipped with a novel reward function that balances goal-reaching performance with path smoothness under fixed-wing dynamic constraints. To enhance policy optimization, we incorporate high-quality experiences from the replay buffer into the gradient computation, introducing a soft imitation mechanism that encourages the agent to align its behavior with previously successful actions. To further balance exploration and exploitation, we integrate an adaptive entropy regularization mechanism into the Proximal Policy Optimization (PPO) algorithm. This module dynamically adjusts policy entropy during training, leading to improved stability, faster convergence, and better generalization to unseen scenarios. Extensive software-in-the-loop (SITL) and hardware-in-the-loop (HITL) experiments demonstrate that our approach outperforms baseline methods in obstacle avoidance success rate and path quality, while remaining lightweight and deployable on resource-constrained aerial platforms.

With the rapid development of renewable energy, power systems face increased flexibility and dispatch requirements, particularly in response to grid load fluctuations and energy reserve security. Pumped hydro storage, a mature energy storage technology, is widely used in power regulation and reserve capacity provision, but its dispatching still presents significant optimization challenges. Traditional dispatching methods struggle to simultaneously meet multi-objective optimization requirements. Therefore, this study proposes a dispatching optimization method based on Proximal Policy Optimization (PPO) combined with Multi-Objective Optimization (MOO) to maximize profit, ensure sufficient reserve capacity, and safeguard grid stability. We construct a multi-objective optimization model for pumped hydro storage and combine it with the Proximal Policy Optimization (PPO) algorithm for dynamic dispatch optimization. A multi-objective reward function is used to weightedly optimize profit, reserve capacity, and grid stability. Experimental results demonstrate that the PPO + MOO method significantly outperforms traditional methods (such as piecewise linear approximation, genetic algorithm, and particle swarm optimization) across multiple evaluation metrics. It effectively improves dispatching accuracy and system stability, particularly in complex power markets and under conditions of large load fluctuations. Furthermore, the PPO algorithm demonstrates excellent convergence and stability during training, ensuring continuous model optimization in dynamic environments. Although computational time is relatively long, optimization techniques such as weight pruning have significantly improved computational efficiency. Ultimately, this study provides a new solution for the scheduling optimization of pumped hydro energy storage systems and provides an important reference for scheduling problems in smart grids and renewable energy systems.

With the rapid development of video streaming services, adaptive bitrate (ABR) algorithms have become a core technology for ensuring optimal viewing experiences. Traditional ABR strategies, predominantly rule-based or reinforcement learning-driven, typically employ uniform quality assessment metrics that overlook users’ subjective preference differences regarding factors such as video quality and stalling. To address this limitation, this paper proposes an adaptive video bitrate selection system that integrates preference modeling with reinforcement learning. By incorporating a preference learning module, the system models and scores user viewing trajectories, using these scores to replace conventional rewards and guide the training of the Proximal Policy Optimization (PPO) algorithm, thereby achieving policy optimization that better aligns with users’ perceived experiences. Simulation results on DASH network bandwidth traces demonstrate that the proposed optimization method improves overall Quality of Experience (QoE) by over 9% compared to other mainstream algorithms.

Water quality plays a vital role in aquaculture sustainability, particularly for sensitive freshwater species such as Salmo letnica, which require strict physicochemical conditions for survival. This study evaluates the suitability of household water sources from eight rural villages in North Macedonia for supporting Salmo letnica in home-based tank environments. Seven key water quality parameters were measured and compared against biological thresholds derived from ecological literature. A Deep Reinforcement Learning (DRL) agent, trained using the Proximal Policy Optimization (PPO) algorithm, was developed to classify the suitability of water samples based on these parameters. While the DRL model exhibited low precision and recall due to the limited sample size, it provided a framework for interpretability through reward dynamics and parameter correlations. Among the eight villages, Forino and Gradec were found to meet all critical biological thresholds, while Kamenjane and Vrapçishte were identified as marginally suitable. The remaining locations exhibited insufficient oxygen levels or excessive nutrient concentrations. These findings demonstrate the potential of AI-based classification models in supporting aquaculture planning and ecological risk assessment. Future work will focus on data expansion, reward function refinement, and field-level model deployment. Unlike traditional supervised classifiers, the DRL agent enables autonomous learning and decision-making without requiring large labeled datasets. This makes the approach suitable for real-time, remote water quality monitoring in resource-limited rural areas.

Self-play has been shown to be effective to provide a proper training curriculum for a reinforcement learning agent in competitive multi-agent environments without direct supervision. However, its performance is still unstable for problems with sparse rewards, e.g., the scoring task with goalkeeper for robots in RoboCup soccer. It is challenging to solve these tasks in reinforcement learning, especially for those that require combining high-level actions with flexible control. To address these challenges, we introduce a distributed self-play training framework for an extended proximal policy optimization (PPO) algorithm that learns to act in parameterized action space and plays against a group of opponents, i.e., a league. Experiments on the domain of simulated RoboCup soccer show that, the approach is effective and learns more robust policies against various opponents compared to existing reinforcement learning methods. A demonstration video is available online at https://youtu.be/BuLli1vND4.

This paper presents an integrated Reinforcement Learning (RL) and Model Predictive Control (MPC) framework for autonomous satellite docking with a partially filled fuel tank. Traditional docking control faces challenges due to fuel sloshing in microgravity, which induces unpredictable forces affecting stability. To address this, we integrate Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) RL algorithms with MPC, leveraging MPC’s predictive capabilities to accelerate RL training and improve control robustness. The proposed approach is validated through Zero-G Lab of SnT experiments for planar stabilization and high-fidelity numerical simulations for 6-DOF docking with fuel sloshing dynamics. Simulation results demonstrate that SAC-MPC achieves superior docking accuracy, higher success rates, and lower control effort, outperforming standalone RL and PPO-MPC methods. This study advances fuel-efficient and disturbance-resilient satellite docking, enhancing the feasibility of on-orbit refueling and servicing missions.

Controlling 6-Degree-of-Freedom (6-DOF) robotic arms for precise manipulation tasks is challenging due to kinematic redundancy and the complexity of existing simulation environments like MuJoCo or ROS-Gazebo. This paper presents ArmReach6DOFEnv, a lightweight, open-source simulation framework built on PyBullet for rapid Reinforcement Learning (RL) prototyping on 6-DOF robotic arms. Using a Universal Robot Description Format (URDF) model, the environment supports a continuous state-action space for a 3D reaching task, with a reward function balancing accuracy and control effort. We evaluate two state-of-the-art RL algorithms, Proximal Policy Optimization (PPO) and Deep Deterministic Policy Gradient (DDPG), implemented via Stable-Baselines3, comparing their convergence, success rate, and motion smoothness. Experimental results demonstrate DDPG’s superior performance (69% success rate vs. PPO’s 34%) and smoother trajectories, despite PPO’s faster convergence. This framework enables accessible RL experimentation on resource-constrained systems, with potential for future sim-to-real transfer.