Policy-based monocular vision autonomous quadrotor obstacle avoidance method

Aiming at the problem of multi-objective path planning (MOPP) and dynamic obstacle avoidance of manipulator in dynamic environment, this paper proposes a solution based on hierarchical reinforcement learning (HRL) framework. Traditional path planning methods have some problems in dynamic scenes, such as poor real-time performance, difficult to balance multi-objective conflicts and insufficient adaptability to environmental changes. Therefore, this paper designs a two-tier architecture including global path planning layer and local obstacle avoidance layer, which is trained by Proximal policy optimization (PPO) and Soft Actor-Critic (SAC) algorithms respectively, and achieves collaborative optimization through an efficient information exchange mechanism between layers. At the same time, a multi-objective reward function based on dynamic weight adjustment strategy is introduced. Fuzzy logic is used to adaptively balance the relationship among path length, obstacle avoidance safety and energy consumption according to environmental complexity. Combined with Long Short-Term Memory (LSTM), the trajectory of obstacles is predicted, and the potential field method is further introduced to modify the obstacle avoidance reward, which improves the real-time response ability and robustness of the algorithm in dynamic environment. The experimental results show that the HRL-SAC-PPO method proposed in this paper shows superior performance in both static and dynamic scenarios. In the static scene, the success rate of this method reaches 100%, the average path length is shortened to 2.13m, no collision occurs, and the energy consumption is reduced to 1.12 kJ, which shows a good multi-objective optimization effect. In the dynamic scene, the trajectory error of obstacles predicted by LSTM is only 4.2%, and the safe distance between the robot arm and obstacles is improved by 35%, which significantly enhances the reliability of obstacle avoidance. In addition, the average decision delay of this method is only 11.3ms, and the peak delay is 23ms, which is much lower than that of the contrast algorithm, showing stronger real-time response ability. The ablation experiment further verified the key role of LSTM trajectory prediction, dynamic weight adjustment and layered structure on the overall performance. In the welding task verification of the real UR5 manipulator, the success rate of the system in dynamic environment is 95.3%, the average path length is 3.41m, and the maximum joint acceleration is only 0.87 radian/s², which is far below the safety threshold, indicating that the algorithm has good stability and obstacle avoidance ability in practical application. The comprehensive performance comparison shows that this method performs well in different industrial scenarios, and has stronger environmental adaptability and comprehensive path planning ability.

In this paper, a simple path correction and obstacle avoidance method with a bipedal robot for the learning stratification and performance, using an ultrasonic sensor and electronic compass sensor, is proposed. This bipedal robot was comprised using the Lego NXT Intelligent Bricks. The proposed method is implemented on an autonomous humanoid robot (the ARSR). One ultrasonic sensor and one electronic compass sensor are installed on the ARSR to detect environmental information including obstacles, the distance to the obstacle, and the directional angle of the robot. Based on the obtained information, an obstacle avoidance and path correct method is proposed to decide the ARSR's behavior so that it can avoid obstacles and move effectively to the destination area. Obstacle avoidance experiments are carried out to confirm the effectiveness of the proposed method and confirm the learning stratification and performance.

Obstacle avoidance during high-speed, low-altitude flight remains a significant challenge for unmanned aerial vehicles (UAVs), particularly in unfamiliar environments where prior maps and heavy onboard sensors are unavailable. To address this, we present an entropy-aware deep reinforcement learning framework that enables fixed-wing UAVs to navigate safely using only monocular onboard cameras. Our system features a lightweight, single-frame depth estimation module optimized for real-time execution on edge computing platforms, followed by a reinforcement learning controller equipped with a novel reward function that balances goal-reaching performance with path smoothness under fixed-wing dynamic constraints. To enhance policy optimization, we incorporate high-quality experiences from the replay buffer into the gradient computation, introducing a soft imitation mechanism that encourages the agent to align its behavior with previously successful actions. To further balance exploration and exploitation, we integrate an adaptive entropy regularization mechanism into the Proximal Policy Optimization (PPO) algorithm. This module dynamically adjusts policy entropy during training, leading to improved stability, faster convergence, and better generalization to unseen scenarios. Extensive software-in-the-loop (SITL) and hardware-in-the-loop (HITL) experiments demonstrate that our approach outperforms baseline methods in obstacle avoidance success rate and path quality, while remaining lightweight and deployable on resource-constrained aerial platforms.

This paper presents an integrated Reinforcement Learning (RL) and Model Predictive Control (MPC) framework for autonomous satellite docking with a partially filled fuel tank. Traditional docking control faces challenges due to fuel sloshing in microgravity, which induces unpredictable forces affecting stability. To address this, we integrate Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) RL algorithms with MPC, leveraging MPC’s predictive capabilities to accelerate RL training and improve control robustness. The proposed approach is validated through Zero-G Lab of SnT experiments for planar stabilization and high-fidelity numerical simulations for 6-DOF docking with fuel sloshing dynamics. Simulation results demonstrate that SAC-MPC achieves superior docking accuracy, higher success rates, and lower control effort, outperforming standalone RL and PPO-MPC methods. This study advances fuel-efficient and disturbance-resilient satellite docking, enhancing the feasibility of on-orbit refueling and servicing missions.

Deep reinforcement learning with dynamic window approach based collision avoidance path planning for maritime autonomous surface ships

Reinforcement Learning (RL) is a technique that enables an agent to learn to behave optimally by repeatedly interacting with the environment and receiving the rewards. RL is widely used in domains such as robotics, game playing and finance. Proximal Policy Optimization (PPO) is the state-of-the-art policy optimization algorithm which achieves superior overall performance on various RL benchmarks. PPO iteratively optimizes its policy - a function which chooses optimal actions, with each iteration consisting of two computationally intensive phases: Inference phase - agents infer actions to interact with the environment and collect data, and Training phase - agents train the policy using the collected data. In this work, we develop the first high-throughput PPO accelerator on CPU-FPGA heterogeneous platform, targeting both phases of the algorithm for acceleration. We implement a systolic-array based architecture coupled with a novel memory-blocked data layout that enables streaming data access in both forward and backward propagations to achieve high-throughput performance. Additionally, we develop a novel systolic array compute sharing technique to mitigate the potential load imbalance in the training of two networks. We develop an accurate performance model of our design, based on which we perform design space exploration to obtain optimal design points. Our design is evaluated on widely used robotics benchmarks, achieving $2.1 \times - 30.5 \times$ and $2 \times - 27.5 \times$ improvements in throughput against state-of-the-art CPU and CPU-GPU implementations, respectively.

This paper investigates the error accumulation problem in centralized training and decentralized execution (CTDE) policy‐based multi‐agent reinforcement learning (MARL) algorithms, which arises from local observation inaccuracies. To address this issue, we propose a novel MARL algorithm that incorporates imitation learning using local observations. Firstly, by analysing the multi‐agent proximal policy optimization algorithm and examining the problems arising when global states are replaced with local observations, it is proved that insufficient observations can lead to information loss, thereby introducing errors of advantage function, and it is demonstrated that the generalized advantage estimation method accumulates errors during the training process. Then, imitation learning is introduced and a novel training framework that combines reinforcement learning and imitation learning is proposed. During the reinforcement learning phase, an MARL agent trained with global observations acts as an expert. Subsequently, imitation learning is applied to train another agent that mimics the expert's decisions using only local observations. Finally, the effectiveness of this algorithm is verified in some commonly used multi‐agent environments, which demonstrates its superior performance compared to traditional multi‐agent reinforcement learning algorithms.

Implementing action mask in proximal policy optimization (PPO) algorithm

The application of reinforcement learning algorithms to motion planning is a research hotspot in robotics in recent years. However, training reinforcement learning agents from scratch has low training efficiency and difficulty in convergence. In this paper, a robot motion planning method based on residual reinforcement learning is proposed. This method divides the agent's policy of motion planning into initial policy and residual policy. The initial policy is composed of a neural network motion planner responsible for guiding the training direction of residual policy. The residual policy is composed of Proximal Policy Optimization (PPO) algorithm in reinforcement learning. A motion planning experiment is carried out in a simulation environment, and the result shows that the method can successfully perform motion planning. The comparison experiment between PPO and the proposed algorithm demonstrates that the proposed algorithm has better motion planning performance.

Obstacle avoidance for mobile robot to reach the desired target from a start location is one of the most interesting research topics. However, until now, few works discuss about working of mobile robot in the dynamic and continuously changing environment. So, this issue is still the research challenge for mobile robots. Traditional algorithm for obstacle avoidance in the dynamic, complex environment had many drawbacks. As known that Q-learning, the type of reinforcement learning, has been successfully applied in computer games. However, it is still rarely used in real world applications. This research presents an effectively method for real time dynamic obstacle avoidance based on Q-learning in the real world by using three-wheeled mobile robot. The position of obstacles including many static and dynamic obstacles and the mobile robot are recognized by fixed camera installed above the working space. The input for the robot is the 2D data from the camera. The output is an action for the robot (velocities, linear and angular parameters). Firstly, the simulation is performed for Q-learning algorithm then based on trained data, The Q-table value is implemented to the real mobile robot to perform the task in the real scene. The results are compared with intelligent control method for both static and dynamic obstacles cases. Through implement experiments, the results show that, after training in dynamic environments and testing in a new environment, the mobile robot is able to reach the target position successfully and have better performance comparing with fuzzy controller.

Existing resource-intensive obstacle avoidance techniques are hard to be applied on the small-size Unmanned Aerial Vehicles (UAVs) that have limited sensing and computation capacity. Therefore, it is necessary to develop an obstacle recognition and avoidance scheme which works under resource-constrained environments. To this backdrop, this paper first presents an obstacle recognition model based on monocular vision feature points. Afterwards, an obstacle recognition algorithm is put forward, whose computational complexity is low. Then, an obstacle avoidance method is proposed to regulate the obstacle-avoidance path of a UAV until it arrives its destination. To evaluate the effectiveness of the presented algorithms, we design and implement a simulation platform on Objective Modular Network Testbed in C++ (OMNeT++), and conduct a series of experiments. Experimental results show that the proposed model and algorithms can effectively guide a micro UAV to its destination using only an embedded processor and 0.5kg extra load. Even in poor communication conditions, the UAV can independently avoid obstacles and reach the destination only by acquiring the destination coordinates from the ground station.

Wireless Power Transfer (WPT) for Electric Vehicles (EVs) offers a promising solution for convenient and efficient charging. However, misalignments, sensor noise, and parameter variability can significantly degrade Power Transfer Efficiency (PTE). This study proposes a novel unified artificial intelligence (AI)-driven optimization and control framework that integrates Genetic Algorithm (GA)-based static optimization, Artificial Neural Network (ANN) surrogate modeling, and Reinforcement Learning (RL) dynamic control using the Proximal Policy Optimization (PPO) algorithm. This unified design bridges the gap between previous static-only optimization methods and dynamic adaptive controllers, enabling both peak efficiency and verified robustness within a single digital twin simulation environment. A high-fidelity MATLAB/Simulink model of the WPT system was developed and validated using an ANN surrogate model (Test MSE: 7.87×10−13). The GA-optimized configuration achieved a peak PTE of 85.47%, representing a 2.11 percentage-point improvement over the baseline. The RL controller, based on PPO, maintained a mean efficiency of approximately 80% under unseen trajectories, ±10% hardware parameter variations, and Gaussian sensor noise (σ=0.56%), demonstrating superior adaptability. Comparative analysis with state-of-the-art studies confirms that the proposed approach not only matches or exceeds the reported efficiency gains, but also uniquely integrates robustness validation and generalization testing. The results suggest that combining offline GA optimization with online RL adaptation provides a scalable, real-time control strategy for practical WPT deployments.

Mobile robots face collision risk avoidance challenges in dynamic environments, necessitating that we address the safety and adaptability shortcomings of traditional navigation methods. Traditional methods rely on predefined rules, making it difficult to achieve flexible, safe, and real-time obstacle avoidance in complex, dynamic environments. To address this issue, a risk-aware, dynamic, adaptive regulation barrier policy optimization (RADAR-BPO) method is proposed, combining proximal policy optimization (PPO) with the control barrier function (CBF). RADAR-BPO generates exploratory actions using PPO and constructs a real-time safety filter using the CBF. This method uses quadratic programming to minimize risky actions, thereby ensuring safe obstacle avoidance while maintaining navigation efficiency. Testing of three phased learning environments in the ROS Gazebo simulation environment demonstrated that the proposed method achieves an obstacle avoidance success rate of nearly 90% in complex, dynamic, multi-obstacle environments and improves the overall mission success rate, validating its robustness and effectiveness in complex dynamic scenarios.

Proximal policy optimization via enhanced exploration efficiency

Aiming at the problem that obstacle avoidance flexibility and formation integrity are difficult to coexist in multi-robot formation motion, a path-deformation mapping mechanism is proposed, which deeply integrates artificial potential field and affine transformation, and drives formation adaptive adjustment in real time through path information. By using the non-uniform scaling characteristics of the affine transformation, the limitation of traditional conformal transformation is broken through, and the unity of flexibility and integrity is realized. The effectiveness of the algorithm is verified by experiments, which provide a practical solution for cooperative obstacle avoidance of multi-robot systems in complex environments. In order to verify the performance of the algorithm, a numerical simulation is carried out, and an experimental platform composed of seven omnidirectional mobile robots is built for physical verification. The simulation and experimental results show that the formation can complete the obstacle avoidance task in the complex static obstacle environment, and the average formation tracking error is maintained below 0.05 m. Compared with the traditional local obstacle avoidance or formation switching method, this algorithm significantly improves the fluency of the obstacle avoidance process and the integrity of the formation while ensuring a success rate of 100% obstacle avoidance.