MPC-based Deep Reinforcement Learning Method for Space Robotic Control with Fuel Sloshing Mitigation

With the continuous progress of UAV technology and the rapid development of mobile edge computing (MEC), the UAV-assisted MEC system has shown great application potential in special fields such as disaster rescue and emergency response. However, traditional deep reinforcement learning (DRL) decision-making methods suffer from limitations such as difficulty in balancing multiple objectives and training convergence when making mixed action space decisions for UAV path planning and task offloading. This article innovatively proposes a hybrid decision framework based on the improved Dynamic Adaptive Genetic Optimization Algorithm (DAGOA) and soft actor–critic with hierarchical action decomposition, an uncertainty-quantified critic ensemble, and adaptive entropy temperature, where DAGOA performs an effective search and optimization in discrete action space, while SAC can perform fine control and adjustment in continuous action space. By combining the above algorithms, the joint optimization of drone path planning and task offloading can be achieved, improving the overall performance of the system. The experimental results show that the framework offers significant advantages in improving system performance, reducing energy consumption, and enhancing task completion efficiency. When the system adopts a hybrid decision framework, the reward score increases by a maximum of 153.53% compared to pure deep reinforcement learning algorithms for decision-making. Moreover, it can achieve an average improvement of 61.09% on the basis of various reinforcement learning algorithms such as proposed SAC, proximal policy optimization (PPO), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3).

The application of reinforcement learning algorithms to motion planning is a research hotspot in robotics in recent years. However, training reinforcement learning agents from scratch has low training efficiency and difficulty in convergence. In this paper, a robot motion planning method based on residual reinforcement learning is proposed. This method divides the agent's policy of motion planning into initial policy and residual policy. The initial policy is composed of a neural network motion planner responsible for guiding the training direction of residual policy. The residual policy is composed of Proximal Policy Optimization (PPO) algorithm in reinforcement learning. A motion planning experiment is carried out in a simulation environment, and the result shows that the method can successfully perform motion planning. The comparison experiment between PPO and the proposed algorithm demonstrates that the proposed algorithm has better motion planning performance.

In response to the deterioration of path-tracking accuracy and driving stability encountered by intelligent vehicles under dynamically varying operating conditions, a multi-objective optimization strategy integrating soft actor-critic (SAC) reinforcement learning with variable-parameter Model Predictive Control (MPC) is proposed in this paper to achieve online adaptive adjustment of path-tracking controller parameters. Based on a three-degree-of-freedom vehicle dynamics model, a linear time-varying (LTV) MPC controller is constructed to jointly optimize the front wheel steering angle. An SAC agent is developed utilizing the actor-critic framework, with a comprehensive reward function designed around tracking accuracy and control smoothness to enable online tuning of the MPC weighting matrices (lateral error weight, heading error weight, and steering control weight) as well as the prediction horizon parameter, thereby realizing adaptive balance between tracking accuracy and stability under different operating conditions. Based on the simulation results, it can be concluded that under normal operating conditions, the proposed integrated SAC-MPC control scheme demonstrates superior tracking performance, with the maximum absolute lateral error and mean lateral error reduced by 44.9% and 67.2%, respectively, and the maximum absolute heading error reduced by 23.5%. When the system operates under nonlinear conditions during the transitional phase, the proposed control scheme not only enhances tracking accuracy—evidenced by reductions of 43.4% and 23.8% in the maximum absolute lateral error and maximum absolute heading error, respectively—but also significantly improves system stability, as indicated by a 20.7% reduction in the sideslip angle at the center of gravity. Experimental validation further confirms these findings. The experimental results reveal that, compared with the fixed-parameter MPC, the maximum absolute value and mean value of the lateral error are reduced by approximately 36.2% and 78.1%, respectively; the maximum absolute heading angle error is decreased by 24.3%; the maximum absolute yaw rate is diminished by 19.6%; and the maximum absolute sideslip angle at the center of gravity is reduced by 30.8%.

Implementing action mask in proximal policy optimization (PPO) algorithm

Energy-efficient and damage-recovery slithering gait design for a snake-like robot based on reinforcement learning and inverse reinforcement learning

The rapid advancement of artificial intelligence (AI) has accelerated the semiconductor industry, driving chip performance requirements to unprecedented levels. However, conventional system-on-chip (SoC) designs are increasingly constrained by the physical and economic limits of monolithic scaling, where continued transistor miniaturization yields diminishing returns and rising fabrication costs. To address these challenges, the industry has shifted toward chiplet-based heterogeneous integration and advanced packaging technologies. By interconnecting multiple specialized chiplets through high-density, high-bandwidth interfaces, this modular approach improves scalability, reusability, and cost efficiency while enabling faster time-to-market. However, chiplet-based systems introduce significant design complexity, particularly in signal integrity (SI) and power integrity (PI), where highly constrained interconnects and power delivery networks must meet stringent electrical requirements. Traditional design methodologies rely heavily on expert-driven manual tuning and extensive full-wave electromagnetic simulations using electronic design automation (EDA) tools, resulting in long design cycles that can span weeks or months. To overcome these limitations, this thesis investigates the application of AI, particularly deep reinforcement learning (RL), to automate and optimize chiplet-based system design with a focus on SI and PI challenges in advanced packaging technologies. The research initially explores a supervised learning approach using convolutional neural networks (CNNs) to classify signal impairments from eye diagrams. While effective, this approach reveals a critical dependence on large labeled datasets, which are scarce in SI/PI domains, motivating a shift toward RL for autonomous design space exploration without labeled data. The first major contribution proposes an RL framework for automated power distribution network (PDN) design in a four-chiplet system. Two RL algorithms, Dueling Double Deep Q-Network (DDDQN) and Proximal Policy Optimization (PPO), are evaluated under different reward strategies. Results show that dense analytical rewards based on power domain squareness and bump proximity enable the RL agent to achieve lower self-impedance than human-designed PDNs, ensuring power integrity across 28 power domains. The complete PDN layout, which typically requires three months of manual effort, is generated in under one hour using a single GPU. The second contribution focuses on signal integrity optimization for chiplet interconnects in CoWoS redistribution layers (RDLs). An RL agent is trained using a reward function that balances eye width, voltage transfer function (VTF) crosstalk, and interconnect density. The agent explores design variations across data rates from 4 to 32 GT/s using a simulation environment integrated with commercial EDA tools. Experimental results demonstrate up to 24.67% improvement in eye width and over 22 dB reduction in VTF crosstalk while satisfying UCIe electrical specifications. The training process also generates labeled datasets for future supervised learning. The third contribution introduces an RL-based routing framework for UCIe bridge interconnects in advanced System-in-Package (SiP). Using Proximal Policy Optimization, the framework learns to generate routing layouts that comply with constraints such as ground-signal-ground structures, layer assignment, and congestion avoidance. It supports all UCIe module configurations and produces design rule check-compliant solutions within 100 training episodes, reducing design time from four weeks to less than one hour while improving signal distribution and reducing via usage. In summary, this thesis presents a unified AI-driven methodology for addressing SI and PI challenges in chiplet-based systems. The proposed RL frameworks significantly outperform traditional design approaches in both efficiency and quality, demonstrating the potential of deep RL to enable scalable and intelligent design automation for advanced packaging and chiplet-based architectures.

Autonomous navigation at unsignalized intersections: A coupled reinforcement learning and model predictive control approach

Aiming at the problem of multi-objective path planning (MOPP) and dynamic obstacle avoidance of manipulator in dynamic environment, this paper proposes a solution based on hierarchical reinforcement learning (HRL) framework. Traditional path planning methods have some problems in dynamic scenes, such as poor real-time performance, difficult to balance multi-objective conflicts and insufficient adaptability to environmental changes. Therefore, this paper designs a two-tier architecture including global path planning layer and local obstacle avoidance layer, which is trained by Proximal policy optimization (PPO) and Soft Actor-Critic (SAC) algorithms respectively, and achieves collaborative optimization through an efficient information exchange mechanism between layers. At the same time, a multi-objective reward function based on dynamic weight adjustment strategy is introduced. Fuzzy logic is used to adaptively balance the relationship among path length, obstacle avoidance safety and energy consumption according to environmental complexity. Combined with Long Short-Term Memory (LSTM), the trajectory of obstacles is predicted, and the potential field method is further introduced to modify the obstacle avoidance reward, which improves the real-time response ability and robustness of the algorithm in dynamic environment. The experimental results show that the HRL-SAC-PPO method proposed in this paper shows superior performance in both static and dynamic scenarios. In the static scene, the success rate of this method reaches 100%, the average path length is shortened to 2.13m, no collision occurs, and the energy consumption is reduced to 1.12 kJ, which shows a good multi-objective optimization effect. In the dynamic scene, the trajectory error of obstacles predicted by LSTM is only 4.2%, and the safe distance between the robot arm and obstacles is improved by 35%, which significantly enhances the reliability of obstacle avoidance. In addition, the average decision delay of this method is only 11.3ms, and the peak delay is 23ms, which is much lower than that of the contrast algorithm, showing stronger real-time response ability. The ablation experiment further verified the key role of LSTM trajectory prediction, dynamic weight adjustment and layered structure on the overall performance. In the welding task verification of the real UR5 manipulator, the success rate of the system in dynamic environment is 95.3%, the average path length is 3.41m, and the maximum joint acceleration is only 0.87 radian/s², which is far below the safety threshold, indicating that the algorithm has good stability and obstacle avoidance ability in practical application. The comprehensive performance comparison shows that this method performs well in different industrial scenarios, and has stronger environmental adaptability and comprehensive path planning ability.

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

Greenhouse climate control is essential for optimizing crop growth while minimizing resource consumption in controlled environment agriculture. Traditional rule-based and fixed-action strategies often struggle to achieve a balance between these objectives. This paper proposes a reinforcement learning (RL) based framework for greenhouse climate control, integrating deep learning models to predict both crop growth and resource consumption. The framework enables an RL agent to optimize greenhouse control setpoints dynamically, maximizing crop yield while ensuring sustainable resource usage. The proposed system incorporates a Multi-Layer Perceptron (MLP) model to predict internal greenhouse climate conditions, a Long Short-Term Memory (LSTM) model for crop parameter estimation, and a separate LSTM model for forecasting daily resource consumption. These models collectively simulate a greenhouse environment where an RL agent learns to regulate temperature, CO2 concentration, and irrigation levels by interacting with the virtual environment. A custom reward function is designed to guide the agent, considering key crop parameters; stem elongation, stem thickness, and cumulative trusses; alongside resource consumption metrics, including heating, electricity, CO2, and irrigation costs. To enhance the adaptability of the RL agent, a feature-selection mechanism identified the most influential climate and control features, reducing observation complexity and accelerating convergence. Retraining under stochastic weather conditions strengthened robustness to dynamic environments, enabling the agent to consistently outperform fixed-action strategies. Evaluation revealed a stable Pareto frontier between yield and resource consumption, confirming that the framework accurately captured the productivity and sustainability trade-off and remained robust across varying reward-weight settings. Comparative analysis of multiple RL algorithms; Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), Soft Actor-Critic (SAC), and Twin Delayed Deep Deterministic Policy Gradient (TD3) demonstrated that TD3 outperforms other algorithms, achieving the highest cumulative rewards and reaching optimal policies faster. Experimental evaluations demonstrate that the proposed TD3 RL-based greenhouse control system achieves higher crop yield growth rates while optimizing resource usage, outperforming conventional greenhouse control strategies. This study presents a novel data-driven, adaptive greenhouse management approach, bridging the gap between crop growth modeling and autonomous climate control, contributing to sustainable and intelligent agricultural practices.

The rapid advancement of generative models and optimization algorithms has opened new opportunities for intelligent garden design systems. However, existing approaches often lack the ability to integrate aesthetic, functional and ecological considerations simultaneously and fail to adapt dynamically to complex design constraints. To address these challenges, this study proposes a dynamic garden design decision-making framework that integrates StyleGAN, Convolutional Neural Networks (CNNs) and the Soft Actor-Critic (SAC) reinforcement learning algorithm. The model combines StyleGAN’s capability to generate diverse and high-quality garden layouts, CNN’s feature extraction for spatial and visual representation and SAC’s optimization for balancing multiple objectives, including ecological benefit, resource efficiency and aesthetic quality. Experimental results on the Places Dataset and AID Dataset demonstrate that the proposed model significantly outperforms baseline methods in image quality, design diversity and optimization effectiveness. Specifically, it achieves superior performance in generating realistic and context-aware garden layouts while maintaining flexibility to adapt to environmental constraints. This research provides an adaptive framework for garden design and offers a foundation for extending dynamic design systems to fields like urban planning and interior design, promoting multi-objective optimization in diverse contexts.

An adaptive PID controller for path following of autonomous underwater vehicle based on Soft Actor–Critic

Despite the fact that deep reinforcement learning (RL) has surpassed human-level performances in various tasks, it still has several fundamental challenges. First, most RL methods require intensive data from the exploration of the environment to achieve satisfactory performance. Second, the use of neural networks in RL renders it hard to interpret the internals of the system in a way that humans can understand. To address these two challenges, we propose a framework that enables an RL agent to reason over its exploration process and distill high-level knowledge for effectively guiding its future explorations. Specifically, we propose a novel RL algorithm that learns high-level knowledge in the form of a finite reward automaton by using the L* learning algorithm. We prove that in episodic RL, a finite reward automaton can express any non-Markovian bounded reward functions with finitely many reward values and approximate any non-Markovian bounded reward function (with infinitely many reward values) with arbitrary precision. We also provide a lower bound for the episode length such that the proposed RL approach almost surely converges to an optimal policy in the limit. We test this approach on two RL environments with non-Markovian reward functions, choosing a variety of tasks with increasing complexity for each environment. We compare our algorithm with the state-of-the-art RL algorithms for non-Markovian reward functions, such as Joint Inference of Reward machines and Policies for RL (JIRP), Learning Reward Machine (LRM), and Proximal Policy Optimization (PPO2). Our results show that our algorithm converges to an optimal policy faster than other baseline methods.

Reinforcement learning and deep reinforcement learning, as a research hotspot in the field of machine learning, have been widely used in our daily life. In this field, game is playing an extremely important role in the developing of reinforcement learning algorithms. Based on the Tennis environment built by Unity ML Agents, this paper used three algorithms, Proximal Policy Optimization (PPO), Multi-Agent Deep Deterministic Policy Gradients (MADDPG) and Soft Actor-Critic (SAC), combined with PyTorch framework, solved the continuous control problem of this environment. Meanwhile, a group of optimal parameters are obtained through multiple trainings, so that Agents can achieve a perfect effect of solving the continuous control problem in Tennis environment. At the end, this paper compared and analyzed the difference among these three algorithms, summarized the application and properties of each algorithm. For different parameters of the algorithm, this paper also maked a comparison and explained the reasons for some special cases as well which can be used for the future work.

A smart grid system that combines together deep learning, reinforcement learning, and genetic algorithms to manage and optimize energy more intelligently. By merging real-time and historical data, the system forecasts power demand using a CNN-LSTM model and then adjusts to it using Soft Actor-Critic (SAC) reinforcement learning, successfully tackling load management concerns. At the same time, genetic algorithms fine-tune the PI controllers of wind turbines, enhancing energy production. Simulation findings demonstrate that hybrid AI strategy gives more accurate forecasts, smoother real-time judgements, and greater energy management than previous techniques. The chapter illustrates how AI might alter smart grids, delivering more dependable, efficient, and sustainable energy management.