Towards Safe Continuing Task Reinforcement Learning

This study proposes a layered hybrid control framework for automation and robotics systems that combines Tube-based Robust MPC (RMPC) as a safety envelope, adaptive parameter estimation to narrow online uncertainty, Sliding Mode Control (SMC) as a high-resilience fail-safe, and Reinforcement Learning (RL) as a performance booster through a warm-start/residual policy scheme that is always constrained by Control Barrier Functions (CBF). The study aims to achieve better tracking performance and computational efficiency without sacrificing safety and constraint compliance. The methodology includes constrained nonlinear plant modeling, design of tube invariance-based RMPC with constraint tightening, RLS/concurrent learning-based adaptive estimator, SMC for infeasibility/model drift conditions, and a CBF-based safety filter formulated as an online small QP. Evaluation is done through ablation studies on multi-domain benchmarks and HIL-style tests, with finite-horizon cost, ITAE, constraint violation rate, and computational latency (average and WCET) metrics. The results show that the RL+CBF configuration reduces the finite-horizon cost by about 15.7% compared to pure RMPC, maintaining a violation rate of ~0.35% (better than RMPC: 0.80%) and significantly below that of unfiltered RL (4.20%). The warm-start RL scheme accelerates the completion time of the MPC solver by ≈15–18% over various prediction horizons and reduces the WCET (26.4 → 23.3 ms), supporting real-time implementation. These findings confirm that the integration of RMPC–Adaptive–SMC–RL(+CBF) effectively bridges model-based optimality, online adaptivity, robust resilience, and learning from data in a single architecture that is safety-certified and feasible for real-world applications demanding high uptime.

International audience

This paper introduces robust control barrier functions for uncertain parameter-varying control affine systems, where the parametric uncertainties can be time-varying and nonlinearly affecting the system dynamics and/or safety sets. In particular, we propose two methods based on mixed-monotone decomposition and concave bounding, where the controlled invariance condition remains linear in the control inputs despite nonlinear uncertainties. Moreover, we design alternative robust control Lyapunov functions where the control inputs also appear linearly; thus, these robust control barrier and Lyapunov functions can be coupled to obtain a quadratic program that can be solved online. Additionally, we propose two set-membership parameter estimation methods using polyhedral intersections and interval observers to reduce the conservatism of the robust approaches. Our robust approaches are observed to have comparable performance with adaptive approaches and robust safety guarantees, even when the (robust) adaptive methods may not.

Robust tracking control of uncertain nonlinear systems with adaptive dynamic programming

In this article, robust control schemes are presented for time-varying trajectory tracking control of Wheeled Mobile Robots (WMRs) in the presence of disturbance. First, the robust model predictive control (RMPC) is implemented for the kinematic subsystem by solving the modified optimization problem with the fixed starting point. Second, the robust control is extended for the dynamic subsystem by the Backstepping technique and additional observer. This paper shows how to ensure the stability of the closed-loop tracking error system. The performance of the proposed robust control is verified through simulation studies using Casadi tool.KeywordsModel predictive control (MPC)Wheeled mobile robotic systems (WMRs)Output feedback controlTerminal controllerFeasibility

We propose a robust control system which has robustness for disturbance and can deal with a nonlinear system with partially unknown structure by fusing reinforcement learning and robust control theory. First, we solved an optimal control problem without using unknown part of functions of the system, using neural network and the repetition learning of reinforcement learning algorithm. Second, we built the robust reinforcement learning control system which permits uncertainty and has robustness for disturbance by fusing the idea of H infinity control theory with above system.

Reinforcement learning has shown encouraging results in decision-making tasks such as those found in gaming and robotics. However, when applied in safety-critical applications, like navigation, reinforcement learning can cause serious safety problems due to its trial-and-error nature. In this paper, we propose a safe reinforcement learning framework for LiDAR-based navigation. The proposed framework incorporates control barrier function theory with reinforcement learning to mitigate safety risks while learning a navigation strategy. Furthermore, by designing an online learning control barrier function, the proposed method can adapt to different obstacle settings through learning from past experiences. As a map-less solution, our approach uses only LiDAR as the perception system and requires no additional localization information. We conduct simulated experiments in ROS Gazebo under different indoor environments. The results show that our method outperforms other approaches by achieving up to twice the reward and five times fewer collisions when compared to the runner-up.

We address the problem of coordination and control of Connected and Automated Vehicles (CAVs) in the presence of imperfect observations in mixed traffic environment. A commonly used approach is learning-based decision-making, such as reinforcement learning (RL). However, most existing safe RL methods suffer from two limitations: (i) they assume accurate state information, and (ii) safety is generally defined over the expectation of the trajectories. It remains challenging to design optimal coordination between multi-agents while ensuring hard safety constraints under system state uncertainties (e.g., those that arise from noisy sensor measurements, communication, or state estimation methods) at every time step. We propose a safety guaranteed hierarchical coordination and control scheme called Safe-RMM to address the challenge. Specifically, the high-level coordination policy of CAVs in mixed traffic environment is trained by the Robust Multi-Agent Proximal Policy Optimization (RMAPPO) method. Though trained without uncertainty, our method leverages a worst-case Q network to ensure the model's robust performances when state uncertainties are present during testing. The low-level controller is implemented using model predictive control (MPC) with robust Control Barrier Functions (CBFs) to guarantee safety through their forward invariance property. We compare our method with baselines in different road networks in the CARLA simulator. Results show that our method provides the best evaluated safety and efficiency in challenging mixed traffic environments with uncertainties.

Robust velocity and height control problem is dealt with for hypersonic vehicles subject to highly nonlinearity and uncertainties. The linear robust controller design for nonlinear systems suffers from the conservation problem, because the nonlinear dynamics are considered as uncertainties. In this paper, a feedback linearization controller is firstly applied to achieve the dynamic decoupling of the velocity and height channels. Then, robust controllers are designed for each channel to restrain the effects of the parametric uncertainties, external disturbances, and nonlinear dynamics, which cannot be counteracted accurately by the feedback linearization technique. This method can guarantee the robust stability and robust tracking properties of the closed-loop control system, while the conservation of the robust controller design can be reduced because the nonlinear information of dynamic system is used. Simulation comparisons are given to show the advantages of the proposed robust nonlinear control method.

The problem of robust velocity and height control is dealt with for hypersonic vehicles subject to high nonlinearity and uncertainties. Robust linear controller design for nonlinear systems suffers from the problem of conservation because the nonlinear dynamics are considered as uncertainties. In this paper, a feedback linearization controller is firstly applied to achieve dynamic decoupling of the velocity and height channels. Robust controllers are then designed for each channel to achieve the desired tracking and restrain the effects of parametric uncertainties, external disturbances, and nonlinear dynamics, which cannot be accurately counteracted by the feedback linearization technique. This method can guarantee the robust stability and robust tracking properties of the closed-loop control system. Moreover, the conservation of the robust controller design can be reduced because the nonlinear information of the dynamic system is used. Simulation results are given to show the advantages of the proposed robust nonlinear control method compared with its linear counterpart.

This article presents a method for guaranteeing the safety of a system with both parametric and nonparametric uncertainties, while at the same time decreasing the conservatism compared to existing approaches. This is obtained by combining robust adaptive control barrier functions (RaCBF) and Gaussian process control barrier functions (GPCBF). We provide a condition under which the considered system is safe with a given probability, and show that the proposed method is less conservative than GPCBF. We evaluate the method through a simulation study, where we consider a force controlled robot manipulator in contact with a partially unknown environment. The results show that our proposed GPRaCBF can guarantee bounds on the contact forces despite parametric and nonparametric uncertainties in the contact dynamics and outperforms GPCBF in terms of the conservatism.

This paper reviews the recent advancements of reinforcement learning (RL) for chemical process control. RL presents a systematic strategy in which the machine learning agent learns a policy of actions based on interactions with the environment. We first provide a brief overview of RL theoretic basis built on Markov decision processes (MDPs) and then move onto its application to process control. With particular interest in chemical processes, we review state-of-the-art research developments on RL for controller tuning and direct control policy learning. This work highlights the importance of safe RL control to incorporate deterministic or probabilistic safety constraints such as constrained MDPs, control barrier functions, etc. We conclude the review with a discussion on some of the outstanding challenges such as sampling efficiency, generalizability, uncertainty, and observability, as well as the emergent and future directions to address these limitations.

The complexity in planning and control of robot compliance tasks mainly results from simultaneous control of both position and force and inevitable contact with environments. It is quite difficult to achieve accurate modeling of the interaction between the robot and the environment during contact. In addition, the interaction with the environment varies even for compliance tasks of the same kind. To deal with these phenomena, in this paper, we propose a reinforcement learning and robust control scheme for robot compliance tasks. A reinforcement learning mechanism is used to tackle variations among compliance tasks of the same kind. A robust compliance controller that guarantees system stability in the presence of modeling uncertainties and external disturbances is used to execute control commands sent from the reinforcement learning mechanism. Simulations based on deburring compliance tasks demonstrate the effectiveness of the proposed scheme.

This publication presents a new method by which control methods based on reinforcement learning can be combined with classical robust control methods. The combination results in a robust management system that meets high-quality criteria. The described method is presented through the control of an autonomous vehicle. By choosing the reward function chosen during reinforcement learning, various driving styles can be realized, e.g., lap time minimization, track tracking, and travel comfort. The neural network was trained using the Proximal Policy Optimization algorithm, and the robust control is based on H∞. The two controllers are combined using a supervisor structure, in which a quadratic optimization task is implemented. The result of the method is a control structure that realizes the longitudinal and lateral control of the vehicle by specifying the reference speed and the steering angle. The effectiveness of the algorithm is demonstrated through simulations.

In this chapter, a novel control method using a reinforcement learning (RL) (Sutton and Barto (1998)) with concept of sliding mode control (SMC) (Slotine and Li (1991)) for unknown dynamical system is considered. In designing the control system for unknown dynamical system, there are three approaches. The first one is the conventional model-based controller design, such as optimal control and robust control, each of which is mathematically elegant, however both controller design procedures present a major disadvantage posed by the requirement of the knowledge of the system dynamics to identify and model it. In such cases, it is usually difficult to model the unknown system, especially, the nonlinear dynamical complex system, to make matters worse, almost all real systems are such cases. The second one is the way to use only the soft-computing, such as neural networks, fuzzy systems, evolutionary systems with learning and so on. However, in these cases it is well known that modeling and identification procedures for the dynamics of the given uncertain nonlinear system and controller design procedures often become time consuming iterative approaches during parameter identification and model validation at each step of the iteration, and in addition, the control system designed through such troubles does not guarantee the stability of the system. The last one is the way to use the method combining the above the soft-computing method with the model-based control theory, such as optimal control, sliding mode control (SMC), H∞ control and so on. The control systems designed through such above control theories have some advantages, that is, the good nature which its adopted theory has originally, robustness, less required iterative learning number which is useful for fragile system controller design not allowed a lot of iterative procedure. This chapter concerns with the last one, that is, RL system, a kind of soft-computing method, supported with robust control theory, especially SMC for uncertain nonlinear systems. RL has been extensively developed in the computational intelligence and machine learning societies, generally to find optimal control policies for Markovian systems with discrete state and action space. RL-based solutions to the continuous-time optimal control problem have been given in Doya (Doya (2000). The main advantage of using RL for solving optimal