A DDPG-based Path Following Control Strategy for Autonomous Vehicles by Integrated Imitation Learning and Feedforward Exploration

This paper presents the implementation of an autonomous electric vehicle (EV) project in the National Taiwan University of Science and Technology (NTUST) campus in Taiwan. The aim of this work was to integrate two important practices of realizing an autonomous vehicle in a campus environment, including vehicle positioning and path tracking. Such a project is helpful to the students to learn and practice key technologies of autonomous vehicles conveniently. Therefore, a laboratory-made EV was equipped with real-time kinematic GPS (RTK-GPS) to provide centimeter position accuracy. Furthermore, the model predictive control (MPC) was proposed to perform the path tracking capability. Nevertheless, the RTK-GPS exhibited some robust positioning concerns in practical application, such as a low update rate, signal obstruction, signal drift, and network instability. To solve this problem, a multisensory fusion approach using an unscented Kalman filter (UKF) was utilized to improve the vehicle positioning performance by further considering an inertial measurement unit (IMU) and wheel odometry. On the other hand, the model predictive control (MPC) is usually used to control autonomous EVs. However, the determination of MPC parameters is a challenging task. Hence, reinforcement learning (RL) was utilized to generalize the pre-trained datum value for the determination of MPC parameters in practice. To evaluate the performance of the RL-based MPC, software simulations using MATLAB and a laboratory-made, full-scale electric vehicle were arranged for experiments and validation. In a 199.27 m campus loop path, the estimated travel distance error was 0.82% in terms of UKF. The MPC parameters generated by RL also achieved a better tracking performance with 0.227 m RMSE in path tracking experiments, and they also achieved a better tracking performance when compared to that of human-tuned MPC parameters.

This paper presents an innovative path tracking method of model predictive control based on multi-point preview (MPP-MPC) that the original equation of state is augmented by considering a dynamics model of the multiple preview points. The path tracking issue is formulated as an optimal control problem with dynamic disturbance, i.e., the future road curvature. Hence, an ameliorative Kalman filter is adopted to fuse the information about camera and vehicle dynamics to accurately estimate the road curvature. To ensure lateral control performance, dynamic constraints are defined by the estimated road curvature to guarantee comfort and safety. In the control domain of model predictive control (MPC), the model can be approximated as a linear time-invariant model (LTIM) to reduce the computational complexity. The MPC problem can be transformed into a standard quadratic programming (QP) problem with dynamic constraints of guaranteed comfort and safety. Finally, the complex path tracking problem can be solved by the QP problem. Through lane keeping experiments, it is shown that the tracking accuracy and steering smoothness can be significantly improved by the proposed method.

Model predictive control (MPC) has established itself as the standard advanced process control method. However, solving the resulting optimization problems in real-time can be challenging, especially when uncertainty is explicitly considered in a robust nonlinear predictive control approach. An increasingly popular alternative to avoid the online solution of the resulting optimization problems is to approximate their solution using neural networks. The networks are trained using many solutions of the MPC problem for different system states and therefore this approach is often called imitation learning. Controllers obtained via imitation learning have important drawbacks, since it is difficult to learn behaviors that are not well represented in the data and they must be redesigned from scratch when the control task changes. In this work, we show that these two drawbacks can be mitigated by combining imitation learning and concepts from reinforcement learning. The central idea is to use imitation learning as a very good initialization of a control policy that is iteratively updated using reinforcement learning, taking advantage of the fact that an explicit and differentiable expression of the approximate MPC controller is available. The efficacy of the combination of the two learning paradigms is highlighted via simulations of a semi-batch industrial polymerization reactor.

This paper presents a path planning and tracking framework for autonomous mining articulated vehicles (AVs). The relation space method is proposed for path planning. In this method, a self-organising, competitive neural network is adopted to identify the space around the vehicle. Then, the vehicle's optimal driving direction is determined by using the spatial geometric relationships of the identified space. A proportional-integral-derivative (PID) controller is used to control the vehicle speed, whereas a model predictive control (MPC) is designed for steering control, where the path tracking error model is used for MPC development. In addition, the AV's dynamics model is built in Adams. The effect and sensitivity of the path-tracking controller are validated through the Matlab-Adams co-simulation. The test results show the satisfactory path-tracking performance.

Integrated path planning and control through proximal policy optimization for a marine current turbine

Robots had a great impact on the manufacturing industry ever since the early seventies when companies such as KUKA and ABB started deploying their first industrial robots. These robots merely performed very specific tasks in specific ways within well-defined environments. Still, they proved to be very useful as they could exceed human performance at these tasks. However, in order to enable robots to enter our daily life, they need to become more versatile and need to operate in much less structured environments. This thesis is partly devoted to stretching these limitations by means of learning, namely imitation learning (IL) and inverse reinforcement learning (IRL). Reinforcement learning (RL) is a powerful approach to enable robots to solve a task in an unknown environment. The practitioner describes a desired behavior by specifying a reward function and the robot autonomously interacts with the environment in order to find a control policy that generates high accumulated reward. However, RL is not suitable for teaching new tasks by non-experts because specifying appropriate reward functions can be difficult. Demonstrating the desired behavior is often easier for non-experts. Imitation learning can be used in order to enable the robot to reproduce the demonstrations. However, without explicitly inferring and modeling the intentions of the demonstrations, it can become difficult to solve the task for unseen situations. Inverse reinforcement learning (IRL) therefore aims to infer a reward function from the demonstrations, such that optimizing this reward function yields the desired behavior even for different situations. This thesis introduces a unifying approach to solve the inverse reinforcement learning problem in the same way as the reinforcement learning problem. This is achieved by framing both problems as information projection problems, i.e., we strive to minimize the relative entropy between a probabilistic model of the robot behavior and a given desired distribution. Furthermore, a trust region on the robot behavior is used to stabilize the optimization. For inverse reinforcement learning, the desired distribution is implicitly given by the expert demonstrations. The resulting optimization can be efficiently solved using state-of-the-art reinforcement learning methods. For reinforcement learning, the log-likelihood of the desired distribution is given by the reward function. The resulting optimization problem corresponds to a standard reinforcement learning formulation, except for an additional objective of maximizing the entropy of the robot behavior. This entropy objective adds little overhead to the optimization, but can lead to better exploration and more diversified policies. Trust-region I-projections are not only useful for training robots, but can also be applied to other machine learning problems. I-projections are typically used for variational inference, in order to approximate an intractable distribution by a simpler model. However, the resulting optimization problems are usually optimized based on stochastic gradient descent which often suffers from high variance in the gradient estimates. As trust-region I-projections where shown to be effective for reinforcement learning and inverse reinforcement learning, this thesis also explores their use for variational inference. More specifically, trust-region I-projections are investigated for the problem of approximating an intractable distribution by a Gaussian mixture model (GMM) with an adaptive number of components. GMMs are highly desirable for variational inference because they can yield arbitrary accurate approximations while inference from GMMs is still relatively cheap. In order to make learning the GMM feasible, we derive a lower bound that enables us to decompose the objective function. The optimization can then be performed by iteratively updating individual components using a technique from reinforcement learning. The resulting method is capable of learning approximations of significantly higher quality than existing variational inference methods. Due to the similarity of the underlying optimization problems, the insights gained from our variational inference method are also useful for IL and IRL. Namely, a similar lower bound can be applied also for the I-projection formulation of imitation learning. However, whereas for variational inference the lower bound serves to decompose the objective function, for imitation learning it allows us to provide a reward signal to the robot that does not depend on its behavior. Compared to reward functions that are relative to the current behavior of the robot---which are typical for popular adversarial methods---behavior-independent reward functions have the advantages that we can show convergence even for greedy optimization. Furthermore, behavior-independent reward functions solve the inverse reinforcement learning problem, thereby closing the gap between imitation learning and IRL. However, algorithms derived from our non-adversarial formulation are actually very similar to existing AIL methods, and we can even show that adversarial inverse reinforcement learning (AIRL) is indeed an instance of our formulation. AIRL was derived from an adversarial formulation, and we point out several problems of that derivation. In contrast, we show that AIRL can be straightforwardly derived from out non-adversarial formulation. Furthermore, we demonstrate that the non-adversarial formulation can be also used to derive novel algorithms by presenting a non-adversarial method for offline imitation learning.

Due to the uncertainty of vehicle model parameters, modeling errors and external disturbances, the performance of path tracking control system is poor, especially under high velocity and large curvature extreme conditions. To address this issue, this paper presents a novel gain-scheduled robust control strategy based on linear parameter varying system with model predictive control (MPC) and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> . Firstly, fully considering the influence of the time-varying characteristics of vehicle velocity and tire cornering stiffness on the path tracking system model, a novel linear parameter varying system model is built for path tracking control of autonomous vehicle. Then, a path tracking robust controller is designed based on gain-scheduled approach, and the linear matrix inequality (LMI) is applied to solve optimization problem, in which MPC and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">H</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> robust control theory are applied to the controller design process. Finally, the simulation experiments have verified that the proposed novel robust control strategy can improve the path tracking accuracy and ensure the vehicle lateral and roll stability, especially under high velocity and large curvature extreme conditions. Meanwhile, the proposed robust controller shows superiority to suppress the parameters uncertainty, modeling error and external disturbance.

To improve the path tracking performance of intelligent hybrid articulated tractors in all working conditions in unmanned operation, a path-tracking control method based on corrected model predictive control is proposed. The kinematic model of the tractor is established by analyzing the tractor’s kinematics. Taking the lateral and longitudinal errors as the target and the speed and articulation angular acceleration as the constraints, a tracking control algorithm based on model predictive control is proposed. In addition, to improve the transient performance of the tractor in the path tracking process, the proportional-integral-derivative controller and fuzzy controller are used to correct the model-predicted output articulation angular acceleration, forming a corrected model predictive control path tracking control method. To verify the effectiveness of the control method, model predictive control is used as a comparison method, and the effectiveness of the proposed method is verified based on the MATLAB 2024a simulation platform. The results show that compared with the MPC algorithm, the speed standard deviation is reduced by 2%, the longitudinal tracking error is reduced by 8%, and the lateral tracking error is reduced by 50%. The proposed method can effectively improve the path-tracking accuracy of the intelligent hybrid tractor.

A novel regional path tracking description is presented in this manuscript, and the moving horizon control method that is model predictive control (MPC) is proposed to discuss the regional path tracking issue which could avoid colliding road boundary when tracking a more complex road effectively. The feasible region for autonomous ground vehicles (AGVs) running is determined first according to the detected road boundaries. Then, in order to keep the actual trajectory of AGVs in the region and satisfy the safety requirements, MPC method is employed to design the path tracking controller considering actuator and road boundary constraints. In order to verify the effectiveness of the proposed method, experiments based on Hongqi AGV HQ430 are carried out, and the results illustrate that the presented method could be successfully applied to Hongqi AGV vehicle HQ 430.

Human drivers have rich and diverse driving characteristics on curved roads. Finding the characteristic quantities of the experienced drivers during curve driving and applying them to the steering control of autonomous vehicles is the research goal of this article. We first recruited 10 taxi drivers, 5 bus drivers, and 5 driving instructors as the representatives of experienced drivers and conducted a real car field experiment on six curves with different lengths and curvatures. After processing the collected driving data in the Frenet frame and considering the free play of a real car’s steering system, it was interesting to observe that the shape enclosed by steering wheel angles and the coordinate axis was a trapezoid. Then, we defined four feature points, four feature distances, and one feature steering wheel angle, and the trapezoidal steering wheel angle (TSWA) model was developed by backpropagation neural network with the inputs were vehicle speeds at four feature points, and road curvature and the outputs were feature distances and feature steering wheel angle. The comparisons between TSWA model and experienced drivers, model predictive control, and preview-based driver model showed that the proposed TSWA model can best reflect the steering features of experienced drivers. What is more, the concise expression and human-like characteristic of TSWA model make it easy to realize human-like steering control for autonomous vehicles. Lastly, an autonomous vehicle composed of a nonlinear vehicle model and electric power steering (EPS) system was established in Simulink, the steering wheel angles generated by TSWA model were tracked by EPS motor directly, and the results showed that the EPS system can track the steering angles with high accuracy at different vehicle speeds.

Reinforcement learning (RL) was first demonstrated to be a feasible approach to controlling heating, ventilation, and air conditioning (HVAC) systems more than a decade ago. However, there has been limited progress towards a practical and scalable RL solution for HVAC control. While one can train an RL agent in simulation, it is not cost-effective to create a model for each thermal zone or building. Likewise, existing RL agents generally take a long time to learn and are opaque to expert interrogation, making them unattractive for real-world deployment. To tackle these challenges, we propose Gnu-RL: a novel approach that enables practical deployment of RL for HVAC control and requires no prior information other than historical data from existing HVAC controllers. To achieve this, Gnu-RL adopts a recently-developed Differentiable Model Predictive Control (MPC) policy, which encodes domain knowledge on planning and system dynamics, making it both sample-efficient and interpretable. Prior to any interaction with the environment, a Gnu-RL agent is pre-trained on historical data using imitation learning, which enables it to match the behavior of the existing controller. Once it is put in charge of controlling the environment, the agent continues to improve its policy end-to-end, using a policy gradient algorithm. We evaluate Gnu-RL on both an EnergyPlus model and a real-world testbed. In both experiments, our agents were directly deployed in the environment after offline pre-training on expert demonstration. In the simulation experiment, our approach saved 6.6% energy compared to the best published RL result for the same environment, while maintaining a higher level of occupant comfort. Next, Gnu-RL was deployed to control the HVAC of a real-world conference room for a three-week period. Our results show that Gnu-RL saved 16.7% of cooling demand compared to the existing controller and tracked temperature set-point better.

In this paper, a novel path tracking method is proposed to improve the accuracy and stability of path tracking for the autonomous driving robotic vehicle (ADRV). This method is established based on an improved model predictive control (IMMPC) method with some optimization algorithms integrated. To apply this method, firstly, the steering system is simplified and a vehicle error dynamic model considering road curvature is used as the algorithm prediction model. Then, the Whale Optimization algorithm is combined with the Backpropagation (WOA-BP) neural network algorithm to predict the optimal dual time domain parameters, which can improve the algorithm’s robustness against variations in speed and road adhesion coefficients. Meanwhile, the constraints on sideslip angles and environmental road conditions are considered, and integrate them into the nonlinear cost function to avert vehicle rollovers. Furthermore, an enhanced hierarchical particle swarm optimization algorithm with warm start is employed to replace the traditional quadratic sequence method to mitigate local optimal issues, this nonlinear cost function, so that the nonlinear cost function can be optimized. In the last part, co-simulations and driving robot hardware-in-the-loop (HIL) experiments are constructed to validate the stability and precision of the algorithm. The results show that the improved MPC can smoothly track the test path under environmental constraints with higher accuracy and enhanced control stability.

This paper investigates the error accumulation problem in centralized training and decentralized execution (CTDE) policy‐based multi‐agent reinforcement learning (MARL) algorithms, which arises from local observation inaccuracies. To address this issue, we propose a novel MARL algorithm that incorporates imitation learning using local observations. Firstly, by analysing the multi‐agent proximal policy optimization algorithm and examining the problems arising when global states are replaced with local observations, it is proved that insufficient observations can lead to information loss, thereby introducing errors of advantage function, and it is demonstrated that the generalized advantage estimation method accumulates errors during the training process. Then, imitation learning is introduced and a novel training framework that combines reinforcement learning and imitation learning is proposed. During the reinforcement learning phase, an MARL agent trained with global observations acts as an expert. Subsequently, imitation learning is applied to train another agent that mimics the expert's decisions using only local observations. Finally, the effectiveness of this algorithm is verified in some commonly used multi‐agent environments, which demonstrates its superior performance compared to traditional multi‐agent reinforcement learning algorithms.

Chemical space is routinely explored by machine learning methods to discover interesting molecules, before time-consuming experimental synthesizing is attempted. However, these methods often rely on a graph representation, ignoring 3D information necessary for determining the stability of the molecules. We propose a reinforcement learning (RL) approach for generating molecules in Cartesian coordinates allowing for quantum chemical prediction of the stability. To improve sample-efficiency we learn basic chemical rules from imitation learning (IL) on the GDB-11 database to create an initial model applicable for all stoichiometries. We then deploy multiple copies of the model conditioned on a specific stoichiometry in a RL setting. The models correctly identify low energy molecules in the database and produce novel isomers not found in the training set. Finally, we apply the model to larger molecules to show how RL further refines the IL model in domains far from the training data.

The model predictive control method can be effective for converter control in distributed power generation but requires a large amount of computation, leading to a considerable time delay in the actuation. If the delay is not reflected, the system performance could get worse. This paper presents a two-step (horizon) prediction algorithm of model predictive control technique for grid-tied inverters used in wind turbine systems. The control objectives such as active and reactive power flow and switching loss reduction are reflected in the objective function of the controller. The proposed model predictive control strategy is verified numerically by using MATLAB/Simulink.