A TD3-Based Reinforcement Learning Algorithm With Curriculum Learning for Adaptive Yaw Control in All-Wheel-Drive Electric Vehicles

This study presents an innovative solution for simultaneous energy optimization and dynamic yaw control of all-wheel-drive (AWD) electric vehicles (EVs) using deep reinforcement learning (DRL) techniques. To this end, three model-free DRL-based methods, based on deep deterministic policy gradient (DDPG), twin delayed deep deterministic policy gradient (TD3), and TD3 enhanced with curriculum learning (CL TD3), are developed for determining optimal yaw moment control and energy optimization online. The proposed DRL controllers are benchmarked against model-based controllers, i.e., linear quadratic regulator with the sequential quadratic programming (LSQP) and sliding mode control with SQP (SSQP). A tailored multi-term reward function is structured to penalize excessive yaw rate error, sideslip angle, tire slip deviations beyond peak grip regions, and power losses based on a realistic electric machine efficiency map. The learning environment is based on a nonlinear double-track vehicle model, incorporating tire-road interactions. To evaluate the generalizability of the algorithms, the agents are tested across various velocities, tire-road friction coefficients, and additional scenarios implemented in IPG CarMaker, a high-fidelity vehicle dynamics simulator. In addition to the deployment without requiring an explicit model of the plant, the simulation results demonstrate that the proposed solution modifies vehicle dynamics and maneuverability in most cases compared to the model-based conventional controller. Furthermore, the reduction in sideslip angle, excellent traction through minimizing tire slip ratio, avoiding oversteering and understeering, and maintaining an acceptable range of energy optimization are demonstrated for DRL controllers, especially for the TD3 and CL TD3 algorithms.

Controlling feedback control systems in continuous action spaces has always been a challenging problem. Nevertheless, reinforcement learning is mainly an area of artificial intelligence (AI) because it has been used in process control for more than a decade. However, the existing algorithms are unable to provide satisfactory results. Therefore, this research uses a reinforcement learning (RL) algorithm to manage the control system. We propose an adaptive speed control of the motor system based on depth deterministic strategy gradient (DDPG). The actor-critic scenario using DDPG is implemented to build the RL agent. In addition, a framework has been created for traditional feedback control systems to make RL implementation easier for control systems. The RL algorithms are robust and proficient in using trial and error to search for the best strategy. Our proposed algorithm is a deep deterministic policy gradient, in which a large amount of training data trains the agent. Once the system is trained, the agent can automatically adjust the control parameters. The algorithm has been developed using Python 3.6 and the simulation results are evaluated in the MATLAB/Simulink environment. The performance of the proposed RL algorithm is compared with a proportional integral derivative (PID) controller and a linear quadratic regulator (LQR) controller. The simulation results of the proposed scheme are promising for the feedback control problems.

Reinforcement learning (RL) algorithms have been successfully applied to control tasks associated with unmanned aerial vehicles and robotics. In recent years, safe RL has been proposed to allow the safe execution of RL algorithms in industrial and mission-critical systems that operate in closed loops. However, if the system operating conditions change, such as when an unknown fault occurs, typical safe RL algorithms cannot adapt while retaining past knowledge. Continual RL algorithms have been proposed to address this issue. However, the impact of continual adaptation on the system’s safety is an understudied problem. In this paper, we study the intersection of safe and continual RL. First, we empirically demonstrate that a popular continual RL algorithm, elastic weight consolidation, does not satisfy safety constraints in nonlinear systems subject to varying operating conditions. Specifically, we study the MuJoCo HalfCheetah and Ant environments with velocity constraints and sudden joint loss non-stationarity. Then, we show that an agent trained using constrained policy optimization, a safe RL algorithm, experiences catastrophic forgetting in continual learning settings. With this in mind, we explore a simple reward-shaping method to ensure that elastic weight consolidation prioritizes remembering both safety and task performance for safety-constrained, nonlinear, and non-stationary systems.

The intrinsic gradient in organisms governs long-term information processing and continual learning in dynamic environments. Reinforcement learning (RL) aims to emulate such temporal regulation to enhance learning efficiency and adaptability. However, the absence of intrinsic gradient construction in existing memristive devices leads to stochastic and abrupt state changes, disrupting the generation of temporally correlated internal states critical for continual RL. In this work, a second-order memristor incorporating a stable intrinsic oxygen gradient was designed via a molecular-coordinated layer, which enables a prolonged dynamic barrier evolution (>102 s). This slow dynamic response facilitates balanced oxygen ion migration and diffusion under unipolar spike stimulation, resulting in a significant conductance modulation (ΔG = -98.1%). These temporally adaptive conductance states were quantitatively mapped to learning rates in the RL algorithm, allowing the learning task timescale to co-evolve with device dynamics. Compared with conventional strategies, intrinsic-gradient-driven modulation reduced training iterations by 68.75% and 35.65% in static and dynamic environments, respectively. These findings underscore the potential of slow-dynamic second-order memristors as physically grounded time-adaptive units bridging device dynamics and algorithmic learning in neuromorphic systems.

Active screening is a common approach in controlling the spread of recurring infectious diseases such as tuberculosis and influenza. In this approach, health workers periodically select a subset of population for screening. However, given the limited number of health workers, only a small subset of the population can be visited in any given time period. Given the recurrent nature of the disease and rapid spreading, the goal is to minimize the number of infections over a long time horizon. Active screening can be formalized as a sequential combinatorial optimization over the network of people and their connections. The main computational challenges in this formalization arise from i) the combinatorial nature of the problem, ii) the need of sequential planning and iii) the uncertainties in the infectiousness states of the population. Previous works on active screening fail to scale to large time horizon while fully considering the future effect of current interventions. In this paper, we propose a novel reinforcement learning (RL) approach based on Deep Q-Networks (DQN), with several innovative adaptations that are designed to address the above challenges. First, we use graph convolutional networks (GCNs) to represent the Q-function that exploit the node correlations of the underlying contact network. Second, to avoid solving a combinatorial optimization problem in each time period, we decompose the node set selection as a sub-sequence of decisions, and further design a two-level RL framework that solves the problem in a hierarchical way. Finally, to speed-up the slow convergence of RL which arises from reward sparseness, we incorporate ideas from curriculum learning into our hierarchical RL approach. We evaluate our RL algorithm on several real-world networks. Results show that our RL algorithm can scale up to 10 times the problem size of state-of-the-art (the variant that considers the effect of future interventions but unscalable) in terms of planning time horizon. Meanwhile, it outperforms state-of-the-art (the variant that scales up but does not consider the effect of future interventions) by up to 33% in solution quality.

P2P power trading based on reinforcement learning for nanogrid clusters

G-Learner and GIRL: Goal Based Wealth Management with Reinforcement Learning

Development and Evaluation of a Smart Charging Strategy for an Electric Vehicle Fleet Based on Reinforcement Learning

One of the well-known methods for improving the lateral stability of a vehicle is to utilize an active steering controller. Common active steering systems use feedback signals from a yaw rate sensor and side-slip angle estimation. However, in some driving conditions, skilled drivers can prevent vehicle rollover using the steering angle of the vehicle. Hence, the effect of an active front-steering controller on the roll angle of the vehicle is studied in this paper. For this study, a three-degree-of-freedom linear model of a vehicle which has the lateral motion, the yaw motion and the roll motion as the degrees of freedom is derived. The effectiveness of the active steering controller on the roll stability of the vehicle is examined by simulations in a wide range of longitudinal velocities of the vehicle using a linear quadratic regulator controller. The results obtained indicate that the active steering controller increases the roll stability of the vehicle only at low vehicle speeds and also that consideration of the roll degree of freedom in the design of the active steering controller is not effective at high vehicle speeds. Therefore a two-degree-of-freedom bicycle model is sufficient for active steering controller design. Another part of this paper is dedicated to designing a practical active steering controller for a vehicle with time-varying and uncertain parameters. In fact, designing a controller which guarantees the stability of the vehicle with simultaneous uncertainties in the cornering stiffnesses of the tyres and in the time-varying velocity is the main goal of this paper. Another important advantage of the proposed controller is its simplicity and static structure. In fact, this controller design is carried out offline and can be implemented for use in a real vehicle. This simple and powerful controller is a robust linear quadratic regulator controller. For designing a robust linear quadratic regulator controller, a polytopic model of the two-degree-of-freedom vehicle is obtained, which considers the uncertainty in the parameters. In addition, a linear matrix inequality is used to design a linear quadratic regulator controller for an uncertain or parameter-varying system. Finally, the performance of the designed robust linear quadratic regulator controller was studied by simulating the vehicle responses in some manoeuvres. A sport utility vehicle model is used for simulation purposes. This non-linear vehicle model has eight degrees of freedom and its tyres are assumed to be both linear and non-linear. The performance of the robust linear quadratic regulator controller was studied for vehicles with both linear and non-linear Pacejka tyres. The proposed controller guarantees robust stability of the closed-loop system in the presence of 50% variation in the vehicle speed and 20% uncertainty in the stiffnesses of the tyres.

Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning

This paper proposes a reinforcement learning (RL) algorithm for the security problem of state estimation of cyber-physical system (CPS) under denial-of-service (DoS) attacks. The security of CPS will inevitably decline when faced with malicious cyber attacks. In order to analyze the impact of cyber attacks on CPS performance, a Kalman filter, as an adaptive state estimation technology, is combined with an RL method to evaluate the issue of system security, where estimation performance is adopted as an evaluation criterion. Then, the transition of estimation error covariance under a DoS attack is described as a Markov decision process, and the RL algorithm could be applied to resolve the optimal countermeasures. Meanwhile, the interactive combat between defender and attacker could be regarded as a two-player zero-sum game, where the Nash equilibrium policy exists but needs to be solved. Considering the energy constraints, the action selection of both sides will be restricted by setting certain cost functions. The proposed RL approach is designed from three different perspectives, including the defender, the attacker and the interactive game of two opposite sides. In addition, the framework of Q-learning and state–action–reward–state–action (SARSA) methods are investigated separately in this paper to analyze the influence of different RL algorithms. The results show that both algorithms obtain the corresponding optimal policy and the Nash equilibrium policy of the zero-sum interactive game. Through comparative analysis of two algorithms, it is verified that the differences between Q-Learning and SARSA could be applied effectively into the secure state estimation in CPS.

ABSTRACTThe handling behaviour of vehicles is an important property for its relation to performance and safety. In 1970s, Pacejka did the groundwork for an objective analysis introducing the handling diagram and the understeer coefficient. In more recent years, the understeer concept is still mentioned but the handling is actively managed by direct yaw control (DYC). In this paper an accurate analysis of the vehicle handling is carried out, considering also the effect of drive forces. This analysis brings to a new formulation of the understeer coefficient, which is almost equivalent to the classical one, but it can be obtained by quasi-steady-state manoeuvres. In addition, it relates the vehicle yaw torque to the understeer coefficient, filling up the gap between the classical handling approach and DYC. A multibody model of a Formula SAE car is then used to perform quasi-steady-state simulations in order to verify the effectiveness of the new formulation. Some vehicle set-ups and wheel drive arrangements are simulated and the results are discussed. In particular, the handling behaviours of the rear wheel drive (RWD) and the front wheel drive (FWD) architectures are compared, finding an apparently surprising result: for the analysed vehicle the FWD is less understeering than for RWD. The relation between the yaw torque and the understeer coefficient allows to understand this behaviour and opens-up the possibility for different yaw control strategies.

Recent advancement on reinforcement learning (RL) algorithms shows that effective learning of parametric action- selection policies can often be achieved through direct opti- mization of a performance lower bound subject to pre-defined policy behavioral constraints. Driven by this understanding, this paper seeks to develop new policy search techniques where RL is achieved through maximizing a performance lower bound ob- tained originally based on an Expectation-Maximization method. For reliable RL, our new learning techniques must also simul- taneously guarantee constrained policy behavioral changes mea- sured through KL divergence. Two separate approaches will be pursued to tackle our constrained policy optimization problems, resulting in two new RL algorithms. The first algorithm utilizes a conjugate gradient technique and a Bayesian learning method for approximate optimization. The second algorithm focuses on minimizing a loss function derived from solving the Lagrangian for constrained policy search. Both algorithms have been experi- mentally examined on several benchmark problems provided by OpenAI GYM. The experiment results clearly demonstrate that our algorithms can be highly effective in comparison to several well-known RL algorithms.

In this work, an Adaptive Linear Quadratic Regulator (LQR) is proposed for the tracking control of a 3 DOF articulated robotic manipulator. The main objective of this work is to drive the robotic manipulator to its equilibrium point from any initial position of the end effector of a robotic manipulator in an adaptive way even if the system dynamics consist of modelling uncertainties. The conventional LQR is able to provide a good measure of robustness against disturbances and achieve good tracking performance. However, it will give degraded tracking performance in case of the presence of model uncertainties. Therefore LQR tracking performance is enhanced by using a Model Reference Adaptive Controller (MRAC) in the literature for enabling a 2 DOF system to achieve good tracking against modelling uncertainties. But it has not been applied for a 3 DOF system till date. Therefore in this work, it is proposed to be applied for such a system for the first time to realize an adaptive control. The forward and inverse kinematics level equations are formulated using an indirect form of the Denavit Hartenberg (DH) convention. The dynamics of robotic manipulator are derived using the Euler-Lagrange formulation. The dynamics of manipulator with the LQR control are considered to be the reference dynamics for the MRAC control and the modelling uncertainty will be added to the system dynamics to test its performance. The MRAC augmented LQR provides faster convergence with improved stability against modelling uncertainties.

Ship welding is a crucial part of ship building, requiring higher levels of robot coordination and working efficiency than ever before. To this end, this paper studies the coordinated ship-welding task, which involves multi-robot welding of multiple weld lines consisting of synchronous ones to be executed by a pair of robots and normal ones that can be executed by one robot. To evaluate working efficiency, the objectives of optimal lazy robot ratio and energy consumption were considered, which are tackled by the proposed dynamic Kuhn–Munkres-based model-free policy gradient (DKM-MFPG) reinforcement learning algorithm. In DKM-MFPG, a dynamic Kuhn–Munkres (DKM) dispatcher is designed based on weld line and co-welding robot position information obtained by the wireless sensors, such that robots always have dispatched weld lines in real-time and the lazy robot ratio is 0. Simultaneously, a model-free policy gradient (MFPG) based on reinforcement learning is designed to achieve the energy-optimal motion control for all robots. The optimal lazy robot ratio of the DKM dispatcher and the network convergence of MFPG are theoretically analyzed. Furthermore, the performance of DKM-MFPG is simulated with variant settings of welding scenarios and compared with baseline optimization methods. Compared to the four baselines, DKM-MFPG owns a slight performance advantage within 1% on energy consumption and reduces the average lazy robot ratio by 11.30%, 10.99%, 8.27%, and 10.39%.