Robust reinforcement learning control with static and dynamic stability

We propose a robust control system which has robustness for disturbance and can deal with a nonlinear system with partially unknown structure by fusing reinforcement learning and robust control theory. First, we solved an optimal control problem without using unknown part of functions of the system, using neural network and the repetition learning of reinforcement learning algorithm. Second, we built the robust reinforcement learning control system which permits uncertainty and has robustness for disturbance by fusing the idea of H infinity control theory with above system.

In this chapter, a novel control method using a reinforcement learning (RL) (Sutton and Barto (1998)) with concept of sliding mode control (SMC) (Slotine and Li (1991)) for unknown dynamical system is considered. In designing the control system for unknown dynamical system, there are three approaches. The first one is the conventional model-based controller design, such as optimal control and robust control, each of which is mathematically elegant, however both controller design procedures present a major disadvantage posed by the requirement of the knowledge of the system dynamics to identify and model it. In such cases, it is usually difficult to model the unknown system, especially, the nonlinear dynamical complex system, to make matters worse, almost all real systems are such cases. The second one is the way to use only the soft-computing, such as neural networks, fuzzy systems, evolutionary systems with learning and so on. However, in these cases it is well known that modeling and identification procedures for the dynamics of the given uncertain nonlinear system and controller design procedures often become time consuming iterative approaches during parameter identification and model validation at each step of the iteration, and in addition, the control system designed through such troubles does not guarantee the stability of the system. The last one is the way to use the method combining the above the soft-computing method with the model-based control theory, such as optimal control, sliding mode control (SMC), H∞ control and so on. The control systems designed through such above control theories have some advantages, that is, the good nature which its adopted theory has originally, robustness, less required iterative learning number which is useful for fragile system controller design not allowed a lot of iterative procedure. This chapter concerns with the last one, that is, RL system, a kind of soft-computing method, supported with robust control theory, especially SMC for uncertain nonlinear systems. RL has been extensively developed in the computational intelligence and machine learning societies, generally to find optimal control policies for Markovian systems with discrete state and action space. RL-based solutions to the continuous-time optimal control problem have been given in Doya (Doya (2000). The main advantage of using RL for solving optimal

Reinforcement learning (RL) is one of the popular methods for intelligent control and decision making in the field of robotics recently. The goal of RL is to learn an optimal policy of the agent by interacting with the environment via trail and error. There are two main algorithms for RL problems, including model-free and model-based methods. Model-free RL is driven by historical trajectories and empirical data of the agent to optimize the policy, which needs to take actions in the environment to collect the trajectory data and may cause the damage of the robot during training in the real environment. The main different between model-based and model-free RL is that a model of the transition probability in the interaction environment is employed. Thus the agent can search the optimal policy through internal simulation. However, the model of the transition probability is usually estimated from historical data in a single environment with statistical errors. Therefore, an issue is faced by the agent is that the optimal policy is sensitive to perturbations in the model of the environment which can lead to serious degradation in performance. Robust RL aims to learn a robust optimal policy that accounts for model uncertainty of the transition probability to systematically mitigate the sensitivity of the optimal policy in perturbed environments. In this overview, we begin with an introduction to the algorithms in RL, then focus on the model uncertainty of the transition probability in robust RL. In parallel, we highlight the current research and challenges of robust RL for robot control. To conclude, we describe some research areas in robust RL and look ahead to the future work about robot control in complex environments.

This publication presents a new method by which control methods based on reinforcement learning can be combined with classical robust control methods. The combination results in a robust management system that meets high-quality criteria. The described method is presented through the control of an autonomous vehicle. By choosing the reward function chosen during reinforcement learning, various driving styles can be realized, e.g., lap time minimization, track tracking, and travel comfort. The neural network was trained using the Proximal Policy Optimization algorithm, and the robust control is based on H∞. The two controllers are combined using a supervisor structure, in which a quadratic optimization task is implemented. The result of the method is a control structure that realizes the longitudinal and lateral control of the vehicle by specifying the reference speed and the steering angle. The effectiveness of the algorithm is demonstrated through simulations.

Background: The conventional method using one-degree-of-freedom (1DOF) controller for Permanent Magnet Synchronous Motor (PMSM) servo system has the trade-off problem between the dynamic performance and the robustness. Methods: In this paper, by using H∞ control theory, a novel robust two-degree-of-freedom (2DOF) controller has been proposed to improve the position control performance of PMSM servo system. Using robust control theory and 2DOF control theory, a H∞ robust position controller has been designed and discussed in detail. Results: The trade-off problem between the dynamic performance and robustness which exists in one-degree-of-freedom (1DOF) control can be dealt with by the application of 2DOF control theory. Then, through H∞ control theory, the design of robust position controller can be translated to H∞ robust standard design problem. Moreover, the control system with robust controller has been proved to be stable. Conclusion: Further simulation results demonstrate that compared with the conventional PID control, the designed control system has better robustness and attenuation to the disturbance of load impact.

Microgrids consist of multiple distributed generators, which are normally integrated via power inverters and can employ various sustainable energy resources. Therefore, the microgrid is a new promising concept for future power systems with improved energy security, reliability, quality of power, and energy efficiency. To this end, microgrids should survive during severe power system abnormalities so that the sub-areas in the microgrids can be protected all the time. This paper focuses on a robust controller design scheme for inverter-interfaced distributed generators considering various operating modes and loading conditions of microgrids. Especially, during the island mode, the voltage and system frequency can easily change due to instant power mismatch between distributed generators and loads in the microgrid. This paper presents the design procedure of a robust optimal controller against exogenous disturbances such as voltage and frequency variation in microgrids. L 1 robust control theory is applied to optimize the robust controller and a Particle Swarm Optimization algorithm is used to solve the optimal problems.

Model predictive control (MPC) has established itself as the standard advanced process control method. However, solving the resulting optimization problems in real-time can be challenging, especially when uncertainty is explicitly considered in a robust nonlinear predictive control approach. An increasingly popular alternative to avoid the online solution of the resulting optimization problems is to approximate their solution using neural networks. The networks are trained using many solutions of the MPC problem for different system states and therefore this approach is often called imitation learning. Controllers obtained via imitation learning have important drawbacks, since it is difficult to learn behaviors that are not well represented in the data and they must be redesigned from scratch when the control task changes. In this work, we show that these two drawbacks can be mitigated by combining imitation learning and concepts from reinforcement learning. The central idea is to use imitation learning as a very good initialization of a control policy that is iteratively updated using reinforcement learning, taking advantage of the fact that an explicit and differentiable expression of the approximate MPC controller is available. The efficacy of the combination of the two learning paradigms is highlighted via simulations of a semi-batch industrial polymerization reactor.

Robust vibration control of smart composite beams using neural networks was studied. Linear quadratic Gaussian with loop transfer recovery (LQG/LTR) methodology was used to design a robust controller on the basis of the state space model of the system. The state space model of the system was obtained using the finite-element method and mode superposition. The finite-element model was based on a higher-order shear deformation theory which included the lateral strains. The mode superposition method was used to transform the coupled finite-element equations of motion in the physical coordinates into a set of reduced uncoupled equations in the modal coordinates. The performance of the LQG/LTR controller was verified for various arbitrary initial conditions. A system of neural networks was then trained to emulate the robust controller. The neural network system was trained using the backpropagation algorithm. After suitable training, the NN (neural network) controller was shown to effectively control the vibrations of the composite beam. A robustness study including the effects of tip mass, structural parameter variation, and loss of a sensor input was performed. The NN controller is shown to provide robustness and control capabilities equivalent to that of the LQG/LTR controller.

The complexity in planning and control of robot compliance tasks mainly results from simultaneous control of both position and force and inevitable contact with environments. It is quite difficult to achieve accurate modeling of the interaction between the robot and the environment during contact. In addition, the interaction with the environment varies even for compliance tasks of the same kind. To deal with these phenomena, in this paper, we propose a reinforcement learning and robust control scheme for robot compliance tasks. A reinforcement learning mechanism is used to tackle variations among compliance tasks of the same kind. A robust compliance controller that guarantees system stability in the presence of modeling uncertainties and external disturbances is used to execute control commands sent from the reinforcement learning mechanism. Simulations based on deburring compliance tasks demonstrate the effectiveness of the proposed scheme.

Adaptive robust servo constraint tracking control for an underactuated quadrotor UAV with mismatched uncertainties

An integrated modeling and robust multivariable control approach is presented for paper machines. An application of this approach is demonstrated on a high-fidelity simulator. Modeling relies on input-output data, collected from an identification experiment at the desired operating condition. An estimate of the process model along with the uncertainty bounds that describe the confidence limits of the model, consistent with the robust control theory, is obtained. The results are used to design an multivariable controller based on loop-shaping principles and guided by the estimated uncertainty bounds. The simulations demonstrate the suitability of the approach and illustrate that the technique can be used to provide high bandwidth performance for both servo and regulatory control.

For the speed control system of autonomous electric vehicle (AEV), challenge happens with how to determine an appropriate driving speed to satisfy the dynamic environment while resisting uncertainty and disturbance. Therefore, this paper proposes a robust optimal speed control approach based on hierarchical architecture for AEV through combining deep reinforcement learning (DRL) and robust control. In decision‐making layer, a deep maximum entropy proximal policy optimization (DMEPPO) algorithm is presented to obtain an optimal speed via dynamic environment information, heuristic target entropy and adaptive entropy constraint. In motion control layer, to track the learned optimal speed while resisting uncertainty and disturbance, a robust speed controller is designed by the linear matrix inequality (LMI). Finally, simulation experiment results show that the proposed robust optimal speed control scheme based on hierarchical architecture for AEV is feasible and effective.

Robust tracking control of uncertain nonlinear systems with adaptive dynamic programming

Fuzzy-based predictive deep reinforcement learning for robust and constrained optimal control of industrial solar thermal plants

Lower back injuries in rowing are responsible for more time loss from training in National level rowers compared to International level rowers. For rib stress fractures International rowers are more at risk