Robust control for a class of nonlinear systems with input constraints based on actor‐critic learning

A formulation of the coupled modeling and robust control design problem is presented. The distinguishing feature is the treatment of the control design model as a degree of design freedom. A simple example highlights the potential improvement in closed-loop performance under the coupled approach. Quantification of the strength of the coupling in terms of performance boundaries is also described. >

In this paper, the direct neural dynamic programming technique is utilized to solve the Hamilton Jacobi-Bellman (HJB) equation online and forward-in-time for the decentralized nearly optimal control of nonlinear interconnected discrete-time systems in affine form with unknown internal subsystem and interconnection dynamics. Only the state vector of the local subsystem is considered measurable. The decentralized optimal controller design for each subsystem consists of an action neural network (NN) that is aimed to provide a nearly optimal control signal, and a critic NN which approximates the cost function. The NN weights are tuned online for both the NNs. It is shown that all subsystems signals are uniformly ultimately bounded (UUB) and that the subsystem inputs approach their corresponding nearly optimal control inputs with bounded error.

ABSTRACTThis paper investigates a nonlinear two‐player non‐zero‐sum differential game with state and input constraints. To solve this problem, this paper constructs a neural network (NN) framework to approximate the solution of the Hamilton‐Jacobi‐Isaacs (HJI) equation. The adaptive dynamic programming (ADP) method is utilized where each player only needs one critic NN. To solve the issue of state and input saturations, this paper develops a novel constrained system for the differential game, firstly to make the states within the predetermined constraint set. Then, the non‐quadratic expression is used to substitute the traditional quadratic expression for the two‐player non‐zero‐sum differential game, and both of the inputs of the two players are constrained. With these treatments, the control input and the system are more in line with real‐world applications. Moreover, the stability of the system is also analyzed using the Lyapunov theorem. Two numerical examples are presented to illustrate that the critic NN weights estimation errors and the system are uniformly ultimately bounded (UUB), and the state and input constraints can be achieved.

This article develops an adaptive finite‐time stable control algorithm for an accurate manipulation of robot arms considering unknown model parameters and external disturbances. First, a state‐dependent twisting control algorithm is developed for the robust manipulation of robot arms under the consideration of bounded unknown uncertainties conditions. The more challenging problem of robust control design under consideration of totally unknown dynamical parameters is investigated in a second stage by supporting the control law with a suitable online parameters estimator. The objective of the parameters estimator is to drive the vector of estimated parameters to some level that ensures the establishment of real second‐order sliding mode without overloading the discontinuous controller gains. This approach based on combining both techniques of robust and adaptive algorithms allows overcoming large levels of uncertainties while preserving a high level of tracking accuracy without exciting the unwanted phenomena of chattering. The system finite‐time stability is proven by using Lyapunov theory and the weighted homogeneity principle. Practical tests are reported to support theoretical study results.

This paper is concerned with the problem of robust controller design for electro-hydraulic servo systems driven by double-rod hydraulic actuators. Firstly, by introducing a linear varying parameter, an electro hydraulic servo system is described as a linear parameter varying(LPV) model. Secondly, based on this LPV model and a parameter-dependent Lyapunov approach, two parameter-dependent controllers are established for the stabilization and tracking problems, respectively. The obtained controllers are simple and their gains can vary with the parameter measured online. Finally, the numerical simulations demonstrate the satisfactory performance of the proposed controllers.

Robust control for nonlinear systems with structured [formula omitted]2-gain bounded uncertainty

Summary form only given. This paper presents a computational approach for an offline measurement based design of an optimal damping controller using adaptive critics for a new type of FACTS device - the gate controlled series capacitor (GCSC). Remote measurements are provided to the controller to damp out system modes. The optimal controller is developed based on the heuristic dynamic programming (HDP) approach. Three multi-layer perceptron (MLP) neural networks are used in the design - the identifier/model network to identify the dynamics of the power system; the critic network to evaluate the performance of the damping controller; and the controller network to provide optimal damping. This measurement based technique provides an alternative to the classical linear model based optimal control design. The eigenvalue analysis of the closed loop system is performed with time domain responses using the Prony method. Analysis of the simulation results show potential of the HDP based optimal damping controller on a GCSC for enhancing the stability of the power system.

Bellman’s discrete dynamic programming is one of the most general approaches to solve optimal control problems. For discrete-time dynamical systems, it is, at least theoretically, capable to determine globally optimal control laws. In most practical cases, both state and control variables are subject to constraints. Due to the necessity for gridding of the range of both state and control variables in numerical implementations of dynamic programming, the computational effort grows exponentially with increasing system dimensions. This fact is well known as the curse of dimensionality. Furthermore, gridding of intervals representing uncertain system parameters is inevitable, if dynamic programming is used for the design of optimal controllers for systems with uncertainties. In this contribution, an interval arithmetic procedure for the design of optimal and robust controllers is presented. This procedure relies on the basic concepts of dynamic programming. Sophisticated techniques for the exclusion of non-optimal control strategies significantly reduce the computational burden. Since interval techniques can be applied to both continuous-time and discrete-time dynamical systems, the interval arithmetic optimization approach presented in this chapter is applicable to both cases. In addition, the inclusion of effects of uncertain parameters in the underlying optimality criteria is demonstrated. For that purpose, interval arithmetic routines for analysis and design of optimal and robust controllers have been developed. Details about computationally efficient implementations of interval arithmetic optimization procedures and numerical results for a mechanical positioning system with statedependent switchings between different dynamical models for viscous and Coulomb friction are summarized.

In this work, an adaptive neural network (NN) optimized output-feedback control problem is studied for a class of stochastic nonlinear systems with unknown nonlinear dynamics, input saturation, and state constraints. A nonlinear state observer is designed to estimate the unmeasured states, and the NNs are used to approximate the unknown nonlinear functions. Under the framework of the backstepping technique, the virtual and actual optimal controllers are developed by employing the actor-critic architecture. Meanwhile, the tan-type Barrier optimal performance index functions are developed to prevent the nonlinear systems from the state constraints, and all the states are confined within the preselected compact sets all the time. It is worth mentioning that the proposed optimized control is clearly simple since the reinforcement learning (RL) algorithm is derived based on the negative gradient of a simple positive function. Furthermore, the proposed optimal control strategy ensures that all the signals in the closed-loop system are bounded. Finally, a practical simulation example is carried out to further illustrate the effectiveness of the proposed optimal control method.

The authors propose a method by which to design an optimum vibration control system composed of continuous body structures using modal analysis and sensitivity analysis. The mechanical structures are modeled first by FEM. This spatial model is of such a high degree-of-freedom (DOF) that a control system cannot be designed. This spatial model is transformed into a modal model in order to reduce DOF. The optimum control system is designed by applying the optimum control theory to this reduced model. However, stability of the control system is not always guaranteed because the modal model ignores high-order modes. An optimization method of the control system which guarantees stability of the actual system is proposed in this paper. We propose also an experimental optimization of the control system based on experimental modal analysis. The optimization is performed using sensitivity analysis and nonlinear programming. The effectiveness of the proposed method is verified by both simulation and experiment.

This paper proposes a cooperative game-oriented optimal design problem of robust control for uncertain mechanical systems. State of the concerned system is affected by (possibly fast) time-varying but bounded uncertainty. The task is to drive the system to obey a set of prescribed constraints. A $$\beta $$ -measure is defined to gauge the constraint-following error; based on which, a robust control with two tunable parameters is then proposed to render the system to be uniform boundedness and uniform ultimate boundedness. For the seeking of the optimal design parameters, two cost functions, each of which is dominated by one tunable parameter, are developed, and thereout a two-player cooperative game is formulated. Finally, the optimal design problem is successfully solved: with the existence, uniqueness, and analytical expression of the Pareto-optimality. This paper is the first ever endeavor to cast both constraint following and cooperative game into control framework for uncertain mechanical systems.

To present an application of optimal controller design to a block-ice production process using dynamic programming. First, we develop mathematical models for such a process by employing system identification techniques. In particular, we build two parametric models, namely, linear model and neural network model. Linear models are built with Auto-Regressive model with exogenous inputs (linear ARX model). Feedforward neural networks (NNARX model) are constructed using the information provided by the linear ARX models. In addition, the Optimal Brain Surgeon (OBS) method is employed to prune the neural network models. Comparing the experimental results obtained from these models, it indicates that linear ARX models yield reasonably good performance in terms of the model fit. Then, using the obtained models, we investigate the energy consumption and operation cost of an ice factory when using time of use (TOU) tariff and time of day (TOD) tariff. Specifically, we develop an optimal control design for block-ice production process. The optimal strategy aims to minimize the electricity cost over a finite-time horizon and is subjected to constraints involving the control input of compressors, the brine temperature, and the number of block-ices ready for sales. Moreover, various factors in the design criteria have been analyzed with respect to the energy consumption and the operation cost. We compare the performance of the optimal controllers and the conventional control strategy. The simulation reveals that the proposed optimal control design has better performance than that of the conventional control. Furthermore, the optimal control using TOU tariff can reduce the operation cost when comparing to the one using TOD tariff.

In this paper, a novel synchronous Q-learning method is proposed for solving discrete-time linear quadratic regulator (LQR) problems. To begin with, the Bellman equation corresponding to the optimal Q-function is reformulated into a consistency equation on the parameters of the optimal Q-function and the optimal controller. Then an actor-critic structure is introduced to learn the optimal Q-function and the optimal controller online in real time by using the state samples generated by the behavior policy. Particularly, the proposed synchronous Q-learning scheme simultaneously updates the Q-function approximation and the optimal controller approximation, rather than iterating between policy evaluation and policy improvement. The proposed control scheme is proved to be uniformly ultimately bounded (UUB) under appropriate learning rates, provided that certain persistence of excitation (PE) conditions are satisfied. Besides, the PE conditions can be easily met by injecting appropriate exploration noise into the behavior policy without causing any excitation noise bias. Finally, one simulation example is provided to verify the effectiveness of the proposed synchronous Q-learning method.

Online event-triggered adaptive critic design for multi-player zero-sum games of partially unknown nonlinear systems with input constraints

A novel neural network (NN) -based output feedback controller with magnitude constraints is designed to deliver a desired tracking performance for a class of multi-input-multi-output (MIMO) discrete-time strict feedback nonlinear systems. Reinforcement learning in discrete time is proposed for the output feedback controller, which uses three NN: 1) a NN observer to estimate the system states with the input-output data; 2) a critic NN to approximate certain strategic utility function; and 3) an action NN to minimize both the strategic utility function and the unknown dynamics estimation errors. The magnitude constraints are manifested as saturation nonlinearities in the output feedback controller design. Using the Lyapunov approach, the uniformly ultimate boundedness (UUB) of the state estimation errors, the tracking errors and weight estimates is shown.