Optimal Iterative Learning Control for Nonlinear Discrete-Time Systems

This paper presents a method for obtaining optimal stable control for general nonlinear nonautonomous dynamical systems. The approach is inspired by recent developments in analytical dynamics and the observation that the Lyapunov criterion for stability of dynamical systems can be recast as a constraint to be imposed on the system. A closed-form expression for control is obtained that minimizes a user-defined control cost at each instant of time and enforces the Lyapunov constraint simultaneously. The derivation of this expression closely mirrors the development of the fundamental equation of motion used in the study of constrained motion. For this control method to work, the positive definite functions used in the Lyapunov constraint should satisfy a consistency condition. A class of positive definite functions has been provided for mechanical systems that meet this criterion. To illustrate the broad scope of the method, for linear systems it is shown that a proper choice of these positive definite functions results in conventional LQR control. Control of the Lorenz system and a multi-degree of freedom nonlinear mechanical system are considered. Numerical examples demonstrating the efficacy and simplicity of the method are provided.

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems. Unlike existing optimal state feedback control, the control input of the optimal parallel control is introduced into the feedback system. However, due to the introduction of control input into the feedback system, the optimal state feedback control methods can not be applied directly. To address this problem, an augmented system and an augmented performance index function are proposed firstly. Thus, the general nonlinear system is transformed into an affine nonlinear system. The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically. It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function. Moreover, an adaptive dynamic programming ( ADP ) technique is utilized to implement the optimal parallel tracking control using a critic neural network ( NN ) to approximate the value function online. The stability analysis of the closed-loop system is performed using the Lyapunov theory, and the tracking error and NN weights errors are uniformly ultimately bounded ( UUB ) . Also, the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals. Finally, the effectiveness of the developed optimal parallel control method is verified in two cases.

This paper investigates the application of associative reinforcement learning techniques to the optimal control of linear discrete-time dynamic systems. Associative reinforcement learning involves the trial and error interaction with a dynamic system to determine the control actions that optimally achieve some desired performance index. The methodology can be applied either online or off-line and in a model based or model free manner. Associative reinforcement learning techniques are applied to the optimal regulator (LQR) control of discrete-time linear systems. Adaptive critic designs are implemented and the convergence speed compared for the different approaches. These methods can determine the optimal state and state/action value function and the optimal policy without requiring system models. (4 pages)

Optimal control of multivariable systems with pure time delays

This paper presents a simple methodology for obtaining the entire set of continuous controllers that cause a nonlinear dynamical system to exactly track a given trajectory. The trajectory is provided as a set of algebraic and/or differential equations that may or may not be explicitly dependent on time. Closed-form results are also provided for the real-time optimal control of such systems when the control cost to be minimized is any given weighted norm of the control, and the minimization is done not just of the integral of this norm over a span of time but also at each instant of time. The method provided is inspired by results from analytical dynamics and the close connection between nonlinear control and analytical dynamics is explored. The paper progressively moves from mechanical systems that are described by the second-order differential equations of Newton and/or Lagrange to the first-order equations of Poincaré, and then on to general first-order nonlinear dynamical systems. A numerical example illustrates the methodology.

In this work, an enhanced data-driven optimal iterative learning control (eDDOILC) is proposed for nonlinear nonaffine systems where a new iterative sliding mode surface (ISMS) is designed to replace the traditional tracking error in the controller design and analysis. It is the first time to extend the sliding mode surface to the iteration domain for systems operate repetitively over a finite time interval. By virtual of the new designed ISMS, the control design becomes more flexible where both the time and the iteration dynamics can be taken into account. Before proceeding to the controller design, an iterative dynamic linear data model is built between two consecutive iterations to formulate the linear input-output data relationship of the repetitive nonlinear nonaffine discrete-time system. The linear data model is virtual and does not have any physical meanings, which is very different to the traditional mechanism mathematical model. In the sequel, the eDDOILC is proposed by designing an objective function with respect to the proposed two-dimensional ISMS. Rigorous proof is provided to show the convergence of the proposed eDDOILC method. Furthermore, the results have been extended to a multiple-input multiple-output (MIMO) nonaffine nonlinear discrete-time repetitive system. In general, the proposed eDDOILC is data-driven where no explicit model information is included. It is illustrated that the presented eDDOILC is effective when applied to the nonlinear nonaffine uncertain systems.

The paper introduces a new method for finding optimal control of algebraic dynamic systems. The structure of algebraic dynamical systems is nonlinear with quadratic and bilinear terms. A new hybrid extended Fourier series is introduced, and state and control variables of the system are expanded by this series. Moreover, properties of new series are presented, and integration and product operational matrices are obtained. Using operational matrices, optimal control of the systems is converted to a set of simultaneous nonlinear algebraic relations. An illustrative example is included to compare our results with those in the literature.

An ADDHP-based Q-learning algorithm for optimal tracking control of linear discrete-time systems with unknown dynamics

A general result in stochastic optimal control of nonlinear discrete-time systems with quadratic performance criteria

In this paper, a novel synchronous Q-learning method is proposed for solving discrete-time linear quadratic regulator (LQR) problems. To begin with, the Bellman equation corresponding to the optimal Q-function is reformulated into a consistency equation on the parameters of the optimal Q-function and the optimal controller. Then an actor-critic structure is introduced to learn the optimal Q-function and the optimal controller online in real time by using the state samples generated by the behavior policy. Particularly, the proposed synchronous Q-learning scheme simultaneously updates the Q-function approximation and the optimal controller approximation, rather than iterating between policy evaluation and policy improvement. The proposed control scheme is proved to be uniformly ultimately bounded (UUB) under appropriate learning rates, provided that certain persistence of excitation (PE) conditions are satisfied. Besides, the PE conditions can be easily met by injecting appropriate exploration noise into the behavior policy without causing any excitation noise bias. Finally, one simulation example is provided to verify the effectiveness of the proposed synchronous Q-learning method.

Model-free finite-horizon optimal tracking control of discrete-time linear systems

This paper considers the global optimal control for nonlinear interconnected large-scale systems with respect to linear quadratic functional. A successive approximation procedure is proposed to design the feedforward feedback nearly optimal control law. By introducing the Successive Approximation Approach of the nonlinear differential equation theory, a nonhomogeneous linear two-point boundary value problem sequence is constructed to approximate the nonlinear large-scale two-point boundary value problem, which is the necessary condition of the nonlinear control problem. By iteratively solving this nonhomogeneous linear approximating sequence, a global nearly optimal control law that composed of a linear closed-loop feedback compensatory term and an open-loop feedforward compensatory term is designed. An algorithm is presented to design certain qualified nearly optimal control law by truncating an infinite iteration result. An illustrative example shows the validity of the algorithm.

Global optimal feedback control for general nonlinear systems with nonquadratic performance criteria

The decomposition of a system into simpler subsystems, followed by the optimization of the individual subsystems, and then co‐ordinating the subsystem optimal solutions to yield the optimal control policy for the original system, is considered. Lasdon's co‐ordination algorithm, in which the subsystems are regarded as being completely independent, is improved by including the interdependence of the adjacent subsystems. For linear systems with a quadratic performance index, this procedure yields the optimal control policy in a single iteration from an arbitrary initial guess of the decomposition parameters. For nonlinear systems, the method includes linearizing the state equations, the calculation of the optimal control for the linearized system, and the adjustment of the corresponding decomposition parameters of the original problem until convergence is achieved. For the optimization of a nonlinear continuous stirred tank reactor such a procedure requires considerably more computation time than the standard techniques, and the optimum policies are less accurate.

MTN optimal control of MIMO non-affine nonlinear time-varying discrete systems for tracking only by output feedback