Adaptive dynamic programming and optimal stabilization for linear systems with time-varying uncertainty

This paper presents a data-driven method for designing optimal controllers and robust controllers for unknown nonlinear systems. Mathematical models for the realization of the control are difficult to develop owing to a lack of knowledge regarding such systems. The proposed multidisciplinary method, based on optimal control theory and machine learning with kernel functions, facilitates designing appropriate controllers using a data set. Kernel-based system models are useful for representing nonlinear systems. An optimal and an H-infinity controller can be designed by solving Hamilton-Jacobi (HJ) equations, which unfortunately, are difficult to solve owing to the nonlinearity and complexity of the kernel-based models. The objective of this study consists of overcoming two challenges. The first challenge is to derive exact solutions to the HJ equations for a class of kernel-based system models. A key technique in overcoming this challenge is to reduce the HJ equations to easily solvable algebraic matrix equations, from which optimal and H-infinity controllers are designed. The second challenge is to control an unknown system using the obtained controllers, wherein the system is identified as a kernel-based model. Additionally, this study analyzes probabilistic stability of the feedback system with the proposed controllers. Numerical simulations demonstrate control performances of both the derived optimal and H-infinity controllers and stability of the feedback system.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

Design of an intelligent optimal neural network-based tracking controller for nonholonomic mobile robot systems

In this paper, we present a global optimal and stable fuzzy controller design method for both continuous- and discrete-time fuzzy systems under both finite and infinite horizons. First, a sufficient condition is proposed which indicates that the global optimal effect can be achieved by the fuzzily combined local optimal controllers. Based on this sufficient condition, we derive a local concept approach to designing the optimal fuzzy controller by applying traditional linear optimal control theory. The stability of the entire closed-loop continuous fuzzy system can be ensured by the designed optimal fuzzy controller. The optimal feedback continuous fuzzy system can not only be guaranteed to be exponentially stable, but also be stabilized to any desired degree. Also, the total energy of system output is absolutely finite. Moreover, the resultant feedback continuous fuzzy system possesses an infinite gain margin; that is, its stability is guaranteed no matter how large the feedback gain becomes. Two examples are given to illustrate the proposed optimal fuzzy controller design approach and to demonstrate the proved stability properties.

Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm

This paper studies data-driven learning-based methods for the finite-horizon optimal control of linear time-varying discrete-time systems. First, a novel finite-horizon Policy Iteration (PI) method for linear time-varying discrete-time systems is presented. Its connections with existing infinite-horizon PI methods are discussed. Then, both data-driven off-policy PI and Value Iteration (VI) algorithms are derived to find approximate optimal controllers when the system dynamics is completely unknown. Under mild conditions, the proposed data-driven off-policy algorithms converge to the optimal solution. Finally, the effectiveness and feasibility of the developed methods are validated by a practical example of spacecraft attitude control.

Summary form only given. This paper presents a computational approach for an offline measurement based design of an optimal damping controller using adaptive critics for a new type of FACTS device - the gate controlled series capacitor (GCSC). Remote measurements are provided to the controller to damp out system modes. The optimal controller is developed based on the heuristic dynamic programming (HDP) approach. Three multi-layer perceptron (MLP) neural networks are used in the design - the identifier/model network to identify the dynamics of the power system; the critic network to evaluate the performance of the damping controller; and the controller network to provide optimal damping. This measurement based technique provides an alternative to the classical linear model based optimal control design. The eigenvalue analysis of the closed loop system is performed with time domain responses using the Prony method. Analysis of the simulation results show potential of the HDP based optimal damping controller on a GCSC for enhancing the stability of the power system.

An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

In this brief, in order to optimize the tracking accuracy and enhance the robustness of the tracking process for permanent magnet synchronous motor (PMSM), an adaptive dynamic programming(ADP)-based optimal control strategy combining with anti-disturbance control method is proposed. We consider the unknown, mismatched and time-varying load torque as a disturbance and a disturbance observer is then designed. Based on the output of the disturbance observer, a feedforward control can compensate the disturbance in the PMSM in real time. For the easy implementation of optimal control, this brief adopts the actor-critic neural network to approximate the value function and the optimal controller, respectively. A composite controller is then proposed to ensure that the speed can track the reference signal in an optimal and robust way. Finally, an experiment based on a motor towed platform is given to prove the optimality and robustness.

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems. Unlike existing optimal state feedback control, the control input of the optimal parallel control is introduced into the feedback system. However, due to the introduction of control input into the feedback system, the optimal state feedback control methods can not be applied directly. To address this problem, an augmented system and an augmented performance index function are proposed firstly. Thus, the general nonlinear system is transformed into an affine nonlinear system. The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically. It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function. Moreover, an adaptive dynamic programming ( ADP ) technique is utilized to implement the optimal parallel tracking control using a critic neural network ( NN ) to approximate the value function online. The stability analysis of the closed-loop system is performed using the Lyapunov theory, and the tracking error and NN weights errors are uniformly ultimately bounded ( UUB ) . Also, the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals. Finally, the effectiveness of the developed optimal parallel control method is verified in two cases.

The authors propose a method by which to design an optimum vibration control system composed of continuous body structures using modal analysis and sensitivity analysis. The mechanical structures are modeled first by FEM. This spatial model is of such a high degree-of-freedom (DOF) that a control system cannot be designed. This spatial model is transformed into a modal model in order to reduce DOF. The optimum control system is designed by applying the optimum control theory to this reduced model. However, stability of the control system is not always guaranteed because the modal model ignores high-order modes. An optimization method of the control system which guarantees stability of the actual system is proposed in this paper. We propose also an experimental optimization of the control system based on experimental modal analysis. The optimization is performed using sensitivity analysis and nonlinear programming. The effectiveness of the proposed method is verified by both simulation and experiment.

This paper deals with optimal tracking control problems for a class of discrete-time nonlinear systems using a generalized policy iteration adaptive dynamic programming (ADP) algorithm. First, by system transformation, the optimal tracking control problem is transformed into an optimal regulation problem. Then the generalized policy iteration ADP algorithm is employed to obtain the optimal tracking controller with convergence and optimality analysis. The developed algorithm uses the idea of two iteration procedures to obtain the iterative tracking control laws and the iterative value functions. Three neural networks, including model network, critic network and action network, are used to implement the developed algorithm. At last, an simulation example is given to demonstrate the effectiveness of the developed method.

In this paper, a learning-based predictive control (LPC) scheme is proposed for adaptive optimal control of discrete-time nonlinear systems under stochastic disturbances. The proposed LPC scheme is different from conventional model predictive control (MPC), which uses open-loop optimization or simplified closed-loop optimal control techniques in each horizon. In LPC, the control task in each horizon is formulated as a closed-loop nonlinear optimal control problem and a finite-horizon iterative reinforcement learning (RL) algorithm is developed to obtain the closed-loop optimal/suboptimal solutions. Therefore, in LPC, RL and adaptive dynamic programming (ADP) are used as a new class of closed-loop learning-based optimization techniques for nonlinear predictive control with stochastic disturbances. Moreover, LPC also decomposes the infinite-horizon optimal control problem in previous RL and ADP methods into a series of finite horizon problems, so that the computational costs are reduced and the learning efficiency can be improved. Convergence of the finite-horizon iterative RL algorithm in each prediction horizon and the Lyapunov stability of the closed-loop control system are proved. Moreover, by using successive policy updates between adjoint time horizons, LPC also has lower computational costs than conventional MPC which has independent optimization procedures between two different prediction horizons. Simulation results illustrate that compared with conventional nonlinear MPC as well as ADP, the proposed LPC scheme can obtain a better performance both in terms of policy optimality and computational efficiency.

<p style='text-indent:20px;'>The imperfect sinusoidal flux distribution, cogging torque, and current measurement errors can cause periodic torque ripple in the permanent magnet synchronous motor (PMSM). These ripples are reflected in the periodic oscillation of the motor speed and torque, causing vibration at low speeds and noise at high speeds. As a high-precision tracking application, ripple degrades the application performance of PMSM. In this paper, an adaptive dynamic programming (ADP) scheme is proposed to reduce the periodic torque ripples. An optimal controller is designed by iterative control algorithm using robust adaptive dynamic programming theory and strategic iterative technique. ADP is combined with the existing Proportional-Integral (PI) current controller and generates compensated reference current iteratively from cycle to cycle so as to minimize the mean square torque error. As a result, an optimization problem is constructed and an optimal controller is obtained. The simulation results show that the robust adaptive dynamic programming achieves lower torque ripple and shorter dynamic adjustment time during steady-state operation, thus meeting the requirements of steady speed state and the dynamic performance of the regulation system.