Optimal control laws for time-delay systems with saturating actuators based on heuristic dynamic programming

A novel multi-objective ADP method is constructed to obtain the optimal controller of a class of nonlinear time-delay systems in this chapter. Using the weighted sum technology, the original multi-objective optimal control problem is transformed to the single one. An ADP method is established for nonlinear time-delay systems to solve the optimal control problem. To demonstrate the presented iterative performance index function sequence is convergent and the closed-loop system is asymptotically stable, the convergence analysis is also given. The neural networks are used to get the approximative control policy and the approximative performance index function, respectively. Two simulation examples are presented to illustrate the performance of the presented optimal control method.

In this chapter, the data-based optimal tracking control approach is developed by involving the iterative dual heuristic dynamic programming (DHP) algorithm for nonlinear systems. According to the iterative DHP method, the updating formula of the costate function and the new optimal control policy for unknown nonlinear systems are provided to solve the optimal tracking control problem. Moreover, three neural networks are used to facilitate the implementation of the proposed algorithm. The unknown nonlinear system dynamics is first identified by establishing a model neural network. To improve the identification precision, biases are introduced to the model network. The model network with biases is trained by the gradient descent algorithm, where the weights and biases across all layers are updated. The uniform ultimate boundedness stability with a proper learning rate is analyzed, by using the Lyapunov approach. The effectiveness of the proposed method is demonstrated through a simulation example.KeywordsAdaptive dynamic programmingData-based optimal tracking controlLyapunov methodNeural networkUniformly ultimately bounded stabilityValue iteration

In this paper, an iterative adaptive dynamic programming (ADP) algorithm is developed to solve the optimal cooperative control problems for residential multi-battery systems. To avoid solving high-dimensional optimal control problems, we first constrain all the batteries at their worst performance, which transforms the multi-input optimal control problem into a single-input one. Based on the worst-performance optimal control law, the optimal cooperative control law for the residential multi-battery systems is obtained, where in each iteration, only a single-input optimization problem is implemented. Finally, numerical results are given to illustrate the performance of the developed algorithm.

In this paper, a new iterative optimal learning control scheme for discrete-time nonlinear systems using iterative adaptive dynamic programming (ADP) approach is developed to obtain the optimal control law for coal gasification control systems. For the unknown coal gasification process, neural networks (NNs) are introduced to reconstruct the dynamics of the gasification process, where the approximation errors of the reconstruction dynamics are considered. Via system transformation, the optimal tracking control problem with approximation errors is transformed into a two-person zero-sum optimal control problem. A new iterative ADP algorithm is then developed to obtain the optimal control law for the transformed system with convergence analysis. Finally, numerical results are given to illustrate the performance of the present method.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

We are honored to organize this special issue of Neural Computation and Applications, based on contributions to the Ninth International Symposium on Neural Networks (ISNN 2012) held on July 11–14, 2012, in Shenyang, China. Evaluated by the contributions and the recommendation of ISNN 2012 organizers, 22 papers were selected for the further review process with their extended form. Each paper has been reviewed thoroughly by three independent experts in related research area. We are very grateful for the hard work of the reviewers, which help to facilitate the outcome of this special issue with greatly improved quality of the contributions. Finally, 20 papers are accepted, to be introduced as follows. In brief, the contributions of this special issue can be roughly classified into three major categories: data-based control, data-based optimization, and data-based modeling. In each category, not only the theoretical foundation is analyzed, but also some prospective applications are demonstrated. In this introduction, we like to borrow some ideas from the contributions, such as hierarchical, and cluster as the subsidiary category idea. The first category consisting of 6 papers is devoted to the data-based control and applications area, by which an optimal or near-optimal control can be derived from the input and output data. The first 4 papers focus on the hot research direction, adaptive dynamic programming and reinforcement learning (ADPRL), and the remaining 2 papers relate to iterative learning control and optimal control. Derong Liu et al. develop an online algorithm based on policy iteration for continuous-time optimal control with infinite horizon cost for nonlinear systems. In the proposed method, a discount value function is employed, which is considered to be a more general case for optimal control problems. Meanwhile, without knowledge of the internal system dynamics, the algorithm can converge uniformly online to the optimal solution of the modified Hamilton– Jacobi–Bellman equation. By means of two neural networks, the algorithm is able to find suitable approximations of both the optimal control and the optimal cost. Dual iterative adaptive dynamic programming for a class of discrete-time nonlinear systems with time-delays is investigated by Qinglai Wei et al. Dual iterative ADP algorithm is introduced to obtain the optimal solutions of the optimal performance index function and control, where in each iteration, the performance index function and the system states are both updated. Convergence analysis is presented to prove the performance index function to reach D. Zhao (&) State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China e-mail: Dongbin.zhao@ia.ac.cn

In this article, the zero‐sum game problem of control and disturbance is studied for discrete affine nonlinear switched systems with unknown dynamical models. A two‐level heuristic dynamic programming iterative algorithm for solving the zero‐sum game problem is proposed, which can be used to solve the Hamilton‐Jacobi‐Isaacs equation associated with the optimal regulation control problem. The convergence analysis based on the value function and control strategy is given. For algorithm implementation, neural networks are used to approximate the control strategy, disturbance strategy, and value function of the game dynamic programming, respectively. A model network is used to approximate the system states with an unknown dynamical model. The iterative algorithm steps are given. Finally, the validity of the method is verified by simulation examples.

In this paper, a novel heuristic dynamic programming (HDP) iteration algorithm is proposed to solve the optimal tracking control problem for a class of nonlinear discrete-time systems with time delays. The novel algorithm contains state updating, control policy iteration, and performance index iteration. To get the optimal states, the states are also updated. Furthermore, the "backward iteration" is applied to state updating. Two neural networks are used to approximate the performance index function and compute the optimal control policy for facilitating the implementation of HDP iteration algorithm. At last, we present two examples to demonstrate the effectiveness of the proposed HDP iteration algorithm.

The optimal control problem for nonlinear time-delay large-scale systems with persistent disturbances is considered. By using the successive approximation approach, the high order, coupled, nonlinear two-point boundary value (TPBV) problem with both time-delay and time-advance terms, which is derived from the necessary condition of the original optimal control problem, is transformed into a sequence of linear decoupled differential equations. By iteratively solving the sequence of linear differential equations, an optimal control law is obtained, which consists of analytic linear feedforward, feedback terms and a dynamic compensator. The feedforward term is used for persistent disturbance attenuation and the compensator is for the purpose of compensating nonlinearities and time delays. A suboptimal control law is obtained by truncating a finite term of the adjoint vector sequence as its compensator. An iterative algorithm to obtain the suboptimal control law is proposed. A simulation example illustrates the validity of the algorithm.

This article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

In general, optimal control problems rely on numerically rather than analytically solving methods, due to their nonlinearities. The direct method, one of the numerically solving methods, is mainly to transform the optimal control problem into a nonlinear optimization problem with finite dimensions, via discretizing the objective functional and the forced dynamical equations directly. However, in the procedure of the direct method, the classical discretizations of the forced equations will reduce or affect the accuracy of the resulting optimization problem as well as the discrete optimal control. In view of this fact, more accurate and efficient numerical algorithms should be employed to approximate the forced dynamical equations. As verified, the discrete variational difference schemes for forced Birkhoffian systems exhibit excellent numerical behaviors in terms of high accuracy, long-time stability and precise energy prediction. Thus, the forced dynamical equations in optimal control problems, after being represented as forced Birkhoffian equations, can be discretized according to the discrete variational difference schemes for forced Birkhoffian systems. Compared with the method of employing traditional difference schemes to discretize the forced dynamical equations, this way yields faithful nonlinear optimization problems and consequently gives accurate and efficient discrete optimal control. Subsequently, in the paper we are to apply the proposed method of numerically solving optimal control problems to the rendezvous and docking problem of spacecrafts. First, we make a reasonable simplification, i.e., the rendezvous and docking process of two spacecrafts is reduced to the problem of optimally transferring the chaser spacecraft with a continuously acting force from one circular orbit around the Earth to another one. During this transfer, the goal is to minimize the control effort. Second, the dynamical equations of the chaser spacecraft are represented as the form of the forced Birkhoffian equation. Then in this case, the discrete variational difference scheme for forced Birkhoffian system can be employed to discretize the chaser spacecraft's equations of motion. With further discretizing the control effort and the boundary conditions, the resulting nonlinear optimization problem is obtained. Finally, the optimization problem is solved directly by the nonlinear programming method and then the discrete optimal control is achieved. The obtained optimal control is efficient enough to realize the rendezvous and docking process, even though it is only an approximation of the continuous one. Simulation results fully verify the efficiency of the proposed method for numerically solving optimal control problems, if the fact that the time step is chosen to be very large to limit the dimension of the optimization problem is noted.

This paper builds up two equivalence theorems for different kinds of optimal control problems of internally controlled Schrodinger equations. The first one is an equivalence of the minimal norm and the minimal time control problems where the target sets are the origin of the state space. (The minimal time control problems are also called the first type of optimal time control problems.) The second one is an equivalence of three optimal control problems: optimal target control problems, optimal norm control problems, and the second type of optimal time control problems.

Turnpike results hold a central position in the area of applied mathematics, offering profound insights into the long-time behavior of optimal control systems. They play an important role in guiding decision-making processes in dynamic systems. Often dynamic systems depend on uncertain data and/or parameters. In this paper, optimal boundary control problems for the transport equation with random initial data and random source term are considered. The paper begins with the numerical analysis of the turnpike phenomenon for an optimal boundary control problem with nonlinear time-varying feedback control, where the initial data are perturbed by randomized Fourier series and the source term is perturbed by a random variable. Further, an integral turnpike result is presented for the optimal controls and the corresponding optimal states of an optimal boundary control problem with random initial data. A turnpike result is also presented for an optimal boundary control problem governed by a transport equation with random source term, where optimal control, optimal expected state, and variance satisfy an integral turnpike inequality. The turnpike constant, the expected value, and the variance are further specified for Gaussian and uniformly distributed random variables.

This paper is concerned with the optimal output synchronization control problem for a class of heterogeneous complex dynamical networks (CDNs) with partially unknown system dynamics. By defining a new quadratic performance index including the information of state couplings, an optimal controller consisting of a feedback control and a compensated feedforward control is then designed. A necessary and sufficient condition for the optimal control minimizing the quadratic performance index is established. Especially, an algebraic Riccati equation (ARE) associated with a discount factor is obtained, and a novel online iteration algorithm by iteratively solving the ARE is proposed to compute the optimal controller gain for the case that the CDNs are with partially unknown system dynamics. Moreover, it is proved that the output synchronization errors converge to zero asymptotically by applying the optimal control laws. Finally, two simulation examples are provided to verify the effectiveness of the proposed control method.

This article presents a unified approach to simultaneous solution of optimization and optimal control problems in batch distillation, operating under different modes of operation: variable, constant, or optimal reflux. The simplified, computationally efficient short‐cut method and a novel algorithm to solve the optimal control problems in batch distillation is the basis of this unified approach. The short‐cut method identifies the feasible region of operation essential for optimization and optimal control problems, and provides analytical partial derivatives of the model parameters crucial to the solution.The new algorithm for the solution of optimal combination of the maximum principle and NLP optimization techniques. It circumvents the problems associated with the maximum principle approach (iterative solution of a two‐point boundary value problem, undounded control variables, and inability to handle the simultaneous optimization and optimal control problem), and the coupled ODE discretization‐NLP optimization scheme for nonlinear models (higher system nonlinearities, multiplicity of solutions, sensitivity of convergence to initial guesses). This algorithm reduces the dimensionality of the problem, and the nature of the algorithm allows a common platform to optimal solutions of different operating conditions. This article also shows that different categories of the optimal control problems in batch distillation essentially involve the solution of the maximum distillate problem.