Discrete‐Time Optimal Control of State‐Constrained Nonlinear Systems Using Approximate Dynamic Programming

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

This article presents a unified approach to simultaneous solution of optimization and optimal control problems in batch distillation, operating under different modes of operation: variable, constant, or optimal reflux. The simplified, computationally efficient short‐cut method and a novel algorithm to solve the optimal control problems in batch distillation is the basis of this unified approach. The short‐cut method identifies the feasible region of operation essential for optimization and optimal control problems, and provides analytical partial derivatives of the model parameters crucial to the solution.The new algorithm for the solution of optimal combination of the maximum principle and NLP optimization techniques. It circumvents the problems associated with the maximum principle approach (iterative solution of a two‐point boundary value problem, undounded control variables, and inability to handle the simultaneous optimization and optimal control problem), and the coupled ODE discretization‐NLP optimization scheme for nonlinear models (higher system nonlinearities, multiplicity of solutions, sensitivity of convergence to initial guesses). This algorithm reduces the dimensionality of the problem, and the nature of the algorithm allows a common platform to optimal solutions of different operating conditions. This article also shows that different categories of the optimal control problems in batch distillation essentially involve the solution of the maximum distillate problem.

We study parametric optimal control problems governed by a system of time-dependent partial differential equations (PDE) and subject to additional control and state constraints. An approach is presented to compute the optimal control functions and the so-called sensitivity differentials of the optimal solution with respect to perturbations. This information plays an important role in the analysis of optimal solutions as well as in real-time optimal control. The method of lines is used to transform the perturbed PDE system into a large system of ordinary differential equations. A subsequent discretization then transcribes parametric ODE optimal control problems into perturbed nonlinear programming problems (NLP), which can be solved efficiently by SQP methods. Second-order sufficient conditions can be checked numerically and we propose to apply an NLP-based approach for the robust computation of the sensitivity differentials of the optimal solutions with respect to the perturbation parameters. The numerical method is illustrated by the optimal control and sensitivity analysis of the Burgers equation.

This paper builds up two equivalence theorems for different kinds of optimal control problems of internally controlled Schrodinger equations. The first one is an equivalence of the minimal norm and the minimal time control problems where the target sets are the origin of the state space. (The minimal time control problems are also called the first type of optimal time control problems.) The second one is an equivalence of three optimal control problems: optimal target control problems, optimal norm control problems, and the second type of optimal time control problems.

A neural network based optimal control synthesis is presented for solving optimal control problems with control and state constraints. The optimal control problem is transcribed into nonlinear programming problem which is implemented with adaptive critic neural network. The proposed simulation methods is illustrated by the optimal control problem of feeding adaptation of filter feeders of Daphnia. Results show that adaptive critic based systematic approach holds promise for obtaining the optimal control with control and state constraints.

An efficient method is introduced for solving optimal control and optimal parameter selection problems of nonlinear differential-algebraic systems involving general constraints. These infinite dimensional problems are converted into the finite dimensional optimization problems by using the control parametrization technique. Moreover, the transformed time locations for the control variables are included in such problems. As a result, optimal control problems become acausal optimal parameter selection problems. A modified version of differential evolution is introduced to solve these problems. The integration of the penalty functions is used to ensure the solution onto the feasible domain of the problems. Two industrial systems, a chemical reactor and a robotics arm, are used to illustrate the robustness and effectiveness of the method. The control effects can be improved by selecting the proper time locations as observed from the simulated results.

This paper describes a real-time model predictive control (MPC) scheme for an overhead crane subject to state and input constraints. The constraints are taken into account by means of a transformation technique which transforms the system dynamics with the corresponding constraints into a new unconstrained representation. This method allows to reformulate the underlying optimal control problem (OCP) of the MPC scheme into an unconstrained counterpart. The unconstrained OCP is then solved by means of a fast MPC algorithm that uses a finite number of iterations per MPC step in order to ensure real-time feasibility. Simulation as well as experimental results demonstrate the computational performance of the MPC scheme.

This article proposes an online data-based inverse reinforcement learning (IRL) scheme to solve optimal control problem for nonlinear systems with control constraints, in which the unknown reward functions are recovered based on the systems’ demonstrated state and input data. To deal with control constraint, we introduce a saturation function to formulate the original constrained optimal control problem into a new unconstrained optimal control problem. Then a data-based identifier using neural network (NN) approximation technique is designed to estimate the system’s dynamics. Subsequently, we develop a data-based IRL approach to learn the unknown reward function and establish the weight tuning law of the value function and reward function using demonstrated state and input data. The proof of the uniform boundedness of the weight estimation error is presented. A simulation example is provided to verify the effectiveness of the proposed approach.

Reinforcement learning (RL) and approximate dynamic programming (ADP) have been recently studied to solve nonlinear optimal control problems (OCPs) of continuous‐time (CT) systems. However, online learning efficiency and reliability are two major concerns to be further improved. Motivated by the above issues, in this paper we propose a receding‐horizon reinforcement learning (RHRL) algorithm for near‐optimal control of CT systems under control constraints. Different from classic RL and ADP, in the proposed approach, the infinite‐horizon OCP is decomposed as a series of finite‐horizon ones solved with an actor‐critic structure according to the receding horizon strategy, which can improve the online learning efficiency and reliability. The unknown dynamics of the system are identified offline using a sparse kernel‐based neural network structure whose weights are also updated online in the RHRL framework to improve the control performance. Moreover, the convergence of the modeling error is proven. To verify the effectiveness of our approach, we apply the RHRL algorithm to the autonomous ground vehicle for realizing near‐optimal path‐tracking control. Compared with CT model predictive control using a nominal model and other model‐free tracking controllers such as pure pursuit, heuristic dual programming, and the soft actor‐critic algorithm, RHRL performs better in terms of control performance.

In this paper, the infinite horizon optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems is investigated using neural-network-based online solution of Hamilton-Jacobi-Bellman (HJB) equation. By establishing an appropriate bounded function and defining a modified cost function, the optimal robust guaranteed cost control problem is transformed into an optimal control problem. It can be observed that the optimal cost function of the nominal system is nothing but the optimal guaranteed cost of the original uncertain system. A critic neural network is constructed to facilitate the solution of the modified HJB equation corresponding to the nominal system. More importantly, an additional stabilizing term is introduced for helping to verify the stability, which reinforces the updating process of the weight vector and reduces the requirement of an initial stabilizing control. The uniform ultimate boundedness of the closed-loop system is analyzed by using the Lyapunov approach as well. Two simulation examples are provided to verify the effectiveness of the present control approach.

Neural-network-based accelerated safe Q-learning for optimal control of discrete-time nonlinear systems with state constraints.

This article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

This paper deals with neural network based optimal control synthesis for solving optimal control problems with control and state constraints and discrete time delay. The optimal control problem is transcribed into nonlinear programming problem which is implemented with adaptive critic neural network. This approach is applicable to a wide class of nonlinear systems. The proposed simulation methods is illustrated by the optimal control problem of photosynthetic production described by discrete time delay differential equations. Results show that adaptive critic based systematic approach holds promise for obtaining the optimal control with control and state constraints.

In this paper, we deal with an obstacle avoidance control for a PVTOL (Planar Vertical Takeoff and Landing) aircraft. One of the effective approaches is to formulate such a problem in the framework of optimal control with penalties in the cost functions, and thus any existing optimal control design method is applicable. Based on the ‘virtual space’ approach, we formulate the obstacle avoidance problem as an unconstrained optimal control problem. We design a transform function for making an unconstrained virtual space. The model of the PVTOL aircract is represented as a descriptor form. Then, we propose a solution to the descriptor form. The effectiveness of our approach is confirmed by numerical experiments.