A Combined Policy Gradient and Q-learning Method for Data-driven Optimal Control Problems

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

Nearly finite-horizon optimal control for a class of nonaffine time-delay nonlinear systems based on adaptive dynamic programming

Time optimal control problems involving high dimensional systems are generally very difficult to solve. Instead of using suboptimal control based on reduced system, iterative dynamic programming is used directly to establish the time optimal control policy of a binary distillation column described by 11 ordinary differential equations. The resulting optimal control policy that is bang‐bang in nature was obtained very rapidly. The minimum time of 2236 seconds obtained here is substantially better than 4120 seconds obtained by a suboptimal control policy. Simulations show that the procedure may be used for on‐line optimal control of the distillation column, since the optimal control policy can be established very fast on a personal computer.

We consider a finite-capacity single-server queuing system in which the problems of capacity design and service quality control are integrated. The amount of time spent on servicing a customer is used as a measure of the quality of service provided. The expected reward from servicing is assumed to be a nondecreasing function of the service duration, while a linear holding cost is assumed for customers waiting in the system. We maximize the long-run average return per unit time by optimally controlling the quality of service to be provided as a function of the workload facing the server for different waiting-room capacities. We show that the optimal control policy for a given capacity is monotone and also examine the effect of varying the system capacity on the optimal control policy. Furthermore, we analyze the design problem of selecting an optimal capacity by assuming that an optimal service control policy will be followed for each system capacity.

Adaptive nearly optimal control for a class of continuous-time nonaffine nonlinear systems with inequality constraints

An incremental iterative Q-learning algorithm (IIQLA) is proposed to tackle the optimal secure control problem for cyber-physical systems under false data injection attacks. Within a zero-sum game framework, the secure control problem is transformed into solving an iterative algebraic Riccati equation. To derive the optimal secure control policy from the equation, the IIQLA is developed, which does not require prior knowledge of the system dynamics. This algorithm utilizes two auxiliary variables to separate the behavior policy and the target policy, thereby improving the exploration of data. As a result, the devised control policies are more conservative, leading to solutions that are closer to the optimal policy. Moreover, by introducing an adaptive learning rate, the proposed IIQLA can accelerate the convergence speed and decrease the number of required iterations, thus alleviating the computation burden. The convergence of the proposed IIQLA with different learning rates is also analyzed. In addition, the closed-loop system is guaranteed to be asymptotically stable, and the exploration noise does not introduce bias into the optimal policies. Finally, a simulation example is provided to demonstrate the effectiveness of the proposed approach.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.

A generalized formulation of optimal control policy for physical stochastic optimal control of structures is proposed in this paper, which extends the classic forms of optimal control policies of passively controlled modalities, actively controlled modalities, semi-actively controlled modalities, and hybridly controlled modalities. This unified scheme features the simultaneous consideration of optimal control force and optimal placement of control devices. A new controllability index associated with the exceedance probability is developed, which characterizes system safety, system serviceability, system comfort, control device workability, and their trade-off. A sequential procedure with a minimum storey controllability index gradient is defined to efficiently search for the optimal topology of control devices. Furthermore, a performance function for parameters optimization is proposed, which exhibits the same practical accounts as the controllability index. The resolution procedure of the generalized optimal control policy is detailed, and two randomly base-excited structural systems are investigated for illustrative purposes, which consist of a ten-storey shear frame equipped with viscoelastic dampers and an eight-storey shear frame controlled by active tendons. Numerical results reveal that the proposed generalized optimal control policy can gain the maximum control effectiveness with the minimum control cost. The controlled system response is significantly reduced and becomes smoother along the height of the structure, which results in a more desirable structural performance. Using the design criterion, meanwhile, of the minimum storey controllability index gradient, the structural stochastic optimal control achieves the objective performance more efficiently than that of the maximum storey controllability index currently in use. Copyright © 2011 John Wiley & Sons, Ltd.

An efficient method is introduced for solving optimal control and optimal parameter selection problems of nonlinear differential-algebraic systems involving general constraints. These infinite dimensional problems are converted into the finite dimensional optimization problems by using the control parametrization technique. Moreover, the transformed time locations for the control variables are included in such problems. As a result, optimal control problems become acausal optimal parameter selection problems. A modified version of differential evolution is introduced to solve these problems. The integration of the penalty functions is used to ensure the solution onto the feasible domain of the problems. Two industrial systems, a chemical reactor and a robotics arm, are used to illustrate the robustness and effectiveness of the method. The control effects can be improved by selecting the proper time locations as observed from the simulated results.

Integrating product and process design with optimal control: A case study of solvent recycling

In general, optimal control problems rely on numerically rather than analytically solving methods, due to their nonlinearities. The direct method, one of the numerically solving methods, is mainly to transform the optimal control problem into a nonlinear optimization problem with finite dimensions, via discretizing the objective functional and the forced dynamical equations directly. However, in the procedure of the direct method, the classical discretizations of the forced equations will reduce or affect the accuracy of the resulting optimization problem as well as the discrete optimal control. In view of this fact, more accurate and efficient numerical algorithms should be employed to approximate the forced dynamical equations. As verified, the discrete variational difference schemes for forced Birkhoffian systems exhibit excellent numerical behaviors in terms of high accuracy, long-time stability and precise energy prediction. Thus, the forced dynamical equations in optimal control problems, after being represented as forced Birkhoffian equations, can be discretized according to the discrete variational difference schemes for forced Birkhoffian systems. Compared with the method of employing traditional difference schemes to discretize the forced dynamical equations, this way yields faithful nonlinear optimization problems and consequently gives accurate and efficient discrete optimal control. Subsequently, in the paper we are to apply the proposed method of numerically solving optimal control problems to the rendezvous and docking problem of spacecrafts. First, we make a reasonable simplification, i.e., the rendezvous and docking process of two spacecrafts is reduced to the problem of optimally transferring the chaser spacecraft with a continuously acting force from one circular orbit around the Earth to another one. During this transfer, the goal is to minimize the control effort. Second, the dynamical equations of the chaser spacecraft are represented as the form of the forced Birkhoffian equation. Then in this case, the discrete variational difference scheme for forced Birkhoffian system can be employed to discretize the chaser spacecraft's equations of motion. With further discretizing the control effort and the boundary conditions, the resulting nonlinear optimization problem is obtained. Finally, the optimization problem is solved directly by the nonlinear programming method and then the discrete optimal control is achieved. The obtained optimal control is efficient enough to realize the rendezvous and docking process, even though it is only an approximation of the continuous one. Simulation results fully verify the efficiency of the proposed method for numerically solving optimal control problems, if the fact that the time step is chosen to be very large to limit the dimension of the optimization problem is noted.

Synthesis of optimal controllers for a class of maximization problems

Integrated production-delivery control policy for an unreliable manufacturing system and multiple retailers

In this paper, optimal synchronization problem of multi-agent systems (MASs) with time-varying input delay is investigated. To remove the effect of input delay, the delayed MAS model is firstly transformed into a new discrete-time delay-free version. Then we establish an equivalence relation on the performance indexes of the two systems so as to formulate a new optimal consensus control problem of the transformed delay-free system. Based on the Bellman optimality theorem, we derive optimal consensus control policies from the coupled HJB equations. A policy iteration algorithm is introduced to learn the solutions to the HJB equations in real-time. To execute iterative ADP method, the critic neural networks (NN) are utilized to approximate the value functions and help to calculate the control policies. Finally, a numerical simulation is provided to show the effectiveness of the proposed approach.