Numerical solution to optimal feedback control by dynamic programming approach: A local approximation algorithm

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

High order optimal feedback control of space trajectories with bounded control

This chapter presents an optimal tracking control methodology for an offshore steel jacket platform subject to external wave force. Based on a dynamic model of an offshore steel jacket platform with an AMD mechanism and a linear exogenous system model of the external wave force on the offshore platform, an optimal tracking control scheme with feedforward compensation is proposed to attenuate wave-induced vibration of the offshore platform. A feedforward and feedback optimal tracking controller (FFOTC) can be obtained by solving an algebraic Riccati equation and a Sylvester equation, respectively. It is demonstrated that the wave-induced vibration amplitudes of the offshore platform under the FFOTC are much smaller than the ones under the feedback optimal tracking controller (FOTC) and the feedforward and feedback optimal controller (FFOC). Furthermore, the required control force under the FFOTC is smaller than the ones under the FOTC and the FFOC.

Wind farms can increase annual energy production (AEP) with advanced control algorithms by coordinating the set points of individual turbine controllers across the farm. However, it remains a significant challenge to achieve performance improvements in practice because of the difficulty of utilizing models that capture pertinent complex aerodynamic phenomena while remaining amenable to control design. We formulate a multi-stage stochastic optimal control problem for wind farm power maximization and show that it can be solved analytically via dynamic programming. In particular, our model incorporates state- and input-dependent multiplicative noise whose distributions capture stochastic wind fluctuations. The optimal control policies and value functions explicitly incorporate the moments of these distributions, establishing a connection between wind flow data and optimal feedback control. We illustrate the results with numerical experiments that demonstrate the advantages of our approach over existing methods based on deterministic models.

This article introduces a droop-free, approximate optimal feedback control strategy to optimally control distributed generators (DGs) in islanded dc microgrids (DCMGs). Each DG is modeled as a control affine dynamical system, and constrained input of each DG is designed to minimize the infinite horizon quadratic state cost and nonquadratic control effort. The approximate dynamic programming (ADP) method is employed to solve the infinite horizon optimal control problem by successive approximation of the value function via a linear in the parameter (LIP) neural network (NN). The NN weights are updated online by a reinforcement learning (RL)-based tuning algorithm, and the convergence of the unknown weights to a neighborhood of the optimal weights is guaranteed without the persistence of excitation (PE). Simulation and experimental results are presented to demonstrate the effectiveness and applicability of the proposed concept.

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

This paper presents a now method of optimal digital PID feedback control for a linear multivariate system by using a, parameter optimization technique. The most difficult points for the design of optimal digital PID feedback control system by the modern control theory are : (1) the control structure of it is specified in advance, (2) unless all the state variables are fed back, the optimal control cannot be obtained by dynamic programming or the maximum principle. However, the optimal digital PID feedback gains are determined by using the parameter optimization technique in this paper. When there are some states which are not fed back in the control system, the optimal feedback gains depend on the initial states. First, the optimal solution is presented in the case where all the initial Btates are known. Next, it is extended to the case where the initial states are random variables and their statistical properties are known,

Distributed Optimal Output Feedback Control of Heterogeneous Multi-agent Systems under a Directed Graph

In this paper, near optimal tracking of a class of nonlinear systems is addressed. Adaptive (approximate) dynamic programming approach is used to calculate the optimal control in closed form. ADP (Adaptive (approximate) dynamic programming) has been widely used to resolve optimal regulation and tracking problems of nonlinear control systems. Despite advances in the so called supervised and unsupervised ADP techniques for optimal tracking, they have a main draw back. That is, the optimal controller needs to be recalculated for every particular reference trajectory. The main goal of this work is to address this issue for a class of nonlinear systems. Finally, this approach is applied on a Delta robot and the performance of the method is analyzed experimentally.

This paper presents a method of designing robust optimal controllers for digital control systems with control delay. A predictor similar to the Smith one is used. The robust optimal state feedback control law is obtained by dynamic programming method. The uncertainty in system parameters is taken care of by an additional term in the Riccati equation. The case of output feedback controller is also dicussed.

Finite-time partial stability and stabilization, and optimal feedback control

Data-based adaptive neural network optimal output feedback control for nonlinear systems with actuator saturation

A Novel Computer Approach to Optimal Feedback Control of Bilinear Systems

Many Cyber-Physical Systems (CPSs) demonstrate high-dimensional environmental uncertainties that modulate the physical system dynamics and complicate the computational decision-making tasks. As an example, strategic air traffic flow management (ATFM) aims to resolve air traffic congestion through managing traffic flows at a long look-ahead time. The convective weather, a dominant factor of traffic congestion, is very uncertain and significantly complicates the decision-making process. Other uncertainties such as traffic demands further expand the uncertainty space and complicate the design of optimal control solutions that are robust to uncertainties. In this paper, we formulate the strategic ATFM as a stochastic optimal control problem, and use scalable sampling based dynamic programming and Q-learning approaches to address high-dimensional uncertainties. Simulation studies demonstrate the effectiveness of these approaches. These approaches generally apply to CPSs that operate under high-dimensional environmental uncertainties.

In this paper, we study the optimal control problem for Takagi–Sugeno (T-S) fuzzy systems with disturbances. Due to the presence of disturbances in the T-S fuzzy systems, a fuzzy optimal state feedback control approach is presented by employing a two-player zero-sum game. Since the analytical optimal control solutions and the worst-case disturbance policies of T-S fuzzy systems can be boiled down to solving the algebraic Riccati equations (AREs), which are difficult to be obtained directly, a model-free policy iteration (PI) learning algorithm is proposed to obtain their approximation solutions. It is proved that the developed fuzzy optimal state feedback controller can ensure the fuzzy systems to be asymptotically stable and satisfy the disturbance attenuation condition simultaneously. Also, the designed PI learning algorithm can converge to their optimal solutions. Finally, we apply the developed fuzzy optimal state feedback control method to the truck-trailer system and the simulation results demonstrated the effectiveness of the developed scheme.