Data-Driven Stochastic Optimal Control With Safety Constraints Using Linear Transfer Operators

The Hamilton Jacobi Bellman (HJB) PDE for the stochastic optimal control (SOC) problem for diffusion SDE dynamics which have affine controls and state and control multiplicative noise is a second order fully nonlinear PDE. The previously known linearly solvable optimal control framework as well as the first order forward backward SDEs (FBSDEs) frameworks therefore have a characteristic inadequacy to support sampling algorithms for this SOC problem. We present the framework of second order FBSDEs for solving this SOC problem in this paper. We derive the nonlinear Feynman Kac representation of the second order fully nonlinear HJB PDE corresponding to diffusions with state and control multiplicative noise. The Feynman Kac representation enables a sampling based scheme for solving this SOC problem. This scheme is then leveraged to develop a least squares Monte Carlo regression based algorithm for implementations. The algorithm is validated by examples of simulated control of an underactuated system and comparison against an analytical characterization.

The stochastic optimal nonlinear control of coupled adjacent building structures is studied based on the stochastic dynamical programming principle and the stochastic averaging method. The coupled structures with control devices under random seismic excitation are first condensed to form a reduced-order structural model for the control analysis. The stochastic averaging method is applied to the reduced model to yield stochastic differential equations for structural modal energies as controlled diffusion processes. Then a dynamical programming equation for the energy processes is established based on the stochastic dynamical programming principle, and solved to determine the optimal nonlinear control law. The seismic response mitigation of the coupled structures is achieved through the structural energy control and the dimension of the optimal control problem is reduced. The seismic excitation spectrum is taken into account according to the stochastic dynamical programming principle. Finally, the nonlinear controlled structural response is predicted by using the stochastic averaging method and compared with the uncontrolled structural response to evaluate the control efficacy. Numerical results are given to demonstrate the response mitigation capabilities of the proposed stochastic optimal control method for coupled adjacent building structures.

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.

A minimax stochastic optimal semi-active control strategy for stochastically excited quasi-integrable Hamiltonian systems with parametric uncertainty by using magneto-rheological (MR) dampers is proposed. Firstly, the control problem is formulated as an n-degree-of-freedom (DOF) controlled, uncertain quasi-integrable Hamiltonian system and the control forces produced by MR dampers are split into the passive part and the semi-active part. Then the passive part is incorporated into the uncontrolled system. After that, the stochastic optimal semi-active control problem is solved by applying the minimax stochastic optimal control strategy based on the stochastic averaging method and stochastic differential game. The worst-case disturbances and the optimal controls are obtained by the minimax dynamical programming equation with the constraints of disturbance bounds and MR damper dynamics. Finally, the system response and the control performance are evaluated by using Monte Carlo simulation. An example of a two-DOF system with coupling damping and parametric uncertainty under Gaussian white noise excitations is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy, which is also compared with the clipped linear-quadratic-Gaussian control strategy to show the advantages.

Uncertainty propagation is an important step in the derivation of optimal control strategies for dynamic systems in the presence of state and parameter uncertainty. Many stochastic control formulations seek to optimize an expected value of a score or cost function, or otherwise enforce a probabilistic constraint through the use of an expected value. The time-varying state density needed for this expected value computation may be quantified through Monte Carlo methods or explicit approaches such as the Frobenius-Perron operator. This paper explores an alternative approach to the computation of the expected value by leveraging the adjoint relationship between the Frobenius-Perron operator and the Koopman operator. This relationship states that the expected value of a forward-propagated density with a score function is equivalent to the expected value of the initial density with the score function mapped back to the initial state space. By computing the expected value through use of the Koopman operator, several computational issues arising when using the Frobenius-Perron operator can be mitigated, while maintaining the benefits of an explicit method over Monte Carlo. A general stochastic optimal control problem is formulated wherein the expected value is computed between the initial state density and a score function using the Koopman operator. The computational benefits of computing the expected value through use of the Koopman operator are discussed, particularly with respect to numerical issues encountered using explicit forward propagation of the state probability density. A series of numerical examples highlight these computational advantages and illustrate the benefit of using this adjoint approach in solution to a range of optimal control problems in the presence of uncertainty.

SummaryIn this article, we study the open‐loop and closed‐loop solvability for indefinite mean‐field stochastic linear quadratic (MF‐SLQ) optimal control problem and its application in finance, where the controlled stochastic system is driven by a Brownian motion and a Poisson random martingale measure and also disturbed by some stochastic processes. The intrinsic property of stochastic systems results in the inequivalence of those two solvabilities, which is different from deterministic case. Based on a well‐posedness result of the problem, it is shown that the uniform convexity of cost functional is sufficient for the open‐loop solvability of the problem. By a matrix minimum principle, the necessity condition of regular solvability for a decoupled Riccati equation is established, meanwhile, the closed‐loop solvability is turned out to be equivalent to the regular solvability of Riccati equations with some constraints on the solutions of disturbances equations, moreover, the optimal closed‐loop strategy is characterized by regular solutions of Riccati equations and adapted solutions of disturbances equations. And then a mean‐variance model is considered to solve optimal portfolio selection strategy problem of the insurance company with liability. Our study fills a gap in the field of solvability for mean‐field stochastic optimal control with random jumps and provides a different way for solving mean‐variance portfolio selection model.

Stochastic bifurcation characteristics and optimal control of giant magnetostrictive film (GMF)-shape memory alloy (SMA) composite plate subjected to in-plane stochastic excitation were studied in this paper. Nonlinear difference item was introduced to interpret the hysteretic phenomena of both GMF and SMA, and then the nonlinear dynamic model of GMF-SMA composite plate subjected to in-plane stochastic excitation was developed. The stochastic stability of the system was analyzed, and the condition of stochastic Hopf bifurcation was discussed. The reliability function of the system was solved from backward Kolmogorov equation, and then the probability density of the first-passage time was obtained. Finally, the stochastic optimal control strategy was obtained in stochastic dynamic programming method. Numerical simulation shows that the stability of the trivial solution varies with bifurcation parameters, and stochastic Hopf bifurcation appears in the process; the reliability of the system is improved by stochastic optimal control, and the first-passage time is delayed. GMF-SMA composite plate combines the advantages of both GMF and SMA, and can reduce vibration through passive control and active control effectively. The results of this paper are helpful to application of GMF-SMA composite plate in engineering fields.

Optimal control of Markovian jump systems via a neural network-based ADP iterative algorithm

The formulation of optimal control problems governed by Cauchy-Riemann equations is presented. A distributed control mechanism through divergence and curl sources is considered with the boundary conditions of mixed type. A Lagrange multiplier framework is introduced to characterize the solution to Cauchy-Riemann optimal control problems as the solution of an optimality system of four first-order partial differential equations and two optimality conditions. To solve the optimality system, staggered grids and multigrid methods are investigated. It results that staggered grids provide a natural collocation of the optimization variables and second-order accurate solutions are obtained. The proposed multigrid scheme is based on a coarsening by a factor of three that results in a nested hierarchy of staggered grids. On these grids a distributed-Gauss-Seidel and gradient-based smoothing scheme is employed. Results of numerical experiments validate the proposed optimal control formulation and demonstrate the effectiveness of the staggered-grids multigrid solution procedure.

On the infinite time solution to state-constrained stochastic optimal control problems

Innovative procedures for the stochastic optimal time-delay control and stabilization are proposed for a quasi-integrable Hamiltonian system subject to Gaussian white noises. First, the problem of stochastic optimal control with time delay is formulated. Then, the problem is converted into a stochastic optimal control without time delay, and the converted control problems are then solved by applying the stochastic averaging method and the stochastic dynamical programming principle. The optimal time-delay stabilization of quasi-integrable Hamiltonian systems is formulated as an ergodic control with a cost function determined by minimizing the largest Lyapunov exponent of the controlled system. As an example, a two-degree-of-freedom quasi-integrable Hamiltonian system with time delay in feedback control forces is investigated in detail to illustrate the procedures and their effectiveness.KeywordsTime-delay feedback controlStochastic averaging methodStochastic optimal controlFeedback stabilization

This article addresses the problem of quantifying the uncertainty in planning aircraft ground movement operations using towbarless robotic tractors taking into account the inherent uncertainties of the problem, specifically, the uncertainties in the weight of the aircraft and in the rolling resistance of the wheels of the main landing gear. The tractor‐aircraft system is represented as a tractor‐trailer system with random parameters characterized by means of probability density functions. The quantification of the uncertainties is conducted within an optimal control framework using the formulation of a stochastic optimal control problem, which is solved through a stochastic collocation method based on generalized polynomial chaos. Specifically, the stochastic optimal control problem is converted into a set of deterministic optimal control problems, in which a reduced number of sample values of the random parameters are employed to solve particular instances of the problem. Using these sample values, it is possible to express the obtained optimal solutions as orthogonal polynomial expansions in terms of the uncertain parameters, which allows both statistical and global sensitivity analysis of the stochastic optimal solutions to be carried out in an efficient way. The objectives of this article are to understand the effects of the uncertainties in the model parameters of the tractor‐aircraft system on the solutions of the optimal control problem and to identify which uncertain parameters have more influence on the variability of these solutions. The ultimate goal is to determine the best approach for executing aircraft ground movement operations in the presence of such uncertainties.

This paper addresses a continuous-time continuous-space chance-constrained stochastic optimal control (SOC) problem via a Hamilton-Jacobi-Bellman (HJB) partial differential equation (PDE). Through Lagrangian relaxation, we convert the chance-constrained (risk-constrained) SOC problem to a risk-minimizing SOC problem, the cost function of which possesses the time-additive Bellman structure. We show that the risk-minimizing control synthesis is equivalent to solving an HJB PDE whose boundary condition can be tuned appropriately to achieve a desired level of safety. Furthermore, it is shown that the proposed risk-minimizing control problem can be viewed as a generalization of the problem of estimating the risk associated with a given control policy. Two numerical techniques are explored, namely the path integral and the finite difference method (FDM), to solve a class of risk-minimizing SOC problems whose associated HJB equation is linearizable via the Cole-Hopf transformation. Using a 2D robot navigation example, we validate the proposed control synthesis framework and compare the solutions obtained using path integral and FDM.