A stochastic optimal control approach to a mathematical drug administration model

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.

We provide a data-driven framework for optimal control of a continuous-time control-affine stochastic dynamical system. The proposed framework relies on the linear operator theory involving Perron-Frobenius (P-F) and Koopman operators. Our first result involving the P-F operator provides a convex formulation to the optimal control problem in the dual space of densities. This convex formulation of the stochastic optimal control problem leads to an infinite-dimensional convex program. The finite-dimensional approximation of the convex program is obtained using a data-driven approximation of the linear operators. We provide rigorous error bounds and convergence rate for the data-driven approximation of the linear operators and comment on the convergence rate of the optimal control w.r.t. data size. Our second results demonstrate using the Koopman operator, dual to the P-F operator, for the stochastic optimal control design. We show that the Hamilton Jacobi Bellman (HJB) equation can be expressed using the Koopman operator. We provide an iterative procedure along the lines of a popular policy iteration algorithm based on the data-driven approximation of the Koopman operator for solving the HJB equation. The two formulations, namely the convex formulation involving the P-F operator and the Koopman-based formulation using the HJB equation, can be viewed as dual to each other where the duality follows due to the dual nature of the P-F and Koopman operators. Finally, we present examples to demonstrate the efficacy of the developed framework and verify the convergence rates for the operator and optimal control numerically.

A minimax stochastic optimal semi-active control strategy for stochastically excited quasi-integrable Hamiltonian systems with parametric uncertainty by using magneto-rheological (MR) dampers is proposed. Firstly, the control problem is formulated as an n-degree-of-freedom (DOF) controlled, uncertain quasi-integrable Hamiltonian system and the control forces produced by MR dampers are split into the passive part and the semi-active part. Then the passive part is incorporated into the uncontrolled system. After that, the stochastic optimal semi-active control problem is solved by applying the minimax stochastic optimal control strategy based on the stochastic averaging method and stochastic differential game. The worst-case disturbances and the optimal controls are obtained by the minimax dynamical programming equation with the constraints of disturbance bounds and MR damper dynamics. Finally, the system response and the control performance are evaluated by using Monte Carlo simulation. An example of a two-DOF system with coupling damping and parametric uncertainty under Gaussian white noise excitations is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy, which is also compared with the clipped linear-quadratic-Gaussian control strategy to show the advantages.

A stochastic control problem is formulated and we get the explicit form of the optimal control for initial-data-parameterized linear quadratic stochastic optimal control problems with random jumps. The optimal control can be proved to be unique. A stochastic Riccati equation is rigorous derived from the stochastic Hamilton system, which provides an optimal feedback control. This completes the the interrelationship between the stochastic Riccati equation and stochastic Hamilton system as two different but equivalent tools for the stochastic linear quadratic problem.

The stochastic optimal control solution to the Linear-Quadratic-Gaussian problem with partial information is considered where the system parameters operating on the control are assumed to be unknown Gaussian random processes. This class of problems has special properties that are exploited. First, the state and the random parameters are conditionally Gaussian with respect to the measurement history. This allows the optimal stochastic control problem to be reformulated as a stochastic optimal control problem of full information where the new state space is the state and parameter estimates and the error variance of the state and parameters. Even in this somewhat simple form, there is no closed form solution. Therefore, it is assumed that there is only a first order dependence on the random fluctuation of the parameter about its mean. In this way an asymptotic series solution is obtained in terms of the new state space. For this problem a closed form solution for the controller is obtained up to second order. The controller is derived for the scalar case and then extended to the case of vector state with single input and multiple outputs. The characteristics of the new solution is investigated and a scalar numerial example is given to assess the performance of this new controller.

This article examines an economic growth model that expresses the interaction between production, technology stock, and research and development (R&D) investments. The goal of this study is to maximize production. Considering the presence of Gaussian white noises, this model is reformulated as a stochastic optimal control problem, where the R&D investment rate is defined as the control input. We aim to explore the efficiency of Kalman filtering approaches for solving this problem. Here, the extended Kalman filter (EKF) and unscented Kalman filter (UKF) are applied for state estimation. The state equation linearization is made in the EKF, while the unscented transform is taken in the UKF for generating a set of sigma points. These approaches aim to estimate the state dynamics from different perspectives. With these state estimates, two different computational algorithms are proposed, the EKF for state-control (EKF4SC) and UKF for state-control (UKF4SC) algorithms. The optimal control policy is designed to minimize the cost function. For illustration, the model's parameters are considered in the simulation experiment. The simulation results showed that the UKF has higher accuracy in the state estimation with the smallest mean squares of error compared with the EKF. Moreover, the optimal control policy based on the state estimate generated from the UKF could optimize the cost function of the problem. Hence, the results show the efficiency of the algorithms proposed. In conclusion, the application of the Kalman filtering algorithms to the economic growth model is presented. The significance of this study is to provide a stochastic optimal control model for the economic growth problem and to suggest an efficient computational approach for solving this stochastic optimal control problem.

The stochastic optimal nonlinear control of coupled adjacent building structures is studied based on the stochastic dynamical programming principle and the stochastic averaging method. The coupled structures with control devices under random seismic excitation are first condensed to form a reduced-order structural model for the control analysis. The stochastic averaging method is applied to the reduced model to yield stochastic differential equations for structural modal energies as controlled diffusion processes. Then a dynamical programming equation for the energy processes is established based on the stochastic dynamical programming principle, and solved to determine the optimal nonlinear control law. The seismic response mitigation of the coupled structures is achieved through the structural energy control and the dimension of the optimal control problem is reduced. The seismic excitation spectrum is taken into account according to the stochastic dynamical programming principle. Finally, the nonlinear controlled structural response is predicted by using the stochastic averaging method and compared with the uncontrolled structural response to evaluate the control efficacy. Numerical results are given to demonstrate the response mitigation capabilities of the proposed stochastic optimal control method for coupled adjacent building structures.

We consider a sufficient maximum principle of optimal control for a stochastic control problem. This problem is governed by a system of fully coupled multi-dimensional forward-backward doubly stochastic differential equation with Poisson jumps. Moreover, all the coefficients appearing in this system are allowed to be random and depend on the control variable. We derive, in particular, sufficient conditions for optimality for this stochastic optimal control problem. We apply our result to treat a kind of forward-backward doubly stochastic linear quadratic optimal control problems with jumps.

In this paper, we investigate the problem of maximizing the expected exponential utility for an insurer. In the problem setting, the insurer can invest his/her wealth into the market and he/she can also purchase the proportional reinsurance. To control the risk exposure, we impose a value-at-risk constraint on the portfolio, which results in a constrained stochastic optimal control problem. It is difficult to solve a constrained stochastic optimal control problem by using traditional dynamic programming or Martingale approach. However, for the frequently used exponential utility function, we show that the problem can be simplified significantly using a decomposition approach. The problem is reduced to a deterministic constrained optimal control problem, and then to a finite dimensional optimization problem. To show the effectiveness of the approach proposed, we consider both complete and incomplete markets; the latter arises when the number of risky assets are fewer than the dimension of uncertainty. We also conduct numerical experiments to demonstrate the effect of the risk constraint on the optimal strategy.

This article addresses the problem of quantifying the uncertainty in planning aircraft ground movement operations using towbarless robotic tractors taking into account the inherent uncertainties of the problem, specifically, the uncertainties in the weight of the aircraft and in the rolling resistance of the wheels of the main landing gear. The tractor‐aircraft system is represented as a tractor‐trailer system with random parameters characterized by means of probability density functions. The quantification of the uncertainties is conducted within an optimal control framework using the formulation of a stochastic optimal control problem, which is solved through a stochastic collocation method based on generalized polynomial chaos. Specifically, the stochastic optimal control problem is converted into a set of deterministic optimal control problems, in which a reduced number of sample values of the random parameters are employed to solve particular instances of the problem. Using these sample values, it is possible to express the obtained optimal solutions as orthogonal polynomial expansions in terms of the uncertain parameters, which allows both statistical and global sensitivity analysis of the stochastic optimal solutions to be carried out in an efficient way. The objectives of this article are to understand the effects of the uncertainties in the model parameters of the tractor‐aircraft system on the solutions of the optimal control problem and to identify which uncertain parameters have more influence on the variability of these solutions. The ultimate goal is to determine the best approach for executing aircraft ground movement operations in the presence of such uncertainties.

In this paper, we use the solutions of forward-backward stochastic differential equations with random jumps to get the optimal controls for forward-backward stochastic linear quadratic optimal control problem and the Nash equilibrium point for nonzero sum differential games problem.

The authors study the asymptotic behavior of stochastic evolutionary systems with Markov-modulated Poisson perturbation in the diffusion approximation scheme. They consider combining the Poisson process with the Markov process, which allows for describing random transitions between different modes of evolution. The ergodic properties of the Markov-modulated Poisson process that ensure the stable behavior of the system on average are presented. Boundary generators for the original system of stochastic differential equations are constructed. The results allow for studying stochastic optimization and optimal control problems. Keywords: stochastic evolution, Markov-modulated Poisson process, diffusion approximation schema.

We consider the stochastic linear quadratic optimal control problem where the state equation is given by a stochastic differential equation of the Ito–Skorokhod type with respect to fractional Brownian motion. The dynamics are driven by strongly continuous semigroups and the cost functional is quadratic. We use the fractional isometry mapping defined between the space of square integrable stochastic processes with respect to fractional Gaussian white noise measure and the space of integrable stochastic processes with respect to the classical Gaussian white noise measure. By this mapping we transform the fractional state equation to a state equation with Brownian motion. Applying the chaos expansion approach, we can solve the optimal control problem with respect to a state equation with the standard Brownian motion. We recover the solution of the original problem by the inverse of the fractional isometry mapping. Finally, we consider a general form of the state equation related to the Gaussian colored noise, we study the control problem, a system with an algebraic constraint and a particular example involving generalized operators from the Malliavin calculus.

1 A Survey on Computational Optimal Control.- Issues in the Direct Transcription of Optimal Control Problems to Sparse Nonlinear Programs.- Optimization in Control of Robots.- Large-scale SQP Methods and their Application in Trajectory Optimization.- Solving Optimal Control and Pursuit-Evasion Game Problems of High Complexity.- 2 Theoretical Aspects of Optimal Control and Nonlinear Programming.- Continuation Methods In Boundary Value Problems.- Second Order Optimality Conditions for Singular Extremals.- Synthesis of Adaptive Optimal Controls for Linear Dynamic Systems.- Control Applications of Reduced SQP Methods.- Time Optimal Control of Mechanical Systems.- 3 Algorithms for Optimal Control Calculations.- Second Order Algorithm for Time Optimal Control of a Linear System.- An SQP-type Solution Method for Constrained Discrete-Time Optimal Control Problems.- Numerical Methods for Solving Differential Games, Prospective Applications to Technical Problems.- Construction of the Optimal Feedback Controller for Constrained Optimal Control Problems with Unknown Disturbances.- Repetitive Optimization for Predictive Control of Dynamic Systems under Uncertainty.- Optimal Control of Multistage Systems Described by High-Index Differential-Algebraic Equations.- A New Class of a High Order Interior Point Method for the Solution of Convex Semiinfinite Optimization Problems.- A Structured Interior Point SQP Method for Nonlinear Optimal Control Problems.- 4 Software for Optimal Control Calculations.- Automated Approach for Optimizing Dynamic Systems.- ANDECS: A Computation Environment for Control Applications of Optimization.- Application of Automatic Differentiation to Optimal Control Problems.- OCCAL: A mixed symbolic-numeric Optimal Control CALculator.- 5 Applications of Optimal Control.- A Robotic Satellite with Simplified Design.- Nonlinear Control under Constraints of a Biological System.- An Object-Oriented Approach to Optimally Describe and Specify a SCADA System Applied to a Power Network.- Near-Optimal Flight Trajectories Generated by Neural Networks.- Performance of a Feedback Method with Respect to Changes in the Air-Density during the Ascent of a Two-Stage-To-Orbit Vehicle.- Linear Optimal Control for Reentry Flight.- Steady-State Modelling of Turbine Engine with Controllers.- Shortest Paths for Satellite Mounted Robot Manipulators.- Optimal Control of the Industrial Robot Manutec r3.