An Iterative Procedure for Constructing Subsolutions of Discrete-Time Optimal Control Problems

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.

This article addresses the problem of quantifying the uncertainty in planning aircraft ground movement operations using towbarless robotic tractors taking into account the inherent uncertainties of the problem, specifically, the uncertainties in the weight of the aircraft and in the rolling resistance of the wheels of the main landing gear. The tractor‐aircraft system is represented as a tractor‐trailer system with random parameters characterized by means of probability density functions. The quantification of the uncertainties is conducted within an optimal control framework using the formulation of a stochastic optimal control problem, which is solved through a stochastic collocation method based on generalized polynomial chaos. Specifically, the stochastic optimal control problem is converted into a set of deterministic optimal control problems, in which a reduced number of sample values of the random parameters are employed to solve particular instances of the problem. Using these sample values, it is possible to express the obtained optimal solutions as orthogonal polynomial expansions in terms of the uncertain parameters, which allows both statistical and global sensitivity analysis of the stochastic optimal solutions to be carried out in an efficient way. The objectives of this article are to understand the effects of the uncertainties in the model parameters of the tractor‐aircraft system on the solutions of the optimal control problem and to identify which uncertain parameters have more influence on the variability of these solutions. The ultimate goal is to determine the best approach for executing aircraft ground movement operations in the presence of such uncertainties.

In this paper, we investigate the problem of maximizing the expected exponential utility for an insurer. In the problem setting, the insurer can invest his/her wealth into the market and he/she can also purchase the proportional reinsurance. To control the risk exposure, we impose a value-at-risk constraint on the portfolio, which results in a constrained stochastic optimal control problem. It is difficult to solve a constrained stochastic optimal control problem by using traditional dynamic programming or Martingale approach. However, for the frequently used exponential utility function, we show that the problem can be simplified significantly using a decomposition approach. The problem is reduced to a deterministic constrained optimal control problem, and then to a finite dimensional optimization problem. To show the effectiveness of the approach proposed, we consider both complete and incomplete markets; the latter arises when the number of risky assets are fewer than the dimension of uncertainty. We also conduct numerical experiments to demonstrate the effect of the risk constraint on the optimal strategy.

In Chapters 10 to 15, we have shown the convergence of properly designed numerical approximations for a wide range of stochastic and deterministic optimal control problems. The approach to proving the convergence has been based on demonstrating the convergence of a sequence of controlled Markov chains to a controlled process (diffusion, jump diffusion, etc.) appropriate to the given stochastic or deterministic optimal control problem.KeywordsMarkov ChainNumerical SchemeViscosity SolutionBellman EquationFinite Difference ApproximationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We review a class of online planning algorithms for deterministic and stochastic optimal control problems, modeled as Markov decision processes. At each discrete time step, these algorithms maximize the predicted value of planning policies from the current state, and apply the first action of the best policy found. An overall receding-horizon algorithm results, which can also be seen as a type of model-predictive control. The space of planning policies is explored optimistically, focusing on areas with largest upper bounds on the value - or upper confidence bounds, in the stochastic case. The resulting optimistic planning framework integrates several types of optimism previously used in planning, optimization, and reinforcement learning, in order to obtain several intuitive algorithms with good performance guarantees. We describe in detail three recent such algorithms, outline the theoretical guarantees on their performance, and illustrate their behavior in a numerical example.

OF THE DISSERTATION Optimal Control in Gene Mutation by Juanyi Yu Doctor of Philosophy in Electrical Engineering Washington University in St. Louis, 2011 Research Advisor: Professor Tzyh-Jong Tarn, Professor Jr-Shin Li Gene mutations are the radical causes of many diseases, including inheritance diseases and cancers. Current medical treatments usually focus on changing the concentrations of related chemicals or mRNAs at the cellular level to stop protein productions or cell duplications, which can only control the diseases under certain circumstances but cannot cure them. Little research work has been done at the molecular level, the fundamental of inheritance, to search possible ways to cure those severe diseases. In this dissertation, we propose a molecular level control system view of the gene mutations in DNA replication from the finite field concept. By treating DNA sequences as state variables, chemical mutagens and radiation as control inputs, one cell cycle as a step increment, and the measurements of the resulting DNA sequence as outputs, we derive system equations for both deterministic and stochastic discrete-time, finite-state systems of different scales. Defining the cost function as a summation of the costs of applying mutagens and the off-trajectory penalty, we solve the deterministic and stochastic optimal control problems by dynamic programming algorithm. In

In this paper, the stochastic optimal control problems, which frequently occur in economic and finance are investigated. First, using Bellman’s dynamic programming method the stochastic optimal control problems are converted to Hamilton-Jacobi-Bellman (HJB) equation. Then, obtained HJB equation is solved through the method of separation of variables by guessing a solution via its terminal condition. Also, the non-linear optimal feedback control law is constructed. Finally, the solution procedure is illustrated for solving some examples that two of them are financial models. In fact, to highlight the applications of stochastic optimal control problems in financial mathematics, some financial models are presented.

A stochastic optimal control approach to a mathematical drug administration model

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

We introduce a novel generic methodology to solve continuous finite-horizon stochastic optimal control problems (SOCPs). We treat controlled stochastic differential equations (SDEs) within the Wiener chaos framework by utilizing Malliavin calculus and developing innovative ideas to preserve the feedback character of optimal Markov decision rules. This allows a direct reformulation of SOCPs into deterministic ones. Hence, it facilitates using Bock’s direct multiple shooting method for solving SOCPs and pioneers the extension of sophisticated methods for deterministic control to the broad context of SDEs. Numerical examples validate this new framework with huge computational advantages compared to standard ideas in SOC.

Acid pre-treatment of bagasse is performed in a batch reactor, in which concentration of components evolve over time. This type of systems is usually studied using optimal control, which is concerned with dynamic systems and their optimization over time. This study applies the theory of optimal control for multi-objective optimization. The objectives include maximization of xylose and acid soluble lignin (ASL) fraction, which is obtained during acid pre-treatment. The Pontryagin's maximum principle in combination with the numerical method of steepest descent of Hamiltonian is used to solve multi-objective optimal control problem. The results showed that maximum xylose and acid soluble lignin (ASL) yield was obtained around 0.79 (g/g of xylan) and 0.3 (g/g of lignin), respectively, at final time (t <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> =300 min.). However, acid pre-treatment is influenced by various sources of uncertainties. Therefore, it is essential to optimize the acid pre-treatment process after considering these variabilities. In this work, variability in feedstock composition has been considered, and the stochastic maximum principle is used to perform stochastic multi-objective optimal control problem. Deterministic and stochastic control profiles were obtained for different weights. An open loop simulation of Ito model, using deterministic and stochastic control profile, was performed for 1000 samples and statistics were generated by assuming normal distribution with 95% confidence bound.

Discrete-time hybrid control in Borel spaces: Average cost optimality criterion

This work presents the approach to solving the functional equation of non-local type that appears in the study of a stochastic optimal control problem. This experience demonstrates the challenges and possible solutions in the analysis of optimal control problems with Poisson uncertainty.

A global maximum principle for stochastic optimal control problems with delay and applications

In this chapter* we show how some of the concepts we have developed in our work on duopoly can be applied to other types of decision problems In particular, we will apply our methodology to some problems in control theory. As indicated in chapter 3, we believe that control theory is appropriate for building mathematical models of the firm. We will develop an extension of the usual linear-quadratic, stochastic control problems in discrete time (Simon 1956; DeGroot 1970, sec. 14.10; Chow 1975; Whittle 1982) to problems in which there are two decision-makers or players, both of whom are trying to control the same stochastic system in order to maximize their own individual payoffs or, equivalently, to minimize their own individual losses. The control problems we shall be considering can be regarded as a special class of non-zero-sum, two-person games (for other applications see Grossman 1975a; Fogelman-Soulie, Munier, and Shakun 1983).