The Viscosity Solution Approach to Proving Convergence of Numerical Schemes

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.

In this paper, we investigate the problem of maximizing the expected exponential utility for an insurer. In the problem setting, the insurer can invest his/her wealth into the market and he/she can also purchase the proportional reinsurance. To control the risk exposure, we impose a value-at-risk constraint on the portfolio, which results in a constrained stochastic optimal control problem. It is difficult to solve a constrained stochastic optimal control problem by using traditional dynamic programming or Martingale approach. However, for the frequently used exponential utility function, we show that the problem can be simplified significantly using a decomposition approach. The problem is reduced to a deterministic constrained optimal control problem, and then to a finite dimensional optimization problem. To show the effectiveness of the approach proposed, we consider both complete and incomplete markets; the latter arises when the number of risky assets are fewer than the dimension of uncertainty. We also conduct numerical experiments to demonstrate the effect of the risk constraint on the optimal strategy.

OF THE DISSERTATION Optimal Control in Gene Mutation by Juanyi Yu Doctor of Philosophy in Electrical Engineering Washington University in St. Louis, 2011 Research Advisor: Professor Tzyh-Jong Tarn, Professor Jr-Shin Li Gene mutations are the radical causes of many diseases, including inheritance diseases and cancers. Current medical treatments usually focus on changing the concentrations of related chemicals or mRNAs at the cellular level to stop protein productions or cell duplications, which can only control the diseases under certain circumstances but cannot cure them. Little research work has been done at the molecular level, the fundamental of inheritance, to search possible ways to cure those severe diseases. In this dissertation, we propose a molecular level control system view of the gene mutations in DNA replication from the finite field concept. By treating DNA sequences as state variables, chemical mutagens and radiation as control inputs, one cell cycle as a step increment, and the measurements of the resulting DNA sequence as outputs, we derive system equations for both deterministic and stochastic discrete-time, finite-state systems of different scales. Defining the cost function as a summation of the costs of applying mutagens and the off-trajectory penalty, we solve the deterministic and stochastic optimal control problems by dynamic programming algorithm. In

An iterative procedure for constructing subsolutions of deterministic or stochastic optimal control problems in discrete time with continuous state space is introduced. The procedure generates a nondecreasing sequence of subsolutions, giving true lower bounds on the minimal costs. Convergence of the values at any fixed initial state is shown.

A stochastic optimal control approach to a mathematical drug administration model

We review a class of online planning algorithms for deterministic and stochastic optimal control problems, modeled as Markov decision processes. At each discrete time step, these algorithms maximize the predicted value of planning policies from the current state, and apply the first action of the best policy found. An overall receding-horizon algorithm results, which can also be seen as a type of model-predictive control. The space of planning policies is explored optimistically, focusing on areas with largest upper bounds on the value - or upper confidence bounds, in the stochastic case. The resulting optimistic planning framework integrates several types of optimism previously used in planning, optimization, and reinforcement learning, in order to obtain several intuitive algorithms with good performance guarantees. We describe in detail three recent such algorithms, outline the theoretical guarantees on their performance, and illustrate their behavior in a numerical example.

This article addresses the problem of quantifying the uncertainty in planning aircraft ground movement operations using towbarless robotic tractors taking into account the inherent uncertainties of the problem, specifically, the uncertainties in the weight of the aircraft and in the rolling resistance of the wheels of the main landing gear. The tractor‐aircraft system is represented as a tractor‐trailer system with random parameters characterized by means of probability density functions. The quantification of the uncertainties is conducted within an optimal control framework using the formulation of a stochastic optimal control problem, which is solved through a stochastic collocation method based on generalized polynomial chaos. Specifically, the stochastic optimal control problem is converted into a set of deterministic optimal control problems, in which a reduced number of sample values of the random parameters are employed to solve particular instances of the problem. Using these sample values, it is possible to express the obtained optimal solutions as orthogonal polynomial expansions in terms of the uncertain parameters, which allows both statistical and global sensitivity analysis of the stochastic optimal solutions to be carried out in an efficient way. The objectives of this article are to understand the effects of the uncertainties in the model parameters of the tractor‐aircraft system on the solutions of the optimal control problem and to identify which uncertain parameters have more influence on the variability of these solutions. The ultimate goal is to determine the best approach for executing aircraft ground movement operations in the presence of such uncertainties.

Acid pre-treatment of bagasse is performed in a batch reactor, in which concentration of components evolve over time. This type of systems is usually studied using optimal control, which is concerned with dynamic systems and their optimization over time. This study applies the theory of optimal control for multi-objective optimization. The objectives include maximization of xylose and acid soluble lignin (ASL) fraction, which is obtained during acid pre-treatment. The Pontryagin's maximum principle in combination with the numerical method of steepest descent of Hamiltonian is used to solve multi-objective optimal control problem. The results showed that maximum xylose and acid soluble lignin (ASL) yield was obtained around 0.79 (g/g of xylan) and 0.3 (g/g of lignin), respectively, at final time (t <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> =300 min.). However, acid pre-treatment is influenced by various sources of uncertainties. Therefore, it is essential to optimize the acid pre-treatment process after considering these variabilities. In this work, variability in feedstock composition has been considered, and the stochastic maximum principle is used to perform stochastic multi-objective optimal control problem. Deterministic and stochastic control profiles were obtained for different weights. An open loop simulation of Ito model, using deterministic and stochastic control profile, was performed for 1000 samples and statistics were generated by assuming normal distribution with 95% confidence bound.

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

This article examines an economic growth model that expresses the interaction between production, technology stock, and research and development (R&amp;D) investments. The goal of this study is to maximize production. Considering the presence of Gaussian white noises, this model is reformulated as a stochastic optimal control problem, where the R&amp;D investment rate is defined as the control input. We aim to explore the efficiency of Kalman filtering approaches for solving this problem. Here, the extended Kalman filter (EKF) and unscented Kalman filter (UKF) are applied for state estimation. The state equation linearization is made in the EKF, while the unscented transform is taken in the UKF for generating a set of sigma points. These approaches aim to estimate the state dynamics from different perspectives. With these state estimates, two different computational algorithms are proposed, the EKF for state-control (EKF4SC) and UKF for state-control (UKF4SC) algorithms. The optimal control policy is designed to minimize the cost function. For illustration, the model's parameters are considered in the simulation experiment. The simulation results showed that the UKF has higher accuracy in the state estimation with the smallest mean squares of error compared with the EKF. Moreover, the optimal control policy based on the state estimate generated from the UKF could optimize the cost function of the problem. Hence, the results show the efficiency of the algorithms proposed. In conclusion, the application of the Kalman filtering algorithms to the economic growth model is presented. The significance of this study is to provide a stochastic optimal control model for the economic growth problem and to suggest an efficient computational approach for solving this stochastic optimal control problem.

In this paper, the stochastic optimal control problems, which frequently occur in economic and finance are investigated. First, using Bellman’s dynamic programming method the stochastic optimal control problems are converted to Hamilton-Jacobi-Bellman (HJB) equation. Then, obtained HJB equation is solved through the method of separation of variables by guessing a solution via its terminal condition. Also, the non-linear optimal feedback control law is constructed. Finally, the solution procedure is illustrated for solving some examples that two of them are financial models. In fact, to highlight the applications of stochastic optimal control problems in financial mathematics, some financial models are presented.

In this research article, we focus on a stochastic optimal control problem with two types of terminal constraints. These specific conditions provide real-valued and stochastic Lagrange multipliers. Our model evolves according to a Markov regime-switching jump diffusion model with memory. In this context, the memory is represented by a Stochastic Differential Delay Equation. We present two theorems for each constraint within the general formulation of stochastic optimal control theory in a Lagrangian environment. We approach to this task from a theoretical perspective and provide mild technical assumptions, which make our theorems applicable for a broad class of stochastic control problems as well as for a wide range of disciplines such as engineering, biology, operations research, medicine, computer science and economics. In this work, we apply Stochastic Maximum Principle to demonstrate an optimal dividend policy corresponding to a time-delayed wealth process of a company. Moreover, we determine the real-valued Lagrange multiplier of this control problem explicitly.

Most of the existing literature concerning lifetime extension of battery energy storage (BES) devices mainly focus on designing controllers in BES devices themselves. In this paper, the problem of BES lifetime extension in an islanded microgrid is considered from a different perspective, in the sense that such target is achieved by adjusting output power of controllable power generation devices. To obtain accurate power models of photovoltaic panels and loads, a new hybrid modeling method utilizing both neural networks and stochastic differential equations is proposed. We formulate the problem of extending the service life of BES devices as a stochastic optimal control problem. Meanwhile, the situation of over-control in micro-turbines is effectively avoided. We solve such stochastic optimization problem using the dynamic programming solver provided by the open source C++ program BOCOPHJB. Finally, the obtained optimal controller is verified with simulations.

A Class of Linear Quadratic Gaussian Hybrid Optimal Control Problems with Realization–Independent Riccati Equations

Integrating product and process design with optimal control: A case study of solvent recycling