A New Approach to Solving Stochastic Optimal Control Problems

A nonlinear stochastic control problem related to ow control is considered. It is assumed that the state of a link is described by a controlled hidden Markov process with a finite state set, while the loss ow is described by a counting process with intensity depending on a current transmission rate and an unobserved link state. The control is the transmission rate, and it has to be chosen as a nonanticipating process depending on the observation of the loss process. The aim of the control is to achieve the maximum of some utility function that takes into account losses of the transmitted information. Originally, the problem belongs to the class of stochastic control problems with incomplete information; however, optimal filtering equations that provide estimation of the current link state based on observations of the loss process allow one to reduce the problem to a standard stochastic control problem with full observations. Then a necessary optimality condition is derived in the form of a stochastic maximum principle, which allows us to obtain explicit analytic expressions for the optimal control in some particular cases. Optimal and suboptimal controls are investigated and compared with the ow control schemes used in TCP/IP (Transmission Control Protocols/Internet Protocols) networks. In particular, the optimal control demonstrates a much smoother behavior than the TCP/IP congestion control currently used.

A global maximum principle for stochastic optimal control problems with delay and applications

Integrating product and process design with optimal control: A case study of solvent recycling

A stochastic control problem is formulated and we get the explicit form of the optimal control for initial-data-parameterized linear quadratic stochastic optimal control problems with random jumps. The optimal control can be proved to be unique. A stochastic Riccati equation is rigorous derived from the stochastic Hamilton system, which provides an optimal feedback control. This completes the the interrelationship between the stochastic Riccati equation and stochastic Hamilton system as two different but equivalent tools for the stochastic linear quadratic problem.

Acid pre-treatment of bagasse is performed in a batch reactor, in which concentration of components evolve over time. This type of systems is usually studied using optimal control, which is concerned with dynamic systems and their optimization over time. This study applies the theory of optimal control for multi-objective optimization. The objectives include maximization of xylose and acid soluble lignin (ASL) fraction, which is obtained during acid pre-treatment. The Pontryagin's maximum principle in combination with the numerical method of steepest descent of Hamiltonian is used to solve multi-objective optimal control problem. The results showed that maximum xylose and acid soluble lignin (ASL) yield was obtained around 0.79 (g/g of xylan) and 0.3 (g/g of lignin), respectively, at final time (t <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> =300 min.). However, acid pre-treatment is influenced by various sources of uncertainties. Therefore, it is essential to optimize the acid pre-treatment process after considering these variabilities. In this work, variability in feedstock composition has been considered, and the stochastic maximum principle is used to perform stochastic multi-objective optimal control problem. Deterministic and stochastic control profiles were obtained for different weights. An open loop simulation of Ito model, using deterministic and stochastic control profile, was performed for 1000 samples and statistics were generated by assuming normal distribution with 95% confidence bound.

In this paper, the stochastic optimal control problems, which frequently occur in economic and finance are investigated. First, using Bellman’s dynamic programming method the stochastic optimal control problems are converted to Hamilton-Jacobi-Bellman (HJB) equation. Then, obtained HJB equation is solved through the method of separation of variables by guessing a solution via its terminal condition. Also, the non-linear optimal feedback control law is constructed. Finally, the solution procedure is illustrated for solving some examples that two of them are financial models. In fact, to highlight the applications of stochastic optimal control problems in financial mathematics, some financial models are presented.

Optimal control of systems with noisy memory and BSDEs with Malliavin derivatives

This paper presents three versions of maximum principle for a stochastic optimal control problem of Markov regime-switching forward–backward stochastic differential equations with jumps. First, a general sufficient maximum principle for optimal control for a system, driven by a Markov regime-switching forward–backward jump–diffusion model, is developed. In the regime-switching case, it might happen that the associated Hamiltonian is not concave and hence the classical maximum principle cannot be applied. Hence, an equivalent type maximum principle is introduced and proved. In view of solving an optimal control problem when the Hamiltonian is not concave, we use a third approach based on Malliavin calculus to derive a general stochastic maximum principle. This approach also enables us to derive an explicit solution of a control problem when the concavity assumption is not satisfied. In addition, the framework we propose allows us to apply our results to solve a recursive utility maximization problem.

In this research article, we focus on a stochastic optimal control problem with two types of terminal constraints. These specific conditions provide real-valued and stochastic Lagrange multipliers. Our model evolves according to a Markov regime-switching jump diffusion model with memory. In this context, the memory is represented by a Stochastic Differential Delay Equation. We present two theorems for each constraint within the general formulation of stochastic optimal control theory in a Lagrangian environment. We approach to this task from a theoretical perspective and provide mild technical assumptions, which make our theorems applicable for a broad class of stochastic control problems as well as for a wide range of disciplines such as engineering, biology, operations research, medicine, computer science and economics. In this work, we apply Stochastic Maximum Principle to demonstrate an optimal dividend policy corresponding to a time-delayed wealth process of a company. Moreover, we determine the real-valued Lagrange multiplier of this control problem explicitly.

This article addresses the problem of quantifying the uncertainty in planning aircraft ground movement operations using towbarless robotic tractors taking into account the inherent uncertainties of the problem, specifically, the uncertainties in the weight of the aircraft and in the rolling resistance of the wheels of the main landing gear. The tractor‐aircraft system is represented as a tractor‐trailer system with random parameters characterized by means of probability density functions. The quantification of the uncertainties is conducted within an optimal control framework using the formulation of a stochastic optimal control problem, which is solved through a stochastic collocation method based on generalized polynomial chaos. Specifically, the stochastic optimal control problem is converted into a set of deterministic optimal control problems, in which a reduced number of sample values of the random parameters are employed to solve particular instances of the problem. Using these sample values, it is possible to express the obtained optimal solutions as orthogonal polynomial expansions in terms of the uncertain parameters, which allows both statistical and global sensitivity analysis of the stochastic optimal solutions to be carried out in an efficient way. The objectives of this article are to understand the effects of the uncertainties in the model parameters of the tractor‐aircraft system on the solutions of the optimal control problem and to identify which uncertain parameters have more influence on the variability of these solutions. The ultimate goal is to determine the best approach for executing aircraft ground movement operations in the presence of such uncertainties.

In this paper, we investigate the problem of maximizing the expected exponential utility for an insurer. In the problem setting, the insurer can invest his/her wealth into the market and he/she can also purchase the proportional reinsurance. To control the risk exposure, we impose a value-at-risk constraint on the portfolio, which results in a constrained stochastic optimal control problem. It is difficult to solve a constrained stochastic optimal control problem by using traditional dynamic programming or Martingale approach. However, for the frequently used exponential utility function, we show that the problem can be simplified significantly using a decomposition approach. The problem is reduced to a deterministic constrained optimal control problem, and then to a finite dimensional optimization problem. To show the effectiveness of the approach proposed, we consider both complete and incomplete markets; the latter arises when the number of risky assets are fewer than the dimension of uncertainty. We also conduct numerical experiments to demonstrate the effect of the risk constraint on the optimal strategy.

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

This work presents the approach to solving the functional equation of non-local type that appears in the study of a stochastic optimal control problem. This experience demonstrates the challenges and possible solutions in the analysis of optimal control problems with Poisson uncertainty.

The stochastic optimal control solution to the Linear-Quadratic-Gaussian problem with partial information is considered where the system parameters operating on the control are assumed to be unknown Gaussian random processes. This class of problems has special properties that are exploited. First, the state and the random parameters are conditionally Gaussian with respect to the measurement history. This allows the optimal stochastic control problem to be reformulated as a stochastic optimal control problem of full information where the new state space is the state and parameter estimates and the error variance of the state and parameters. Even in this somewhat simple form, there is no closed form solution. Therefore, it is assumed that there is only a first order dependence on the random fluctuation of the parameter about its mean. In this way an asymptotic series solution is obtained in terms of the new state space. For this problem a closed form solution for the controller is obtained up to second order. The controller is derived for the scalar case and then extended to the case of vector state with single input and multiple outputs. The characteristics of the new solution is investigated and a scalar numerial example is given to assess the performance of this new controller.

In this paper, necessary conditions of optimality, in the form of a maximum principle, are obtained for singular stochastic control problems. This maximum principle is derived for a state process satisfying a general stochastic differential equation where the coefficient associated to the control process can be dependent on the state. We consider the class of so called robust nonlinear impulsive systems, those discontinuous solutions can be considered also as point-wise limits of ordinary solutions. The special conditions of robustness permit to derive the backward equations for adjoint variables in concise form of differential equation with measure and thereby to derive the optimality condition in the form of strong (point-wise) maximum principle.