Continuity of the value function for deterministic optimal impulse control with terminal state constraint

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

It has been suggested by many supply chain practitioners that in certain cases inventory can have a stimulating effect on the demand. In mathematical terms this amounts to the demand being a function of the inventory level alone. In this work we propose a logistic growth model for the inventory dependent demand rate and solve first the continuous time deterministic optimal control problem of maximising the present value of the total net profit over an infinite horizon. It is shown that under a strict condition there is a unique optimal stock level which the inventory planner should maintain in order to satisfy demand. The stochastic version of the optimal control problem is considered next. A bang-bang type of optimal control problem is formulated and the associated Hamilton-Jacobi-Bellman equation is solved. The inventory level that signifies a switch in the ordering strategy is worked out in the stochastic case.

We consider an optimal control problem for discrete-time linear time-invariant systems subject to unknown state-space model, constrained inputs and noisy output measurements. Since traditional model-based optimal control problem formulations and methods are not applicable to the problem under consideration, we propose a data-driven robust formulation based on the explicit model description derived from a single measured trajectory of the system. Then we propose an open-loop optimal feedback control scheme and show that its efficient implementation requires solution of a number of optimal estimation problems and a deterministic optimal control problem, all in data-driven formulations. The main contributions of this paper are the separation of the estimation and control processes in the data-driven context and the resulting robust feedback control scheme.

In this paper, we investigate the problem of maximizing the expected exponential utility for an insurer. In the problem setting, the insurer can invest his/her wealth into the market and he/she can also purchase the proportional reinsurance. To control the risk exposure, we impose a value-at-risk constraint on the portfolio, which results in a constrained stochastic optimal control problem. It is difficult to solve a constrained stochastic optimal control problem by using traditional dynamic programming or Martingale approach. However, for the frequently used exponential utility function, we show that the problem can be simplified significantly using a decomposition approach. The problem is reduced to a deterministic constrained optimal control problem, and then to a finite dimensional optimization problem. To show the effectiveness of the approach proposed, we consider both complete and incomplete markets; the latter arises when the number of risky assets are fewer than the dimension of uncertainty. We also conduct numerical experiments to demonstrate the effect of the risk constraint on the optimal strategy.

This paper investigates a reliability-based trajectory optimization method for the design of soft landing trajectory on an irregular shape asteroid with highly uncertain gravitational field. First, the gravitational field of the irregular asteroid is described by the finite particle model. Second, to avoid the singularity and reduce the sensitivity, the original finite particle model is modified to an “N-body/two-body” switching dynamic model. The trajectory optimization problem in the switching dynamic model is summarized as an optimal control problem and is then transformed into a two-point boundary value problem by Pontryagin’s maximum principle. By solving the two-point boundary value problem with a homotopic continuation procedure, the nominal optimal trajectory is obtained. Third, the uncertainty caused by the nonuniform mass distribution of the asteroid is considered. With high uncertainty, the deterministic optimal control problem becomes a parameter optimization problem with reliability constraints. This problem is then solved by a sequential optimization and reliability assessment, and the parallel computation technique is adopted. Finally, two soft landing trajectories are optimized using the proposed method to demonstrate the effectiveness of the technique.

SummaryA method is developed to numerically solve chance constrained optimal control problems. The chance constraints are reformulated as nonlinear constraints that retain the probability properties of the original constraint. The reformulation transforms the chance constrained optimal control problem into a deterministic optimal control problem that can be solved numerically. The new method developed in this paper approximates the chance constraints using Markov Chain Monte Carlo sampling and kernel density estimators whose kernels have integral functions that bound the indicator function. The nonlinear constraints resulting from the application of kernel density estimators are designed with bounds that do not violate the bounds of the original chance constraint. The method is tested on a nontrivial chance constrained modification of a soft lunar landing optimal control problem and the results are compared with results obtained using a conservative deterministic formulation of the optimal control problem. Additionally, the method is tested on a complex chance constrained unmanned aerial vehicle problem. The results show that this new method can be used to reliably solve chance constrained optimal control problems.

A counter-example to the characterization of the discontinuous value function of control problems with reflection

An optimal control approach to the design of vibrating elastic-viscoelastic sandwich beams

Optimal feedback production planning in a stochastic N-machine flowshop

In Chapters 10 to 15, we have shown the convergence of properly designed numerical approximations for a wide range of stochastic and deterministic optimal control problems. The approach to proving the convergence has been based on demonstrating the convergence of a sequence of controlled Markov chains to a controlled process (diffusion, jump diffusion, etc.) appropriate to the given stochastic or deterministic optimal control problem.KeywordsMarkov ChainNumerical SchemeViscosity SolutionBellman EquationFinite Difference ApproximationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A stochastic optimal control approach to a mathematical drug administration model

Optimal control simulations have shown that both musculoskeletal dynamics and physiological noise are important determinants of movement. However, due to the limited efficiency of available computational tools, deterministic simulations of movement focus on accurately modelling the musculoskeletal system while neglecting physiological noise, and stochastic simulations account for noise while simplifying the dynamics. We took advantage of recent approaches where stochastic optimal control problems are approximated using deterministic optimal control problems, which can be solved efficiently using direct collocation. We were thus able to extend predictions of stochastic optimal control as a theory of motor coordination to include muscle coordination and movement patterns emerging from non-linear musculoskeletal dynamics. In stochastic optimal control simulations of human standing balance, we demonstrated that the inclusion of muscle dynamics can predict muscle co-contraction as minimal effort strategy that complements sensorimotor feedback control in the presence of sensory noise. In simulations of reaching, we demonstrated that nonlinear multi-segment musculoskeletal dynamics enables complex perturbed and unperturbed reach trajectories under a variety of task conditions to be predicted. In both behaviors, we demonstrated how interactions between task constraint, sensory noise, and the intrinsic properties of muscle influence optimal muscle coordination patterns, including muscle co-contraction, and the resulting movement trajectories. Our approach enables a true minimum effort solution to be identified as task constraints, such as movement accuracy, can be explicitly imposed, rather than being approximated using penalty terms in the cost function. Our approximate stochastic optimal control framework predicts complex features, not captured by previous simulation approaches, providing a generalizable and valuable tool to study how musculoskeletal dynamics and physiological noise may alter neural control of movement in both healthy and pathological movements.

We study the problem of optimal control of a deterministic system with final state constraints using dynamic programming methods. We present a penalization scheme and regularity results for the optimal cost function.

In this paper, an iterative computational method is proposed to handle constrained nonlinear quadratic optimal control problems. with terminal state constraints, saturation state and saturation control variables constraints. The method is based on replacing the original constrained nonlinear optimal control problem by a sequence of constrained linear time-varying quadratic optimal control problems. To this end, an iterative technique is used to replace the original constrained nonlinear dynamic system by a sequence of constrained linear time-varying systems. Then each of constrained linear time-varying quadratic optimal control problems is approximated by a quadratic programming problem by parameterizing each of the state variable by a finite length Legendre polynomials with unknown parameters. To show the effectiveness of the proposed method, simulation results of a constrained nonlinear optimal control problem are presented.

OF THE DISSERTATION Optimal Control in Gene Mutation by Juanyi Yu Doctor of Philosophy in Electrical Engineering Washington University in St. Louis, 2011 Research Advisor: Professor Tzyh-Jong Tarn, Professor Jr-Shin Li Gene mutations are the radical causes of many diseases, including inheritance diseases and cancers. Current medical treatments usually focus on changing the concentrations of related chemicals or mRNAs at the cellular level to stop protein productions or cell duplications, which can only control the diseases under certain circumstances but cannot cure them. Little research work has been done at the molecular level, the fundamental of inheritance, to search possible ways to cure those severe diseases. In this dissertation, we propose a molecular level control system view of the gene mutations in DNA replication from the finite field concept. By treating DNA sequences as state variables, chemical mutagens and radiation as control inputs, one cell cycle as a step increment, and the measurements of the resulting DNA sequence as outputs, we derive system equations for both deterministic and stochastic discrete-time, finite-state systems of different scales. Defining the cost function as a summation of the costs of applying mutagens and the off-trajectory penalty, we solve the deterministic and stochastic optimal control problems by dynamic programming algorithm. In