Optimal Consumption Plans—A Dynamic Programming Approach

Optimal consumption and portfolio policies when asset prices follow a diffusion process

In this paper, the application of the dynamic programming approach to constrained stochastic control problems with expected value constraints is demonstrated. Specifically, two such problems are analyzed using this approach. The problems analyzed are the problem of minimizing a discounted cost infinite horizon expectation objective subject to an identically structured constraint, and the problem of minimizing a discounted cost infinite horizon minimax objective subject to a discounted expectation constraint. Using the dynamic programming approach, optimality equations, which are the chief contribution of this paper, are obtained for these problems. In particular, the dynamic programming operators for problems with expectation constraints differ significantly from those of standard dynamic programming and problems with worst-case constraints. For the discounted cost infinite horizon cases, existence and uniqueness of solutions to the dynamic programming equations are explicitly shown by using the Banach fixed point theorem to show that the corresponding dynamic programming operators are contractions. The theory developed is illustrated by numerically solving the constrained stochastic control dynamic programming equations derived for simple example problems. The example problems are based on a two-state Markov model that represents an error prone system that is to be maintained.

This article presents a novel dynamic programming approach to determine the robust controllability of Boolean control networks (BCNs) subject to stochastic disturbances. By applying Bellman's optimality principle, we derive the recurrence relation for computing the optimal time matrix, a crucial concept characterizing robust reachability between two arbitrary states. We develop a finite-termination dynamic programming algorithm to calculate the optimal time matrix exactly and efficiently, with a rigorously certified iteration count. Sufficient and necessary conditions for robust controllability are then established based on the optimal time matrix. Furthermore, for any pair of reachable states, we construct time-optimal state feedback control laws to steer the system from the initial state to the target state, regardless of disturbances. Finally, extensive numerical experiments with biological networks validate the effectiveness of the proposed approach, showing significant improvements in computational efficiency. Additionally, we introduce a Q-learning-based algorithm and compare its performance, highlighting the advantages of our dynamic programming approach in terms of both efficiency and solution quality.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

This research is about the development of a dynamic programming model for solving fuzzy linear programming problems. Initially, fuzzy dynamic linear programming model FDLP is developed. This research revises the established dynamic programming model for solving linear programming problems in a crisp environment. The mentioned approach is upgraded to address the problem in an uncertain environment. Dynamic programming model can either be passing forward or backward. In the proposed approach backward dynamic programming approach is adopted to address the problem. It is then followed by implementing the proposed method on the education system of Pakistan. The education system of Pakistan comprises of the Primary, Middle, Secondary, and Tertiary education stages. The problem is to maximize the efficiency of the education system while achieving the targets with minimum usage of the constrained resources. Likewise the model tries to maximize the enrollment in the Primary, Middle, Secondary and Tertiary educational categories, subject to the total available resources in a fuzzy uncertain environment. The solution proposes that the enrollment can be increased by an amount 9997130, by increasing the enrollment in the Middle and Tertiary educational categories. Thus the proposed method contributes to increase the objective function value by 30%. Moreover, the proposed solutions violate none of the constraints. In other words, the problem of resources allocation in education system is efficiently managed to increase efficiency while remaining in the available constrained resources. The motivation behind using the dynamic programming methodology is that it always possesses a numerical solution, unlike the other approaches having no solution at certain times. The proposed fuzzy model takes into account uncertainty in the linear programming modeling process and is more robust, flexible and practicable.

For Meyer type optimal control problems the principle of optimality states that every subarc of an optimal solution that extends to the final time furnishes again a solution to the same optimal control problem, only with different initial conditions. Through a simple counter example it is shown in this paper that this principle is not satisfied, in general, for solutions obtained from a standard collocation approach. Introduction The principle of optimality [1] is one of the most intuitive and yet most fundamental theorems in the theory of optimal control. Its basic statement is that for Meyer type optimal control problems every subarc of an optimal solution that extends to the final time furnishes again a solution to the same optimal control problem, only with different initial conditions. This theorem has significant implications. For example, it is the starting point for the derivation of the Hamilton-Jacobi-Bellman equations [2] and it is the underlying principle in Bellman's dynamic programming approach [1], [3]. The present paper shows that the principle of optimality is not satisfied, in general, for solutions obtained from a standard collocation approach[4]. For a simple example whose collocation solution can be generated analytically it is shown that further improvements are possible along an optimal collocation solution if the design parameters along a number of leading nodes are frozen and the remainder of the trajectory is re-optimized. I. Continuous Optimal Control Problem In this section of the paper we introduce a simple class of optimal control problems without state and control constraints. In Section 1.2 we present the associated necessary conditions for optimality [2], [5]. For ease of notation we assume that the right-hand side of the 'Supervising Engineer, Member AIAA, t Senior Engineer, Member AIAA, *17 Research Drive, Hampton, VA 23666. ^Graduate Student, Member AIAA, Department, 304 Town Engineering, Ames, IA 50011 equations of motion does not depend explicitly on time t. This does not represent a loss of generality, as it is always possible to transform explicit time dependence away through the introduction of an additional, timelike state variable. 1. Problem Formulation Let us consider the following class of optimal control problems: min «e(Pwc[*o,*/]), (1) subject to the conditions ±(t) = /(*(*),«(*)), tf.(z(*oMo) = 0, (2)

In modern production systems, it is critical to perform maintenance, calibration, installation, and upgrade tasks during planned downtime. Otherwise, the systems become unreliable and new product introductions are delayed. For reasons of safety, testing, and access, task performance often requires the vicinity of impacted equipment to be left in a specific “end state” when production halts. Therefore, planning the shutdown of a production system to balance production goals against enabling non-production tasks yields a challenging optimization problem. This article proposes a mathematical formulation of this problem and a dynamic programming approach that efficiently finds optimal shutdown policies for deterministic serial production lines. An event-triggered re-optimization procedure that is based on the proposed deterministic dynamic programming approach is also introduced for handling uncertainties in the production line for the stochastic case. It is demonstrated numerically that in these cases with random breakdowns and repairs, the re-optimization procedure is efficient and even obtains results that are optimal or nearly optimal.

This study formulates and solves an advertising pulsation problem for a monopolistic firm using dynamic programming (DP). The firm aims at maximising profit through an optimal allocation of the advertising budget in terms of rectangular pulses over a finite planning horizon. Aggregate sales response to the advertising effort is assumed to be governed by a modified version of the Vidale–Wolfe model in continuous time proposed by Little. Using a numerical example in which a planning horizon of one year is divided into one, two through ten equal time periods, computing routines are developed to solve 150 DP problems. Computational results show among other findings that the performance yielded by the DP policy dominates the uniform advertising policy (constant spending) for a concave advertising response function and the advertising pulsing policy (turning advertising on and off) for a linear or convex response function.

This chapter is devoted to the modification of an extension of dynamic programming approach for optimization of decision rules relative to length. “Classical” dynamic programming approach allows one to obtain optimal rules, i.e., rules with the minimum length. This fact is important from the point of view of knowledge representation. The idea of the dynamic programming approach for optimization of decision rules is based on a partitioning of a decision table into subtables. The algorithm constructs a directed acyclic graph. Basing on the constructed graph, sets of rules with the minimum length, attached to each row of a decision table, can be described. Proposed modification is based on the idea that not the complete graph is constructed but its part. It allows one to obtain values of length of decision rules close to optimal ones, and the size of the graph is smaller than in case of “classical” dynamic programming approach. The chapter also contains results of experiments with decision tables from UCI Machine Learning Repository.

Opaque pricing is a form of pricing where certain characteristics of the product or service are hidden from the consumer until after purchase. In essence, opaque selling transforms a differentiated good into a commodity. Opaque pricing has become popular in service pricing as it allows firms to sell their differentiated product at higher prices to regular brand loyal customers while simultaneously selling to non‐brand loyal customers at discounted prices. We use a nested logit model in combination with logistic regression and dynamic programming to illustrate how a service firm can optimally set prices on an opaque sales channel. The choice model allows the characterization of consumer trade‐offs when purchasing opaque products while the dynamic programming approach allows the characterization of the optimal pricing policy as a function of inventory and time remaining. We compare optimal prices and expected revenues when dynamic pricing is restricted to daily price changes. We provide an illustrative example using data from an opaque selling mechanism ( Hotwire.com ) and a Washington DC‐based hotel.

This paper investigates an optimal decentralized control problem for a system with B-matrices dependent on stochastic parameters. It is assumed that these parameters are independent in time and available locally to each controller. The objective is to find a decentralized state feedback control policy that minimizes a multi step quadratic cost function. We first solve this problem for the case of one time step and show that the optimal policy can be reached through iterating the best responses of each player. For the optimal multiple step problem, a dynamic programming approach is employed while using the result of the one step control at each step. Finally in the infinite horizon case, we provide sufficient conditions under which the optimal control policy is unique and stabilizes the system in a mean square sense.

Ship voyage optimization for safe and energy-efficient navigation: A dynamic programming approach

This paper considers optimal feedback control for a general continuous time finite-dimensional deterministic system with finite horizon cost functional. A practically feasible algorithm to calculate the numerical solution of the optimal feedback control by dynamic programming approach is developed. The highlights of this algorithm are: a) It is based on a convergent constructive algorithm for optimal feedback control law which was proposed by the authors before through an approximation for the viscosity solution of the time-space discretization scheme developed by dynamic programming method; b) The computation complexity is significantly reduced since only values of viscosity solution on some local cones around the optimal trajectory are calculated. Two numerical experiments are presented to illustrate the effectiveness and fastness of the algorithm.

Optimal policy structure characterization for a two-period dual-sourcing inventory control model with forecast updating

We consider complete partitioning of bandwidth among multiple services. When class bandwidth is an integer multiple of the next lower class and total bandwidth is an integer multiple of the largest class bandwidth, we develop a recursive algorithm that determines the optimal complete partitioning policy with a significantly lower complexity than that of known dynamic programming or mixed integer programming approaches.