The Scope for Dynamic Programming Applied to Resource Management

This research is about the development of a dynamic programming model for solving fuzzy linear programming problems. Initially, fuzzy dynamic linear programming model FDLP is developed. This research revises the established dynamic programming model for solving linear programming problems in a crisp environment. The mentioned approach is upgraded to address the problem in an uncertain environment. Dynamic programming model can either be passing forward or backward. In the proposed approach backward dynamic programming approach is adopted to address the problem. It is then followed by implementing the proposed method on the education system of Pakistan. The education system of Pakistan comprises of the Primary, Middle, Secondary, and Tertiary education stages. The problem is to maximize the efficiency of the education system while achieving the targets with minimum usage of the constrained resources. Likewise the model tries to maximize the enrollment in the Primary, Middle, Secondary and Tertiary educational categories, subject to the total available resources in a fuzzy uncertain environment. The solution proposes that the enrollment can be increased by an amount 9997130, by increasing the enrollment in the Middle and Tertiary educational categories. Thus the proposed method contributes to increase the objective function value by 30%. Moreover, the proposed solutions violate none of the constraints. In other words, the problem of resources allocation in education system is efficiently managed to increase efficiency while remaining in the available constrained resources. The motivation behind using the dynamic programming methodology is that it always possesses a numerical solution, unlike the other approaches having no solution at certain times. The proposed fuzzy model takes into account uncertainty in the linear programming modeling process and is more robust, flexible and practicable.

In this paper, the application of the dynamic programming approach to constrained stochastic control problems with expected value constraints is demonstrated. Specifically, two such problems are analyzed using this approach. The problems analyzed are the problem of minimizing a discounted cost infinite horizon expectation objective subject to an identically structured constraint, and the problem of minimizing a discounted cost infinite horizon minimax objective subject to a discounted expectation constraint. Using the dynamic programming approach, optimality equations, which are the chief contribution of this paper, are obtained for these problems. In particular, the dynamic programming operators for problems with expectation constraints differ significantly from those of standard dynamic programming and problems with worst-case constraints. For the discounted cost infinite horizon cases, existence and uniqueness of solutions to the dynamic programming equations are explicitly shown by using the Banach fixed point theorem to show that the corresponding dynamic programming operators are contractions. The theory developed is illustrated by numerically solving the constrained stochastic control dynamic programming equations derived for simple example problems. The example problems are based on a two-state Markov model that represents an error prone system that is to be maintained.

In this paper, we propose a resource allocation scheme for interference management in heterogeneous networks. We particularly consider downlink interference from a Home eNB (HeNB) to macrocell user equipments (MUEs) in the coverage of the traditional macrocell basestation (MBS). By overhearing uplink feedback information from MUEs and Home User Equipments (HUEs) together with downlink control information (DCI) from the MBS, the HeNB formulates a dynamic programming problem with the objective of maximizing the total reward over an infinite horizon. By exploiting the feedback information as well as the DCI at the HeNB, we show that the solution of the dynamic programming problem follows a greedy policy at the HeNB, which simplifies the solution of the infinite horizon dynamic programming problem. We also examine the effect of uncertainty in the feedback information on the performance of our proposed scheme.

In the next decades, many public infrastructure assets will reach the end of their life that they were originally designed for. Replacement costs are high, and therefore increasing effort is put into lifetime-extending maintenance, including major overhauls and renovations. A key question is whether the investments in lifetime-extending maintenance justify the postponement of a full replacement. This question becomes more complicated when future life cycle cash flows are non-repeatable. Differential inflation and technological change, including multiple intervention strategies to maintain a desired functionality, cause such non-repeatability. In this case, classic replacement analysis techniques will not suffice in answering this question. Literature demonstrates that case-specific modelling with dynamic or linear programming techniques is required to find economic optimisation. However, such literature primarily addresses replacement interval optimisation of new investments within relative short time horizons, whereas the current research develops a nested dynamic programming (DP) approach for typical ageing infrastructure assets over long service life periods. The model can deal with multiple and various successive intervention strategies and addresses differential inflation and age-related cost increases. Finally, it is shown in an infrastructure case study that this DP approach leads to a better decision in comparison to the application of classical replacement techniques.

Ship voyage optimization for safe and energy-efficient navigation: A dynamic programming approach

Quiet Achiever It is always sad when we lose colleagues. But it is particularly sad when they die too young. We worked with John for many years, and he was always highly regarded as a professional in agricultural and environmental economics. He was somewhat shy and retiring, always a gentle person, quietly spoken, very thoughtful and scholarly and very caring towards his colleagues, and undergraduate and postgraduate students. Former postgraduate students have commented that John's style as a supervisor was to guide and encourage, rather than to direct. He was an absolute joy to work with. We received numerous responses from his colleagues in agricultural and environmental economics following the announcement of his death, as we expected. But it was only when we read responses from colleagues around the world in fisheries resources and economics that we realised how widespread his impact has been. John was a member of the International Institute of Fisheries Economics and Trade. Following John's death, the Institute circulated to its membership an appreciation of John's contributions and forwarded their comments to us. He was indeed a ‘Quiet Achiever’. John was born in Scotland and raised in London. He obtained a BSc in Economics and Accounting at Bristol University and a PhD in Agricultural Economics at Wye College, University of London in 1971. He came to Australia in 1971 and for 3 years was a Research Fellow in the Department of Agricultural Economics and Business Management at the University of New England (UNE). Here, John commenced the building of an Aggregative Programming Model for Australian Agriculture (APMAA) under the supervision of Professors John Dillon and John Guise. The project grew to support a team of four full-time researchers and was ahead of its time. It continued after John left UNE. John joined La Trobe University in 1975 as a lecturer in the School of Economics where he stayed for 37 years, reaching positions of Reader and Associate Professor. John's wife Ann wrote after his death that when John was still well he remembered with great fondness the halcyon days at La Trobe with his agricultural economics colleagues. His main research and teaching interests at La Trobe University were in agricultural, fisheries, forestry and environmental economics. He employed and further developed decision-making tools – in particular dynamic programming. He initiated and taught a wide array of courses at undergraduate and postgraduate levels, including microeconomic theory, cost-benefit analysis, global environmental issues, resource and environmental economics, environmental law and management, and location and transport economics, to name a few. He supervised around a dozen PhD theses including ‘Methodological issues in contingent valuation: An application to endangered species’; ‘Sustainable and conventional agriculture: An economic analysis of Australian cereal/livestock farming’; ‘Trade games: non-cooperative strategies in the international wheat market’; ‘Incentives for switching from fossil fuels to renewable energy sources by the North and the South: a game-theoretic analysis of response to global warming’; and ‘The effect of trade in carbon credits on global timber demand’. John published around 50 journal articles, a number of books and book chapters, including the book ‘Dynamic Programming: Applications to Agriculture and Natural Resources’. His journal articles ranged from ‘A model for determining optimal feeding and marketing policies for beef cattle’ to ‘The impact of quotas on the Southern Bluefin Tuna industry’. The article ‘Within-season rents: maximized or dissipated in an open-access fishery?’ co-authored by John with Rognvaldur Hannesson won the outstanding article award in Marine Resource Economics for 2006. His book on Dynamic Programming is a seminal work that has been widely quoted in the literature. A reviewer in the Australian Journal of Agricultural Economics (1988) concluded that the book would make the tools of dynamic analysis available to a much wider audience of applied economists and that its greatest use would be for researchers and applied economists in the analysis of stochastic and dynamic problems. A valuable feature of the book was its inclusion of generalised computer programs in BASIC for different classes of dynamic programming problems. John won research grants including a significant grant by The Australian Research Council for a project on the ‘Determination of Optimal Annual Fish Catches allowing for Multicohort Stocks and Adjustments in Fleet Size’. He was also appointed Project Leader by the Australian Centre for International Agricultural Research for the project ‘Maximising the economic benefits to Pacific Island Nations from management of migratory tuna stocks’ with a budget of approximately $600,000. Professional positions held by John have included Book Review Editor and Joint Editor, Australian Journal of Agricultural Economics; Associate Dean for Research and Postgraduate Studies, School of Economics and Commerce, La Trobe University; President, Victorian Branch of the Australian Agricultural Economics Society; Distinguished Research Fellow, Centre for Fisheries Economics, Norwegian School of Economics and Business Administration, Bergen; and Associate Editor, Marine Resource Economics. John also participated in a number of consultancies including for the Bureau of Immigration Research on ‘Immigration, Population Growth and the Environment’; and the Victorian Department of Conservation and Environment on ‘Economic Principles for the Establishment and Optimisation of Conjunctive Water Use Schemes’. John had several periods of sabbatical leave in the United States, Norway, England and Scotland, as well as at Australian universities. He and his wife Ann were especially fond of Edinburgh and planned to make their home there in retirement. Unfortunately, John's retirement years were few and beset by the rapid onset of dementia. John is survived by Ann and their daughters Jessica and Vanessa. John will be sorely missed by all those who knew him.

Optimal Consumption Plans—A Dynamic Programming Approach

This study formulates and solves an advertising pulsation problem for a monopolistic firm using dynamic programming (DP). The firm aims at maximising profit through an optimal allocation of the advertising budget in terms of rectangular pulses over a finite planning horizon. Aggregate sales response to the advertising effort is assumed to be governed by a modified version of the Vidale–Wolfe model in continuous time proposed by Little. Using a numerical example in which a planning horizon of one year is divided into one, two through ten equal time periods, computing routines are developed to solve 150 DP problems. Computational results show among other findings that the performance yielded by the DP policy dominates the uniform advertising policy (constant spending) for a concave advertising response function and the advertising pulsing policy (turning advertising on and off) for a linear or convex response function.

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

Dynamic programming is a mathematical technique for optimizing decisions in stages by breaking a problem inton stages mthat solve recursively. It display the quantity and arrangement of invemtory to maintain production and ,businness revenue efficiently when used in inventory control management. This production planning uses a dynamic programming method based on the results of advanced calculations start from stage 1 and move forward to stage 12. the dynamic programming approach in this writing is deterministic because the pattern of the number of requests for bread is known absolutely. Decisions involving a problem are simplified little by little rather than directly. This research uses linear regression with a mathematical formula that is: followed by the defect percetage formula to forecast demand for the next 12 months and apply a dynamic program to achieve the minimum total quantity.

This paper considers optimal feedback control for a general continuous time finite-dimensional deterministic system with finite horizon cost functional. A practically feasible algorithm to calculate the numerical solution of the optimal feedback control by dynamic programming approach is developed. The highlights of this algorithm are: a) It is based on a convergent constructive algorithm for optimal feedback control law which was proposed by the authors before through an approximation for the viscosity solution of the time-space discretization scheme developed by dynamic programming method; b) The computation complexity is significantly reduced since only values of viscosity solution on some local cones around the optimal trajectory are calculated. Two numerical experiments are presented to illustrate the effectiveness and fastness of the algorithm.

This article presents a novel dynamic programming approach to determine the robust controllability of Boolean control networks (BCNs) subject to stochastic disturbances. By applying Bellman's optimality principle, we derive the recurrence relation for computing the optimal time matrix, a crucial concept characterizing robust reachability between two arbitrary states. We develop a finite-termination dynamic programming algorithm to calculate the optimal time matrix exactly and efficiently, with a rigorously certified iteration count. Sufficient and necessary conditions for robust controllability are then established based on the optimal time matrix. Furthermore, for any pair of reachable states, we construct time-optimal state feedback control laws to steer the system from the initial state to the target state, regardless of disturbances. Finally, extensive numerical experiments with biological networks validate the effectiveness of the proposed approach, showing significant improvements in computational efficiency. Additionally, we introduce a Q-learning-based algorithm and compare its performance, highlighting the advantages of our dynamic programming approach in terms of both efficiency and solution quality.

In the future, automated smart home energy management systems (SHEMSs) will assist residential energy users to schedule and coordinate their energy use. In order to undertake efficient and robust scheduling of distributed energy resources, such a SHEMS needs to consider the stochastic nature of the household's energy use and the intermittent nature of its distributed generation. Currently, stochastic mixed-integer linear programming (MILP), particle swarm optimization and dynamic programming approaches have been proposed for incorporating these stochastic variables. However, these approaches result in a SHEMS with very costly computational requirements or lower quality solutions. Given this context, this paper discusses the drawbacks associated with these existing methods by comparing a SHEMS using stochastic MILP with heuristic scenario reduction techniques to one using a dynamic programming approach. Then, drawing on analysis of the two methods above, this paper discusses ways of reducing the computational burden of the stochastic optimization framework by using approximate dynamic programming to implement a SHEMS.

Many Cyber-Physical Systems (CPSs) demonstrate high-dimensional environmental uncertainties that modulate the physical system dynamics and complicate the computational decision-making tasks. As an example, strategic air traffic flow management (ATFM) aims to resolve air traffic congestion through managing traffic flows at a long look-ahead time. The convective weather, a dominant factor of traffic congestion, is very uncertain and significantly complicates the decision-making process. Other uncertainties such as traffic demands further expand the uncertainty space and complicate the design of optimal control solutions that are robust to uncertainties. In this paper, we formulate the strategic ATFM as a stochastic optimal control problem, and use scalable sampling based dynamic programming and Q-learning approaches to address high-dimensional uncertainties. Simulation studies demonstrate the effectiveness of these approaches. These approaches generally apply to CPSs that operate under high-dimensional environmental uncertainties.

Parallel dynamic programming and automata theory