An approximate dynamic programming approach to tackling mass evacuation operations

Many public funding agencies adopt the one-stage call-based mode to select R&D projects. The funding decision-making in this mode is characterized by two primary features: 1) it is static, meaning that decision-makers solely concentrate on projects submitted within the current call, and 2) it is decentralized, indicating that funding decisions are made independently across diverse sectors/disciplines. As a result, the administrative burden on funding agencies is reduced. However, due to the budget constraint, the funding agencies may fail to fund better projects that emerge in future calls or appear in other sectors/disciplines. Therefore, we investigate whether a better project portfolio can be achieved by selecting public R&D projects in a dynamic and centralized manner. We formulate a Markov decision process (MDP) model for the dynamic centralized public R&D project portfolio selection problem. We develop an approximate dynamic programming (ADP) approach that combines learning-based Monte Carlo simulation and a two-stage rollout algorithm to efficiently solve the MDP model. Based on extensive computational experiments, we compare our ADP approach with a threshold heuristic that is representative of the static and decentralized funding mode, as well as two baseline algorithms. The results indicate that our ADP approach is effective, and it is beneficial to adopt the dynamic centralized decision-making mode in public R&D project portfolio selection. We also investigate our ADP approach utilizing a case study based on real-world data from the National Natural Science Foundation of China. Our ADP approach can be integrated into the decision support systems of funding agencies, enabling automated, dynamic, and centralized selection of public R&D projects.

An approximate dynamic programming approach for solving an air combat maneuvering problem

This paper proposes an Approximate Dynamic Programming (ADP) approach for obtaining the optimal real-time operation strategy of microgrid with power-to-hydrogen (P2H) device. Firstly, the real-time operation model of microgrid with P2H is established, then a piece-wise linear function based approximate dynamic programming is proposed to solve the operation model. With the direction of the slope update and training policies under various training scenarios, the proposed ADP approach can obtain the approximate optimal real-time operation strategies to cope with uncertainties. Case studies verify the superiority of the proposed ADP approach compared with Model Predictive Control approach (MPC).

Unmanned aircraft systems can perform some more dangerous and difficult missions which manned aircraft systems cannot perform. For tasks with high complexity, such as air combat, maneuvering decision mechanism is required to sense the combat environment and make the optimal strategy in real time. This paper formulates one-to-one air combat maneuvering problem in 3D environment, and proposes an approximate dynamic programming approach to make optimal maneuvering decisions automatically. The aircraft searches for combat strategies based-on Reinforcement Leaning, while sensing the environment, taking available maneuvering actions and receiving feedback reward signals. To solve the problem of dimensional explosion in the air combat, the proposed method is implemented through feature selection, trajectory sampling, function approximation and Bellman backup operation in the air combat simulation environment. This approximate dynamic programming approach provides a fast response to rapid changing tactical situations, and learns effective strategies to fight against the opponent aircraft.

Optimal Exercise Patterns and Valuation of Employee Stock Options: An Approximate Dynamic Programming Approach

Admission Control in UMTS Networks based on Approximate Dynamic Programming

Simple or proxy production models are potentially very useful and tractable because they are computationally attractive whilst still providing insight to the decision at hand. Useful and tractable models are required for supporting high-quality decisions in uncertain, complex, and computationally demanding contexts. A key decision for development planning is: what is the optimal time to initiate an Improved-Oil-Recovery (IOR) process. We aim to illustrate the implementation and application of a useful and tractable approach for the analysis of the optimal IOR switch time using a two-factor production model and Least-Squares Monte Carlo (LSM) simulation. The two-factor production model contains only two parameters for each recovery phases. One parameter describes how much recovery efficiency a recovery mechanism can ultimately achieve whilst the other describes how fast the recovery efficiency increases. The simplicity of the model makes it computationally attractive. The LSM algorithm is an approximate dynamic programming approach, which allows for learning over time. It provides a near-optimal solution for the IOR switch time problem. The Value-Of-Information (VOI) framework—a powerful decision-analysis tool—provides an estimate of the value of learning. Closed-Loop Reservoir Management (CLRM) is considered to be a state-of-the-art approach to solving for the optimal IOR switch time. However, this approach can produce a suboptimal solution as the CLRM approach considers only uncertainties and actions reflecting currently available information but not those uncertainties and actions arising from future information. The dynamic programming approach used here considers both the impact of the information obtained before a decision is made and the impact of the information that might be obtained to support future decisions. We conclude that a dynamic programming approach, such as the LSM algorithm, can significantly improve both the timing and value of decisions, leading to a significant increase in a field's economic performance. Furthermore, the two-factor model combined with the LSM algorithm is tractable and provides useful insight into the IOR switch time problem. The novelties provided by this work are: developing and illustrating the structure of the IOR switch time problem in a decision tree, demonstrating and discussing the suboptimality of the CLRM solution, developing and illustrating the detailed steps of applying the LSM algorithm for the IOR switch time decision, and implementing the two-factor model combined with the LSM algorithm for analyzing the optimal IOR switch time.

There is a growing interest in using electric vehicles (EVs) and drones for many applications. However, battery-oriented issues, including range anxiety and battery degradation, impede adoption. Battery swap stations are one alternative to reduce these concerns that allow the swap of depleted for full batteries in minutes. We consider the problem of deriving actions at a battery swap station when explicitly considering the uncertain arrival of swap demand, battery degradation, and replacement. We model the operations at a battery swap station using a finite horizon Markov decision process model for the stochastic scheduling, allocation, and inventory replenishment problem (SAIRP), which determines when and how many batteries are charged, discharged, and replaced over time. We present theoretical proofs for the monotonicity of the value function and monotone structure of an optimal policy for special SAIRP cases. Because of the curses of dimensionality, we develop a new monotone approximate dynamic programming (ADP) method, which intelligently initializes a value function approximation using regression. In computational tests, we demonstrate the superior performance of the new regression-based monotone ADP method compared with exact methods and other monotone ADP methods. Furthermore, with the tests, we deduce policy insights for drone swap stations.

This report formulates a dual-source model that considers artificial blood along with natural blood as sources of blood platelet supply. The model is formulated for a single hospital by considering deterministic lead time while ordering, uncertainty in supply, and demand. A newsvendor model is used to find the optimal order quantity for each day and approximate dynamic programming (ADP) is used to deal with uncertainty over a year. Three different step size rules are being studied and compared in terms of average value function approximation (VFA) per timestep, with two being deterministic step size and the other being an optimal stochastic step size to deal with non-stationary data. The project aims is to select a policy that can reduce shortages, outages, and maximize the rewards for the developed model. Eleven policies are being investigated while introducing artificial blood platelet rewards. Numerical analysis is implemented by considering varied demand and donations for different acceptance percentages and inventory levels of artificial blood under each policy. Another analysis is done to show how shortage and outage values vary for a single source supply vs a dual source (0% acceptance vs 50% acceptance) and are computed for three policies. Computed results show that first in first out (FIFO) is the best policy with the least shortage and outage for various acceptance rates, for varied demand and donations. As the acceptance rate and the inventory level for artificial blood are increased, for 10% a shortage percentage of 5.8% along with 5.9% of outage for natural and 8.8% for artificial is observed. When increased to 30%, 40%, we see an increase in the outage of natural platelets from5.9% to 6.1% along with shortage levels slightly falling. For 50% shortage values falls below 5.2%, outage values fall below 5.7% with outage of artificial from 9% to 8.6%. For the second analysis 0% vs 50%, considering a secondary source can help reduces shortage from 5.8% to 5.1% and outage from 6.3% to 5.7%.

Approximate dynamic programming for an energy-efficient parallel machine scheduling problem

In this paper, a novel adaptive optimal control algorithm based on approximate dynamic programming (ADP) approach is proposed for integrated active front steering (AFS) and direct yaw moment control (DYC). The corrective yaw moment and active steering angle are generated online without knowing system dynamics, which is realised by using a neural network (NN) identifier to identify the unknown system dynamics and a critic NN to calculate the optimal control action, respectively. Control commands are executed via active steering angle on front wheels and proper brake torque distribution on the effective wheels. Computer simulations under three different driving manoeuvres, i.e., lane change manoeuvre, step steer manoeuvre and sine with dwell manoeuvre, are carried out to evaluate the proposed control method. Simulation results show that the proposed ADP-based control method demonstrates improved tracking performance in terms of enhancing vehicle handling and stability performance when encountering the varying longitudinal velocity, the uncertain cornering stiffness and the different road/tyre friction coefficients. Model-free and self-adaptive properties of the proposed method provide a new solution to vehicle stability controller design instead of the commonly used model-based methods.

Abstract: Online resource management in cloud computing must respond to rapidly changing demand and system conditions, often exceeding the adaptability of static or periodically recomputed optimisation plans. This study formulates cloud resource management as a Markov decision process (MDP) and investigates deep reinforcement learning (DRL) as an approximate dynamic programming (ADP) approach for deriving adaptive control policies. The system state encompasses workload, capacity utilisation, and performance indicators, while actions govern virtual machine placement, scaling, and server power-state transitions. Reward functions combine energy and operating costs with penalties associated with service-level agreement violations. DRL algorithms, including deep Q-networks, advantage actor–critic, and proximal policy optimization, are implemented in PyTorch and trained through interaction with a CloudSim-based simulation environment calibrated to realistic load and power profiles. The learned policies are evaluated against deterministic and scenario-based stochastic optimisation baselines using metrics such as average cost, energy consumption, SLA violations, and robustness to non-stationary demand. Results show that DRL policies can approach or surpass the performance of stochastic optimisation solutions in highly dynamic scenarios, while maintaining feasible control actions. The study demonstrates how DRL can be positioned within an OR/ADP framework and highlights conditions under which learning-based methods offer significant advantages for online cloud control. Keywords: deep reinforcement learning, approximate dynamic programming, cloud computing, Markov decision processes, online resource management, energy-aware control, PyTorch, simulation

Using the Approximate Dynamic Programming (ADP) approach, nonlinear optimal control problems can be solved by using a dual neural network architecture called the Adaptive Critic (AC). A Single Network Adaptive Critic (SNAC) architecture has been shown to have much faster convergence than AC. Even though the SNAC offers a systematic optimal control design method for optimal state regulation for a class of nonlinear control systems, it lacks the generality necessary for command tracking as it is practically difficult to anticipate a proper training domain in state space when the command sequence is not known a priori. Dynamic Inversion (DI), on the other hand, is a very popular nonlinear control design method for both output regulation and command tracking that offers a closed form expression for control. However, it lacks the interpretation and advantages of optimal control design. In this paper we propose a systematic technique for selecting the DI gains with the help of a pre-synthesized SNAC network in order that the resulting DI solution closely mimics the SNAC solution for output regulation and can be extended to command tracking problems. Moreover, it retains all the benefits of DI design. The potential of this new methodology of near-optimal dynamic inversion design is demonstrated in this paper by considering a benchmark nonlinear system.

Predictive Control with Learning-Based Terminal Costs Using Approximate Value Iteration

Dynamic pricing and reservation for intelligent urban parking management