Optimal Exercise Patterns and Valuation of Employee Stock Options: An Approximate Dynamic Programming Approach

Many public funding agencies adopt the one-stage call-based mode to select R&D projects. The funding decision-making in this mode is characterized by two primary features: 1) it is static, meaning that decision-makers solely concentrate on projects submitted within the current call, and 2) it is decentralized, indicating that funding decisions are made independently across diverse sectors/disciplines. As a result, the administrative burden on funding agencies is reduced. However, due to the budget constraint, the funding agencies may fail to fund better projects that emerge in future calls or appear in other sectors/disciplines. Therefore, we investigate whether a better project portfolio can be achieved by selecting public R&D projects in a dynamic and centralized manner. We formulate a Markov decision process (MDP) model for the dynamic centralized public R&D project portfolio selection problem. We develop an approximate dynamic programming (ADP) approach that combines learning-based Monte Carlo simulation and a two-stage rollout algorithm to efficiently solve the MDP model. Based on extensive computational experiments, we compare our ADP approach with a threshold heuristic that is representative of the static and decentralized funding mode, as well as two baseline algorithms. The results indicate that our ADP approach is effective, and it is beneficial to adopt the dynamic centralized decision-making mode in public R&D project portfolio selection. We also investigate our ADP approach utilizing a case study based on real-world data from the National Natural Science Foundation of China. Our ADP approach can be integrated into the decision support systems of funding agencies, enabling automated, dynamic, and centralized selection of public R&D projects.

This paper proposes an Approximate Dynamic Programming (ADP) approach for obtaining the optimal real-time operation strategy of microgrid with power-to-hydrogen (P2H) device. Firstly, the real-time operation model of microgrid with P2H is established, then a piece-wise linear function based approximate dynamic programming is proposed to solve the operation model. With the direction of the slope update and training policies under various training scenarios, the proposed ADP approach can obtain the approximate optimal real-time operation strategies to cope with uncertainties. Case studies verify the superiority of the proposed ADP approach compared with Model Predictive Control approach (MPC).

The IASC recently recommended that employee compensation in the form of stock options be measured at the ‘fair value’ based on an option pricing model and the value should be recognized in financial statements. This follows adoption of SFAS No. 123 in the United States, which requires firms to estimate the value of employee stock options using either a Black‐Scholes or binomial model. Most US firms used the B‐S model for their 1996 financial statements. This study assumes that option life follows a Gamma distribution, allowing the variance of option life to be separate from its expected life. The results indicate the adjusted Black‐Scholes model could overvalue employee stock options on the grant date by as much as 72 percent for nondividend paying firms and by as much as 84 percent for dividend paying firms. The results further demonstrate the sensitivity of ESO values to the volatility of the expected option life, a parameter that the B‐S model or a Poisson process cannot accommodate. The variability of option life has an especially big impact on ESO value for firms whose ESOs have a relatively short life (5 years, for example) and high employee turnover. For such firms, the results indicate a binomial option pricing model is more appropriate for estimating ESO value than the B‐S type model.

Admission Control in UMTS Networks based on Approximate Dynamic Programming

Employee Incentive Stock Options: To Be Expensed or Not to Be, that Is Not the Question

We study the problem of hedging early exercise (American) options with respect to exponential utility within a general incomplete market model. This leads us to construct a duality formula involving relative entropy minimization and optimal stopping. We further consider claims with multiple exercises, and static-dynamic hedges of American claims with other European and American options. The problem is important for accurate valuation of employee stock options (ESOs), and we demonstrate this in a standard diffusion model. We find that incorporating static hedges with market-traded options induces the holder to delay exercises and increases the ESO cost to the firm.

In this paper, a novel adaptive optimal control algorithm based on approximate dynamic programming (ADP) approach is proposed for integrated active front steering (AFS) and direct yaw moment control (DYC). The corrective yaw moment and active steering angle are generated online without knowing system dynamics, which is realised by using a neural network (NN) identifier to identify the unknown system dynamics and a critic NN to calculate the optimal control action, respectively. Control commands are executed via active steering angle on front wheels and proper brake torque distribution on the effective wheels. Computer simulations under three different driving manoeuvres, i.e., lane change manoeuvre, step steer manoeuvre and sine with dwell manoeuvre, are carried out to evaluate the proposed control method. Simulation results show that the proposed ADP-based control method demonstrates improved tracking performance in terms of enhancing vehicle handling and stability performance when encountering the varying longitudinal velocity, the uncertain cornering stiffness and the different road/tyre friction coefficients. Model-free and self-adaptive properties of the proposed method provide a new solution to vehicle stability controller design instead of the commonly used model-based methods.

Abstract: Online resource management in cloud computing must respond to rapidly changing demand and system conditions, often exceeding the adaptability of static or periodically recomputed optimisation plans. This study formulates cloud resource management as a Markov decision process (MDP) and investigates deep reinforcement learning (DRL) as an approximate dynamic programming (ADP) approach for deriving adaptive control policies. The system state encompasses workload, capacity utilisation, and performance indicators, while actions govern virtual machine placement, scaling, and server power-state transitions. Reward functions combine energy and operating costs with penalties associated with service-level agreement violations. DRL algorithms, including deep Q-networks, advantage actor–critic, and proximal policy optimization, are implemented in PyTorch and trained through interaction with a CloudSim-based simulation environment calibrated to realistic load and power profiles. The learned policies are evaluated against deterministic and scenario-based stochastic optimisation baselines using metrics such as average cost, energy consumption, SLA violations, and robustness to non-stationary demand. Results show that DRL policies can approach or surpass the performance of stochastic optimisation solutions in highly dynamic scenarios, while maintaining feasible control actions. The study demonstrates how DRL can be positioned within an OR/ADP framework and highlights conditions under which learning-based methods offer significant advantages for online cloud control. Keywords: deep reinforcement learning, approximate dynamic programming, cloud computing, Markov decision processes, online resource management, energy-aware control, PyTorch, simulation

Dynamic programming is a very useful tool in solving optimization and optimal control problems. Here, the Approximate Dynamic Programming (ADP) and the notion of neural networks based predictive control are combined with a model-free control method based on SPSA (Simultaneous perturbation stochastic approximation), and a novel ADP based model-free predictive control strategy for nonlinear systems is proposed. Dynamic programming is used to adjust the control parameters in the novel model-free control method and the notion of predictive control is introduced to modify the whole control structure. Finally, the proposed ADP based model-free predictive control strategy is applied to solve nonlinear tracking problems and the effectiveness of this novel control method is fully illustrated though simulation tests on two typical nonlinear systems.

An approximate dynamic programming approach for solving an air combat maneuvering problem

This article examines a major maritime disaster scenario in which the objective is to maximize the number of survivors. To achieve this objective, we seek to optimize the decision policy to load individuals onto a helicopter for transport from an evacuation site to a forward operating base. Our contributions are twofold. First, we formulate the loading problem as a finite-horizon Markov Decision Process. Second, since the curse of dimensionality renders exact methods not applicable, we use an Approximate Dynamic Programming (ADP) approach to explore the efficacy of these methods to the problem. We generate three ADP policies, each using a value function approximation based on a unique post-decision state variable and a lookup table representation. We compare these ADP policies to a random policy, a myopic policy, and Policy Function Approximation (PFA) which evacuates survivors in order of triage category starting with the most critically ill. Our results show that an ADP-generated policy which prioritizes the evacuation of healthy individuals improved performance by 34 ± 5% versus the random policy, 15 ± 3% versus the myopic policy, and 42 ± 3% versus the PFA policy.

Unmanned aircraft systems can perform some more dangerous and difficult missions which manned aircraft systems cannot perform. For tasks with high complexity, such as air combat, maneuvering decision mechanism is required to sense the combat environment and make the optimal strategy in real time. This paper formulates one-to-one air combat maneuvering problem in 3D environment, and proposes an approximate dynamic programming approach to make optimal maneuvering decisions automatically. The aircraft searches for combat strategies based-on Reinforcement Leaning, while sensing the environment, taking available maneuvering actions and receiving feedback reward signals. To solve the problem of dimensional explosion in the air combat, the proposed method is implemented through feature selection, trajectory sampling, function approximation and Bellman backup operation in the air combat simulation environment. This approximate dynamic programming approach provides a fast response to rapid changing tactical situations, and learns effective strategies to fight against the opponent aircraft.

This report formulates a dual-source model that considers artificial blood along with natural blood as sources of blood platelet supply. The model is formulated for a single hospital by considering deterministic lead time while ordering, uncertainty in supply, and demand. A newsvendor model is used to find the optimal order quantity for each day and approximate dynamic programming (ADP) is used to deal with uncertainty over a year. Three different step size rules are being studied and compared in terms of average value function approximation (VFA) per timestep, with two being deterministic step size and the other being an optimal stochastic step size to deal with non-stationary data. The project aims is to select a policy that can reduce shortages, outages, and maximize the rewards for the developed model. Eleven policies are being investigated while introducing artificial blood platelet rewards. Numerical analysis is implemented by considering varied demand and donations for different acceptance percentages and inventory levels of artificial blood under each policy. Another analysis is done to show how shortage and outage values vary for a single source supply vs a dual source (0% acceptance vs 50% acceptance) and are computed for three policies. Computed results show that first in first out (FIFO) is the best policy with the least shortage and outage for various acceptance rates, for varied demand and donations. As the acceptance rate and the inventory level for artificial blood are increased, for 10% a shortage percentage of 5.8% along with 5.9% of outage for natural and 8.8% for artificial is observed. When increased to 30%, 40%, we see an increase in the outage of natural platelets from5.9% to 6.1% along with shortage levels slightly falling. For 50% shortage values falls below 5.2%, outage values fall below 5.7% with outage of artificial from 9% to 8.6%. For the second analysis 0% vs 50%, considering a secondary source can help reduces shortage from 5.8% to 5.1% and outage from 6.3% to 5.7%.

With globalization and rapid technological-economic development accelerating the market dynamics, consumers' demand is becoming more volatile and diverse. In this situation, capacity adjustment as an operational strategic decision plays a major role to ensure supply chain responsiveness while maintaining costs at a reasonable norm. This study contributes to the literature by developing computationally efficient approximate dynamic programming approaches for production capacity planning considering uncertainties and demand interdependence in a multi-factory multi-product supply chain setting. For this purpose, the k-Nearest-Neighbor-based Approximate Dynamic Programming and the Rolling-Horizon-based Approximate Dynamic Programming are developed to enable real-time decision support while ensuring the robustness of the outcomes in stochastic decision environments. Given the market volatilities in the Thin Film Transistor-Liquid Crystal Display industry, a real case from this sector is investigated to evaluate the applicability of the developed approach and provide insights for other industry situations. The developed method is less complex to implement, and numerical experiments showed that it is also computationally more efficient compared to Stochastic Dynamic Programming.

Sensory data have becoming widely available in large volume and variety due to the increasing presence and adoption of the Internet of Things. Such data can be tremendously useful if they are processed properly in a timely fashion. They could play a key role in the coordination of industrial production. It is thus desirable to explore an effective and efficient scheme to support data tracking and monitoring. This paper intends to propose a novel automatic learning scheme to improve the tracking efficiency while maintaining or improving the data tracking accuracy. A core strategy in the proposed scheme is the design of Laguerre neural network (LaNN)-based approximate dynamic programming (ADP). As a traditional optimal learning strategy, ADP is a popular approach for data processing. The action neural network (NN) and the critic NN as two important components in ADP have big impact on the performance of ADP. In this paper, a LaNN is employed as the implementation of the action NN in ADP considering Laguerre polynomials' approximation capability. In addition, this LaNN-based ADP is integrated into an online parameter-tuning framework to optimize those parameters of characteristic model that is used to trace the data in the tracking control system. Meanwhile, this article provides an associated Lyapunov convergence analysis to guarantee a uniformly ultimately boundedness property for tracking errors in the proposed approach. Furthermore, the proposed LaNN-based ADP optimal online parameter-tuning scheme is validated using a temperature dynamic tracking control task. The simulation results demonstrate that the scheme has satisfactory learning performance over time.