An Efficient Simulation-Based Policy Improvement with Optimal Computing Budget Allocation Based on Accumulated Samples

Many of simulation based learning algorithms have been developed to obtain near optimal policies for Markov decision processes (MDPs) with large state space. However, most of them are for unichain problems. In view that some applications involve multichain processes and it is NP-hard to determine whether a MDP is unichain or not, it is desirable to obtain an algorithm that is applicable to multichain problems as well. This paper presents a rollout algorithm for multichain MDPs with average cost. Preliminary analysis of the estimation error and parameter settings are provided based on the problem structures, i.e., mixing time of transition matrix. Ordinal optimization and Optimal Computing Budget Allocation are also suggested to improve the efficiency of the algorithm.

Minimizing Risk Models in Markov Decision Processes with Policies Depending on Target Values

Simulation models of emergency departments (EDs) are often built based on incomplete data, for example, missing arrival times or service-time durations. The difficulty in collecting reliable and complete data can subsequently lead to invalid simulation results. To tackle this problem, we propose a simulation and optimisation method to characterise the unavailable durations of service times. Since many services in an ED are sequential and dependent on each other, this paper considers these multiple process steps cooperatively. We first use lognormal distributions to characterise the key service durations. Then we propose a new meta-heuristic approach, which combines an Improved Adaptive Genetic Algorithm (AGA) and Simulated Annealing (SA), IAGASA, to search for the optimal set of service-time distribution parameters. To address the difficulties of applying IAGASA when noise is involved in the performance measures and improve the simulation efficiency, we jointly apply IAGASA and Optimal Computing Budget Allocation (OCBA) technology. OCBA minimises the total simulation cost for achieving a desired level of probability of correctly selecting the best set of distribution parameters, which improves the search efficiency significantly. The experimental results indicate that our proposed method can find accurate estimates of service-time distribution parameters within a relatively short time.

Stochastic point location (SPL) is to search for a target point on the line in stochastic environment. An SPL solver can be described as a Learning Machine (LM) attempting to locate a target point on a line. By using the prompts from stochastic environment, possibly erroneous, the LM moves along the line yielding updated estimates to approximate the target point. This paper proposes an SPL algorithm based on Optimal Computing Budget Allocation (OCBA), named as SPL-OCBA, which employs OCBA and the historical sample information to guide to the location of a target point in stationary and stochastic environment. The proposed algorithm partitions or combines the subintervals of the target line adaptively. Then, OCBA considers such subintervals as its designs and allocates the sample budget for them based on the historical data, thereby resulting in a new method. Extensive experiments show that the newly proposed algorithm is significantly more efficient than the existing ones.

In this paper, a simulation-based optimization approach, named NHOCBA, for a typical resource allocation problem is presented. The hybrid algorithm based on neighborhood algorithm is applied to explore toward optimal direction in design space. For increasing efficiency, an optimal computing budget allocation (OCBA) is adopted to compute the optimal number of replications and to provide reliable evaluation of variance. In addition, we deal with the resource allocation problem which exist multiple global optima in design space. Therefore, a trim procedure which prevents to allocate extra numbers of replication to local optima has been proposed to enhance efficiency. Then, we use confidence interval at the end of algorithm procedure to find an optimal set instead of an optimal solution from design space. Finally, we compare the NHOCBA with different algorithms by experimentation study which shows that NHOCBA approach can perform better than the other algorithms under certain conditions.

Particle Swarm Optimization (PSO) is a population-based algorithm designed to find good solutions to optimization problems. Its characteristics have encouraged its adoption to tackle a variety of problems in different fields. However, when such problems are subject to noise, the performance of PSO suffers an immediate deterioration which demands the incorporation of noise handling mechanisms. One such mechanism comprises resampling methods, which re-evaluate the solutions multiple times in order to estimate their true objective values. The state-of-the-art integration with which the best results have been obtained utilizes the resampling method named Optimal Computing Budget Allocation (OCBA). This resampling method starts by estimating the objective values of all the solutions via Equal Resampling (ER), and then sequentially allocating further re-evaluations to the estimated best solutions. However, after having a first estimate via ER, we question the importance of the additional efforts to correctly select the true best solutions when a good-enough and accurate one can be selected. In this paper, we propose a new PSO algorithm based on ER in which the additional evaluations are allocated at once to the estimated best solutions, thus skipping the complexity of using OCBA. Experiments on 20 large-scale benchmark functions subject to different levels of noise show that the proposed algorithm produces similar results to PSO with OCBA in most cases.

For the design of call admission control (CAC) policy in wireless networks, how to decrease the average system cost is a key issue. In this paper, we study the optimal admission policy for minimizing the system cost. By modeling the admission control problem into a Markov decision process (MDP) and analyzing the corresponding value function, we obtain some monotonicity properties of the optimal policy. These properties suggest that the optimal admission control policy for the bandwidth asymmetry wireless networks should have a threshold structure and the threshold specified for a class of calls may change with the system state. Due to the prohibitively high complexity for computing the thresholds in a system with large state space, we propose a heuristic CAC policy called call-rate- based dynamic threshold (CRDT) policy to approximate the theoretical optimal policy based on the insights we obtain from the modeling and the analytical study on the properties of the optimal policy. The CRDT policy is efficient and can be easily implemented. The numerical results show that the performance of average system cost of the proposed CRDT policy is close to that of the optimal policy from the MDP model and is better than that of some known existing CAC schemes, including those performing well in bandwidth asymmetry wireless networks.

We consider the optimal computing budget allocation (OCBA) problem where the simulated designs are correlated. The exact optimal allocation is presented for two designs, and an approximation proposed for more than two designs. Numerical experiments for several examples compare the approximation with some other allocation procedures. Results on convergence rates and the non-Gaussian setting are also discussed.

Although particle swarm optimization (PSO) is a powerful evolutionary algorithm for solving nonlinear optimization problems in deterministic environments, many practical problems have some stochastic noise. The optimal computing budget allocation (OCBA) has been integrated into PSO in various ways to cope with this. The OCBA can mitigate the effect of noise on PSO by selecting the best solution under a limited evaluation budget. Recently, with the increasing complexity of PSO applications, the evaluation costs are also increasing rapidly, which has sparked the need for more efficient PSO in stochastic environments. This article proposes a simple yet effective adjustment to the integration of OCBA to further improve the efficiency of PSO. The proposed adjustment allows OCBA to expand its search space to find the global best position more correctly such that the entire swarm can move on a better direction under stochastic noise. The experimental results on various benchmarks demonstrate the improved performance of PSO by the proposed adjustment under a limited budget compared with the latest studies. In addition, the results regarding fighters' evasion flight optimization emphasize the practical need for the proposed adjustment.

We investigate the management of a merchant wind energy farm co‐located with a grid‐level storage facility and connected to a market through a transmission line. We formulate this problem as a Markov decision process (MDP) with stochastic wind speed and electricity prices. Consistent with most deregulated electricity markets, our model allows these prices to be negative. As this feature makes it difficult to characterize any optimal policy of our MDP, we show the optimality of a stage‐ and partial‐state‐dependent‐threshold policy when prices can only be positive. We extend this structure when prices can also be negative to develop heuristic one (H1) that approximately solves a stochastic dynamic program. We then simplify H1 to obtain heuristic two (H2) that relies on a price‐dependent‐threshold policy and derivative‐free deterministic optimization embedded within a Monte Carlo simulation of the random processes of our MDP. We conduct an extensive and data‐calibrated numerical study to assess the performance of these heuristics and variants of known ones against the optimal policy, as well as to quantify the effect of negative prices on the value added by and environmental benefit of storage. We find that (i) H1 computes an optimal policy and on average is about 17 times faster to execute than directly obtaining an optimal policy; (ii) H2 has a near optimal policy (with a 2.86% average optimality gap), exhibits a two orders of magnitude average speed advantage over H1, and outperforms the remaining considered heuristics; (iii) storage brings in more value but its environmental benefit falls as negative electricity prices occur more frequently in our model.

Given a training environment which follows Markov decision process for a specific task, a deep reinforcement learning (DRL) agent is able to find possible optimal policies which map states of the environment to appropriate actions by repeatedly trying various actions to maximize training rewards. However, the learned policies cannot be reused directly in the training process for other new tasks resulting wasted precious time and resources. To solve this problem, we propose a DRL-based method for training an agent capable of selecting the appropriate policy for current state of the environment from a set of previously trained optimal policies for a given task which can be decomposed into other sub tasks. We implement our proposed method to a person-following robot task training that can be broken down into three sub tasks, namely: navigation, left attending, and right attending. Using the proposed method, the previously learned optimal navigation policy obtained from our previous work is integrated to attending policies which are trained in this study. We also introduce the use of weight-scheduled action smoothing which is able to stabilize actions generated by the agent in the attending task training. Our experiment results show that the proposed method is able to integrate all sub policies using the action smoothing method even though the navigation and the attending policies have dissimilar input structures, unalike output ranges, and are trained in different ways. Moreover, our proposed method shows better results compared to training from scratch and training using transfer learning strategy.

Stochastic optimal control of agrochemical pollutant loads in reservoirs for irrigation

Content caching provides a rare trade-off opportunity between low-cost memory and energy consumption, yet finding the optimal causal policy with low computational complexity remains a challenge. In this paper, we formulate the problem of joint pushing and caching at end users in a Markov decision process (MDP) framework with the aim of minimizing energy cost under causal knowledge of the user request delay information and strict delay constraint. We put forth a novel approach to decouple the influence of buffer occupancy and user request, allowing one to equivalently run value iteration in a much smaller state space. Furthermore, by exploiting a structural property we discovered as generalized monotonicity , we design a computationally efficient non-iterative algorithm, fast assignment of state transition, to map the optimal policy from the smaller state space back to the complete state space. This, in turn, significantly simplifies the search for optimal policy. The results attain close performance in comparison with theoretical bounds from non-causal policies, while benefiting from better time efficiency than the unadapted MDP solutions.

This paper presents a hierarchical approach for controlling the spread of an epidemic disease. The approach consists of a three-layer architecture where a set of two-layer multiple social networks is governed by a (third) top-layer consisting of an optimal control policy. Each of the two-layer social networks is modeled by a microscopic Markov chain. On top of all the two-layer networks is an optimal control policy that has been developed by using an underlying Markov Decision Process (MDP) model. Mathematical models pertaining to the top-level MDP as well as two-layer microscopic Markov chains have been presented. Practical implementation methodology using the proposed models has also been discussed along with a numerical example. The results in the numerical example illustrate the control of an epidemic using the optimal policy. Directions for further research and characterization of the optimal policy have also been discussed with the help of the same numerical example.Article HighlightsAn optimal approach for controlling the spread of an epidemic infection.The approach is able to model the uncertainties involved in the problem.The approach is able to cater for the underlying social network.

Dynamic selling of quality-graded products under demand uncertainties