Actor-critic algorithms for hierarchical Markov decision processes

In recent years, simulation optimization has attracted a great deal of attention because simulation can model the real systems in fidelity and capture complex dynamics. Among numerous simulation optimization algorithms, Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm is an attractive approach because of its simplicity and efficiency. Although SPSA has been applied in several problems, it does not converge for some. This research proposes Augmented Simultaneous Perturbation Stochastic Approximation (ASPSA) algorithm in which SPSA is augmented to include presearch, ordinal optimization, non-uniform gain, and line search. Performances of ASPSA are tested on complex discrete supply chain inventory optimization problems. The tests results show that ASPSA not only achieves speed up, but also improves solution quality and converges faster than SPSA. Experiments also show that ASPSA is comparable to Genetic Algorithms in solution quality (6% to 15% worse) but is much more efficient computationally (over 12x faster).Request access from your librarian to read this chapter's full text.

The most widely used method applied in the context of off-line dynamic demand calibration is Simultaneous Perturbation Stochastic Approximation (SPSA). In the research following the SPSA approach single origin-destination (O-D) demand components were mostly considered as calibration parameters. However, basic SPSA, especially in high dimensions, shows convergence issues, as proven by various authors. To overcome this drawback, some authors suggested modifications of basic SPSA to improve its performance. In this paper, we investigate various techniques and approaches to improve the SPSA performance, and overcome, or at least alleviate, its shortcomings. We concentrate our analysis mostly on SPSA coefficients and gradient control. The comparison of investigated settings is conducted on a real-world network. This establishes a path to identify critical aspects that influence the calibration process and suggests an optimal SPSA configuration for practice. The contribution of this paper is to provide a detailed analysis of the SPSA behavior in cases its configuration is subject to various modifications. The findings are primarily intended for the offline context. However, the insights can also be used for the selection of the most efficient SPSA configuration given time constraint, particularly suitable for on-line applications.

Considering the problem of the highly dependence on the accuracy model information of the industrial control process, an online PID parameter tuning method is proposed based on the simultaneous perturbation stochastic approximation (SPSA) algorithm and the traditional PID control only by the input and output data of the system. Interfering with each component of the PID controller parameters simultaneously, and the gradient information of the control performance loss function is estimated via the output value after disturbance by the global optimization method of stochastic approximation algorithm. The PID controller parameter is iterative updated according to the gradient estimation value. Finally, we applied this method to the control process of cement grinding particle size, and the simulation results by MATLAB shown it can lead a high control precision which proves the feasibility and effectiveness of this adaptive PID control method.

This paper describes the implementation of a self- optimizing embedded control scheme for an induction motor drive. The online design problem is formulated as a search problem and solved with a stochastic optimization algorithm. The objective function takes in account the tracking error, and is directly measured on the hardware bench. The online optimization is performed with the simultaneous perturbation stochastic approximation (SPSA) algorithms, which offer a very effective tradeoff between simplicity of implementation, speed of convergence and quality of the final solutions. Among the known SPSA algorithms considered in this paper, we also propose a novel variant inspired to the concept of elitism frequently used in evolutionary computation. To assess the relative performances of the various algorithms, the paper carries out a comprehensive analysis of a control scheme for an induction motor drive subject to time-varying load disturbances.

The purpose of this study is to determine the optimum design of arch longitudinal no prismatic single-cell section. The FE model of the Cetina open spandrel arch bridge was constructed using the ANSYS. In this present work the optimum design is carried out by taking total material volume of substructure of bridge as objective function. Substructure includes of column and reinforced concrete arch. Finally, the optimization technique is performed by Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm. It is concluded that SPSA can be effectively used in the shape optimization of the bridges. Keywords: Long Span, Longitudinal Section Optimization, Open Spandrel Bridge, SPSA Algorithm, Substructure

The simultaneous perturbation stochastic approximation (SPSA) algorithm has recently attracted considerable attention for optimization problems where it is difficult or impossible to obtain a direct gradient of the objective (say, loss) function. The approach is based on a highly efficient simultaneous perturbation approximation to the gradient based on loss function measurements. SPSA is based on picking a simultaneous perturbation (random) vector in a Monte Carlo fashion as part of generating the approximation to the gradient. This paper derives the optimal distribution for the Monte Carlo process. The objective is to minimize the mean square error of the estimate. We also consider maximization of the likelihood that the estimate be confined within a bounded symmetric region of the true parameter. The optimal distribution for the components of the simultaneous perturbation vector is found to be a symmetric Bernoulli in both cases. We end the paper with a numerical study related to the area of experiment design.

A stochastic approximation approach to spatio-temporal anchorage planning with multiple objectives

The minimization of a convex function defined over the grid Z/sup p/ is considered. A few relevant mathematical devices such as integer convexity, Markov chain Monte Carlo (MCMC) methods, including stochastic comparison (SC), and simultaneous perturbation stochastic approximation (SPSA) are summarized. A truncated fixed gain SPSA method is proposed and investigated in combination with devices borrowed from the MCMC literature. The main contribution of the paper is the development and testing a number of devices that may eventually improve the convergence properties of the algorithm, such as various truncation techniques, averaging and choices of acceptance probabilities. The basis for comparison of performances is accuracy vs. number of function evaluations. We present experimental evidence for the superiority of an SC method allowing moves in wrong directions with small probability, where the underlying method is an SPSA method using averaging and adaptive truncation.

Calibration and validation have long been a significant topic in traffic model development. In fact, when moving to dynamic traffic assignment (DTA) models, the need to dynamically update the demand and supply components creates a considerable burden on the existing calibration algorithms, often rendering them impractical. These calibration approaches are mostly restricted either due to non-linearity or increasing problem dimensionality. Simultaneous perturbation stochastic approximation (SPSA) has been proposed for the DTA model calibration, with encouraging results, for more than a decade. However, it often fails to converge reasonably with the increase in problem size and complexity. In this paper, we combine SPSA with principal components analysis (PCA) to form a new algorithm, we call, PC–SPSA. The PCA limits the search area of SPSA within the structural relationships captured from historical estimates in lower dimensions, reducing the problem size and complexity. We formulate the algorithm, demonstrate its operation, and explore its performance using an urban network of Vitoria, Spain. The practical issues that emerge from the scale of different variables and bounding their values are also analyzed through a sensitivity analysis using a non-linear synthetic function.

In this paper we consider a possible application of simultaneous perturbation stochastic approximation (SPSA) method to problems of control of educational processes. SPSA is an efficient tool for optimization problems and uncertain parameters estimation. What is more important, such SPSA procedures actually work under arbitrary external observation noise, while most other optimization methods work only with Gaussian noises. The illustrative example is also presented.

B034 PARALLEL WELL LOCATION OPTIMIZATION USING STOCHASTIC ALGORITHMS ON THE GRID COMPUTATIONAL FRAMEWORK 1 Hector Klie 1 Wolfgang Bangerth 1 2 Mary F. Wheeler 1 Manish Parashar 3 and Vincent Matossian 3 1 Center for Subsurface Modeling The University of Texas at Austin Austin TX. 2 Institute for Geophysics The University of Texas at Austin Austin TX. 3 The Applied Software Systems Laboratory Rutgers University Piscataway NJ. Abstract The determination of optimal well locations is a challenging problem in oil production since it depends on geological and fluid properties as well as on economic parameters. This work addresses the efficient

Daniel C. ChinThe Johns Hopkins University AppliedPhysics Laboratory11100 Johns Hopkins RoadLaurel, Maryland 20723-6099E-mail: daniel.chin@jhuapl.eduAbstract. Extracting a multivariate nonlinear physical model from a setof satellite images is considered as a multivariate nonlinear regressionproblem. Multiple local solutions often prevent gradient type algorithmsfrom obtaining global optimal solutions. A method of solving this problemis presented based on the simultaneous perturbation stochastic approxi-mation (SPSA) algorithm. The method is applied to a problem of estimat-ing the distribution of energetic ion populations in the magnetospherefrom global images of the magnetosphere. The approach uses multipleobjective functions: single image errors and the summation of squareimage errors. The algorithm is demonstrated on simulated energetic-neutral atom (ENA) images. Within a reasonable number of functionevaluations, the process converges and reconstructs the images with amean square error less than or equal to 0.1% of the original image. Also,the SPSA method is compared with results obtained from simulated an-nealing (SAN) in a single objective function setting. In the comparisonstudy, SPSA has a 3:1 advantage over SAN in both accuracy and effi-ciency measures.

Summary The main objective of the present work is to numerically determine how sensor information may aid in reducing the ill-posedness associated with permeability estimation via 4D seismic history matching. These sensors are assumed to provide timely information of pressures, concentrations and fluid velocities at given locations in a reliable fashion. This information is incorporated into an objective function that additionally includes production and seismic components that are mismatched between observed and predicted data. In order to efficiently perform large-scale permeability estimation, a coupled multilevel, stochastic and learning search methodology is proposed. At a given resolution level, the parameter space is globally explored and sampled by the simultaneous perturbation stochastic approximation (SPSA) algorithm. The estimation and sampling performed by SPSA is further enhanced by a neural learning engine that estimates sensitivities in the vicinity of the most promising optimal solutions. Preliminary results shed light on future research avenues for optimizing the frequency and localization of 4-D seismic surveys when sensor data is

Joint Calibration for DTA Model Using Islands-GA and PC-SPSA

Adaptive complementary filter using fuzzy logic and simultaneous perturbation stochastic approximation algorithm