Revisiting the Cross Entropy Method with Applications in Stochastic Global Optimization and Reinforcement Learning

In order to deal with the sequential decision problems with large or continuous state spaces, feature representation and function approximation have been a major research topic in reinforcement learning (RL). In this paper, a clustering-based graph Laplacian framework is presented for feature representation and value function approximation (VFA) in RL. By making use of clustering-based techniques, that is, K-means clustering or fuzzy C-means clustering, a graph Laplacian is constructed by subsampling in Markov decision processes (MDPs) with continuous state spaces. The basis functions for VFA can be automatically generated from spectral analysis of the graph Laplacian. The clustering-based graph Laplacian is integrated with a class of approximation policy iteration algorithms called representation policy iteration (RPI) for RL in MDPs with continuous state spaces. Simulation and experimental results show that, compared with previous RPI methods, the proposed approach needs fewer sample points to compute an efficient set of basis functions and the learning control performance can be improved for a variety of parameter settings.

New optimisation algorithm for planar antenna array synthesis

In order to increase productivity and reduce energy consumption of steelmaking-continuous casting (SCC) production process, especially with complicated technological routes, the cross entropy (CE) method was adopted to optimize the SCC production scheduling (SCCPS) problem. Based on the CE method, a matrix encoding scheme was proposed and a backward decoding method was used to generate a reasonable schedule. To describe the distribution of the solution space, a probability distribution model was built and used to generate individuals. In addition, the probability updating mechanism of the probability distribution model was proposed which helps to find the optimal individual gradually. Because of the poor stability and premature convergence of the standard cross entropy (SCE) algorithm, the improved cross entropy (ICE) algorithm was proposed with the following improvements: individual generation mechanism combined with heuristic rules, retention mechanism of the optimal individual, local search mechanism and dynamic parameters of the algorithm. Simulation experiments validate that the CE method is effective in solving the SCCPS problem with complicated technological routes and the ICE algorithm proposed has superior performance to the SCE algorithm and the genetic algorithm (GA).

This paper discusses an alternative approach to parameter optimization of prototype-based learning algorithms that aim to minimize an objective function based on gradient search. The proposed approach is a stochastic optimization method called the Cross Entropy (CE) method. The CE method is used to tackle the initialization sensitiveness problem associated with the original generalized Learning Vector Quantization (GLVQ) algorithm and its variants and to locate the globally optimal solutions. We will focus our study on a variant which deals with a weighted norm instead of the Euclidean norm in order to select the most relevant features. The results in this paper indicate that the CE method can successfully be applied to this kind of problems and efficiently generate high quality solutions. Also, highly competitive numerical results on real world data sets are reported.KeywordsDirichlet DistributionHigh Quality SolutionLearn Vector QuantizationCross EntropyNatural Exponential Function FamilyThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Project scheduling is essential to be planned before activities. Standard methods of project scheduling based on precedence constraints scheduling events and the assumption that resources are not unlimited capacity. But in fact the project activity has resource limits. The main objective is to minimize the total duration of the project with precedence constraints and resource constraints for all project activities. Many optimization methods are used to improve the quality of scheduling and speed up the calculation time. This paper proposed the use of Cross Entropy (CE) method to solve resource constraints project scheduling problem, as well as comparing the advantages and disadvantages between the CE methods with Differential Evolution (DE) method. The purpose CE method consist of four critical steps including generating a sample of random solution, calculating the performance according to the specific fitness function, selecting elite sample and finally updating the previous parameters to get a better sample in the next iteration. To speed up the computation time, in this study decrease the number of samples for each iteration in Cross Entropy algorithm. Numerical experiments with several data sets from the Project Scheduling library (PSPLIB) showed that CE can provide the optimal total duration of the project same as the DE with calculation time is faster than DE.

A reconsideration on the resolution of phase stability analysis using stochastic global optimization methods: Proposal of a reliable set of benchmark problems

In reinforcement learning, when the state space is enormous or infinite, it is not feasible to find the exact value for each state in the memory. A common way to tackle this problem is to adopt linear value function approximation technique. In this paper, we review some commonly used linear value function approximation algorithms, which divided into two classes. The first class is the so-called on-policy algorithms that mainly include temporal difference and least-squares-based methods. The second is the off-policy algorithms that use two strategies to solve the exploration-exploitation problem. From the perspective of algorithm design, on-policy is simple, but it does not take both exploration and exploitation into consideration when collecting data, while the off-policy can well balance the relation between exploration and exploitation. In the end, we first present some experiments to demonstrate the usefulness of linear value function approximation technique in the continuous and high dimensional scenes, and then in-class and out-class experiments are respectively conducted to further illustrate the performance of different approximation algorithms.KeywordsReinforcement learningLinear value function approximationLeast-squares temporal difference

Feature representation is critical not only for pattern recognition tasks but also for reinforcement learning (RL) methods to solve learning control problems under uncertainties. In this paper, a manifold-based RL approach using the principle of locally linear reconstruction (LLR) is proposed for Markov decision processes with large or continuous state spaces. In the proposed approach, an LLR-based feature learning scheme is developed for value function approximation in RL, where a set of smooth feature vectors is generated by preserving the local approximation properties of neighboring points in the original state space. By using the proposed feature learning scheme, an LLR-based approximate policy iteration (API) algorithm is designed for learning control problems with large or continuous state spaces. The relationship between the value approximation error of a new data point and the estimated values of its nearest neighbors is analyzed. In order to compare different feature representation and learning approaches for RL, a comprehensive simulation and experimental study was conducted on three benchmark learning control problems. It is illustrated that under a wide range of parameter settings, the LLR-based API algorithm can obtain better learning control performance than the previous API methods with different feature representation schemes.

With the increase of the number of IP cores integrated in SOC, the functions of SOC ar e becoming more complexed. Test time and test cost grow rapidly, therefore it becomes bo ttleneck of SOC test. Test scheduling is one of the efficient approaches to solve the forenamed ques tion. Cross-entropy method, which is ba sed on probability density function, has been used to solve the SOC test scheduling problem. Experimental results on ITC’02 benchmarks show that the proposed method provides better test time results compared to the Linear-Programming. Keywords: System on chip, test scheduling, cross-entropy method 1. INTRODUCTION In order to shorten the time to market, core based system-on-chip (SOC) design has becomed a widely used technique of integrated circuit design. Compared with system-on-board, work frequency of SOC has improved observably because signal transfers delay is avoid. However, the testing of SOC and the reusable cores becomes a bottleneck of the SOC development. The SOC test includes test information exchange mechanism, test wrapper generation, test access mechanism (TAM) design, test controller design, test schedulin g and test integration, etc. Among the SOC test issues, the test scheduling is one of the most important challenges. Many works have forced on core test scheduling to minimize the overall testing time, in [7], the test scheduling is mapped to the linear programming (LP) problem, but the computation time is too long. In this paper, we use a cross-entropy (CE) method, which is based on probability density function, to solve the test scheduling problem. The paper is organized as follows. In section II, the test scheduling problem is briefly described. We formulate the problem of test scheduling using CE method in section III. Experimental results are given in section IV, and section V concludes the paper.

In this dissertation we present several papers about two optimization methods that are derived from rare-event simulation techniques, i.e., the splitting method and the cross-entropy (CE) method. The splitting method is a well-known method for rare-event simulation, where sample paths of a Markov process are split into multiple copies during the simulation, so as to make the occurrence of a rare event more frequent. Motivated by the splitting algorithm we introduce a novel global optimization method for continuous optimization (SCO) that is both very fast and accurate. We also introduce a new multi-objective optimization (MOO) methodology based the SCO algorithm. The method, called MOS, generalizes the elite set selection of the traditional splitting framework, and uses both local and global sampling to sample in the decision space. Both SCO and MOS were compared with state-of-the art algorithms using a prevailing set of benchmark problems. Numerical experiments demonstrate that the new splitting methods are competitive with famous methods in the fields. The thesis also contains two new applications of the CE method on Laguerre fitting problems. The Laguerre fitting problems are about recovering the weighted generator points from a given Laguerre tessellation. When the description of the Laguerre tessellation is exact, the solutions are not unique and different weighted generator points can create the same tessellation. To recover pertinent generator points we formulate the problem as an optimization problem and apply the modified CE method to solve it. However, the representation of a Laguerre tessellation is typically not exact when working with real data, such as tomographic image data. In this case, we formulate an optimization problem where we minimize the discrepancy between the image data and our approximated tessellations. We then solve the optimization problem using the CE method. Last, we introduce a new R package that implements the general CE methodology for optimization and well as some useful modifications. The usage and efficacy of CEoptim is demonstrated through a variety of optimization examples, including model fitting, combinatorial optimization, and maximum likelihood estimation.

The logistics of moving grain from silos to ports is constrained by the number of trains available and the capacity of loading and unloading facilities. We aim to schedule the available trains to move grain from silos to the port as quickly as possible. The sequence of trips that minimises the time required to complete all trips is a permutation of a basic sequence that has the required number of trips to each silo. We use the Cross Entropy Optimisation method to search for a permutation that minimises span. For small problems, where the optimal solution can be found by enumeration, the Cross Entropy Optimisation method achieves solutions within 5% of the optimum. It can also find good solutions for large problems. References Unveren A. and A. Acan. Multi-objective optimisation with cross entropy method: Stochastic learning with clustered Pareto fronts. IEEE Congress on Evolutionary Computations, pages 3065--3071, 2007. Boer P. T., Kroese D. P., Mannor S., and Rubinstein R. Y. A tutorial on the cross-entropy method. http://www.springerlink.com/index/KPW596202975755N.pdf. Rubinstein R. Y. and D. P. Kroese. The cross entropy method: A unified approach to combinatorial optimization, monte-carlo simulation, and machine learning. Springer--Verlag, New York, 2004.

Global likelihood maximization is an important aspect of many statistical analyses. Often the likelihood function is highly multiextremal. This presents a significant challenge to standard search procedures, which often settle too quickly into an inferior local maximum. We present a new approach based on the cross-entropy (CE) method, and illustrate its use for the analysis of mixture models.

With large-scale wind farms integrated into power systems, the computational requirements for reliability evaluation of composite power systems are increasing. This paper introduces a new concept of dynamic orthogonal list, which is a form of multiple failure retrievals to avoid repeatedly calling the optimal power flow (OPF) program in the stage of state evaluation. An existing algorithm of cross-entropy (CE) method was developed to deal with the rare events in sampling. The major contribution of CE is to achieve convergence faster and reduce the number of samples, thus shortening the consuming time in the stage of sampling. This paper proposes a fast reliability evaluation process which combines the CE method with dynamic orthogonal list (CE-DOL), comprehensively improving the computational efficiency of Monte Carlo simulation (MCS). The proposed method is tested on a modified IEEE-RTS 79 system. In addition, for comparison purposes, the efficiencies of MCS, CE, and CE-DOL are estimated in the cases of different system adequacies. It is shown that, in range of certain reliability level, the proposed method is much more efficient compared with the CE method. Finally, the applicability of different methods is analyzed under different system reliability levels.

We investigate the problem of using function approximation in reinforcement learning where the agent's policy is represented as a classifier mapping states to actions. High classification accuracy is usually deemed to correlate with high policy quality. But this is not necessarily the case as increasing classification accuracy can actually decrease the policy's quality. This phenomenon takes place when the learning process begins to focus on classifying less important states. In this paper, we introduce a measure of state's decision-making importance that can be used to improve policy learning. As a result, the focused learning process is shown to converge faster to better policies.

Temporal difference (TD) learning is a powerful framework for value function approximation in reinforcement learning. However, standard TD methods often struggle with feature representation and off-policy learning challenges. In this paper, we propose a novel framework, online attentive kernel-based off-policy TD learning, and in combination with well-known algorithms, introduce OAKGTD2, OAKTDC, and OAKETD. This framework uses two-timescale optimization. In the slow-timescale, a sparse representation of state features is learned using an online attentive kernel-based method. In the fast-timescale, auxiliary variables are used to update the value function parameters under the off-policy setting. We theoretically prove the convergence of all three algorithms. Through experiments conducted in several standard reinforcement learning environments, we demonstrate the effectiveness of the improved algorithms and compare their performance with existing algorithms. Specifically, from the perspective of cumulative rewards, the proposed algorithm achieves an average improvement of 15% compared to on-policy algorithms and an average improvement of 25% compared to common off-policy algorithms.