Efficient Real-Parameter Single Objective Optimizer Using Hierarchical CMA-ES Solvers

Monte-Carlo Tree Search (MCTS) grows a partial game tree and uses a large number of random simulations to approximate the values of the nodes. It has proven effective in games with such as Go and Hex where the large search space and difficulty of evaluating positions cause difficulties for standard methods. The best MCTS players use carefully hand-crafted rules to bias the random simulations. Obtaining good hand-crafting rules is a very difficult process, as even rules promoting better simulation play can result in a weaker MCTS system [12]. Our Hivemind system uses evolution strategies to automatically learn effective rules for biasing the random simulations. We have built a MCTS player using Hivemind for the game Hex. The Hivemind learned rules result in a 90% win rate against a baseline MCTS system, and significant improvement against the computer Hex world champion, MoHex.

Monte Carlo Tree Search (MCTS) is widely used for planning in domains where the potential actions can be represented as a tree of sequential decisions. To efficiently select an action, MCTS usually needs to perform many simulations to build a reliable tree representation of the decision space. As such, a bottleneck to MCTS arises when enough simulations cannot be performed between action selections. This is particularly highlighted in continuously running tasks, for which the time available to perform simulations between actions tends to be limited due to the environment's state constantly changing. In this paper, we present an approach that extends the time available for Monte Carlo simulations when allowed. Our approach is to effectively balance the prospect of selecting the right action with the time that can be spared to perform MCTS simulations before the next action selection. For that, we considered the simulation time as a decision variable to be selected alongside an action. We extended the Hierarchical Optimistic Optimization applied to Tree (HOOT) method to adapt our approach to environments with a continuous decision space. We evaluated our approach on tasks with a continuous decision space using OpenAI gym's Pendulum and Continuous Mountain Car environments and on those with discrete action space using the arcade learning environment (ALE) platform. The evaluation results show that, with variable simulation times, the proposed approach outperforms the conventional MCTS in the evaluated continuous decision space tasks and improves the performance of MCTS in most of the ALE tasks.

Randomized direct search algorithms for continuous domains, such as evolution strategies, are basic tools in machine learning. They are especially needed when the gradient of an objective function (e.g., loss, energy, or reward function) cannot be computed or estimated efficiently. Application areas include supervised and reinforcement learning as well as model selection. These randomized search strategies often rely on normally distributed additive variations of candidate solutions. In order to efficiently search in non-separable and ill-conditioned landscapes the covariance matrix of the normal distribution must be adapted, amounting to a variable metric method. Consequently, covariance matrix adaptation (CMA) is considered state-of-the-art in evolution strategies. In order to sample the normal distribution, the adapted covariance matrix needs to be decomposed, requiring in general Θ(n 3) operations, where n is the search space dimension. We propose a new update mechanism which can replace a rank-one covariance matrix update and the computationally expensive decomposition of the covariance matrix. The newly developed update rule reduces the computational complexity of the rank-one covariance matrix adaptation to Θ(n 2) without resorting to outdated distributions. We derive new versions of the elitist covariance matrix adaptation evolution strategy (CMA-ES) and the multi-objective CMA-ES. These algorithms are equivalent to the original procedures except that the update step for the variable metric distribution scales better in the problem dimension. We also introduce a simplified variant of the non-elitist CMA-ES with the incremental covariance matrix update and investigate its performance. Apart from the reduced time-complexity of the distribution update, the algebraic computations involved in all new algorithms are simpler compared to the original versions. The new update rule improves the performance of the CMA-ES for large scale machine learning problems in which the objective function can be evaluated fast.

A comparison of the Covariance Matrix Adaptation Evolution Strategy and the Levenberg–Marquardt method for solving multidimensional inverse transport problems

This research is based on the idea that certain cognitive-intensive tasks typically carried out by structural engineers—such as choosing how to effectively arrange a building’s structure—can be successfully automated. In this article we investigate two techniques widely used in the field of Artificial Intelligence, namely Monte Carlo Tree Search (MCTS) and Genetic Algorithm (GA). Following a tabula rasa approach, according to which no hints nor external data are used as a reference for navigating the search space, we demonstrate how structural designs of two-dimensional multi-storey reinforced concrete structures can be generated, with no human intervention, by implementing and combining the two techniques from a reinforcement-learning perspective. The design tasks assigned to the developed software agents concern civil structures under static and seismic loads, and the basis for comparison is represented by a combination of construction cost and greenhouse gas emissions associated with the making of the structures. In the article, based on the main concepts of Computational Mechanics, a theoretical framework is introduced, which allows to represent both structures and design tasks in a simple, analytical way. The process of gamification, to which MCTS is often associated, is then described, so that structural design is reduced to the concepts of state, actions and payoff.. Finally, case studies are presented in which different agents—based respectively on GA, MCTS, and a combination of both—are tested. The results suggest that hybrid approaches, where GA operates first followed by MCTS, are the most effective.

The performance of evolutionary algorithms can be heavily undermined when\nconstraints limit the feasible areas of the search space. For instance, while\nCovariance Matrix Adaptation Evolution Strategy is one of the most efficient\nalgorithms for unconstrained optimization problems, it cannot be readily\napplied to constrained ones. Here, we used concepts from Memetic Computing,\ni.e. the harmonious combination of multiple units of algorithmic information,\nand Viability Evolution, an alternative abstraction of artificial evolution, to\ndevise a novel approach for solving optimization problems with inequality\nconstraints. Viability Evolution emphasizes elimination of solutions not\nsatisfying viability criteria, defined as boundaries on objectives and\nconstraints. These boundaries are adapted during the search to drive a\npopulation of local search units, based on Covariance Matrix Adaptation\nEvolution Strategy, towards feasible regions. These units can be recombined by\nmeans of Differential Evolution operators. Of crucial importance for the\nperformance of our method, an adaptive scheduler toggles between exploitation\nand exploration by selecting to advance one of the local search units and/or\nrecombine them. The proposed algorithm can outperform several state-of-the-art\nmethods on a diverse set of benchmark and engineering problems, both for\nquality of solutions and computational resources needed.\n

The most widely used methods for toolpath planning in 3D printing slice the input model into successive 2D layers to construct the toolpath. Unfortunately the methods can incur a substantial amount of wasted motion (i.e., the extruder is moving while not printing). In recent years we have introduced a new paradigm that characterizes the space of feasible toolpaths using a dependency graph on the input model, along with several algorithms that optimize objective functions (wasted motion or print time). A natural question that arises is, under what circumstances can we efficiently compute an optimal toolpath? In this paper, we give an algorithm for computing fused deposition modeling (FDM) toolpaths that utilizes Monte Carlo Tree Search (MCTS), a powerful generalpurpose method for navigating large search spaces that is guaranteed to converge to the optimal solution. Under reasonable assumptions on printer geometry that allow us to compress the dependency graph, our MCTS-based algorithm converges to find the optimal toolpath. We validate our algorithm on a dataset of 75 models and examine the performance on MCTS against our previous best local search-based algorithm in terms of toolpath quality. We show that a relatively short time budget for MCTS yields results on par with local search, while a larger time budget yields a 15% improvement in quality over local search. Additionally, we examine the properties of the models and MCTS executions that lead to better or worse results.

Multi-objective Estimation of Distribution Algorithms (MOEDAS) have been successfully applied to solve Multi-objective Optimization Problems (MOPs) since they are able to model dependencies between variables of the problem and then sample new solutions to guide the search to promising areas. A state-of-the-art optimizer for single-objective continuous functions that also uses probabilistic modeling is the Covariance Matrix Adaptation Evolution Strategy (CMA-ES). Different variants of CMA-ES have been proposed for MOPs however most of them are based on Pareto dominance as the main selection criterion. Recently, a new multi-objective CMA-ES called MOEA/D-CMA was proposed combining the strengths of CMA-ES with those of the multi-objective evolutionary algorithm based on decomposition (MOEA/D). Nowadays, however, researchers on MOEAs agree that combining Pareto and decomposition can be beneficial for the search on MOPs. As a result, a new MOEA has been proposed, called MOEA/DD. This algorithm modifies the MOEA/D by including a new Pareto dominance update mechanism that brings more diversity into the search. In this study, we extend the MOEA/D-CMA by replacing its update mechanism by the one of MOEA/DD. The hypothesis is that this update mechanism will improve the performance of MOEA/D-CMA as it improved MOEA/D. MOEA/D-CMA and MOEA/DD-CMA are implemented and evaluated through an experimental study. The experimental study involves two well-known families of benchmark problems whose objective numbers scale from two to fifteen. Then, an extensive statistical analysis of the results is made to extract sound, statistically supported conclusions about the performance of the algorithms as the number of objectives scales.

Evolution-based neural architecture search methods have shown promising results, but they require high computational resources because these methods involve training each candidate architecture from scratch and then evaluating its fitness, which results in long search time. Covariance Matrix Adaptation Evolution Strategy (CMA-ES) has shown promising results in tuning hyperparameters of neural networks but has not been used for neural architecture search. In this work, we propose a framework called CMANAS which applies the faster convergence property of CMA-ES to the deep neural architecture search problem. Instead of training each individual architecture seperately, we used the accuracy of a trained one shot model (OSM) on the validation data as a prediction of the fitness of the architecture, resulting in reduced search time. We also used an architecture-fitness table (AF table) for keeping a record of the already evaluated architecture, thus further reducing the search time. The architectures are modeled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. Experimentally, CMANAS achieves better results than previous evolution-based methods while reducing the search time significantly. The effectiveness of CMANAS is shown on two different search spaces using four datasets: CIFAR-10, CIFAR-100, ImageNet, and ImageNet16-120. All the results show that CMANAS is a viable alternative to previous evolution-based methods and extends the application of CMA-ES to the deep neural architecture search field.

Short-term solar radiation forecasting using hybrid deep residual learning and gated LSTM recurrent network with differential covariance matrix adaptation evolution strategy

This paper describes a method for generative player modeling and its application to the automatic testing of game content using archetypal player models called procedural personas. Theoretically grounded in psychological decision theory, procedural personas are implemented using a variation of Monte Carlo tree search (MCTS) where the node selection criteria are developed using evolutionary computation, replacing the standard UCB1 criterion of MCTS. Using these personas, we demonstrate how generative player models can be applied to a varied corpus of game levels and demonstrate how different playstyles can be enacted in each level. In short, we use artificially intelligent personas to construct synthetic playtesters. The proposed approach could be used as a tool for automatic play testing when human feedback is not readily available or when quick visualization of potential interactions is necessary. Possible applications include interactive tools during game development or procedural content generation systems where many evaluations must be conducted within a short time span.

Information capture and reuse strategies in Monte Carlo Tree Search, with applications to games of hidden information

Monte Carlo Tree Search (MCTS) is a general search algorithm that has improved the state of the art for multiple games and optimization problems. Nested Rollout Policy Adaptation (NRPA) is an MCTS variant that has found record-breaking solutions for puzzles and optimization problems. It learns a playout policy online that dynamically adapts the playouts to the problem at hand. We propose to enhance NRPA using more selectivity in the playouts. The idea is applied to three different problems: Bus regulation, SameGame and Weak Schur numbers. We improve on standard NRPA for all three problems.

Car classification, using different machine learning models with optimization frameworks, is done for evaluation in this work. We used different models, namely, Extra Trees, XGBoost, Gaussian Naive Bayes, K-Nearest Neighbors, Histogram-based Gradient Boosting (Hist Gradient Boosting), and Linear Discriminant Analysis, for classification. The car samples are classified as “very good,” “good,” “acceptable,” and “unacceptable.” Among these, Hist Gradient Boosting has the highest value for precision, accuracy, recall, and F1 score. We further tuned this model using Evolutionary Strategies, Evolutionary Programming, Covariance Matrix Adaptation Evolution Strategy, and the Flower Pollination Algorithm. Our outcomes indicate that the Covariance Matrix Adaptation Evolution Strategy and Flower Pollination Algorithm significantly enhanced the performance of the model and outperformed Evolutionary Programming. This work investigates the potential of integrating advanced machine learning models with sophisticated optimization strategies to deliver an effective car evaluation classification process that will be useful in this industry and, perhaps, in many other classification tasks.

This paper presents an extended variant of the distance-weighted exponential natural evolution strategy (DXNES) that works well in uncertain environments. Since we often face objective functions with uncertain parameters in real-world problems, function optimization in uncertain environments is an important problem. The covariance matrix adaptation evolution strategy (CMA-ES) and DX-NES have been proposed as promising methods for function optimization in deterministic environments. The performance of these methods, however, deteriorates in uncertain environments. The uncertain handling CMA-ES (TIH-CMA-ES) has been proposed as an extended variant of CMA-ES for uncertain environments and has shown relatively good performance on problems with uncertain parameters. In this paper, we propose an extended variant of DX-NES named DX-NES for uncertain environments (DX-NES-TIE). DX-NES-TIE approximates the objective function by a quadratic function. DXNES-TIE uses approximation function values for updating the mutation distribution if the noise is strong; otherwise it uses observed objective function values. The strength of the noise is quantified by using the approximation function and the evolution path. Through numerical experiments on 20-dimensional uncertain benchmark problems, we demonstrate that DX-NES-TIE can find ten to 2,000 times as accurate solutions as TIH-CMA-ES can. We also apply DX-NES-TIE to 80-dimensional problems and confirm that DX-NES-TIE is scalable with respect to problem dimensionality.