Nested Rollout Policy Adaptation with Selective Policies

Monte Carlo Tree Search (MCTS) is a novel machine learning paradigm that is used to find good solutions for complex optimization problems with very large search spaces (like playing GO). We combine MCTS with Covariance Matrix Adaptation Evolution Strategies (CMA-ES) to efficiently optimize real-parameter single objective problems by balancing the exploitation of promising areas with the exploration of new regions of the search space. The novel algorithm is called hierarchical CMA-ES and it is influenced by both machine learning and evolutionary computation research areas. Like in evolutionary computation, we use a population of individuals to explore the commonalities of CMA-ES solvers. These CMA-ES solvers are structured using a MCTS tree like structure. Our experiments compare the performance of hierarchical CMA-ES solvers with two other algorithms: the standard CMA-ES optimizer, and an adaptation of MCTS to solve real-parameter problems. The hierarchical CMA-ES optimizer has the best empirical performance on several benchmark problems.

Today's automated vehicles lack the ability to cooperate implicitly with others. This work presents a Monte Carlo Tree Search (MCTS) based approach for decentralized cooperative planning using macro-actions for automated vehicles in heterogeneous environments. Based on cooperative modeling of other agents and Decoupled-UCT (a variant of MCTS), the algorithm evaluates the state-action-values of each agent in a cooperative and decentralized manner, explicitly modeling the interdependence of actions between traffic participants. Macro-actions allow for temporal extension over multiple time steps and increase the effective search depth requiring fewer iterations to plan over longer horizons. Without predefined policies for macro-actions, the algorithm simultaneously learns policies over and within macro-actions. The proposed method is evaluated under several conflict scenarios, showing that the algorithm can achieve effective cooperative planning with learned macro-actions in heterogeneous environments.

Tree search algorithms are widely applied methods to model and solve sequential decision problems. In particular, the family of sampling-based tree search algorithms called Monte Carlo Tree Search (MCTS) has had great success in problems with large branching factors. However, Real-Time Strategy (RTS) games offer a challenging testbed for tree search algorithms due to their large combinatorial action spaces, partial observability, simultaneous moves, and other factors, making them beyond the grasp of even current MCTS algorithms. This thesis makes contributions towards scaling MCTS algorithms to become more effective and efficient in the domain of RTS games. Specifically, this thesis contributes on the following problems. Firstly, we explore the problem of the integration of MCTS and domain knowledge, in the form of unit-action probability distributions, state evaluation functions, and scripted bots. Secondly, we investigate the optimization of gameplay/rollout policies for MCTS. Third, we study methods for self-learning in MCTS, where tree and/or rollout policies are bootstrapped directly from MCTS behavior iteratively.

We explore how a general AI algorithm can be used for 3D scene understanding to reduce the need for training data. More exactly, we propose a modification of the Monte Carlo Tree Search (MCTS) algorithm to retrieve objects and room layouts from noisy RGB-D scans. While MCTS was developed as a game-playing algorithm, we show it can also be used for complex perception problems. Our adapted MCTS algorithm has few easy-to-tune hyperparameters and can optimise general losses. We use it to optimise the posterior probability of objects and room layout hypotheses given the RGB-D data. This results in an analysis-by-synthesis approach that explores the solution space by rendering the current solution and comparing it to the RGB-D observations. To perform this exploration even more efficiently, we propose simple changes to the standard MCTS’ tree construction and exploration policy. We demonstrate our approach on the ScanNet dataset. Our method often retrieves configurations that are better than some manual annotations, especially on layouts.

International audience

This paper puts forward a constructive heuristic algorithm called the method of inserting the minimum neighbor edge from outside to the center (IMNEFOTC) that can be applied to solve large-scale and ultra-large-scale travelling salesman problems. Through it and the randomized greedy heuristic algorithm (RGH) which greedy heuristic algorithm is modified, a general meta-heuristic search algorithm framework is built. The general search algorithm (GSA) is based on a set of initial solutions, and continuous 2-opt operations are performed, so as to search for solutions in better quality. The data from the experiment reveals that the performance of IMNEFOTC is better than the traditional greedy algorithm, and the GSA can obtain a pretty satisfactory solution in a reasonable range of time.

Generative design for complex floorplans in high-rise residential buildings: A Monte Carlo tree search-based self-organizing multi-agent system (MCTS-MAS) solution

Flexible job-shop scheduling problem (FJSP) is very important in both fields of production management and combinatorial optimization. This paper focuses on the multiobjective flexible job shop scheduling problem (MO-FJSP) with three objectives which minimizing make span, total workload and maximal workload, respectively, with Pareto manner. In addition, Monte-Carlo Tree Search (MCTS) is successful in computer Go and many other games. Hence, solving FJSP by MCTS is a new attempt. In this paper, we propose an MCTS algorithm for FJSP, by incorporating Variable Neighborhood Descent Algorithm and other techniques like Rapid Action Value Estimates Heuristic and Transposition Table. Our algorithm finds Pareto solutions of the benchmark problems proposed by Kacem et al. within 116 seconds: 4 solutions in 4×5, 3 in 10×7, 4 in 8×8, 4 in 10×10 and 2 in 15×10. These solutions are the same as the best found to date. Although one article claimed to have an extra 8×8 solution, that article did not find some of the above solutions.

Monte Carlo Tree Search has been successfully applied to complex domains such as computer Go. However, despite its success in building game-playing agents, there is little understanding of general principles to design or learn its playout policy. Many systems, such as AlphaGo, use a policy optimized to mimic human expert as the playout policy. But are strong policies good playout policies? In this paper, we take a case study in real-time strategy games. We use bandit algorithms to optimize stochastic policies as both gameplay policies and playout policies for MCTS in the context of RTS games. Our results show that strong policies do not make the best playout policies, and that policies that maximize MCTS performance as playout policies are actually weak in terms of gameplay strength

Software acting on complex data structures can be challenging to test: it is difficult to generate diverse test data that satisfies structural constraints while simultaneously exhibiting properties, such as a particular size, that the test engineer believes will be effective in detecting faults. In our previous work we introduced GodelTest, a framework for generating such data structures using non-deterministic programs, and combined it with Differential Evolution to optimize the generation process. Monte-Carlo Tree Search (MCTS) is a search technique that has shown great success in playing games that can be represented as a sequence of decisions. In this paper we apply Nested Monte-Carlo Search, a single-player variant of MCTS, to the sequence of decisions made by the generating programs used by GodelTest, and show that this combination can efficiently generate random data structures which exhibit the specific properties that the test engineer requires. We compare the results to Boltzmann sampling, an analytical approach to generating random combinatorial data structures.

The rise in EVs popularity, combined with reducing emissions and cutting costs, encouraged delivery companies to integrate them in their fleets. The fleet heterogeneity brings new challenges to the Capacitated Vehicle Routing Problem (CVRP). Driving range and different vehicles' capacity constraints must be considered. A cluster-first, route-second heuristic approach is proposed to maximise the number of parcels delivered. Clustering is achieved with two algorithms: a capacitated k-median algorithm that groups parcel drop-offs based on customer location and parcel weight; and a hierarchical constrained minimum weight matching clustering algorithm which considers EVs' range. This reduces the solution space in a meaningful way for the routing. The routing heuristic introduces a novel Monte–Carlo Tree Search enriched with a custom objective function, rollout policy and a tree pruning technique. Moreover, EVs are preferred overother vehicles when assigning parcels to vehicles. A Tabu Search step further optimises the solution. This two-step procedure allows problems with thousands of customers and hundreds of vehicles to be solved accomodating customised vehicle constraints. The solution quality is similar to other CVRP implementations when applied to classic VRP test-instances; and its quality is superior in real-world scenarios when constraints for EVs must be used.

The supply chain nowadays is more dynamic than before partly due to the information techniques and changes of consumer behaviors. This proposal analyzes the multiple objective optimization problems existing in the supply chain management, and the stochastic and dynamic circumstance of current supply chain. It presents a general overview of multi-objective optimization techniques, stochastic and dynamic programming. We attempt to apply these techniques to solve some optimization problems in supply chain, under changing environment. It can be a guideline of future research. There are mainly three parts of the proposal; the first is multiple objective optimization problems in the real supply chain management, and the stochastic and dynamic environment of the supply chain. The second part is the suggesting approaches dealing with multi-objective optimization problems and the stochastic, dynamic circumstance. The last part is attempting to apply these methods into a real case existing in the supply chain.

Generalized gradient projection neural network models are proposed to solve nonsmooth convex and nonconvex nonlinear programming problems over a closed convex subset of ℝ n . By using Clarke’s generalized gradient, the neural network modeled by a differential inclusion is developed, and its dynamical behavior and optimization capabilities both for convex and nonconvex problems are rigorously analyzed in the framework of nonsmooth analysis and the differential inclusion theory. First for nonconvex optimization problems, the quasiconvergence results similar to those achieved by previous work are proved. For convex optimization problems, the global convergence results are proved, i.e. any trajectory of the neural network converges to an asymptotically stable equilibrium point, which is an optimal solution of the primal problem whenever it exists. This result remains valid even if the initial condition is chosen outside the feasible set. In addition, an asymptotic control result involving a Tykhonov-like regularization shows that any trajectory of the revised neural network can be forced to converge toward a particular optimal solution of the primal problem. Finally, two simulation optimization algorithms for solving the optimization and control problems are designed respectively, and three typical simulation experiments are illustrated to present the accuracy and efficiency of theoretical convergence results of this paper.

Generative AI has made remarkable progress in addressing various design challenges. One prominent area where generative AI could bring significant value is in engineering design. In particular, selecting an optimal set of components and their interfaces to create a mechanical system that meets design requirements is one of the most challenging and time-consuming tasks for engineers. This configuration design task is inherently challenging due to its categorical nature, multiple design requirements a solution must satisfy, and the reliance on physics simulations for evaluating potential solutions. These characteristics entail solving a combinatorial optimization problem with multiple constraints involving black-box functions. To address this challenge, we propose a deep generative model to predict the optimal combination of components and interfaces for a given design problem. To demonstrate our approach, we solve a gear train synthesis problem by first creating a synthetic dataset using a domain-specific language, a parts catalogue, and a physics simulator. We then train a Transformer-based model using this dataset, named GearFormer, which can not only generate quality solutions on its own, but also augment traditional search methods such as an evolutionary algorithm and Monte Carlo tree search. We show that GearFormer outperforms such search methods on their own in terms of satisfying the specified design requirements with orders of magnitude faster generation time. Additionally, we showcase the benefit of hybrid methods that leverage both GearFormer and search methods, which further improve the quality of the solutions.

In order to enlarge the IoT ecosystem, there is a growing need for microbatteries with high areal energy density. A promising approach to improve areal energy density includes using three-dimensional (3D) electrodes, which allow for the holding of more active materials without increasing the internal resistance. Nonetheless, pinpointing the optimal 3D geometry is a challenging task, reliant on the printable materials, the resolution of the production machinery, and the usage of the battery. This constitutes a multiobjective optimization problem, introducing substantial difficulties even when computer simulations are employed because of their high computational cost.To overcome this challenge, we propose a novel approach for the automated identification of optimal 3D battery geometries, which combines quick and accurate energy prediction models with an automatic geometry generator (AGG). [1-4] The energy prediction models employ simple principal component regression, which results in a significantly reduced computational cost. [2] AGG constructs a 3D geometry by filling the empty cell space with cuboid-shaped electrodes (electrode elements). The placement of each electrode element is determined automatically through the Monte Carlo tree search (MCTS) algorithm, leading to the designation of this method as AGG-MCTS. Given that the dimensions of the electrode elements are determined following the resolution of using fabrication equipment, such as 3D printers, our method ensures the generation of only manufacturable geometries. The combination of the energy prediction models and AGG-MCTS enables the rapid identification of optimal 3D battery geometries, without using human intuition and experience.In this study, we demonstrate the efficacy of our approach through its application to two standard electrode pairs: LiFePO4/Li4Ti5O12 (LFP/LTO) and LiNi0.5Mn0.3Co0.2O2/graphite (NMC/Graphite). Despite the distinct discharging behaviors of these systems, our regression models accurately predict the discharge-current-dependent energies of both systems.[4] By integrating these models with the AGG-MCTS, we successfully identify frontier solutions and obtain optimal geometries that display 30% - 40% greater energy than the current state of the art. Notably, optimal geometries vary in response to the discharge current and the specific electrode combination. Our approach uniquely facilitates the discovery of material- and discharge-current-dependent optimal geometries for the first time. This advancement underscores the significance of custom-designed batteries for a variety of IoT applications and highlights our method’s potential to enable such designs.[1] Miyamoto, et al., iScience, 23, 101317 (2020).[2] Miyamoto, et al., Cell. Rep. Phys. Sci., 2, 100504 (2021).[3] Miyamoto, et al., J. Power. Sources, 536, 231473 (2022).[4] Miyamoto, in preparation (2024).