Predicting the structure of large protein complexes using AlphaFold and Monte Carlo tree search

Monte Carlo Tree Search (MCTS) is a sampling best-first method to search for optimal decisions. The success of MCTS depends heavily on how the tree is built and the selection process plays a fundamental role in this. One particular selection mechanism that has proved to be reliable is based on the Upper Confidence Bounds for Trees (UCT). The UCT attempts to balance exploration and exploitation by considering the values stored in the statistical tree of the MCTS. However, some tuning of the MCTS UCT is necessary for this to work well. In this work, we use Evolutionary Algorithms (EAs) to evolve mathematical expressions with the goal to substitute the UCT formula and use the evolved expressions in MCTS. More specifically, we evolve expressions by means of our proposed Semantic-inspired Evolutionary Algorithm in MCTS approach (SIEA-MCTS). This is inspired by semantics in Genetic Programming (GP), where the use of fitness cases is seen as a requirement to be adopted in GP. Fitness cases are normally used to determine the fitness of individuals and can be used to compute the semantic similarity (or dissimilarity) of individuals. However, fitness cases are not available in MCTS. We extend this notion by using multiple reward values from MCTS that allow us to determine both the fitness of an individual and its semantics. By doing so, we show how SIEA-MCTS is able to successfully evolve mathematical expressions that yield better or competitive results compared to UCT without the need of tuning these evolved expressions. We compare the performance of the proposed SIEA-MCTS against MCTS algorithms, MCTS Rapid Action Value Estimation algorithms, three variants of the *-minimax family of algorithms, a random controller and two more EA approaches. We consistently show how SIEA-MCTS outperforms most of these intelligent controllers in the challenging game of Carcassonne, whose state-space complexity is, approx., 10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{40}$</tex-math></inline-formula> .

Tree search algorithms are widely applied methods to model and solve sequential decision problems. In particular, the family of sampling-based tree search algorithms called Monte Carlo Tree Search (MCTS) has had great success in problems with large branching factors. However, Real-Time Strategy (RTS) games offer a challenging testbed for tree search algorithms due to their large combinatorial action spaces, partial observability, simultaneous moves, and other factors, making them beyond the grasp of even current MCTS algorithms. This thesis makes contributions towards scaling MCTS algorithms to become more effective and efficient in the domain of RTS games. Specifically, this thesis contributes on the following problems. Firstly, we explore the problem of the integration of MCTS and domain knowledge, in the form of unit-action probability distributions, state evaluation functions, and scripted bots. Secondly, we investigate the optimization of gameplay/rollout policies for MCTS. Third, we study methods for self-learning in MCTS, where tree and/or rollout policies are bootstrapped directly from MCTS behavior iteratively.

The construction of approximate replication strategies for pricing and hedging of derivative contracts in incomplete markets is a key problem of financial engineering. Recently Reinforcement Learning algorithms for hedging under realistic market conditions have attracted significant interest. While research in the derivatives area mostly focused on variations of $Q$-learning, in artificial intelligence Monte Carlo Tree Search is the recognized state-of-the-art method for various planning problems, such as the games of Hex, Chess, Go,... This article introduces Monte Carlo Tree Search as a method to solve the stochastic optimal control problem behind the pricing and hedging tasks. As compared to $Q$-learning it combines Reinforcement Learning with tree search techniques. As a consequence Monte Carlo Tree Search has higher sample efficiency, is less prone to over-fitting to specific market models and generally learns stronger policies faster. In our experiments we find that Monte Carlo Tree Search, being the world-champion in games like Chess and Go, is easily capable of maximizing the utility of investor's terminal wealth without setting up an auxiliary mathematical framework.

Monte Carlo Tree Search (MCTS) is widely used for planning in domains where the potential actions can be represented as a tree of sequential decisions. To efficiently select an action, MCTS usually needs to perform many simulations to build a reliable tree representation of the decision space. As such, a bottleneck to MCTS arises when enough simulations cannot be performed between action selections. This is particularly highlighted in continuously running tasks, for which the time available to perform simulations between actions tends to be limited due to the environment's state constantly changing. In this paper, we present an approach that extends the time available for Monte Carlo simulations when allowed. Our approach is to effectively balance the prospect of selecting the right action with the time that can be spared to perform MCTS simulations before the next action selection. For that, we considered the simulation time as a decision variable to be selected alongside an action. We extended the Hierarchical Optimistic Optimization applied to Tree (HOOT) method to adapt our approach to environments with a continuous decision space. We evaluated our approach on tasks with a continuous decision space using OpenAI gym's Pendulum and Continuous Mountain Car environments and on those with discrete action space using the arcade learning environment (ALE) platform. The evaluation results show that, with variable simulation times, the proposed approach outperforms the conventional MCTS in the evaluated continuous decision space tasks and improves the performance of MCTS in most of the ALE tasks.

Recently, theory of collective intelligence (CI) evolution is proposed as a meta algorithm toward artificial general intelligence. But the only implementation of the CI algorithm of the theory is the Monte Carlo tree search (MCTS) used by AlphaZero. Since ant colony optimization (ACO) is an extensively used CI algorithm, it is useful to implement CI evolution using ACO. A genetic version of ACO is adapted to satisfy the CI evolution theory by two methods. One method is realized by using a policy network, namely policy network guided ACO (P-ACO). The other method is realized by using a policy network and a value network, namely policy and value network guided ACO (PV-ACO). Both methods of ACO evolution algorithm are applied to Tic-Tac-Toe and Four in a Row, where traditional ACO played poorly compared to the tree search algorithm, e.g., MCTS. Computational experiments are done to compare both methods with pure ACO and MCTS. As a result, the intelligence level of ACO evolution algorithm quickly exceeds pure ACO and MCTS. In this article, the performance of ACO evolution algorithm is analyzed and the feasibility of applying the CI evolution theory to a specific application is verified.

In this paper, Monte Carlo tree search (MCTS) is introduced for controlling the Pac-Man character in the real-time game Ms Pac-Man. MCTS is used to find an optimal path for an agent at each turn, determining the move to make based on the results of numerous randomized simulations. Several enhancements are introduced in order to adapt MCTS to the real-time domain. Ms Pac-Man is an arcade game, in which the protagonist has several goals but no conclusive terminal state. Unlike games such as Chess or Go there is no state in which the player wins the game. Instead, the game has two subgoals, 1) surviving and 2) scoring as many points as possible. Decisions must be made in a strict time constraint of 40 ms. The Pac-Man agent has to compete with a range of different ghost teams, hence limited assumptions can be made about their behavior. In order to expand the capabilities of existing MCTS agents, four enhancements are discussed: 1) a variable-depth tree; 2) simulation strategies for the ghost team and Pac-Man; 3) including long-term goals in scoring; and 4) reusing the search tree for several moves with a decay factor gamma. The agent described in this paper was entered in both the 2012 World Congress on Computational Intelligence (WCCI'12, Brisbane, Qld., Australia) and the 2012 IEEE Conference on Computational Intelligence and Games (CIG'12, Granada, Spain) Pac-Man Versus Ghost Team competitions, where it achieved second and first places, respectively. In the experiments, we show that using MCTS is a viable technique for the Pac-Man agent. Moreover, the enhancements improve overall performance against four different ghost teams.

Monte Carlo Tree Search (MCTS) parallelization is one of the many possible enhancements for MCTS algorithms. However, no work has been done on evaluating these methods in the rather new area of General Video Game Playing (GVGP), an area that challenges the creation of agents that can play any videogame even without prior knowledge about the video game they are going to play. To address this gap, this paper proposes the implementation and evaluation of the three main MCTS parallelization methods as agents of the General Video Game AI framework, a popular framework for GVGP agents evaluation. This paper is not focused on comparing the parallel MCTS agents to other existing GVGP agents, but rather on exploring how the MCTS parallelization methods compare between themselves. This paper also presents a testing methodology for evaluating these agents, which is based on a set of three experiments focused on different aspects of the parallel MCTS algorithms. In these experiments, the overall best results were achieved by the root parallelization method using the sum merging technique and the UCT’s sigma value of \( \sqrt 2 \).

Monte Carlo Tree Search (MCTS) is a novel machine learning paradigm that is used to find good solutions for complex optimization problems with very large search spaces (like playing GO). We combine MCTS with Covariance Matrix Adaptation Evolution Strategies (CMA-ES) to efficiently optimize real-parameter single objective problems by balancing the exploitation of promising areas with the exploration of new regions of the search space. The novel algorithm is called hierarchical CMA-ES and it is influenced by both machine learning and evolutionary computation research areas. Like in evolutionary computation, we use a population of individuals to explore the commonalities of CMA-ES solvers. These CMA-ES solvers are structured using a MCTS tree like structure. Our experiments compare the performance of hierarchical CMA-ES solvers with two other algorithms: the standard CMA-ES optimizer, and an adaptation of MCTS to solve real-parameter problems. The hierarchical CMA-ES optimizer has the best empirical performance on several benchmark problems.

This research is based on the idea that certain cognitive-intensive tasks typically carried out by structural engineers—such as choosing how to effectively arrange a building’s structure—can be successfully automated. In this article we investigate two techniques widely used in the field of Artificial Intelligence, namely Monte Carlo Tree Search (MCTS) and Genetic Algorithm (GA). Following a tabula rasa approach, according to which no hints nor external data are used as a reference for navigating the search space, we demonstrate how structural designs of two-dimensional multi-storey reinforced concrete structures can be generated, with no human intervention, by implementing and combining the two techniques from a reinforcement-learning perspective. The design tasks assigned to the developed software agents concern civil structures under static and seismic loads, and the basis for comparison is represented by a combination of construction cost and greenhouse gas emissions associated with the making of the structures. In the article, based on the main concepts of Computational Mechanics, a theoretical framework is introduced, which allows to represent both structures and design tasks in a simple, analytical way. The process of gamification, to which MCTS is often associated, is then described, so that structural design is reduced to the concepts of state, actions and payoff.. Finally, case studies are presented in which different agents—based respectively on GA, MCTS, and a combination of both—are tested. The results suggest that hybrid approaches, where GA operates first followed by MCTS, are the most effective.

Many biological processes are governed by large assemblies of protein molecules. However, it is often very difficult to determine the three-dimensional structures of these assemblies using experimental biophysical techniques. Hence there is a need to develop computational approaches to fill this gap. This article presents an ant colony optimization approach to predict the structure of large multi-component protein complexes. Starting from pair-wise docking predictions, a multi-graph consisting of vertices representing the component proteins and edges representing candidate interactions is constructed. This allows the assembly problem to be expressed in terms of searching for a minimum weight spanning tree. However, because the problem remains highly combinatorial, the search space cannot be enumerated exhaustively and therefore heuristic optimisation techniques must be used. The utility of the ant colony based approach is demonstrated by re-assembling known protein complexes from the Protein Data Bank. The algorithm is able to identify near-native solutions for five of the six cases tested. This demonstrates that the ant colony approach provides a useful way to deal with the highly combinatorial multi-component protein assembly problem.

Monte Carlo Tree Search (MCTS) is a powerful approach to designing game-playing bots or solving sequential decision problems. The method relies on intelligent tree search that balances exploration and exploitation. MCTS performs random sampling in the form of simulations and stores statistics of actions to make more educated choices in each subsequent iteration. The method has become a state-of-the-art technique for combinatorial games. However, in more complex games (e.g. those with a high branching factor or real-time ones) as well as in various practical domains (e.g. transportation, scheduling or security) an efficient MCTS application often requires its problem-dependent modification or integration with other techniques. Such domain-specific modifications and hybrid approaches are the main focus of this survey. The last major MCTS survey was published in 2012. Contributions that appeared since its release are of particular interest for this review.

Many of the strongest game playing programs use a combination of Monte Carlo tree search (MCTS) and deep neural networks (DNN), where the DNNs are used as policy or value evaluators. Given a limited budget, such as online playing or during the self-play phase of AlphaZero (AZ) training, a balance needs to be reached between accurate state estimation and more MCTS simulations, both of which are critical for a strong game playing agent. Typically, larger DNNs are better at generalization and accurate evaluation, while smaller DNNs are less costly, and therefore can lead to more MCTS simulations and bigger search trees with the same budget. This paper introduces a new method called the multiple policy value MCTS (MPV-MCTS), which combines multiple policy value neural networks (PV-NNs) of various sizes to retain advantages of each network, where two PV-NNs f_S and f_L are used in this paper. We show through experiments on the game NoGo that a combined f_S and f_L MPV-MCTS outperforms single PV-NN with policy value MCTS, called PV-MCTS. Additionally, MPV-MCTS also outperforms PV-MCTS for AZ training.

An important yet challenging problem in LINAC-based rotational arc radiation therapy is the design of beam trajectory, which requires simultaneous consideration of delivery efficiency and final dose distribution. In this work, we propose a novel trajectory selection strategy by developing a Monte Carlo tree search (MCTS) algorithm during the beam trajectory selection process.To search through the vast number of possible trajectories, the MCTS algorithm was implemented. In this approach, a candidate trajectory is explored by starting from a leaf node and sequentially examining the next level of linked nodes with consideration of geometric and physical constraints. The maximum Upper Confidence Bounds for Trees, which is a function of average objective function value and the number of times the node under testing has been visited, was employed to intelligently select the trajectory. For each candidate trajectory, we run an inverse fluence map optimization with an infinity norm regularization. The ranking of the plan as measured by the corresponding objective function value was then fed back to update the statistics of the nodes on the trajectory. The method was evaluated with a chest wall and a brain case, and the results were compared with the coplanar and noncoplanar 4pi beam configurations.For both clinical cases, the MCTS method found effective and easy-to-deliver trajectories within an hour. As compared with the coplanar plans, it offers much better sparing of the OARs while maintaining the PTV coverage. The quality of the MCTS-generated plan is found to be comparable to the 4pi plans.Artificial intelligence based on MCTS is valuable to facilitate the design of beam trajectory and paves the way for future clinical use of non-coplanar treatment delivery.

In this paper, we address the task of rearranging items with a robot. A rearrangement task is challenging because it requires us to solve the following issues: determine how to pick the items and plan how and where to place the items. In our previous work, we proposed to solve a rearrangement task by combining the symbolic and motion planners using a Motion Feasibility Checker (MFC) and a Monte Carlo Tree Search (MCTS). The MCTS searches for the goal while it collaborates with the MFC to accept or reject instructions. We could solve the rearrangement task, but one drawback is the time it takes to find a solution. In this study, we focus on quickly accepting or rejecting tentative instructions obtained from an MCTS. We propose using a Neural Network-based Motion Feasibility Checker (NN-MFC), a Fully Connected Neural Network trained with data obtained from the MFC. This NN-MFC quickly decides if the instruction is valid or not, reducing the time the MCTS uses to find a solution to the task. The NN-MFC determines the validity of the instruction based on the initial and target poses of the item. Before the final execution of the instructions, we re-validate the instructions with the MFC as a confirmation before execution. We tested the proposed method in a simulation environment by doing an item rearrangement task in a convenience store setup.

Nowadays manipulators are widely used in logistics systems, thus being important to improve the efficiency. In this paper, we transform the task sequencing problem of manipulator into a dynamic traveling salesman problem (DTSP), and use the Monte Carlo tree search (MCTS) approach to determine the execution order of the tasks. MCTS is an effective self-learning algorithm, which consists of four phases, i.e., initialization, simulation, selection and back propagation. Furthermore, to fit the dynamic feature of real-life applications, we introduce a dynamic mechanism using the information of historically found solutions. Finally, we carry out experiments based on 30 randomly generated task sets. The results show that, compared to the most popular method which executes all the tasks in sequence, MCTS is able to save 14.37%, 15.68%, 18.10% execution time (respectively on the small-scale, mid-scale and large-scale task sets), indicating its ability in improving the efficiency of manipulator.