Belief-State Monte Carlo Tree Search for Phantom Go

Monte Carlo Tree Search (MCTS) parallelization is one of the many possible enhancements for MCTS algorithms. However, no work has been done on evaluating these methods in the rather new area of General Video Game Playing (GVGP), an area that challenges the creation of agents that can play any videogame even without prior knowledge about the video game they are going to play. To address this gap, this paper proposes the implementation and evaluation of the three main MCTS parallelization methods as agents of the General Video Game AI framework, a popular framework for GVGP agents evaluation. This paper is not focused on comparing the parallel MCTS agents to other existing GVGP agents, but rather on exploring how the MCTS parallelization methods compare between themselves. This paper also presents a testing methodology for evaluating these agents, which is based on a set of three experiments focused on different aspects of the parallel MCTS algorithms. In these experiments, the overall best results were achieved by the root parallelization method using the sum merging technique and the UCT’s sigma value of \( \sqrt 2 \).

Information capture and reuse strategies in Monte Carlo Tree Search, with applications to games of hidden information

AlphaFold can predict the structure of single- and multiple-chain proteins with very high accuracy. However, the accuracy decreases with the number of chains, and the available GPU memory limits the size of protein complexes which can be predicted. Here we show that one can predict the structure of large complexes starting from predictions of subcomponents. We assemble 91 out of 175 complexes with 10–30 chains from predicted subcomponents using Monte Carlo tree search, with a median TM-score of 0.51. There are 30 highly accurate complexes (TM-score ≥0.8, 33% of complete assemblies). We create a scoring function, mpDockQ, that can distinguish if assemblies are complete and predict their accuracy. We find that complexes containing symmetry are accurately assembled, while asymmetrical complexes remain challenging. The method is freely available and accesible as a Colab notebook https://colab.research.google.com/github/patrickbryant1/MoLPC/blob/master/MoLPC.ipynb.

Extensive-form games are a powerful framework for modeling sequential multi-agent interactions. In extensive-form games with imperfect information, Nash equilibria are generally used as a solution concept, but computing a Nash equilibrium can be intractable in large games. Instead, a variety of techniques are used to find strategies that approximate Nash equilibria. Traditionally, an approximate Nash equilibrium strategy is evaluated by measuring the strategy's worst-case performance, or exploitability. However, because exploitability fails to capture how likely the worst-case is to be realized, it provides only a limited picture of strategy strength, and there is extensive empirical evidence showing that exploitability can correlate poorly with one-on-one performance against a variety of opponents. In this thesis, we introduce a class of adaptive opponents called pretty-good responses that exploit a strategy but only have limited exploitative power. By playing a strategy against a variety of counter-strategies created with pretty-good responses, we get a more complete picture of strategy strength than that offered by exploitability alone. In addition, we show how standard no-regret algorithms can me modified to learn strategies that are strong against adaptive opponents. We prove that this technique can produce optimal strategies for playing against pretty-good responses. We empirically demonstrate the effectiveness of the technique by finding static strategies that are strong against Monte Carlo opponents who learn by sampling our strategy, including the UCT Monte Carlo tree search algorithm.

바둑 게임은 가장 오래된 게임 중의 하나이며 적어도 2,500년 전에 기원되었다. 게임프로그래밍에서 대부분의 성공적인 접근법은 평가함수를 활용한 게임트리 탐색을 사용하는 것이다. 그러나 컴퓨터바둑에서 그럴싸한 평가함수를 구축한다는 것은 매우 어렵다. 몬테카를로 트리탐색(MCTS)은 9줄 바둑에서 프로기사를 제압한 MoGo와 CrazyStone과 같은 강력한 컴퓨터바둑프로그램을 만들어 내었다. 몬테카를로 트리탐색은 몬테카를로 시뮬레이션에 의해 계산된 승률을 근간으로 한다. 몬테카를로 트리탐색을 컴퓨터바둑에 구현하기에 앞서 삼목에서 최상의 첫 수로 중앙, 귀, 변의 세 수에 대한 각각의 승률을 측정하려고 했다. 실험 결과로 최상의 첫 수는 중앙이 우선하고, 다음은 귀, 마지막으로는 변이라는 사실이 밝혀졌다.

Monte-Carlo Tree Search (MCTS) grows a partial game tree and uses a large number of random simulations to approximate the values of the nodes. It has proven effective in games with such as Go and Hex where the large search space and difficulty of evaluating positions cause difficulties for standard methods. The best MCTS players use carefully hand-crafted rules to bias the random simulations. Obtaining good hand-crafting rules is a very difficult process, as even rules promoting better simulation play can result in a weaker MCTS system [12]. Our Hivemind system uses evolution strategies to automatically learn effective rules for biasing the random simulations. We have built a MCTS player using Hivemind for the game Hex. The Hivemind learned rules result in a 90% win rate against a baseline MCTS system, and significant improvement against the computer Hex world champion, MoHex.

Fuegito is an educational software package for learning about programming two player games. The package provides a simple, yet flexible and extensible framework which allows students to study the core search algorithms of computer game-playing, and extend them easily in projects. The current version of Fuegito supports alphabeta and Monte Carlo Tree Search as main game tree search methods, as well as a number of simple playing algorithms.

An agent, or a coalition of agents, faces an ethical dilemma between several statements if she is forced to make a conscious choice between which of these statements will be true. This paper proposes to capture ethical dilemmas as a modality in strategic game settings with and without limit on sacrifice and for perfect and imperfect information games. The authors show that the dilemma modality cannot be defined through the earlier proposed blameworthiness modality. The main technical result is a sound and complete axiomatization of the properties of this modality with sacrifice in games with perfect information.

In this paper, we introduce a Monte-Carlo tree search (MCTS) approach for the game Hearthstone: Heroes of Warcraft. We argue that, in light of the challenges posed by the game (such as uncertainty and hidden information), Monte Carlo tree search offers an appealing alternative to existing AI players. Additionally, by enriching MCTS with a properly constructed heuristic, it is possible to introduce significant gains in performance. We illustrate through extensive empirical validation the superior performance of our approach against vanilla MCTS and the current state-of-the art AI for Hearthstone.

Imperfect information game in multiplayer no-limit Texas Hold'em is a critical challenge in AI research. Recent advanced solving approaches, such as deep CounterFactual Value networks(CFVnet) combined with continual resolving, provide a way to conduct depth-limited search in imperfect-information games. However, CFVnet has limited deployment in Heads-Up No-Limit Texas Hold'em, and is hard to scale to multiplayer setting. In this paper, we propose a novel algorithm, mean approximation, that effectively converting multi-agent interactions to two-agent interactions, and introduce a useful trick virtual action generation to solve conflicts occur in this conversion. Furthermore, we introduce several improvements to deep CFVnet applied in Texas Hold'em Poker. We combined all above improvements and extensions of CFVnet to create our poker AI MuCFVnet, unlocking the potential of deep CFVnet. Experimental results show that MuCFVnet has strong performance, successfully beats some available public multiplayer poker AI.

Finding Nash equilibrium in the domain of imperfect information games as a challenging problem has received much attention. Neural Fictitious Self-Play (NFSP) is a popular model-free machine learning algorithm and has computed approximate Nash equilibrium on such games. However, the deep reinforcement learning method used to approximate the best response in NFSP requires reaching a fully observable Markov state, while the states in imperfect information games are partially observable and non-Markovian, which results in a poor approximation of the best response. Thus, NFSP needs more iterations to converge. In this study, we present a new reinforcement learning method that is inspired by counterfactual regret minimization to relax the Markov requirement by iteratively updating policy according to the regret matching process. Combining this new reinforcement learning algorithm with fictitious play, we further present a novel algorithm to find approximate Nash equilibrium in zero-sum imperfect information games. Experimental results in three benchmark games show that this new algorithm can find approximate Nash equilibrium effectively and converge much faster compared with baseline.

Given an agent with limited sensing capabilities, we analyze whether it is possible to deploy a new agent in the operational space of the preexisting agent in a safe manner. One approach for modeling the interaction of the introduced agent with its environment, which contains the preexisting agent, is through a two-player game of imperfect information. However, the computational cost of solving this game is prohibitive. Restricting the preexisting agent's strategy to just memoryless strategies and assuming that the introduced agent has perfect information alleviates the computational cost while still modeling realistic environments. The proposed algorithm for solving the game finds a winning strategy for the introduced agent by solving a quantified Boolean formula (QBF) for the game. We justify this approach by establishing a matching PSPACE lower bound. We also show that this result holds even when the preexisting agent uses bounded history to condition its play.

Auto-encoder neural network based prediction of Texas poker opponent's behavior

Decision-making problems with two agents can be modeled as two player games, and a Nash equilibrium is the basic solution concept describing good play in adversarial games. Computing this equilibrium solution for imperfect information games, where players have private, hidden information, is harder than solving perfect information games. Both domains may technically be classified as easy, with algorithms that require polynomial time and space, but the difference in degree means that for extremely large games, both computation time and storage requirements may exceed available resources. In this thesis, I present four main contributions towards fast and memory efficient techniques for solving extensive-form games, which describe a class of imperfect information games with sequential decision making. First, the thesis introduces an analysis of counterfactual regret minimisation (CFR), an algorithm for solving extensive-form games, and presents tighter regret bounds that describe the rate of progress. CFR is a popular, state-of-the-art algorithm, and the improved bounds give some indication as to why the algorithm has good practical performance. Second, I describe and analyse a game-solving algorithm, CFR+ , which has faster empirical performance than CFR, and compare it to a number of theoretically faster algorithms. We wrote and released an efficient, distributed implementation of CFR+ to solve heads-up limit Texas hold'em, making it the first competitively played imperfect information game to be solved. Third, the thesis presents a series of theoretical tools for using decomposition, and creating algorithms which operate on small portions of a game at a time. I describe an algorithm, CFR-D, which can solve games without ever storing a complete strategy, and an algorithm for re-solving a portion of a game to complete an incomplete equilibrium within the full game. Both algorithms have a formal guarantee of correctness. Finally, I describe continual resolving, an algorithm which is an imperfect information analogue to heuristic search in perfect information games. Continual re-solving uses decomposition to play games by doing depth-limited solving with a heuristic evaluation, with a theoretical bound on solution quality. We implemented continual re-solving in the game of no-limit heads-up no-limit Texas hold'em, and beat a group of 33 professional players in a human trial.

This research was conducted by an interdisciplinary team of two undergraduate students and a faculty to explore solutions to the Birds of a Feather (BoF) Research Challenge. BoF is a newly-designed perfect-information solitaire-type game. The focus of the study was to design and implement different algorithms and evaluate their effectiveness. The team compared the provided depth-first search (DFS) to heuristic algorithms such as Monte Carlo tree search (MCTS), as well as a novel heuristic search algorithm guided by machine learning. Since all of the studied algorithms converge to a solution from a solvable deal, effectiveness of each approach was measured by how quickly a solution was reached, and how many nodes were traversed until a solution was reached. The employed methods have a potential to provide artificial intelligence enthusiasts with a better understanding of BoF and novel ways to solve perfect-information games and puzzles in general. The results indicate that the proposed heuristic search algorithms guided by machine learning provide a significant improvement in terms of number of nodes traversed over the provided DFS algorithm.