Wavefront-MCTS

The goal of search result diversification is to select a subset of documents from the candidate set to satisfy as many different subtopics as possible. In general, it is a problem of subset selection and selecting an optimal subset of documents is NP-hard. Existing methods usually formalize the problem as ranking the documents with greedy sequential document selection. At each of the ranking position the document that can provide the largest amount of additional information is selected. It is obvious that the greedy selections inevitably produce suboptimal rankings. In this paper we propose to partially alleviate the problem with a Monte Carlo tree search (MCTS) enhanced Markov decision process (MDP), referred to as M$^2$Div. In M$^2$Div, the construction of diverse ranking is formalized as an MDP process where each action corresponds to selecting a document for one ranking position. Given an MDP state which consists of the query, selected documents, and candidates, a recurrent neural network is utilized to produce the policy function for guiding the document selection and the value function for predicting the whole ranking quality. The produced raw policy and value are then strengthened with MCTS through exploring the possible rankings at the subsequent positions, achieving a better search policy for decision-making. Experimental results based on the TREC benchmarks showed that M$^2$Div can significantly outperform the state-of-the-art baselines based on greedy sequential document selection, indicating the effectiveness of the exploratory decision-making mechanism in M$^2$Div.

The Rubik's Cube is a complex puzzle with an enormous number of possible configurations, making it a challenging problem for both humans and computational methods to solve. While traditional solving algorithms rely on predefined strategies, this study explores the application of reinforcement learning (RL) to develop an adaptive and efficient solution model. This study aims to create an RL_based solver using the Markov Decision Process (MDP) framework, optimizing for speed, move efficiency, and solving steps. The proposed model employs Q-learning and Monte Carlo Tree Search (MCTS) to determine the optimal actions at each game state, which are trained through extensive Rubik's Cube simulations. The key novelty of this study lies in the integration of MCTS with Q-learning to enhance decision-making efficiency by reducing the number of moves compared with conventional methods. The experimental results demonstrate that the model achieves near-optimal solutions with fewer moves, outperforming basic rule-based approaches. Additionally, a web-based application was developed to provide real-time solving strategies based on user-input cube configurations. This study contributes to the advancement of RL applications in combinatorial puzzles and offers a practical tool for Rubik's Cube enthusiasts seeking to improve their solving techniques.

This paper introduces a novel model for online dynamic pricing of electric vehicle charging services that integrates reservation, parking, and charging into a comprehensive bundle priced as a whole. Our approach focuses on the individual high-demand, fast-charging location, employing a Poisson process as a model of charging reservation arrivals, and develops an online dynamic pricing strategy optimized through a Markov Decision Process (MDP). A key contribution is the novel analysis of discretization error introduced when incorporating the continuous-time Poisson process into the discrete MDP framework. The MDP model’s feasibility is demonstrated with a heuristic dynamic pricing method based on Monte-Carlo tree search, offering a viable path for real-world applications.

Although the development of AI in games is remarkable, intelligent machines still lag behind humans in games that require the ability of language understanding. In this paper, we focus on the crossword puzzle resolution task. Solving crossword puzzles is a challenging task since it requires the ability to understand natural language and the ability to execute a search over possible answers to find an optimal set of solutions for the grid. Previous solutions are devoted to exploiting heuristic strategies in search to find solutions while having limited ability to explore the search space. We propose a solution for crossword puzzle resolution based on Monte Carlo tree search (MCTS). As far as we know, we are the first to model the crossword puzzle resolution problem as a Markov Decision Process and apply the MCTS to solve it. We construct a dataset for crossword puzzle resolution based on daily puzzles from New York Times with detailed specifications on both the puzzle and clue database selection. Our method can achieve an accuracy of 97% on the dataset.

AlphaGo and its successors AlphaGo Zero and AlphaZero made international headlines with their incredible successes in game playing, which have been touted as further evidence of the immense potential of artificial intelligence, and in particular, machine learning. AlphaGo defeated the reigning human world champion Go player Lee Sedol 4 games to 1, in March 2016 in Seoul, Korea, an achievement that surpassed previous computer game-playing program milestones by IBM’s Deep Blue in chess and by IBM’s Watson in the U.S. TV game show Jeopardy. AlphaGo then followed this up by defeating the world’s number one Go player Ke Jie 3-0 at the Future of Go Summit in Wuzhen, China in May 2017. Then, in December 2017, AlphaZero stunned the chess world by dominating the top computer chess program Stockfish (which has a far higher rating than any human) in a 100-game match by winning 28 games and losing none (72 draws) after training from scratch for just four hours! The deep neural networks of AlphaGo, AlphaZero, and all their incarnations are trained using a technique called Monte Carlo tree search (MCTS), whose roots can be traced back to an adaptive multistage sampling (AMS) simulation-based algorithm for Markov decision processes (MDPs) published in Operations Research back in 2005 [Chang, HS, MC Fu, J Hu and SI Marcus (2005). An adaptive sampling algorithm for solving Markov decision processes. Operations Research, 53, 126–139.] (and introduced even earlier in 2002). After reviewing the history and background of AlphaGo through AlphaZero, the origins of MCTS are traced back to simulation-based algorithms for MDPs, and its role in training the neural networks that essentially carry out the value/policy function approximation used in approximate dynamic programming, reinforcement learning, and neuro-dynamic programming is discussed, including some recently proposed enhancements building on statistical ranking & selection research in the operations research simulation community.

Markov Decision Processes, AlphaGo, and Monte Carlo Tree Search: Back to the Future

Optimising stochastic task allocation and scheduling plans for mission workers subject to learning-forgetting, fatigue-recovery, and stress-recovery effects

Monte-Carlo Tree Search and specifically the variants of the UCT algorithm have been a break-through in AI of the board game Go. However, UCT has had limited applicability to other domains. We study the limitations of some of the existing variants of UCT in a small-scale Markov decision process (MDP), and propose new variants that can reduce those limitations. Our experiments show great improvements in performance against traditional UCT and comparable performance to estabilished reinforcement learning algorithm, thus opening possibilities for applying UCT in other problem domains.

Task allocation is an important problem in multi-robot systems, particularly in dynamic and unpredictable environments such as offshore oil platforms, large-scale factories, or disaster response scenarios, where high change rates, uncertain state transitions, and varying task demands challenge the predictability and stability of robot operations. Traditional static task allocation strategies often struggle to meet the efficiency and responsiveness demands of these complex settings, while optimization heuristics, though improving planning time, exhibit limited scalability. To address these limitations, this paper proposes a task allocation method based on the Monte Carlo Tree Search (MCTS) algorithm, which leverages the anytime property of MCTS to achieve a balance between fast response and continuous optimization. Firstly, the centralized adaptive MCTS algorithm generates preliminary solutions and monitors the state of the robots in minimal time. It utilizes dynamic Upper Confidence Bounds for Trees (UCT) values to accommodate varying task dimensions, outperforming the heuristic Multi-Robot Goal Assignment (MRGA) method in both planning time and overall task completion time. Furthermore, the parallelized distributed MCTS algorithm reduces algorithmic complexity and enhances computational efficiency through importance sampling and parallel processing. Experimental results demonstrate that the proposed method significantly reduces computation time while maintaining task allocation performance, decreasing the variance of planning results and improving algorithmic stability. Our approach enables more flexible and efficient task allocation in dynamically evolving and complex environments, providing robust support for the deployment of multi-robot systems.

Coarse-grained reconfigurable architectures (CGRAs) are gaining increasing attention as domain-specific accelerators due to their high flexibility and energy efficiency. These architectures offer a compelling solution for applications that require custom hardware performance while retaining a degree of programmability. However, the design space of CGRAs is inherently vast and complex, presenting significant challenges for architects to explore design choices efficiently and systematically. Existing design space exploration (DSE) methodologies for CGRAs are often time-demanding and struggle to deliver optimal solutions when confronted with high-dimensional and multi-objective design space. Therefore, we consider constructing a CGRA parameter tuning framework called MoDAF. MoDAF initializes the design space using the most representative and diverse samples. It adopts a divide-and-conquer approach, utilizing Monte Carlo Tree Search (MCTS) and space partitioning techniques to dynamically break down the complex design space into more manageable subspaces. A hybrid model handles local fluctuations within each subspace, while a dual sampling algorithm is designed to increase sampling efficiency. MoDAF also incorporates a fast evaluation model to estimate CGRA throughput and area, significantly speeding up the exploration process. Compared with previous approaches, experiments show that our proposed framework reduces the average distance from the reference set by 53.0% and the hypervolume deviation by 64.2%, while also cutting wall time by 57.5%.

Since the emergence of Artificial Intelligence as a field, planning under uncertainty has been viewed as one of its crucial subareas. Accordingly, the current lack of scalability of probabilistic planning techniques is a major reason why the grand vision of AI has not been fulfilled yet. Besides being an obstacle to advances in AI, scalability issues also hamper the applicability of probabilistic planning to real-world problems. A powerful framework for describing probabilistic planning problems is Markov Decision Processes (MDPs). Informally, an MDP specifies the objectives the agent is trying to achieve, the actions the agent can perform, and the states in which it can end up while working towards the objective. Solving an MDP means finding a policy, i.e. an assignment of actions to states, that allows the agent to achieve the objective. Optimal solution methods, those that look for the “best” policy according to some criterion, typically try to analyze all possible states or a large fraction of them. Since state space sizes of realistic scenarios can be astronomical, these algorithms quickly run out of memory. Fortunately, the mathematical structure of some classes of MDPs has allowed for inventing more efficient algorithms. This is the case for Stochastic Shortest Path (SSP) problems, whose mathematical properties gave rise to a family of algorithms called Find-and-Revise. When used in combination with a heuristic, the members of this family can find a near-optimal, or even optimal, policy while avoiding the analysis of many states. Nonetheless, the sheer number of states in real-world problems forces algorithms based on state-level analysis to exhaust their capabilities much too early, calling for a fundamentally different approach. Moreover, several expressive and potentially useful MDP classes are currently not known to have a mathematical structure for creating efficient approximation techniques. This dissertation advances the state of the art in probabilistic planning in three complementary ways. For SSP MDPs, it derives a novel class of approximate algorithms based on generalizing state analysis information across many states. This information is stored in the form of basis functions that are generated automatically and the number of which is much smaller than the number of states. As a result, the proposed algorithms have a very compact memory footprint and arrive at high-quality solutions faster than their state-based counterparts. In a parallel effort, the dissertation also builds up mathematical apparatus for classes of MDPs that previously had no applicable approximation algorithms. The developed theory enables the extension of the powerful Find-and-Revise paradigm to these MDP classes as well, providing the first memory-efficient algorithms for solving them. Last but not least, the dissertation will apply the proposed theoretical techniques to a large-scale real-world problem, urban traffic routing being one of the candidates.

An increasing number of devices, from coffee makers to electric kettles, are becoming connected to the Internet. These are all a part of the Internet of Things, or IoT. Each device generates unique network traffic and power consumption patterns. Until now, there has not been a comprehensive set of data that captures these traffic and power patterns. This thesis documents how we collected 10 to 15 weeks of network traffic and power consumption data from 15 different IoT devices and provides an analysis of a subset of 6 devices. Devices including an Amazon Echo Dot, Google Home Mini, and Google Chromecast were used on a regular basis and all of their network traffic and power consumption was logged to a MySQL database. The database currently contains 64 million packets and 71 gigabytes of data and is still growing in size as more data is collected 24/7 from each device. We show that it is possible to see when users are asking their smart speaker a question or whether the lights in their home are on or off based on power consumption and network traffic from the devices. These trends can be seen even if the data being sent is encrypted.

A regression-based design space exploration methodology is proposed that models the impacts of the memory hierarchy and the network-on-chip (NoC) on the overall chip multiprocessor (CMP) performance. Designers cannot explore all possible designs for a NoC without considering interactions with the rest of the uncore, in particular the cache configuration and memory hierarchy which determine the amount and pattern of the traffic on the NoC. The proposed regression model is able to capture the salient design points of the uncore for a comprehensive design space exploration by designing memory and NoC-specific regression models and leveraging recent advances in uncore simulation. To show the utility of our methodology, Uncore RPD, two case studies are presented: i) analyzing and refining regression models for an 8-core CMP and ii) performing a rapid design space exploration to find best performing designs of a NoC-based CMP given area-constraints for CMPs of up to 64 cores. Through these case studies, it is shown that i) simultaneous consideration of the memory and NoC parameters in the NoC design space exploration can refine uncore-based regression models, ii) sampling techniques must consider the dynamic design space of the uncore, and iii) overall, the proposed regression models reduce the amount of simulations required to characterize the NoC design space by up to four orders of magnitude.

Networks-on-Chip (NoC) with low-radix switches forming a simple and planar topology is typically accepted as the right interconnection infrastructure for current Chip Multi Processor and high-end Multi Processor System-on-Chip. This is mainly due to its simplicity in the physical mapping on the chip. However, as the network diameter increases, latency and power consumption are increased due to the rapidly growing queuing delay in each switch do not scale with system size. In this context, topologies with high-radix switches have been recently proposed in the NoC scenario to keep message latency low when interconnecting a large number of devices. However, the use of high-radix switches present several well-known drawbacks, being the most important the scalability in area and frequency when implemented. In addition, average and maximum wire length is increased. In this paper we present a distributed crossbar NoC architecture that reduces network latency, increases network throughput significantly. For the distributed crossbar implementation trees of 2-to-1 multiplexers with arbitration and buffer capabilities are spread over the chip, avoiding the negative impact of a high radix switch degree on NoC operating frequency, and minimizing the impact of long wires. Results show that in a 64-node NoC our most aggressive distributed crossbar configuration reduces flit latency by 42% and increases throughput by 544% with respect to the low latency flattened butterfly architecture, meanwhile area is increased a 110%. A more conservative distributed crossbar configuration obtains an increment in throughput of 276.1%, latency is decreased a 29.7%, but area is also decreased by 7%.

Symbolic Execution is a widely used technique for program testing and analysis. When a program executes a trace symbolically, it simulates all possible paths. This results in an exponential growth of the number of states within the problem, which is commonly referred to as path explosion. We therefore propose novel strategies that only require limited resources to give priority to more valuable paths. In this paper, we utilize a method based on the Monte Carlo tree search (MCTS) strategy to resolve the problem. We then compare the proposed MCTS-based strategy with other methods such as depth-first search (DFS) and breadth-first search (BFS). We also perform different scales of experiments based on time and space resource constraints. For smaller test cases, we found that MCTS performs on average 1.4 times better than BFS and DFS in terms of the block discovery rate. In addition, for larger test cases, MCTS performs on average 2.8 times better than DFS and BFS in terms of the block discovery rate.