A note on the advantage of context in Thompson sampling

Personalized medicine and adaptive clinical trials aim to match optimal treatment plans to individual patients while validating new therapies. Traditional fixed-design trials have limitations, including resource wastage and scarce data. Artificial intelligence has led to the development of dynamic decision algorithms like the Multi-Armed Bandit (MAB) algorithm, which balances exploration and exploitation in treatment allocation. Researchers aim to integrate MAB with clinical needs while adhering to ethical guidelines. This paper discusses the use of Multi-armed Bandit (MAB) algorithms in medicine, highlighting their potential for optimizing treatment allocation and improving patient outcomes, despite ethical constraints and limited data, and their application in contextual and reinforcement learning settings. This research highlight key clinical applications such as adaptive dose-finding, personalized treatment selection, and digital health interventions, supported by both trial-based data and large-scale public datasets like MIMIC-III. Simulation studies are also discussed as a necessary complement to real-world data, facilitating algorithm validation under ethical and logistical constraints. Comparative evaluation of algorithms demonstrates that Bayesian methods, particularly Thompson Sampling and contextual bandits, often provide a more robust balance between efficiency and safety. However, challenges remain in scalability, interpretability, and regulatory acceptance. This research conclude by identifying promising directions for future research, including the integration of deep reinforcement learning and causal inference, which may further enhance the role of MABs in advancing personalized medicine and adaptive clinical trial design.

The importance of multi-armed bandit (MAB) problems is on the rise due to their recent application in a large variety of areas such as online advertising, news article selection, wireless networks, and medicinal trials, to name a few. The most common assumption made when solving such MAB problems is that the unknown reward probability θ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</sup> of each bandit arm k is fixed. However, this assumption rarely holds in practice simply because real-life problems often involve underlying processes that are dynamically evolving. In this paper, we model problems where reward probabilities θ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</sup> are drifting, and introduce a new method called Dynamic Thompson Sampling (DTS) that facilitates Order Statistics based Thompson Sampling for these dynamically evolving MABs. The DTS algorithm adapts its success probability estimates, hat θ <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</sup> , faster than traditional Thompson Sampling schemes and thus leads to improved performance in terms of lower regret. Extensive experiments demonstrate that DTS outperforms current state-of-the-art approaches, namely pure Thompson Sampling, UCB-Normal and UCB <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">f</sub> , for the case of dynamic reward probabilities. Furthermore, this performance advantage increases persistently with the number of bandit arms.

Multi-Armed Bandit (MAB) algorithms are classic algorithms that address sequential decision-making under uncertainty by solving the exploration-exploitation trade-off dilemma. This study investigates the performance comparison of multiple MAB algorithms using a simulated multi-armed bandit machine with Bernoulli reward distribution as the experimental environment. This study compares the performance differences among Upper Confidence Bound (UCB), Thompson Sampling (TS) and -Greedy Thompson Sampling (-TS) in this environment, and attempts to set different parameters, namely the number of arms and the number of experimental rounds and calculate the corresponding cumulative regret of the algorithm under various conditions. The size of the cumulative regret reflects the performance of each algorithm in the simulated slot machine model with rewards that conform to the Bernoulli distribution. In addition, the algorithm running time under the same conditions is also recorded to analyze from the perspective of algorithm efficiency. The experimental results show that under the experimental environment of this study, the cumulative regret produced by the UCB algorithm is more than three times that of the other two algorithms. When the number of trials is small, the cumulative regret generated by the TS algorithm is small, but overall, the performance of the TS algorithm and the -TS algorithm set in this experiment in minimizing the cumulative regret is not much different. However, TS runs in a shorter time under the same conditions. The results of this experiment show that after the number of rounds of experimental operation reaches a large enough number, the operating efficiency of the TS algorithm will be significantly higher than that of the -TS algorithm. TS algorithms have higher randomness, so they show better performance under this experimental condition. The -TS under the parameter setting of this study encourages exploration more in the early stage of the experiment, so it will produce greater cumulative regret than the traditional TS algorithm. In the long run, the performance difference between the two in the multi-armed bandit problem with Bernoulli distribution of rewards is very small. However, the TS algorithm has more advantages in algorithm execution efficiency, so when solving similar problems, the TS algorithm is a better choice.

Unmanned aerial vehicles (UAVs) are expected to be integrated into future wireless networks to offer services, especially in unreachable or congested areas. To improve the spectral efficiency, non-orthogonal multiple access (NOMA) scheme can be utilised within the UAV communication to allow more users to be covered and associated. The performance of the NOMA-UAVs network is governed by several factors including power allocation, user association and pairing methods. This paper presents an approach that uses multi-armed bandit (MAB) and two-sided matching frameworks to maximize the throughput of multi-UAV-assisted NOMA networks in a decentralized manner. The approach enables the UAVs to propose to the ground users (GUs) without explicit cooperation among the UAVs while the GUs can accept or reject the proposals. To this end, we propose a modified Thompson sampling algorithm that we named decaying epsilon Thompson sampling (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula>) MAB algorithm that is designed to improve the exploration-exploitation tradeoff in the MAB. The performance of the proposed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> MAB algorithm is evaluated against other existing MAB techniques. Simulation results show that the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> algorithm attains faster convergence and improved performance in terms of smaller regret and increased achievable system throughput. The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> MAB algorithm particularly excels in regards of the convergence rate when the number of available action spaces increases.

This article explores the application of Multi-Armed Bandit (MAB) algorithms in optimizing consumer tech products, with a particular focus on eCommerce platforms. It provides a comprehensive article overview of the theoretical framework behind MAB algorithms, including the exploration-exploitation trade-off and comparisons with traditional A/B testing methods. The article delves into various MAB strategies commonly used in eCommerce, such as epsilon-greedy, Upper Confidence Bound (UCB), and Thompson Sampling, and examines their applications in personalized product recommendations, dynamic pricing, ad placement optimization, and website content delivery. Implementation considerations, including integration with existing machine learning infrastructure and data processing in high-throughput scenarios, are discussed in detail. The article also addresses the impact of MAB algorithms on key performance metrics like user engagement, conversion rates, and revenue optimization. Ethical considerations, including transparency in automated decision-making and fairness in consumer-facing applications, are explored. Finally, the article presents case studies of successful implementations, discusses current challenges and limitations, and outlines future directions for MAB algorithms in eCommerce and potential cross-industry applications.

While most chapters in this book cover situations in which data is available to draw inferences from, this chapter focuses on situations in which the learner selects actions that will generate new data which allow for future learning. Critically, it is assumed that only the data associated with the action taken by the learner is revealed: the potential outcomes of alternative actions are not disclosed. Generally, such sequential learning approaches are studied under the heading of reinforcement learning, a framework which we briefly introduce. Next, we discuss an extremely common, and useful, special case of reinforcement learning called the contextual multi-armed bandit (cMAB) problem. The cMAB problem formalizes the type of sequential experimentation encountered when, e.g., trying to choose between competing medical treatments or trying to select web content. We will describe several real-world problems that can be studied as a cMAB problem, and we will review effective solutions to this problem such as the UCB algorithm, Thompson sampling, and bootstrapped Thompson sampling. We focus on application but provide references for readers interested in the underlying theoretical results. Our focus on application is further amplified by our discussion of two recently developed software packages designed to experiment with (or solve) bandit problems. First, we detail contextual, an elaborate [R] package that allows users to quickly run simulations of various solutions to bandit problems and conduct offline policy evaluation. Second, we discuss streaming bandit, a popular python package to execute bandit policies in the field. This chapter should allow readers to get started with sequential experimentation methods in their own field or application area of interest.

Recommender systems play a critical role in e-commerce, streaming media, and other fields, where user engagement is vital to platform success. These systems, however, face challenges like the cold start problem, which limits recommendation accuracy for new users and items due to sparse data. Although multi-armed bandit (MAB) algorithms are considered effective for balancing exploration and exploitation, existing studies lack simulations of algorithm performance in scenarios where recommender systems indirectly influence user choices, limiting insights for performance improvements in real-world applications. This study explores the performance of key MAB algorithmsExplore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS)under a novel framework that models user behavior as an unreliable intermediate using Markov state transitions. This Markov perturbation-based approach provides a foundation for indirectly influencing user decisions. Two algorithmic frameworks are examined: one considering only the recommendation system's influence on user choices, and another also considering user feedback as a dynamic influence on state transition probabilities. Results show that, while considering user feedback increases adaptability, it also causes greater volatility. TS achieves the lowest cumulative regret with strong stability, while Asymptotic-UCB achieves a better balance between exploration and exploitation than standard UCB, with consistent adaptability under varying feedback. In contrast, ETC demonstrates significant volatility, highlighting its limitations in feedback-driven scenarios. These findings provide valuable insights for enhancing MAB algorithms in real-world cold start recommendation applications, especially where systems influence user choices indirectly.

Thompson Sampling provides an efficient technique to introduce prior knowledge in the multi-armed bandit problem, along with providing remarkable empirical performance. In this paper, we revisit the Thompson Sampling algorithm under rewards drawn from symmetric alpha-stable distributions, which are a class of heavy-tailed probability distributions utilized in finance and economics, in problems such as modeling stock prices and human behavior. We present an efficient framework for posterior inference, which leads to two algorithms for Thompson Sampling in this setting. We prove finite-time regret bounds for both algorithms, and demonstrate through a series of experiments the stronger performance of Thompson Sampling in this setting. With our results, we provide an exposition of symmetric alpha-stable distributions in sequential decision-making, and enable sequential Bayesian inference in applications from diverse fields in finance and complex systems that operate on heavy-tailed features.

We consider the problem of transmitting at the optimal rate over a rapidly-varying wireless channel with unknown statistics when the feedback about channel quality is very limited. One motivation for this problem is that, in emerging wireless networks, the use of mm Wave bands means that the channel quality can fluctuate rapidly and thus, one cannot rely on full channel-state feedback to make transmission rate decisions. Inspired by related problems in the context of multi-armed bandits, we consider a well-known algorithm called Thompson sampling to address this problem. However, unlike the traditional multi-armed bandit problem, a direct application of Thompson sampling results in a computational and storage complexity that grows exponentially with time. Therefore, we propose an algorithm called Modified Thompson sampling (MTS), whose computational and storage complexity is simply linear in the number of channel states and which achieves at most logarithmic regret as a function of time when compared to an optimal algorithm which knows the probability distribution of the channel states.

Non-stationary multi-armed bandit (MAB) problems have recently attracted extensive attention. We focus on the abruptly changing scenario where reward distributions remain constant for a certain period and change at unknown time steps. Although Thompson sampling (TS) has shown success in non-stationary settings, there is currently no regret bound analysis for TS with uninformative priors. To address this, we propose two algorithms, discounted TS and sliding-window TS, designed for sub-Gaussian reward distributions. For these algorithms, we establish an upper bound for the expected regret by bounding the expected number of times a suboptimal arm is played. We show that the regret upper bounds of both algorithms are O~(TBT), where T is the time horizon and BT is the number of breakpoints. This upper bound matches the lower bound for abruptly changing problems up to a logarithmic factor. Empirical comparisons with other non-stationary bandit algorithms highlight the competitive performance of our proposed methods.

The Multi-Armed Bandit (MAB) problem is a fundamental framework in sequential decision-making. The main goal is to balance exploration of uncertain options with exploitation of known rewards. Originating from classical probability theory, MAB has evolved into a significant modern reinforcement learning. This problem can be applied to all walks of life in reality, from online advertising and recommendation systems to healthcare and finance. Over the years, algorithms such as Explore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS) have emerged as leading strategies, each offering unique trade-offs between exploration efficiency and theoretical regret bounds. This paper gives an overview of the history, main algorithms, applications, and challenges of MAB, and discusses possible future research directions. Furthermore, it highlights the practical implications of these algorithms for real-world decision-making scenarios.

With many gaming AI being developed and being able to defeat top human players in recent years, AI has once again become the hottest topic in research and even in our daily life. This paper also researches on gaming AI and chess. Instead of using deep learning and the Monte Carlo Search algorithm, this paper focuses on the opening only with multi-armed bandit algorithms to find the best moves and opening. Specifically, the method used in this paper is epsilon greedy and Thompson sampling. The dataset used in this paper is from Kaggle. This paper considers each move as a set of choices one needs to make and considers the big picture as a multi-armed bandit problem. This paper aims to develop a relative best strategy to counter those opening or make changes to a disadvantaged situation.

Self-accelerated Thompson sampling with near-optimal regret upper bound

Collaborative bandit learning, i.e., bandit algorithms that utilize collaborative filtering techniques to improve sample efficiency in online interactive recommendation, has attracted much research attention as it enjoys the best of both worlds. However, all existing collaborative bandit learning solutions impose a stationary assumption about the environment, i.e., both user preferences and the dependency among users are assumed static over time. Unfortunately, this assumption hardly holds in practice due to users' ever-changing interests and dependency relations, which inevitably costs a recommender system sub-optimal performance in practice. In this work, we develop a collaborative dynamic bandit solution to handle a changing environment for recommendation. We explicitly model the underlying changes in both user preferences and their dependency relation as a stochastic process. Individual user's preference is modeled by a mixture of globally shared contextual bandit models with a Dirichlet process prior. Collaboration among users is thus achieved via Bayesian inference over the global bandit models. To balance exploitation and exploration during the interactions, Thompson sampling is used for both model selection and arm selection. Our solution is proved to maintain a standard ~O(√T) Bayesian regret in this challenging environment. Extensive empirical evaluations on both synthetic and real-world datasets further confirmed the necessity of modeling a changing environment and our algorithm's practical advantages against several state-of-the-art online learning solutions.

We study multi-armed bandit (MAB) problems with additional observations, where in each round, the decision maker selects an arm to play and can also observe rewards of additional arms (within a given budget) by paying certain costs. In the case of stochastic rewards, we develop a new algorithm KL-UCB-AO which is asymptotically optimal when the time horizon grows large, by smartly identifying the optimal set of the arms to be explored using the given budget of additional observations. In the case of adversarial rewards, we propose H-INF, an algorithm with order-optimal regret. H-INF exploits a two-layered structure where in each layer, we run a known optimal MAB algorithm. Such a hierarchical structure facilitates the regret analysis of the algorithm, and in turn, yields order-optimal regret. We apply the framework of MAB with additional observations to the design of rate adaptation schemes in 802.11-like wireless systems, and to that of online advertisement systems. In both cases, we demonstrate that our algorithms leverage additional observations to significantly improve the system performance. We believe the techniques developed in this paper are of independent interest for other MAB problems, e.g., contextual or graph-structured MAB.