A General Framework for Bandit Problems Beyond Cumulative Objectives

In this paper, we consider the restless bandit problem, which is one of the most well-studied generalizations of the celebrated stochastic multi-armed bandit problem in decision theory. In its ultimate generality, the restless bandit problem is known to be PSPACE-Hard to approximate to any non-trivial factor, and little progress has been made on this problem despite its significance in modeling activity allocation under uncertainty. We make progress on this problem by showing that for an interesting and general subclass that we term Monotone bandits, a surprisingly simple and intuitive greedy policy yields a factor 2 approximation. Such greedy policies are termed index policies, and are popular due to their simplicity and their optimality for the stochastic multi-armed bandit problem. The Monotone bandit problem strictly generalizes the stochastic multi-armed bandit problem, and naturally models multi-project scheduling where the state of a project becomes increasingly uncertain when the project is not scheduled. We develop several novel techniques in the design and analysis of the index policy. Our algorithm proceeds by introducing a novel “balance” constraint to the dual of a well-known LP relaxation to the restless bandit problem. This is followed by a structural characterization of the optimal solution by using both the exact primal as well as dual complementary slackness conditions. This yields an interpretation of the dual variables as potential functions from which we derive the index policy and the associated analysis.

Multi-Armed Bandit (MAB) algorithms are classic algorithms that address sequential decision-making under uncertainty by solving the exploration-exploitation trade-off dilemma. This study investigates the performance comparison of multiple MAB algorithms using a simulated multi-armed bandit machine with Bernoulli reward distribution as the experimental environment. This study compares the performance differences among Upper Confidence Bound (UCB), Thompson Sampling (TS) and -Greedy Thompson Sampling (-TS) in this environment, and attempts to set different parameters, namely the number of arms and the number of experimental rounds and calculate the corresponding cumulative regret of the algorithm under various conditions. The size of the cumulative regret reflects the performance of each algorithm in the simulated slot machine model with rewards that conform to the Bernoulli distribution. In addition, the algorithm running time under the same conditions is also recorded to analyze from the perspective of algorithm efficiency. The experimental results show that under the experimental environment of this study, the cumulative regret produced by the UCB algorithm is more than three times that of the other two algorithms. When the number of trials is small, the cumulative regret generated by the TS algorithm is small, but overall, the performance of the TS algorithm and the -TS algorithm set in this experiment in minimizing the cumulative regret is not much different. However, TS runs in a shorter time under the same conditions. The results of this experiment show that after the number of rounds of experimental operation reaches a large enough number, the operating efficiency of the TS algorithm will be significantly higher than that of the -TS algorithm. TS algorithms have higher randomness, so they show better performance under this experimental condition. The -TS under the parameter setting of this study encourages exploration more in the early stage of the experiment, so it will produce greater cumulative regret than the traditional TS algorithm. In the long run, the performance difference between the two in the multi-armed bandit problem with Bernoulli distribution of rewards is very small. However, the TS algorithm has more advantages in algorithm execution efficiency, so when solving similar problems, the TS algorithm is a better choice.

This paper introduces the upper confidence bound (UCB) applied to the tree (UCT) algorithm for noisy optimization problems. Although there are many studies and effective algorithms regarding optimization problems like PSO and DE, the UCB value enables us to balance exploration and exploitation in the search process and to detect the most probable area containing the optimum solution. Thus, UCB applied to tree can search the area more precisely by dividing the area, and is well known for the bandit problem and computation igo. Under the noisy condition, optimization problems are not so much deterministic problems as probabilistic ones like the bandit problem. Therefore, we propose a novel approach to the situation based on the bandit algorithm as typified by UCT. We performed comparative experiments in continuous optimization problems with additive noise to confirm its convergence speed and ability for best area identification.

Federated learning (FL) is an appealing paradigm for learning a global model among distributed clients while preserving data privacy. Driven by the demand for high-quality user experiences, evaluating the well-trained global model after the FL process is crucial. In this paper, we propose a closed-loop model analytics framework that allows for effective evaluation of the trained global model using clients' local data. To address the challenges posed by system and data heterogeneities in the FL process, we study a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">goal-directed</i> client selection problem based on the model analytics framework by selecting a subset of clients for the model training. This problem is formulated as a stochastic multi-armed bandit (SMAB) problem. We first put forth a quick initial upper confidence bound (Quick-Init UCB) algorithm to solve this SMAB problem under the federated analytics (FA) framework. Then, we further propose a belief propagation-based UCB (BP-UCB) algorithm under the democratized analytics (DA) framework. Moreover, we derive two regret upper bounds for the proposed algorithms, which increase logarithmically over the time horizon. The numerical results demonstrate that the proposed algorithms achieve nearly optimal performance, with a gap of less than 1.44% and 3.12% under the FA and DA frameworks, respectively.

Gap-free Bounds for Stochastic Multi-Armed Bandit

This paper aims to solve the optimization problems in far-field wireless power transfer systems using machine learning techniques. We assembled the RF power transfer robot, which can emit the electromagnetic wave to charge the energy harvesters that are deployed in the experimental field. The wireless transmitter intends to charge all the energy harvesters in a fair manner. Since the energy harvesters can be either stationary or mobile, a multi-armed bandit(MAB) problem is formulated and we use Upper Confidence Bound(UCB) algorithm to determine the optimal transmission strategy. As the number of the transmitters is increased, multiple wireless transmitters coordinate with each other to boost the levels of energy harvesting at all energy harvesters. Correspondingly, we formulate a combinational MAB problem and UCB algorithm is applied to determine the optimal transmission strategy for each transmitter. The simulation results prove the superiority of the Multi-armed bandit approach in solving the proposed optimization problems.

The Multi-Armed Bandit (MAB) problem is a fundamental framework in sequential decision-making. The main goal is to balance exploration of uncertain options with exploitation of known rewards. Originating from classical probability theory, MAB has evolved into a significant modern reinforcement learning. This problem can be applied to all walks of life in reality, from online advertising and recommendation systems to healthcare and finance. Over the years, algorithms such as Explore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS) have emerged as leading strategies, each offering unique trade-offs between exploration efficiency and theoretical regret bounds. This paper gives an overview of the history, main algorithms, applications, and challenges of MAB, and discusses possible future research directions. Furthermore, it highlights the practical implications of these algorithms for real-world decision-making scenarios.

Recently, deep neural networks have been adopted in many application domains showing impressive performances that sometimes even exceed those of humans. However, the considerable size of deep learning models and their computational cost limits their use in mobile and embedded systems. Pruning is an important technique that has been used in the literature to compress a trained model by removing some of its elements (weights, neurons, filters, etc.) that do not notably contribute to its performance. In this paper, we propose a new block pruning method for residual networks that removes holes layers of the network based on the Multi-Armed Bandits (MAB) framework. The proposed method takes advantage of the existence of multiple trainable paths in residual models to identify and prune the less important residual blocks for inference. To evaluate our approach, we conducted extensive experiments using three different MAB algorithms, namely ε -greedy, Win Stay Lose Shift (WSLS), and the Upper Confidence Bound (UCB). The obtained results on different datasets show that our block pruning method can considerably reduce the model size and FLOPs with little to no impact on overall accuracy. We conducted a comparative analysis of our method with three existing techniques to demonstrate that our approach is highly competitive in terms of accuracy, while significantly reducing both model size and FLOPs. Moreover, our experimental results demonstrate that when our block pruning method is combined with a filter-level pruning technique, it leads to even greater reductions in model size and FLOPs while maintaining a relatively high accuracy.

Recommender systems play a critical role in e-commerce, streaming media, and other fields, where user engagement is vital to platform success. These systems, however, face challenges like the cold start problem, which limits recommendation accuracy for new users and items due to sparse data. Although multi-armed bandit (MAB) algorithms are considered effective for balancing exploration and exploitation, existing studies lack simulations of algorithm performance in scenarios where recommender systems indirectly influence user choices, limiting insights for performance improvements in real-world applications. This study explores the performance of key MAB algorithmsExplore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS)under a novel framework that models user behavior as an unreliable intermediate using Markov state transitions. This Markov perturbation-based approach provides a foundation for indirectly influencing user decisions. Two algorithmic frameworks are examined: one considering only the recommendation system's influence on user choices, and another also considering user feedback as a dynamic influence on state transition probabilities. Results show that, while considering user feedback increases adaptability, it also causes greater volatility. TS achieves the lowest cumulative regret with strong stability, while Asymptotic-UCB achieves a better balance between exploration and exploitation than standard UCB, with consistent adaptability under varying feedback. In contrast, ETC demonstrates significant volatility, highlighting its limitations in feedback-driven scenarios. These findings provide valuable insights for enhancing MAB algorithms in real-world cold start recommendation applications, especially where systems influence user choices indirectly.

This article explores the application of Multi-Armed Bandit (MAB) algorithms in optimizing consumer tech products, with a particular focus on eCommerce platforms. It provides a comprehensive article overview of the theoretical framework behind MAB algorithms, including the exploration-exploitation trade-off and comparisons with traditional A/B testing methods. The article delves into various MAB strategies commonly used in eCommerce, such as epsilon-greedy, Upper Confidence Bound (UCB), and Thompson Sampling, and examines their applications in personalized product recommendations, dynamic pricing, ad placement optimization, and website content delivery. Implementation considerations, including integration with existing machine learning infrastructure and data processing in high-throughput scenarios, are discussed in detail. The article also addresses the impact of MAB algorithms on key performance metrics like user engagement, conversion rates, and revenue optimization. Ethical considerations, including transparency in automated decision-making and fairness in consumer-facing applications, are explored. Finally, the article presents case studies of successful implementations, discusses current challenges and limitations, and outlines future directions for MAB algorithms in eCommerce and potential cross-industry applications.

In recent years, the advertising sector has increasingly embraced Multi-armed Bandit Algorithms (MAB) for their versatile applications. This article delves into four stochastic MAB algorithms: the Explore-Then-Commit (ETC), the Upper Confidence Bound (UCB), the Thompson-Sampling (TS), and their respective variants, applied in an online shopping platform's advertising campaign to optimize click-through rates. Our findings indicate that each of these algorithms successfully identifies the most effective advertisement. Both the UCB and TS algorithms demonstrate logarithmic regret relative to the time horizon. The ETC algorithm exhibits a delayed onset of logarithmic regret but surprisingly holds up well against the UCB algorithms. Notably, the TS algorithm class outshines the UCB class due to its inherent randomness, offering a more robust performance. Furthermore, two variants, the UCB algorithm with the Mini-max Optimal Strategy in the Stochastic case (MOSS) and the paralleled TS algorithm, show promising results. They not only maintain logarithmic regret but also enhance efficiency, indicating further potential in the UCB and TS algorithm classes for advanced applications.

Generalized Bandits with Learning and Queueing in Split Liver Transplantation

This paper studies the deviations of the regret in a stochastic multi-armed bandit problem. When the total number of plays n is known beforehand by the agent, Audibert et al. (2009) exhibit a policy such that with probability at least 1-1/n, the regret of the policy is of order logn. They have also shown that such a property is not shared by the popular ucb1 policy of Auer et al. (2002). This work first answers an open question: it extends this negative result to any anytime policy. The second contribution of this paper is to design anytime robust policies for specific multi-armed bandit problems in which some restrictions are put on the set of possible distributions of the different arms.

Online social networking has caused profound changes in the way people communicate and interact. Preserving information privacy is indispensable in such social applications as the shared information would be sensitive. The issue becomes more challenging because of participation of multiple parties on the same shared data. Here propose an efficient trust collection based data sharing technique to allow or disallow the shared resources considering the authorization requirements of all the multiple parties. A logical representation of the proposed data sharing technique is prepared to analyze the privacy of data before sharing to public users. A user in this system is associated with a few trusted users that were selected from the user’s friends. When the user wants to share information to the account, the service provider sends information to the user’s trustees. The user must obtain at least k (i.e., recovery threshold) threshold values from the trustees before being directed to public share. Considering that a user continually posts data items in an OSN, here model the threshold selecting problem as a sequential decision-making problem. More specifically, we formulate the problem as a multi-armed bandit problem and apply the upper confidence bound (UCB) policy to solve the problem. This shows that dynamically adjusting the threshold according to the UCB policy can lead to a higher payoff than using a fixed threshold.

We consider a single-hop wireless network with sources transmitting time-sensitive information to the destination over multiple unreliable channels. Packets from each source are generated according to a stochastic process with known statistics and the state of each wireless channel (ON/OFF) varies according to a stochastic process with unknown statistics. The reliability of the wireless channels is to be learned through observation. At every time-slot, the learning algorithm selects a single pair (source, channel) and the selected source attempts to transmit its packet via the selected channel. The probability of a successful transmission to the destination depends on the reliability of the selected channel. The goal of the learning algorithm is to minimize the Age-of-Information (AoI) in the network over T time-slots. To analyze its performance, we introduce the notion of AoI-regret, which is the difference between the expected cumulative AoI of the learning algorithm under consideration and the expected cumulative AoI of a genie algorithm that knows the reliability of the channels a priori. The AoI-regret captures the penalty incurred by having to learn the statistics of the channels over the T time-slots. The results are two-fold: first, we consider learning algorithms that employ well-known solutions to the stochastic multi-armed bandit problem (such as ϵ-Greedy, Upper Confidence Bound, and Thompson Sampling) and show that their AoI-regret scales as Θ(log T); second, we develop a novel learning algorithm and show that it has O(1) regret. To the best of our knowledge, this is the first learning algorithm with bounded AoI-regret.