Piggy-back Network to Enable Beyond 5G Society Supported by Autonomous Mobilities: Application of MAB Algorithm for Data Synchronization Between Distributed Collaborative Robots

This article explores the application of Multi-Armed Bandit (MAB) algorithms in optimizing consumer tech products, with a particular focus on eCommerce platforms. It provides a comprehensive article overview of the theoretical framework behind MAB algorithms, including the exploration-exploitation trade-off and comparisons with traditional A/B testing methods. The article delves into various MAB strategies commonly used in eCommerce, such as epsilon-greedy, Upper Confidence Bound (UCB), and Thompson Sampling, and examines their applications in personalized product recommendations, dynamic pricing, ad placement optimization, and website content delivery. Implementation considerations, including integration with existing machine learning infrastructure and data processing in high-throughput scenarios, are discussed in detail. The article also addresses the impact of MAB algorithms on key performance metrics like user engagement, conversion rates, and revenue optimization. Ethical considerations, including transparency in automated decision-making and fairness in consumer-facing applications, are explored. Finally, the article presents case studies of successful implementations, discusses current challenges and limitations, and outlines future directions for MAB algorithms in eCommerce and potential cross-industry applications.

Unmanned aerial vehicles (UAVs) are expected to be integrated into future wireless networks to offer services, especially in unreachable or congested areas. To improve the spectral efficiency, non-orthogonal multiple access (NOMA) scheme can be utilised within the UAV communication to allow more users to be covered and associated. The performance of the NOMA-UAVs network is governed by several factors including power allocation, user association and pairing methods. This paper presents an approach that uses multi-armed bandit (MAB) and two-sided matching frameworks to maximize the throughput of multi-UAV-assisted NOMA networks in a decentralized manner. The approach enables the UAVs to propose to the ground users (GUs) without explicit cooperation among the UAVs while the GUs can accept or reject the proposals. To this end, we propose a modified Thompson sampling algorithm that we named decaying epsilon Thompson sampling (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula>) MAB algorithm that is designed to improve the exploration-exploitation tradeoff in the MAB. The performance of the proposed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> MAB algorithm is evaluated against other existing MAB techniques. Simulation results show that the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> algorithm attains faster convergence and improved performance in terms of smaller regret and increased achievable system throughput. The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> MAB algorithm particularly excels in regards of the convergence rate when the number of available action spaces increases.

Personalization has become a focal point of modern revenue management. However, it is often the case that minimal data are available to appropriately make suggestions tailored to each customer. This has led to many products making use of reinforcement learning-based algorithms to explore sets of offerings to find the best suggestions to improve conversion and revenue. Arguably the most popular of these algorithms are built on the foundation of the multi-arm bandit framework, which has shown great success across a variety of use cases. A general multi-arm bandit algorithm aims to trade-off adaptively exploring available, but under observed, recommendations, with the current known best offering. While much success has been achieved with these relatively understandable procedures, much of the airline industry is losing out on better personalized offers by ignoring the context of the transaction, as is the case in the traditional multi-arm bandit setup. Here, we explore a popular exploration heuristic, Thompson sampling, and note implementation details for multi-arm and contextual bandit variants. While the contextual bandit requires greater computational and technical complexity to include contextual features in the decision process, we illustrate the value it brings by the improvement in overall expected

This paper aims to solve the optimization problems in far-field wireless power transfer systems using machine learning techniques. We assembled the RF power transfer robot, which can emit the electromagnetic wave to charge the energy harvesters that are deployed in the experimental field. The wireless transmitter intends to charge all the energy harvesters in a fair manner. Since the energy harvesters can be either stationary or mobile, a multi-armed bandit(MAB) problem is formulated and we use Upper Confidence Bound(UCB) algorithm to determine the optimal transmission strategy. As the number of the transmitters is increased, multiple wireless transmitters coordinate with each other to boost the levels of energy harvesting at all energy harvesters. Correspondingly, we formulate a combinational MAB problem and UCB algorithm is applied to determine the optimal transmission strategy for each transmitter. The simulation results prove the superiority of the Multi-armed bandit approach in solving the proposed optimization problems.

Multi-Armed Bandit (MAB) algorithms are classic algorithms that address sequential decision-making under uncertainty by solving the exploration-exploitation trade-off dilemma. This study investigates the performance comparison of multiple MAB algorithms using a simulated multi-armed bandit machine with Bernoulli reward distribution as the experimental environment. This study compares the performance differences among Upper Confidence Bound (UCB), Thompson Sampling (TS) and -Greedy Thompson Sampling (-TS) in this environment, and attempts to set different parameters, namely the number of arms and the number of experimental rounds and calculate the corresponding cumulative regret of the algorithm under various conditions. The size of the cumulative regret reflects the performance of each algorithm in the simulated slot machine model with rewards that conform to the Bernoulli distribution. In addition, the algorithm running time under the same conditions is also recorded to analyze from the perspective of algorithm efficiency. The experimental results show that under the experimental environment of this study, the cumulative regret produced by the UCB algorithm is more than three times that of the other two algorithms. When the number of trials is small, the cumulative regret generated by the TS algorithm is small, but overall, the performance of the TS algorithm and the -TS algorithm set in this experiment in minimizing the cumulative regret is not much different. However, TS runs in a shorter time under the same conditions. The results of this experiment show that after the number of rounds of experimental operation reaches a large enough number, the operating efficiency of the TS algorithm will be significantly higher than that of the -TS algorithm. TS algorithms have higher randomness, so they show better performance under this experimental condition. The -TS under the parameter setting of this study encourages exploration more in the early stage of the experiment, so it will produce greater cumulative regret than the traditional TS algorithm. In the long run, the performance difference between the two in the multi-armed bandit problem with Bernoulli distribution of rewards is very small. However, the TS algorithm has more advantages in algorithm execution efficiency, so when solving similar problems, the TS algorithm is a better choice.

Simulations have shown multi-armed bandit (MAB) algorithms to be suitable for optimizing channel hopping in IEEE 802.15.4 networks. Thus far, however, there appears to be no practical implementation of this approach, presumably because typical IEEE 802.15.4 nodes lack both floating-point unit (FPUs) and big amounts of random-access memory (RAM). In this paper, we propose fixed-point arithmetic and implementation shortcuts to circumvent these constraints. We focus on two specific multi-armed bandit (MAB) algorithms, namely sliding-window upper confidence bound (SW-UCB) and its predecessor discounted UCB (D-UCB). SW-UCB is particularly promising since it requires only tractable fixed-point arithmetic, while yielding high packet delivery ratios (PDRs) according to prior work. D-UCB, on the other hand, additionally opens up an implementation shortcut that saves RAM. Our implementations of SW-UCB and D-UCB are integrated into Contiki-NG, yet can also be used out-of-tree in a simulation environment. We show our SW-UCB (resp. D-UCB) implementation to attain PDRs of 98.6% (resp. 99.2%) under appropriate parameter settings in the context of intra-body communication. Also, we demonstrate D-UCB to incur a moderate RAM, program memory, and processing overhead on CC2538 SoCs, whereas we find SW-UCB too RAM-consuming for these chips. Finally, using Monte Carlo simulations, we show our SW-UCB and D-UCB implementations to perform equally well as floating-point counterparts.

The multi-armed bandit (MAB) model is one of the most classical models to study decision-making in an uncertain environment. In this model, a player chooses one of K possible arms of a bandit machine to play at each time step, where the corresponding arm returns a random reward to the player, potentially from a specific unknown distribution. The target of the player is to collect as many rewards as possible during the process. Despite its simplicity, the MAB model offers an excellent playground for studying the trade-off between exploration vs exploitation and designing effective algorithms for sequential decision-making under uncertainty. Although many asymptotically optimal algorithms have been established, the finite-time behaviors of the stochastic dynamics of the MAB model appear much more challenging to analyze due to the intertwine between the decision-making and the rewards being collected. In this paper, we employ techniques in statistical physics to analyze the MAB model, which facilitates the characterization of the distribution of cumulative regrets at a finite short time, the central quantity of interest in an MAB algorithm, as well as the intricate dynamical behaviors of the model. Our analytical results, in good agreement with simulations, point to the emergence of an interesting multimodal regret distribution, with large regrets resulting from excess exploitation of sub-optimal arms due to an initial unlucky output from the optimal one.

Recommender systems play a critical role in e-commerce, streaming media, and other fields, where user engagement is vital to platform success. These systems, however, face challenges like the cold start problem, which limits recommendation accuracy for new users and items due to sparse data. Although multi-armed bandit (MAB) algorithms are considered effective for balancing exploration and exploitation, existing studies lack simulations of algorithm performance in scenarios where recommender systems indirectly influence user choices, limiting insights for performance improvements in real-world applications. This study explores the performance of key MAB algorithmsExplore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS)under a novel framework that models user behavior as an unreliable intermediate using Markov state transitions. This Markov perturbation-based approach provides a foundation for indirectly influencing user decisions. Two algorithmic frameworks are examined: one considering only the recommendation system's influence on user choices, and another also considering user feedback as a dynamic influence on state transition probabilities. Results show that, while considering user feedback increases adaptability, it also causes greater volatility. TS achieves the lowest cumulative regret with strong stability, while Asymptotic-UCB achieves a better balance between exploration and exploitation than standard UCB, with consistent adaptability under varying feedback. In contrast, ETC demonstrates significant volatility, highlighting its limitations in feedback-driven scenarios. These findings provide valuable insights for enhancing MAB algorithms in real-world cold start recommendation applications, especially where systems influence user choices indirectly.

We study multi-armed bandit (MAB) problems with additional observations, where in each round, the decision maker selects an arm to play and can also observe rewards of additional arms (within a given budget) by paying certain costs. In the case of stochastic rewards, we develop a new algorithm KL-UCB-AO which is asymptotically optimal when the time horizon grows large, by smartly identifying the optimal set of the arms to be explored using the given budget of additional observations. In the case of adversarial rewards, we propose H-INF, an algorithm with order-optimal regret. H-INF exploits a two-layered structure where in each layer, we run a known optimal MAB algorithm. Such a hierarchical structure facilitates the regret analysis of the algorithm, and in turn, yields order-optimal regret. We apply the framework of MAB with additional observations to the design of rate adaptation schemes in 802.11-like wireless systems, and to that of online advertisement systems. In both cases, we demonstrate that our algorithms leverage additional observations to significantly improve the system performance. We believe the techniques developed in this paper are of independent interest for other MAB problems, e.g., contextual or graph-structured MAB.

Personalized medicine and adaptive clinical trials aim to match optimal treatment plans to individual patients while validating new therapies. Traditional fixed-design trials have limitations, including resource wastage and scarce data. Artificial intelligence has led to the development of dynamic decision algorithms like the Multi-Armed Bandit (MAB) algorithm, which balances exploration and exploitation in treatment allocation. Researchers aim to integrate MAB with clinical needs while adhering to ethical guidelines. This paper discusses the use of Multi-armed Bandit (MAB) algorithms in medicine, highlighting their potential for optimizing treatment allocation and improving patient outcomes, despite ethical constraints and limited data, and their application in contextual and reinforcement learning settings. This research highlight key clinical applications such as adaptive dose-finding, personalized treatment selection, and digital health interventions, supported by both trial-based data and large-scale public datasets like MIMIC-III. Simulation studies are also discussed as a necessary complement to real-world data, facilitating algorithm validation under ethical and logistical constraints. Comparative evaluation of algorithms demonstrates that Bayesian methods, particularly Thompson Sampling and contextual bandits, often provide a more robust balance between efficiency and safety. However, challenges remain in scalability, interpretability, and regulatory acceptance. This research conclude by identifying promising directions for future research, including the integration of deep reinforcement learning and causal inference, which may further enhance the role of MABs in advancing personalized medicine and adaptive clinical trial design.

Dynamic spectrum access (DSA) is an emerging notion in cognitive radio, aiming to improve the spectrum usage with reliable secondary access to the spectral resources. The main challenge in DSA is the detection of spectral opportunities and their efficient utilization without causing interference to the primary users. For this goal, we propose to make use of a reinforcement learning approach: the Multi Armed Bandit (MAB). The MAB approach provides the secondary users with the rules and policies necessary to achieve a tradeoff between exploitation and exploration in DSA. Different MAB strategies are tested on an IEEE802.11 medium access model and evaluated in dynamic environment. Our study shows that the MAB constitute a viable solution for the DSA. Adding to that, the performances of the MAB algorithms can be improved with a finite tuning of the internal parameters.

The construction of a smart autonomous mobile robot system that can operate in an anonymous environment can be difficult. Building an autonomous mobile robot requires collecting data from sensors and cameras in order to recognize the surrounding environment, get knowledge of existing landmarks, and figure out its position and orientation. Many research studies have been carried out in recent years providing algorithms for achieving autonomous motion of mobile robot systems and object detection. The most common platform utilized for robot motion design is ROS (Robot Operating System) and for object detection YOLO (You Only Look Once) is a common choice. Additionally, the autonomous system needs motion planning in a systematic way to avoid obstacles by using LIDAR or other sensing methods generating the feedback signals required for control of motion around occupied spaces. One of the most effective and versatile feedback systems for mobile robot control can be constructed using visual feedback, which relies on a high-resolution camera that processes the image in real-time and the result is fed back to the control system for decision making. Visual feedback can be employed alone without sensors for feedback control of autonomous mobile robots, and it can be used for multiple tasks like path planning, obstacle avoidance, object detection, object recognition in addition to motion control. In this research, results of design and construction of an autonomous mobile rescue robot for finding victims through video data are presented. It is aimed to find victims via applying video as a data collection and feedback generation tool in the autonomous mobile robot control system. The visual feedback and YOLO v3 are used not only for finding victims but also for motion control. Our autonomous mobile robot is designed to move smoothly to the victim's direction and avoids obstacles by using ROS Global Planner. The robot is tested in multiple scenarios in different locations, various obstacle and victim numbers, and satisfactory results with good performances are obtained.

With many gaming AI being developed and being able to defeat top human players in recent years, AI has once again become the hottest topic in research and even in our daily life. This paper also researches on gaming AI and chess. Instead of using deep learning and the Monte Carlo Search algorithm, this paper focuses on the opening only with multi-armed bandit algorithms to find the best moves and opening. Specifically, the method used in this paper is epsilon greedy and Thompson sampling. The dataset used in this paper is from Kaggle. This paper considers each move as a set of choices one needs to make and considers the big picture as a multi-armed bandit problem. This paper aims to develop a relative best strategy to counter those opening or make changes to a disadvantaged situation.

In the current digital age, software programs and applications provide convenience and added value in a number of ways. Multi-armed Bandit algorithms (MAB) are a prime example of this; In capital markets, they assist in the adaptive design of trading strategies that adjust to market shifts and investor behavior. In e-commerce, MAB helps optimize product recommendations by learning customer preferences in real-time. In SaaS, MAB can optimize user experience by personalizing services or pricing models based on user behavior, continuously adjusting to maximize engagement or revenue. In cloud engineering MAB are indispensable for optimizing resource allocation based on user demand. The objective of this paper is to demonstrate the adaptability of MAB algorithms through an examination of their applications, specifically focusing on the Digital Marketing domain and the insight they offer for optimal decision-making. The article highlights the usefulness of major MAB algorithms in digital advertising, e-commerce content suggestion, SaaS and strategic pricing by carefully examining Thompson Sampling, UCB, Restless Bandit, and Structured Bandit as key types of MAB algorithms. The study emphasizes how these algorithms adjust to fluctuating conditions, balance a trade-off between exploration and exploitation, and eventually improve marketing strategies. This article aims to improve knowledge of MAB algorithms and encourage more research in this promising area by providing a detailed analysis of their applications.

Identifying the topic of a search query is a challenging problem, for which a solution would be valuable in diverse situations. In this work, we formulate the problem as a ranking task where various rankers order queries in terms of likelihood of being related to a specific topic of interest. In doing so, an explore-exploit trade-off is established whereby exploiting effective rankers may result in more on-topic queries being discovered, but exploring weaker rankers might also offer value for the overall judgement process. We show empirically that multi-armed bandit algorithms can utilise signals from divergent query rankers, resulting in improved performance in extracting on-topic queries. In particular we find Bayesian non-stationary approaches to offer high utility. We explain why the results offer promise for several use-cases both within the field of information retrieval and for data-driven science, generally.