Optimization of Transmission Strategy for Wireless Power Transfer Using Multi-Armed Bandit Algorithm

This article explores the application of Multi-Armed Bandit (MAB) algorithms in optimizing consumer tech products, with a particular focus on eCommerce platforms. It provides a comprehensive article overview of the theoretical framework behind MAB algorithms, including the exploration-exploitation trade-off and comparisons with traditional A/B testing methods. The article delves into various MAB strategies commonly used in eCommerce, such as epsilon-greedy, Upper Confidence Bound (UCB), and Thompson Sampling, and examines their applications in personalized product recommendations, dynamic pricing, ad placement optimization, and website content delivery. Implementation considerations, including integration with existing machine learning infrastructure and data processing in high-throughput scenarios, are discussed in detail. The article also addresses the impact of MAB algorithms on key performance metrics like user engagement, conversion rates, and revenue optimization. Ethical considerations, including transparency in automated decision-making and fairness in consumer-facing applications, are explored. Finally, the article presents case studies of successful implementations, discusses current challenges and limitations, and outlines future directions for MAB algorithms in eCommerce and potential cross-industry applications.

Energy harvesting (EH), which refers to replenishing energy from the environment, is demonstrated to be a promising approach in reducing the operational expenses (OPEX) and increasing the network lifetime of emerging wireless communication technologies by eliminating the need for frequent battery replacing or recharging of wireless devices. Among various ambient energy sources, the focus of this dissertation is on wireless power transfer (WPT) considering the unique characteristic of radio frequency (RF) signals; that is, they inherently carry both information and energy. A particularly interesting scenario arises when sources support simultaneous wireless information and power transfer (SWIPT) to provide a remote, controllable, and on-demand energy source. In this context, one of the promising deployment approaches is SWIPT-enabled cognitive radio networks (CRNs), which offer significant gains in terms of spectral and energy efficiencies. One of the practical scenarios of RF-powered CRN is deploying a number of distributed cognitive sensor nodes, equipped with RF EH modules, to sense a specific area and send the sensed data to an access point while coexisting with a primary licensed network. However, exploiting the full potentials of SWIPT-enabled CRNs is subject to carefully aligning the requirements of the EH unlicensed secondary receivers (SRs) with those of the legitimate primary receivers (PRs). In light of this, the first contribution of this thesis focuses on investigating the problem of beamforming for the downlink of multi-user multiple-input single-output (MU-MISO) CRNs. With the objective of minimising the transmission power of the secondary base station (SBS), optimal and suboptimal solutions for the formulated optimisation problem are provided by jointly optimising the transmit beamforming vector at the SBS and adjusting the parameters of the energy harvesters at the SRs. It is shown that the obtained solutions are efficient in meeting the EH and quality-of-service (QoS) requirements of the SRs and the levels of interference accepted by the PRs. Apart from SWIPT-enabled CRNs, relay-assisted SWIPT networks are envisioned to be a promising framework offering extended coverage, diversity gains, and enhanced energy efficiency. In this case, the relay network itself can benefit from the relayed transmissions in terms of saving energy, and the harvested energy can be used to charge relay nodes and extend their lifetime as compensation for their role of data forwarding. Nonetheless, a concrete performance analysis on the impact of the involved system parameters such as, the energy conversion efficiency and the location of the EH relay terminal, on the trade-off between the achievable information transfer efficiency and the harvested energy level is crucial for the successful implementation of SWIPT in this context. This inspired the research work in the next two contributions with the main focus on developing novel comprehensive analytical frameworks for the investigation and evaluation of SWIPT relaying systems. Specifically, the second contribution is dedicated to examining the application of noncoherent modulation, which is recognised as an energy efficient modulation scheme for SWIPT, due to its ability to eliminate the need of instantaneous channel state information (CSI) estimation/tracking. Through adopting a moments-based approach, novel expressions are derived for the outage probability, achievable throughput, and average symbol error rate (ASER) of dual-hop SWIPT relaying systems. Furthermore, new asymptotic analytical results are derived for the high SNR regime and are then utilised to analytically quantify the achievable diversity order. The proposed mathematical tools are demonstrated to be an accurate and efficient means by which one can conduct a thorough analysis on the system performance without the burden of Monte Carlo simulations. Finally, the third contribution focuses on SWIPT relaying systems operating in the presence of impulsive non-Gaussian noise, which is typical in several practical scenarios. Several studies show sufficient evidences that impulse man-made noise is encountered in various metropolitan, indoor, and underwater wireless applications. Examples of these sources include automotive ignition, electronic devices, household appliances, medical equipment, and industrial equipment. To characterise the behavior of the system performance under this type of noise, an efficient analytical framework is developed where novel closed-form xpressions for the pairwise error probability (PEP) are derived for two relaying schemes, namely, blind relaying and CSI-assisted relaying, employed under two assumptions imposed on the deployed EH process; namely, instantaneous EH (IEH) and average EH (AEH). Apart from being accurate, the derived expressions are shown to be efficient in quantifying the diversity order of the system and providing a comprehensive study on the impact of the severity of impulsive noise on the behavior of the system.

Multi-Armed Bandit (MAB) algorithms are classic algorithms that address sequential decision-making under uncertainty by solving the exploration-exploitation trade-off dilemma. This study investigates the performance comparison of multiple MAB algorithms using a simulated multi-armed bandit machine with Bernoulli reward distribution as the experimental environment. This study compares the performance differences among Upper Confidence Bound (UCB), Thompson Sampling (TS) and -Greedy Thompson Sampling (-TS) in this environment, and attempts to set different parameters, namely the number of arms and the number of experimental rounds and calculate the corresponding cumulative regret of the algorithm under various conditions. The size of the cumulative regret reflects the performance of each algorithm in the simulated slot machine model with rewards that conform to the Bernoulli distribution. In addition, the algorithm running time under the same conditions is also recorded to analyze from the perspective of algorithm efficiency. The experimental results show that under the experimental environment of this study, the cumulative regret produced by the UCB algorithm is more than three times that of the other two algorithms. When the number of trials is small, the cumulative regret generated by the TS algorithm is small, but overall, the performance of the TS algorithm and the -TS algorithm set in this experiment in minimizing the cumulative regret is not much different. However, TS runs in a shorter time under the same conditions. The results of this experiment show that after the number of rounds of experimental operation reaches a large enough number, the operating efficiency of the TS algorithm will be significantly higher than that of the -TS algorithm. TS algorithms have higher randomness, so they show better performance under this experimental condition. The -TS under the parameter setting of this study encourages exploration more in the early stage of the experiment, so it will produce greater cumulative regret than the traditional TS algorithm. In the long run, the performance difference between the two in the multi-armed bandit problem with Bernoulli distribution of rewards is very small. However, the TS algorithm has more advantages in algorithm execution efficiency, so when solving similar problems, the TS algorithm is a better choice.

Recommender systems play a critical role in e-commerce, streaming media, and other fields, where user engagement is vital to platform success. These systems, however, face challenges like the cold start problem, which limits recommendation accuracy for new users and items due to sparse data. Although multi-armed bandit (MAB) algorithms are considered effective for balancing exploration and exploitation, existing studies lack simulations of algorithm performance in scenarios where recommender systems indirectly influence user choices, limiting insights for performance improvements in real-world applications. This study explores the performance of key MAB algorithmsExplore-Then-Commit (ETC), Upper Confidence Bound (UCB), and Thompson Sampling (TS)under a novel framework that models user behavior as an unreliable intermediate using Markov state transitions. This Markov perturbation-based approach provides a foundation for indirectly influencing user decisions. Two algorithmic frameworks are examined: one considering only the recommendation system's influence on user choices, and another also considering user feedback as a dynamic influence on state transition probabilities. Results show that, while considering user feedback increases adaptability, it also causes greater volatility. TS achieves the lowest cumulative regret with strong stability, while Asymptotic-UCB achieves a better balance between exploration and exploitation than standard UCB, with consistent adaptability under varying feedback. In contrast, ETC demonstrates significant volatility, highlighting its limitations in feedback-driven scenarios. These findings provide valuable insights for enhancing MAB algorithms in real-world cold start recommendation applications, especially where systems influence user choices indirectly.

In recent years, the advertising sector has increasingly embraced Multi-armed Bandit Algorithms (MAB) for their versatile applications. This article delves into four stochastic MAB algorithms: the Explore-Then-Commit (ETC), the Upper Confidence Bound (UCB), the Thompson-Sampling (TS), and their respective variants, applied in an online shopping platform's advertising campaign to optimize click-through rates. Our findings indicate that each of these algorithms successfully identifies the most effective advertisement. Both the UCB and TS algorithms demonstrate logarithmic regret relative to the time horizon. The ETC algorithm exhibits a delayed onset of logarithmic regret but surprisingly holds up well against the UCB algorithms. Notably, the TS algorithm class outshines the UCB class due to its inherent randomness, offering a more robust performance. Furthermore, two variants, the UCB algorithm with the Mini-max Optimal Strategy in the Stochastic case (MOSS) and the paralleled TS algorithm, show promising results. They not only maintain logarithmic regret but also enhance efficiency, indicating further potential in the UCB and TS algorithm classes for advanced applications.

Generalized Bandits with Learning and Queueing in Split Liver Transplantation

Far-field Wireless Power Transfer (WPT) and Simultaneous Wireless Information and Power Transfer (SWIPT) have attracted significant attention in the RF and communication communities. Despite the rapid progress, the problem of waveform design to enhance the output DC power of wireless energy harvester has received limited attention so far. In this paper, we bridge communication and RF design and derive novel multisine waveforms for multi-antenna wireless power transfer. The waveforms are adaptive to the channel state information and result from a posynomial maximization problem that originates from the non-linearity of the energy harvester. They are shown through realistic simulations to provide significant gains (in terms of harvested DC power) over state-of-the-art waveforms under a fixed transmit power constraint.

Unmanned aerial vehicles (UAVs) have recently been envisaged as an enabling technology of 5G. UAVs act as an intermediate relay node to facilitate uninterrupted, high quality communication between information sources and their destination. However, UAV energy management has been a major issue of consideration due to limited power supply, affecting flight duration. Thus, we introduce in this paper a unified energy management framework by resorting to wireless power transfer (WPT), simultaneous wireless information and power transfer (SWIPT) and self-interference (SI) energy harvesting (EH) schemes, in cooperative relay communications. In our new technique, UAVs are deployed as relays equipped with a decode and forward protocol and EH capability operating in a full-duplex (FD) mode. The UAV assists information transmission between a terrestrial base station and a user. The UAV's transmission capability is powered exclusively by the energy harvested from WPT, radio frequency signal transmitted from the source via time-switching SWIPT protocol and SI exploitation. We improve the overall system throughput by the use of FD based UAV-assisted cooperative system. In this proposed system, we formulate two optimization problems to minimize end-to-end outage probability, subject to UAV's power profile and trajectory for a DF relay scheme, respectively. The KKT conditions have been used to obtain closed-form solutions for the two formulated problems. Numerical simulation results validate all the theoretical results. We demonstrate that the performance of our proposed unified EH scheme outperforms that of existing techniques in the literature.

In cognitive Wireless Radio networks, information transfer and power transfer are given special consideration in next-generation wireless networking devices like Simultaneous Wireless Information and Power Transfer. The benefit of integrating cognitive network concepts is that it provides greater efficiency on energy and spectrum to address interference, fading, and path loss, among many other issues, in certain critical applications. Energy harvesting (EH) is a secure and cost-effective method of powering up cluster heads when needed. The proposed method considers two phases. The first phase is energy harvesting of SUs transferred in power signals send by the hybrid access point during wireless power transfer (WPT). While in the second phrase, Wireless Information Transfer (WIT) phase, the captured energy is used while simultaneously acquiring transmission possibilities. In addition, a fuzzy-based cluster using a greedy algorithm is used to reduce PU secrecy outages and to provide the best ideal result. These may be identified using a convolutional neural network (CNN) to detect and classify distinct attack types. This paper presents an efficient resource allocation using fuzzy logic-based clustering and greedy algorithms, which can provide better performance than other technologies. The subsystem for cognitive radio networks is configured, and the primary nodes’ secrecy is maintained and free of three attacks, and the results are simulated using MATLAB tools.

Numerous efforts in wireless power transfer have concentrated on simultaneous wireless information and power transfer (SWIPT), enabling terminal devices (TDs) to achieve permanent operation and reducing the limitation of energy on information transmission. Nevertheless, most existing studies focus on the amount of energy transferred, the estimation of the harvest states of TDs, and the optimization of energy efficiency for a system, failing to investigate the timeliness of energy replenishment for TDs through wireless energy harvesting. Additionally, due to channel fading and the influence of nonlinear circuits, serious estimation errors will occur during wireless energy transmission and state of charge (SoC) evaluation, resulting in the failure of system strategies and degraded system performance. To address this issue, a novel wireless energy access quality of service (QoS) index, age of energy harvesting (AoEH), is proposed to measure the timeliness of wireless energy transfer for TDs. Based on AoEH, this paper establishes a SWIPT access channel model and revises the time sequence of access channels. Furthermore, an energy management strategy is proposed for the permanent operation of TDs, and relevant theoretical analysis on the boundlessness of remaining energy is provided. As a typical case, this paper formulates the age of information (AoI) minimization problem for SWIPT-enabled sensor networks to find the optimal energy packet generation interval at the sink node based on AoEH expressions. Corresponding solutions are proposed for the AoI minimization problem with and without maximum transmission frequency constraints. Simulation results indicate that TD’s energy is bounded and stable under the proposed energy management strategy, and the AoI optimized by the optimal energy packet generation interval algorithm is up to 30% lower compared to existing algorithms.

Far-field wireless power transfer (WPT) and simultaneous wireless information and power transfer (SWIPT) have become increasingly important in radio frequency (RF) and communication communities recently. The problem of modulation design for SWIPT has however been scarcely addressed. In this paper, a modulation scheme based on asymmetric phase-shift keying (PSK) is considered, which improves the SWIPT rate-energy tradeoff region significantly. The nonlinear rectifier model, which accurately models the energy harvester, is adopted for evaluating the output direct current (DC) power at the receiver. The harvested DC power is maximized under an average power constraint at the transmitter and a constraint on the rate of information transmitted via a multi-carrier signal over a flat fading channel. As a consequence of the rectifier nonlinearity, this work highlights that asymmetric PSK modulation provides benefits over conventional symmetric PSK modulation in SWIPT and opens the door to systematic modulation design tailored for SWIPT.

Spatial crowdsensing is a special kind of crowdsourcing which allocates tasks to workers in some special places where workers can sense data for them. Due to the lack of priori information about the quality of workers and the ground truth, selecting the most suitable workers, which can guarantee the quality of the sensing tasks, remains a great challenge. In this paper, we propose a novel framework which can choose the most reliable workers among available workers under constraint budget. We model the quality of workers through two factors, bias and variance, which describe the continuous feature of sensing tasks. Our framework first allocate some calibration tasks to calibrate the bias and then iteratively estimate the workers variance more and more accurately. To choose more reliable workers, we face the exploration and exploitation dilemma. Therefore, we design a novel Multi-Armed Bandit (MAB) algorithm which based on Upper Confidence Bounds (UCB) scheme and combined with a weighted data aggregation scheme to estimate a more accurate ground truth of a sensing task. Futhermore, a dynamic budget allocation algorithm is designed to achieve global optimization. Then, we prove the expected sensing error can be bounded according to the regret bound of the MAB. In simulation experiments, we compare our algorithm with several baselines with real-world data set and it shows the effectiveness in inferring the ground truth with limited budget.

Unmanned aerial vehicles (UAVs) are expected to be integrated into future wireless networks to offer services, especially in unreachable or congested areas. To improve the spectral efficiency, non-orthogonal multiple access (NOMA) scheme can be utilised within the UAV communication to allow more users to be covered and associated. The performance of the NOMA-UAVs network is governed by several factors including power allocation, user association and pairing methods. This paper presents an approach that uses multi-armed bandit (MAB) and two-sided matching frameworks to maximize the throughput of multi-UAV-assisted NOMA networks in a decentralized manner. The approach enables the UAVs to propose to the ground users (GUs) without explicit cooperation among the UAVs while the GUs can accept or reject the proposals. To this end, we propose a modified Thompson sampling algorithm that we named decaying epsilon Thompson sampling (<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula>) MAB algorithm that is designed to improve the exploration-exploitation tradeoff in the MAB. The performance of the proposed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> MAB algorithm is evaluated against other existing MAB techniques. Simulation results show that the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> algorithm attains faster convergence and improved performance in terms of smaller regret and increased achievable system throughput. The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\text {D}\epsilon \text {TS}$ </tex-math></inline-formula> MAB algorithm particularly excels in regards of the convergence rate when the number of available action spaces increases.

Realising both wireless information transfer (WIT) and wireless power transfer (WPT) in the same RF spectral band has imposed great challenges on the system design but the spectral efficiency can be substantially improved, which triggers the significant research interest in the simultaneous wireless information and power transfer (SWIPT). In order to fully exploit the limited spectral resources, a non-orthogonal-multiple-access (NOMA) technique aided SWIPT system is considered for simultaneously realising both of WIT and WPT, which consists of a single access point (AP), a single WPT user and a pair of WIT users. The AP delivers the requested information to the pair of WIT users by adopting the NOMA technique. In order to maximise the amount of energy received by the WPT user, a novel constellation rotation based modulation scheme is proposed for constructively combining the symbols requested by the pair of the WIT users by optimising the constellation rotation angle. By considering the sensitivity of the energy harvester, the simulation results demonstrate that our modulation scheme is capable of efficiently increasing the energy harvested by the WPT user without incurring additional symbol detection errors.

This paper proposes a technique for pre-negotiation of transfer data for a millimeter-wave (mmW) Store-Carry-Forwarding (SCF) data transfer network called the Piggy-back network between autonomous mobile robots. Each robot has its own database on this network, and the purpose is to synchronize the entire network when a part of the data is updated from time to time. While mmW communication is capable of high-speed and large-capacity data communication, the communication range is short and the communication time is extremely limited when the robots are moving. In order to make the best use of communication time and transfer data, it is appropriate to decide in advance which data to receive or transmit. Therefore, by using medium-speed communication, which has a relatively large communication range, and exchanging meta information of each data before the robots approach each other, it is decided which data should be handled. This paper models the difference and update check of data managed in segments for data synchronization as the Multi Armed Bandit (MAB) problem, which is a bandit problem and propose a Tug-of-War (TOW) dynamics-based data query scheme, which is a MAB algorithm. In order to verify the effectiveness of the proposed method, some simulations are performed on the data hit rate, which means the detection rate of update data with a limited number of queries in the situation where the update data occurs randomly. The results show that when the data consists of 10 segments and there is a large bias in the update frequency of each data, the data query method using TOW achieves detection of updated data up to about 40% higher than the random query method.