Acoustic Cloak Design Using Generative Modeling and Reinforcement Learning

Reinforcement learning (RL) models have been broadly used in modeling the choice behavior of humans and other animals. In standard RL models, the action values are assumed to be updated according to the reward prediction error (RPE), i.e., the difference between the obtained reward and the expected reward. Numerous studies have noted that the magnitude of the update is biased depending on the sign of the RPE. The bias is represented in RL models by differential learning rates for positive and negative RPEs. However, which aspect of behavioral data that the estimated differential learning rates reflect is not well understood. In this study, we investigate how the differential learning rates influence the statistical properties of choice behavior (i.e., the relation between past experiences and the current choice) based on theoretical considerations and numerical simulations. We clarify that when the learning rates differ, the impact of a past outcome depends on the subsequent outcomes, in contrast to standard RL models with symmetric value updates. Based on the results, we propose a model-neutral statistical test to validate the hypothesis that value updates are asymmetric. The asymmetry in the value updates induces the autocorrelation of choice (i.e., the tendency to repeat the same choice or to switch the choice irrespective of past rewards). Conversely, if an RL model without an intrinsic autocorrelation factor is fitted to data that possess an intrinsic autocorrelation, a statistical bias to overestimate the difference in learning rates arises. We demonstrate that this bias can cause a statistical artifact in RL-model fitting leading to a “pseudo-positivity bias” and a “pseudo-confirmation bias.”

The relation between reinforcement learning parameters and the influence of reinforcement history on choice behavior

Experimental research such as cell cultures and carbon nanotube (CNT) growth are largely governed by following predefined execution protocols with fine-tuned control of parameters. There are promising opportunities to apply reinforcement learning (RL), an established learning technique in the area of artificial intelligence, in order to automate CNT growth process and accelerate related scientific breakthroughs in material discovery. Although there are benefits in RL-based exploration and exploitation methodologies, there are also challenges in developing relevant learning policies in experimental settings relating to CNT growth. In this paper, we present a novel data-driven RL approach for assisting experimental CNT growth. Our approach focuses on developing an RL model to learn from simulation-based images and characteristics of temporal CNT growth, considering various growth parameters. Our RL model learns from CNT growth variation in a simulation-based environment where critical control parameters, such as density, growth rate, tube radius, tube stiffness, and Van der Waals forces, are used. Our RL model enables CNT growth automation in order to explore of a wider range of growth conditions, and improve reproducibility. The ultimate goal of our RL model is to achieve desired CNT growth by dynamically controlling growth parameters throughout a sequence of experiments. We evaluate the effectiveness of our RL approach by measuring the improvement in the maximum compressive strength of a carbon nanotube ‘with’ and ‘without’ the RL model. Our results show the effectiveness of course correction recommended by our RL approach when controlling the parameters of angular deviation and rate of growth of carbon nanotubes, when compared against non-regulated CNT growth.

Fog computing extends cloud services to the network edge, enabling low-latency processing for Internet of Things (IoT) applications. However, this distributed approach is vulnerable to a wide range of attacks, necessitating advanced intrusion detection systems (IDSs) that operate under resource constraints. This study proposes integrating self-awareness (online learning and concept drift adaptation) into a lightweight RL (reinforcement learning)-based IDS for fog networks and quantitatively comparing it with non-RL static thresholds and bandit-based approaches in real time. Novel self-aware reinforcement learning (RL) models, the Hierarchical Adaptive Thompson Sampling–Reinforcement Learning (HATS-RL) model, and the Federated Hierarchical Adaptive Thompson Sampling–Reinforcement Learning (F-HATS-RL), were proposed for real-time intrusion detection in a fog network. These self-aware RL policies integrated online uncertainty estimation and concept-drift detection to adapt to evolving attacks. The RL models were benchmarked against the static threshold (ST) model and a widely adopted linear bandit (Linear Upper Confidence Bound/LinUCB). A realistic fog network simulator with heterogeneous nodes and streaming traffic, including multi-type attack bursts and gradual concept drift, was established. The models’ detection performance was compared using metrics including latency, energy consumption, detection accuracy, and the area under the precision–recall curve (AUPR) and the area under the receiver operating characteristic curve (AUROC). Notably, the federated self-aware agent (F-HATS-RL) achieved the best AUROC (0.933) and AUPR (0.857), with a latency of 0.27 ms and the lowest energy consumption of 0.0137 mJ, indicating its ability to detect intrusions in fog networks with minimal energy. The findings suggest that self-aware RL agents can detect traffic–dynamic attack methods and adapt accordingly, resulting in more stable long-term performance. By contrast, a static model’s accuracy degrades under drift.

Reinforcement learning (RL) models have been influential in characterizing human learning and decision making, but few studies apply them to characterizing human spatial navigation and even fewer systematically compare RL models under different navigation requirements. Because RL can characterize one’s learning strategies quantitatively and in a continuous manner, and one’s consistency of using such strategies, it can provide a novel and important perspective for understanding the marked individual differences in human navigation and disentangle navigation strategies from navigation performance. One-hundred and fourteen participants completed wayfinding tasks in a virtual environment where different phases manipulated navigation requirements. We compared performance of five RL models (3 model-free, 1 model-based and 1 “hybrid”) at fitting navigation behaviors in different phases. Supporting implications from prior literature, the hybrid model provided the best fit regardless of navigation requirements, suggesting the majority of participants rely on a blend of model-free (route-following) and model-based (cognitive mapping) learning in such navigation scenarios. Furthermore, consistent with a key prediction, there was a correlation in the hybrid model between the weight on model-based learning (i.e., navigation strategy) and the navigator’s exploration vs. exploitation tendency (i.e., consistency of using such navigation strategy), which was modulated by navigation task requirements. Together, we not only show how computational findings from RL align with the spatial navigation literature, but also reveal how the relationship between navigation strategy and a person’s consistency using such strategies changes as navigation requirements change.

Stress and genetic background regulate different aspects of behavioral learning through the action of stress hormones and neuromodulators. In reinforcement learning (RL) models, meta-parameters such as learning rate, future reward discount factor, and exploitation-exploration factor, control learning dynamics and performance. They are hypothesized to be related to neuromodulatory levels in the brain. We found that many aspects of animal learning and performance can be described by simple RL models using dynamic control of the meta-parameters. To study the effects of stress and genotype, we carried out 5-hole-box light conditioning and Morris water maze experiments with C57BL/6 and DBA/2 mouse strains. The animals were exposed to different kinds of stress to evaluate its effects on immediate performance as well as on long-term memory. Then, we used RL models to simulate their behavior. For each experimental session, we estimated a set of model meta-parameters that produced the best fit between the model and the animal performance. The dynamics of several estimated meta-parameters were qualitatively similar for the two simulated experiments, and with statistically significant differences between different genetic strains and stress conditions.

Data quality is a critical aspect of data analytics since it directly influences the accuracy and effectiveness of insights and predictions generated from data. Artificial Intelligence (AI) schemes have grown in the existing era of technological advancement, which provides innovative exposure to healthcare applications. Reinforcement Learning (RL) is a subfield and an influential Machine Learning (ML) model aimed at optimizing decision-making by association with dynamic environments. In healthcare applications, RL can modify conduct strategies, enhance source application, and improve patient investigation history by using various data modalities. The worth of the data quality regulates how effective RL is in healthcare applications. In healthcare, the model predictions have a direct impact on patient's lives, and poor data quality often leads to wrong evaluations that expose patient safety and treatment quality. Biases in data quality have also presented a challenging influence on the RL model's effectiveness and accuracy. RL models have enormous potential in healthcare; however, various strategic limitations prevent their widespread acceptance and deployment. The implementation of RL in healthcare faces serious issues, mostly around data quality, bias, and tactical difficulties. This study delivers a broad survey of these challenges, emphasizing how imbalanced, imperfect, and biased data can affect the generalizability and performance of RL models. We critically assessed the sources of data bias, comprising demographic imbalances and irregularities in electronic health records (EHRs), and their impact on RL algorithms. This survey aims to present a detailed study of the complex circumstances relating to data quality, data biases, and strategic barriers in RL models deploying in healthcare applications. However, the main contribution of the proposed study is that it provides a systematic review of these challenges and delivers a roadmap for future work intended to refine the consistency, fairness, and scalability of RL in healthcare sectors.

Modeling human behavior in dynamic tasks can be challenging. As human beings possess a common set of cognitive processes, there should be certain robust cognitive mechanisms that capture human behavior in these dynamic tasks. This paper argues for a learning model of human behavior that uses a reinforcement learning (RL) mechanism that has been widely used in the fields of cognitive modeling, and judgement and decision making. The RL model has a generic decision-making structure that is well suited to explaining human behavior in dynamic tasks. The RL model is used to model human behavior in a popular dynamic control task called Dynamic Stock and Flows (DSF) that was used in a recent Model Comparison Challenge (MCC). The RL model’s performance is compared to a winner model that won the MCC, that also uses the RL mechanism, and that is the best known model to explain human behavior in the DSF task. Results of comparison reveal that the RL model generalizes to explain human behavior better than the winner model. Furthermore, the RL model is able to generalize to human data of best and worst performers better than the winner model. Implications of this research highlight the potential of using experienced-based mechanisms like reinforcement learning to explain human behavior in dynamic tasks. 

Infrastructure‐to‐infrastructure (I2I) communication enables the exchange of traffic data between intersections, which brings a new challenge to urban traffic control. This paper proposes a novel deep reinforcement learning (DRL) framework for urban traffic signal control within the vehicle‐to‐everything (V2X) ecosystem. The framework incorporates a joint‐state representation integrating traffic data from both I2I and vehicle‐to‐infrastructure (V2I) communications, while considering communication range effects. The reward function is designed to optimize both local intersection conditions and global network performance, facilitating adaptive signal coordination. A simulation case study in Huzhou, China, evaluates the proposed model against conventional pretimed, actuated, nonjoint‐state DRL, and joint‐state reinforcement learning (RL) models. Results demonstrate superior performance of the joint‐state DRL model in episode rewards, average speed, and average time loss, particularly during peak traffic periods. To address data limitations, Monte Carlo cross‐validation (MCCV) is employed, further validating the model’s robustness. Results show consistent performance advantages in average speed and time loss across peak and off‐peak periods, with slight variations compared to joint‐state RL models in certain intervals. The impacts of the communication range are also discussed with the proposed model. Pearson correlation analysis reveals a strong positive correlation between the communication range and convergence time across all traffic periods. Meanwhile, correlations between the communication range and reward, average speed, and average time loss vary by traffic period. Findings highlight the transformative potential of integrating DRL with V2X communication technologies for enhancing traffic signal control in complex urban environments. The proposed model offers a flexible, adaptive approach to traffic management, optimizing flow while maintaining safety standards, with implications for future smart city developments.

The need for self-adaptive systems is increasing with the growth in automation to improve accuracy and performance. Such self-adaptive systems can be built using Reinforcement Learning (RL) models with the help of simulated and recorded data. Deep Reinforcement Learning (DRL) is proposed by infusing NN into RL models and can be used to improve performance in use cases with sparse reward distributions. Traditional Actor-Critic DRL models act with a single critic and single actor network in case of on-policy execution. In contrast, the performance of the agent which makes decisions in the environment improves when considering the information exchange from multiple critics. In this work, the proposed Teacher Tutor Trainee a version of Multi Critic architecture provides the impact of information exchange from multiple critics for a channel estimation use case. Channel Estimation is a method of calculating the impact of the propagation medium and other external factors that deteriorate a signal and its optimization. The performance improvement can be observed from the increase in the explained variance. On implementation, it is observed that the Proximal Policy Optimization model has shown better performance than the Advantage Actor-Critic model.

A reinforcement-learning model of active avoidance behavior: Differences between Sprague Dawley and Wistar-Kyoto rats

Vehicle travel is on the rise across the world, particularly in metropolitan cities. As a result, simulating and optimizing traffic control algorithms is required to better handle this growing demand. In this research, we investigate the simulation and optimization of traffic light controllers in a city and provide a reinforcement learning-based approach. The algorithm that is used is deep Q-learning. There are four processes in this study. The first is data collection. Next, a simulation is built. Then, a reinforcement learning model is trained and tested in the model. Last, the results are compared between the traditional traffic lights and traffic lights that were applied with the reinforcement learning model. The results obtained in this study are queue length of vehicles in front of traffic light and delay time of vehicles after given green signal in two scenarios, which are an environment that uses traditional traffic light and an environment that uses traffic light that applied with reinforcement learning model. From the result, the environment that applied with reinforcement learning agent has shorter time delay and queue length of vehicles. Queue length of vehicles is reduced from 58 to 18 and time delay is reduced from 3900 seconds to 380 seconds.

Top-down inputs from prefrontal cortex impact on sensory neurons [1,2], enhancing their selectivity to attended stimuli, while sensory processing of distractors is suppressed [1,3,4]. However, what are the neuro-computational mechanisms that identify the behaviorally relevant information that is worth to bias? Previous studies linked selective attention to learned value [5], suggesting that attentional selection relies on internal mechanisms that track the relevance of sensory information. Consistent with this view, we recently introduced a reinforcement learning (RL) approach for the deployment of selective attention [6]. We tested model-free and model-based versions of RL to identify the mechanisms that most accurately predict behavior: whereas model-based prioritizes attentional selection to task features that are systematically associated with reward, model-free considers all available features. Our results proved that the optimal task-set representation significantly improved predictive power, suggesting that monkeys benefited from model-based mechanisms [6]. Yet, model-based presents two important limitations: i) it is unable to adapt to changes in the association of reward with sensory features that are not included in the learned model, and ii) it excludes the mechanism of learning by which subjects derive the proper task-set representation. The question remains then of how a prioritized task set can be learned to exploit the benefits of employing a model. With the aim to workaround model-based limitations and shed light on the underlying mechanisms that make model-based benefits possible, we propose here the “adaptive learning” mechanism for flexible task-set representation. The mechanism is dynamically tuned according to the statistics of association between sensory features and reward outcome, flexibly adjusting its regime of operation between model-free and modelbased systems. To test the adaptive learning mechanism, we developed it in a RL model and extended our previous analysis of monkey behavior to both, cued [5] and uncued [6], versions of the same attention task. This RL model was able to transition from a naive starting point to an optimal task-set representation, and to flexibly adapt among optimal task-set representations upon changes in reward contingencies. The model achieved so by tracking a separate learning rate a for each stimulus feature in the environment. If the selection of a stimulus feature was systematically correlated with a particular outcome (reward or no-reward), a increased. In contrast, when a feature was unable to consistently predict reward, a decayed. Thus, a changed dynamically, and over a large number of trials any a associated with a non-predictive feature went to zero, effectively eliminating it from the task set, which tuned performance towards that of the model-based system. The adaptive learning mechanism introduced here represents a step further in our understanding of the origins of selective attention. Notably, our results prove that the adaptive learning is an optimal mechanistic candidate to support arbitrary prioritized model-based formation under generic conditions of covert attentional selection, and regardless of whether attentional selection operates on cued or uncued tasks.

Prior research has used reinforcement-learning (RL) models like Expectancy-Valence-Learning (EVL) and Prospect-Valence-Learning (PVL) to investigate human decisions in choice games. However, currently little is known on how RL models would account for human decisions in games where people face a collective risk social dilemma (CRSD) against societal problems like climate change. In CRSD game, a group of players invested some part of their private incomes to a public fund over several rounds with the goal of collectively reaching a climate target, failing which climate change would occur with a certain probability and players would lose their remaining incomes. Next EVL and PVL models were calibrated to human decisions across two between-subject conditions in CRSD (Info-all: N=120; No-Info: N=120), where half of the players in each condition possessed lesser wealth (poor) compared to the other half (rich). A symmetric Nash model was also run in both conditions as a benchmark. In Info-all condition, players possessed complete information on investments of other players after every round; whereas, in the No-info condition, players did not possess this information. Our results showed that for both rich and poor players, the EVL model performed better than the PVL model in No-info condition; however, the PVL model performed better than the EVL model in the Info condition. Both the EVL and PVL models outperformed the symmetric Nash model. Model parameters showed reliance on recency, reward-seeking, and exploitative behaviours. We highlight the implications of our model results for situations involving a collective risk social dilemma.

IntroductionComputational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses.MethodsSimulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses.ResultsInitial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise.ConclusionOverall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.