A Basal Ganglia–Inspired Reinforcement Learning Framework Using Dopamine-Modulated Plasticity

Reinforcement learning (RL) models have been broadly used in modeling the choice behavior of humans and other animals. In standard RL models, the action values are assumed to be updated according to the reward prediction error (RPE), i.e., the difference between the obtained reward and the expected reward. Numerous studies have noted that the magnitude of the update is biased depending on the sign of the RPE. The bias is represented in RL models by differential learning rates for positive and negative RPEs. However, which aspect of behavioral data that the estimated differential learning rates reflect is not well understood. In this study, we investigate how the differential learning rates influence the statistical properties of choice behavior (i.e., the relation between past experiences and the current choice) based on theoretical considerations and numerical simulations. We clarify that when the learning rates differ, the impact of a past outcome depends on the subsequent outcomes, in contrast to standard RL models with symmetric value updates. Based on the results, we propose a model-neutral statistical test to validate the hypothesis that value updates are asymmetric. The asymmetry in the value updates induces the autocorrelation of choice (i.e., the tendency to repeat the same choice or to switch the choice irrespective of past rewards). Conversely, if an RL model without an intrinsic autocorrelation factor is fitted to data that possess an intrinsic autocorrelation, a statistical bias to overestimate the difference in learning rates arises. We demonstrate that this bias can cause a statistical artifact in RL-model fitting leading to a “pseudo-positivity bias” and a “pseudo-confirmation bias.”

Key brain networks underlying cognition and emotion are modulated by major neurotransmitter systems through reward/punishment circuits. Notably, the basal ganglia are thought to funnel emotional information towards motor outcomes. However, the mechanisms that would allow emotional stimuli to guide action selection have yet to be identified. Computational models may contribute to this goal. Here, by using a computer simulation of the guidedpropagation deterministic model, we show that emotional channels can quickly and selectively modulate actionoriented channels, by instantly retrieving all the emotional stimuli paired in the past with a current cue. In agreement with animal based data, the transient modulation signals that implement emotional anticipation appear to be more useful when targeting either newly formed or remote memory traces. The timing and evolution of these signals both suggest a new interpretation of the dopaminergic neuron activity in the basal ganglia during conditioning, usually regarded as coding for a ‘reward prediction error’ in the frame of reinforcement learning. After additional computer trainings involving ‘emotions’ of extreme values, the diversity of actions selected under the influence of a conditioned cue is shown to decrease through either compulsive or avoidance behaviors. Indeed, in the proposed functional model, similar modulation mechanisms account for the development of either drug addiction or posttraumatic stress disorder. Furthermore, spontaneous relapse into these dysfunctions is attributed here to local modulating deficits. The latter can partly be overcome by selectively shifting one of the few control parameters of the model, akin to neuromodulators.

IntroductionComputational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses.MethodsSimulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses.ResultsInitial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise.ConclusionOverall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus–reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.

Estrogenic and androgenic steroids can be synthesised in the brain and rapidly modulate synaptic transmission and plasticity through direct interaction with membrane receptors for estrogens (ERs) and androgens (ARs). We used whole cell patch clamp recordings in brainstem slices of male rats to explore the influence of ER and AR activation and local synthesis of 17β-estradiol (E2) and 5α-dihydrotestosterone (DHT) on the long-term synaptic changes induced in the neurons of the medial vestibular nucleus (MVN). Long-term depression (LTD) and long-term potentiation (LTP) caused by different patterns of high frequency stimulation (HFS) of the primary vestibular afferents were assayed under the blockade of ARs and ERs or in the presence of inhibitors for enzymes synthesizing DHT (5α-reductase) and E2 (P450-aromatase) from testosterone (T). We found that LTD is mediated by interaction of locally produced androgens with ARs and LTP by interaction of locally synthesized E2 with ERs. In fact, the AR block with flutamide prevented LTD while did not affect LTP, and the blockade of ERs with ICI 182,780 abolished LTP without influencing LTD. Moreover, the block of P450-aromatase with letrozole not only prevented the LTP induction, but inverted LTP into LTD. This LTD is likely due to the local activation of androgens, since it was abolished under blockade of ARs. Conversely, LTD was still induced in the presence of finasteride the inhibitor of 5α-reductase demonstrating that T is able to activate ARs and induce LTD even when DHT is not synthesized. This study demonstrates a key and opposite role of sex neurosteroids in the long-term synaptic changes of the MVN with a specific role of T-DHT for LTD and of E2 for LTP. Moreover, it suggests that different stimulation patterns can lead to LTD or LTP by specifically activating the enzymes involved in the synthesis of androgenic or estrogenic neurosteroids.

Striatal Plasticity and Basal Ganglia Circuit Function

It has been suggested that the midbrain dopamine (DA) neurons, receiving inputs from the cortico-basal ganglia (CBG) circuits and the brainstem, compute reward prediction error (RPE), the difference between reward obtained or expected to be obtained and reward that had been expected to be obtained. These reward expectations are suggested to be stored in the CBG synapses and updated according to RPE through synaptic plasticity, which is induced by released DA. These together constitute the “DA=RPE” hypothesis, which describes the mutual interaction between DA and the CBG circuits and serves as the primary working hypothesis in studying reward learning and value-based decision-making. However, recent work has revealed a new type of DA signal that appears not to represent RPE. Specifically, it has been found in a reward-associated maze task that striatal DA concentration primarily shows a gradual increase toward the goal. We explored whether such ramping DA could be explained by extending the “DA=RPE” hypothesis by taking into account biological properties of the CBG circuits. In particular, we examined effects of possible time-dependent decay of DA-dependent plastic changes of synaptic strengths by incorporating decay of learned values into the RPE-based reinforcement learning model and simulating reward learning tasks. We then found that incorporation of such a decay dramatically changes the model's behavior, causing gradual ramping of RPE. Moreover, we further incorporated magnitude-dependence of the rate of decay, which could potentially be in accord with some past observations, and found that near-sigmoidal ramping of RPE, resembling the observed DA ramping, could then occur. Given that synaptic decay can be useful for flexibly reversing and updating the learned reward associations, especially in case the baseline DA is low and encoding of negative RPE by DA is limited, the observed DA ramping would be indicative of the operation of such flexible reward learning.

Attributing outcomes to your own actions or to external causes is essential for appropriately learning which actions lead to reward and which actions do not. Our previous work showed that this type of credit assignment is best explained by a Bayesian reinforcement learning model which posits that beliefs about the causal structure of the environment modulate reward prediction errors (RPEs) during action value updating. In this study, we investigated the brain networks underlying reinforcement learning that are influenced by causal beliefs using functional magnetic resonance imaging while human participants (n = 31; 13 males, 18 females) completed a behavioral task that manipulated beliefs about causal structure. We found evidence that RPEs modulated by causal beliefs are represented in dorsal striatum, while standard (unmodulated) RPEs are represented in ventral striatum. Further analyses revealed that beliefs about causal structure are represented in anterior insula and inferior frontal gyrus. Finally, structural equation modeling revealed effective connectivity from anterior insula to dorsal striatum. Together, these results are consistent with a possible neural architecture in which causal beliefs in anterior insula are integrated with prediction error signals in dorsal striatum to update action values.SIGNIFICANCE STATEMENT Learning which actions lead to reward-a process known as reinforcement learning-is essential for survival. Inferring the causes of observed outcomes-a process known as causal inference-is crucial for appropriately assigning credit to one's own actions and restricting learning to effective action-outcome contingencies. Previous studies have linked reinforcement learning to the striatum, and causal inference to prefrontal regions, yet how these neural processes interact to guide adaptive behavior remains poorly understood. Here, we found evidence that causal beliefs represented in the prefrontal cortex modulate action value updating in posterior striatum, separately from the unmodulated action value update in ventral striatum posited by standard reinforcement learning models.

Storage or processing of information underlies a mechanism able to change the efficacy of certain synapses depending on its activity. Long-term potentiation (LTP) and long-term depression (LTD) are two types of experimentally induced activity dependent change in synaptic efficacy (plasticity). The aim of this study was to determine the subtype specific involvement of group I metabotropic glutamate receptors (mGlus) in both forms of synaptic plasticity, namely LTP and LTD. The mGlu activation underlying pathological processes was also investigated. For all investigations, rats were chronically implanted with recording and stimulating electrodes to enable measurement of evoked potentials from medial perforant path -- dentate gyrus granule cell synapses. An injection cannula was inserted into the ipsilateral cerebral ventricle to enable drug application. Experiments were begun 10d subsequent to the implantation procedure. Histological investigations of hippocampal tissue was used to determine possible changes in cell viability after the induction of chemical LTD. Robust LTP which lasted for over 25h was generated using 200Hz tetanisation (HFT), whereas robust LTD was induced either by 1Hz low-frequency stimulation (LFS) or by pharmacological mGlu activation. The potent mGlu5 receptor agonist MPEP (2-methyl-6-(phenylethynyl)pyridine), applied in concentrations which did not affect basal synaptic transmission, dose-dependently impaired the induction and expression of LTP. Application of MPEP 5 min after tetanisation inhibited late LTP (> 24h). Injections of the potent mGlu1 agonist LY367385 ((S)-(+)-α-Amino-4-carboxy-2-methylbenzenacetic acid), in the concentrations investigated, did not effect basal synaptic transmission. In contrast, a dose- dependent impairment of LTP expression was observed. The lowest concentration used, had no effect on LTP induction, whereas higher concentrations reduced both LTP induction and expression. In contrast to MPEP application, the injection of LY367385 5min after LTP induction did not effect LTP expression. The results strongly implicate an involvement of mGlu1 in LTP induction, whereas the group I mGlu subtype 5 seems to be more involved in LTP expression. Investigating the subtype specific group I mGlu involvement in LTD afford differential results. Whereas LFS-induced LTD strongly required mGlu1 activation during LFS, chemical LTD is induced by activation of mGlu5. Induction of chemical LTD with the group I mGlu receptor agonist, (RS)-3,5-Dihydroxyphenylglycine (DHPG) occluded subsequent induction of LFS- induced LTD. Application of LFS following selective pharmacological activation of mGlu5 by (RS)-2-Chloro-5-hydroxyphenylglycine (CHPG) led to de-depression. Further analysis revealed that concurrent induction of chemically-induced and LFS-induced LTD led to an increase of intracellular calcium sufficient for the induction of LTP by LFS. Whereas LFS-induced LTD required mGlu1 activation during LFS, activation of mGlu5-mediated chemical LTD did not influence responses during LFS. In a next step, it was possible to induce stable LTD by application of the potent group III mGlu agonist AP4 (L-(+)-2-Amino-4-phosphonobutyric acid). The persistent LTD induced by the group I mGlu receptor agonist, DHPG, or the group III mGlu receptor agonist, AP4 were compared. Application of the protein synthesis inhibitor Anisomycin 2h prior to DHPG-injection inhibited the expression of LTD from 6h following DHPG injection. In contrast, Anisomycin did not affect LTD induced by AP4. A paired pulse paradigm determined that AP4-LTD is presynaptic whereas DHPG-LTD has both pre-and postsynaptic components. Histological evaluation at either 4h or 7d following AP4-injection showed that AP4-mediated LTD was ... [abstract truncated]

Reward expectation and reward prediction errors are thought to be critical for dynamic adjustments in decision-making and reward-seeking behavior, but little is known about their representation in the brain during uncertainty and risk-taking. Furthermore, little is known about what role individual differences might play in such reinforcement processes. In this study, it is shown behavioral and neural responses during a decision-making task can be characterized by a computational reinforcement learning model and that individual differences in learning parameters in the model are critical for elucidating these processes. In the fMRI experiment, subjects chose between high- and low-risk rewards. A computational reinforcement learning model computed expected values and prediction errors that each subject might experience on each trial. These outputs predicted subjects' trial-to-trial choice strategies and neural activity in several limbic and prefrontal regions during the task. Individual differences in estimated reinforcement learning parameters proved critical for characterizing these processes, because models that incorporated individual learning parameters explained significantly more variance in the fMRI data than did a model using fixed learning parameters. These findings suggest that the brain engages a reinforcement learning process during risk-taking and that individual differences play a crucial role in modeling this process.

Cannabinoids, the active constituents of marijuana, are known to impair learning and memory. Receptors for cannabinoids are highly expressed in the hippocampus, a brain region that is believed to play an important role in certain forms of learning and memory. To investigate the possible contribution of cannabinoid receptor-mediated deficits in hippocampal function to the learning and memory impairments produced by marijuana, we studied the effects of cannabinoid receptor activation on two models of learning and memory, long-term potentiation (LTP) and long-term depression (LTD), in hippocampal slices. Although LTP and LTD of CA1 field potentials were blocked by cannabinoid receptor activation in the presence of Mg(2+), they could be induced after Mg(2+) was removed. Similarly, LTP and LTD of whole-cell EPSCs were unimpaired in the presence of cannabinoid receptor agonist when the postsynaptic membrane was depolarized during the LTP or LTD induction protocol. Cannabinoid receptor activation also reduced EPSCs and enhanced paired-pulse facilitation, while having no effect on the amplitude of spontaneous miniature EPSCs. Finally, as with cannabinoid receptor activation, inhibition of LTP by adenosine receptor activation could be overcome by removal of Mg(2+) or depolarization of the postsynaptic membrane during tetanus. Our results indicate that cannabinoid receptor activation does not directly inhibit the molecular mechanisms responsible for long-term synaptic plasticity but instead impairs LTP and LTD by reducing presynaptic neurotransmitter release to a level below that required to depolarize the postsynaptic membrane to relieve Mg(2+) blockade of NMDA receptors.

In contemporary reinforcement learning models, reward prediction error (RPE), the difference between the expected and actual reward, is thought to guide action value learning through the firing activity of dopaminergic neurons. Given the importance of dopamine in reward learning and the involvement of Akt1 in dopamine-dependent behaviors, the aim of this study was to investigate whether Akt1 deficiency modulates reward learning and the magnitude of RPE using Akt1 mutant mice as a model. In comparison to wild-type littermate controls, the expression of Akt1 proteins in mouse brains occurred in a gene-dosage-dependent manner and Akt1 heterozygous (HET) mice exhibited impaired striatal Akt1 activity under methamphetamine challenge. No genotypic difference was found in the basal levels of dopamine and its metabolites. In a series of reward-related learning tasks, HET mice displayed a relatively efficient method of updating reward information from the environment during the acquisition phase of the two natural reward tasks and in the reverse section of the dynamic foraging T-maze but not in methamphetamine-induced or aversive-related reward learning. The implementation of a standard reinforcement learning model and the Bayesian hierarchical parameter estimation show that HET mice have higher RPE magnitudes and that their action values are updated more rapidly among all three test sections in T-maze. These results indicate that Akt1 deficiency modulates natural reward learning and RPE. This study showed a promising avenue for investigating RPE in mutant mice and provided evidence for the potential link from genetic deficiency, to neurobiological abnormalities, to impairment in higher-order cognitive functioning.

Differential roles of NR2A and NR2B-containing NMDA receptors in LTP and LTD in the CA1 region of two-week old rat hippocampus

The striatum is the main input structure of the basal ganglia circuits and the functions of this subcortical region are modified with dopaminergic innervation released from Substantia Nigra pars Compacta [1]. We have focused on the formation of this subcortical region and its relation to the dopamine neurotransmitter since striatum has a key role in the basal ganglia functions, such as action selection and reward-based learning. The aim of this study is to find a plausible way to explain the effect of dopamine release on the functions and formations of striatal microcircuits. This work also focuses on the pathological synchrony within D2 MSN network which causes the repetitive projections of the Globus Pallidus External and Subthalamic Nucleus network [2]. This pathological synchrony would be related to parkinsonian behavior as this network is primarily responsible for motor behaviors [3]. It is now claimed that structure of the striatum is modified with long term depression (LTD) and long term potentiation (LTP) and dopamine regulates this structure [4]. So, besides examining D2 network, we also have focused on D1 network to be able to explain how increased level of dopamine release causes the new formation within this region. In this work, we are following the Fast-Slow dynamical system approach to be able to model the Medium Spiny neurons. The striatal network comprises of 400 D1 MSNs, 400 D2 MSNs modeled as a conductance-based compartmental neuron model with 20 Fast-spiking inhibitory (FSIN) Adaptive and Exponential neuron models (%2) which matches the experimental observations [2]. Here, we show that dopamine has the ability to modify both D1 and D2 MSNs as it can be followed in Figure ​Figure1.1. With a biased dopamine release, both network reach stabilize firing regime while increased dopamine causes an overreacted firing pattern in D1 network, it suppresses D2 network firing rate, see Figure ​Figure1.1. In the proposed network FSINs have all-to-all connections to the D1 and D2 MSNs to understand how these feedforward inhibition controls the pathological neuronal oscillations. We have examined the contribution of different dopamine levels to control pathological neuronal oscillations and explained whether Poisson distributed spike trains correlate with dopamine level, significantly. Figure 1 Raster plot of striatal microcircuit which comprises 400 D1 MSN (top), 20 FSIN (middle) and 400 D2 MSN (bottom).

Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies