Neuronal implementation of the temporal difference learning algorithm in the midbrain dopaminergic system

The canonical two neuron model of opioid reward posits that mu opioid receptor (MOR) activation produces reward by disinhibiting midbrain ventral tegmental area (VTA) dopamine neurons through inhibition of local GABAergic interneurons. Although indirect evidence supports the neural circuit postulated by this model, its validity has been called into question by growing evidence for VTA neuronal heterogeneity and the recent demonstration that MOR agonists inhibit GABAergic terminals in the VTA arising from extrinsic neurons. In addition, VTA MOR reward can be dopamine-independent. To directly test the assumption that MOR activation directly inhibits local GABAergic neurons, we investigated the properties of rat VTA GABA neurons directly identified with either immunocytochemistry for GABA or GAD65/67, or in situ hybridization for GAD65/67 mRNA. Utilizing co-labeling with an antibody for the neural marker NeuN and in situ hybridization against GAD65/67, we found that 23±3% of VTA neurons are GAD65/67(+). In contrast to the assumptions of the two neuron model, VTA GABAergic neurons are heterogeneous, both physiologically and pharmacologically. Importantly, only 7/13 confirmed VTA GABA neurons were inhibited by the MOR selective agonist DAMGO. Interestingly, all confirmed VTA GABA neurons were insensitive to the GABAB receptor agonist baclofen (0/6 inhibited), while all confirmed dopamine neurons were inhibited (19/19). The heterogeneity of opioid responses we found in VTA GABAergic neurons, and the fact that GABA terminals arising from neurons outside the VTA are inhibited by MOR agonists, make further studies essential to determine the local circuit mechanisms underlying VTA MOR reward.

<p>The glutamatergic and GABAergic neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) mediated diverse brain functions. However, their whole-brain neural connectivity has not been comprehensively mapped. Here we used the virus tracers to characterize the whole-brain inputs and outputs of glutamatergic and GABAergic neurons in VTA and SNc. We found that these neurons received similar inputs from upstream brain regions, but some quantitative differences were also observed. Neocortex and dorsal striatum provided a greater share of input to VTA glutamatergic neurons. Periaqueductal gray and lateral hypothalamic area preferentially innervated VTA GABAergic neurons. Specifically, superior colliculus provided the largest input to SNc glutamatergic neurons. Compared to input patterns, the output patterns of glutamatergic and GABAergic neurons in the VTA and SNc showed significant preference to different brain regions. Our results laid the anatomical foundation for understanding the functions of cell-type-specific neurons in VTA and SNc.</p>

Can the apparent adaptation of dopamine neurons’ mismatch sensitivities be reconciled with their computation of reward prediction errors?

Activation of mu-opioid receptor (MOR) disinhibits dopaminergic neurons in the ventral tegmental area (VTA) through inhibition of γ-aminobutyric acid (GABA)ergic neurons. This mechanism is thought to play a pivotal role in mediating reward behaviors. Here, we characterised VTA-projecting enkephalinergic neurons in the anterior division of the bed nucleus of the stria terminalis (BST) and investigated their targets by examining MOR expression in the VTA. In the BST, neurons expressing preproenkephalin mRNA were exclusively GABAergic, and constituted 37.2% of the total GABAergic neurons. Using retrograde tracer injected into the VTA, 21.6% of VTA-projecting BST neurons were shown to express preproenkephalin mRNA. Enkephalinergic projections from the BST exclusively formed symmetrical synapses onto the dendrites of VTA neurons. In the VTA, 74.1% of MOR mRNA-expressing neurons were GABAergic, with the rest being glutamatergic neurons expressing type-2 vesicular glutamate transporter mRNA. However, MOR mRNA was below the detection threshold in dopaminergic neurons. By immunohistochemistry, MOR was highly expressed on the extrasynaptic membranes of dendrites in GABAergic VTA neurons, including dendrites innervated by BST-VTA projection terminals. MOR was also expressed weakly on GABAergic and glutamatergic terminals in the VTA. Given that GABAA α1 is expressed at GABAergic BST-VTA synapses on dendrites of GABAergic neurons [T. Kudo etal. (2012) J. Neurosci., 32, 18035-18046], our results collectively indicate that the BST sends dual inhibitory outputs targeting GABAergic VTA neurons; GABAergic inhibition via 'wired' transmission, and enkephalinergic inhibition via 'volume' transmission. This dual inhibitory system provides the neural substrate underlying the potent disinhibitory control over dopaminergic VTA neurons exerted by the BST.

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus–reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.

Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways

A bone-derived hormone permits rapid visual escape via GPR37 receptor in a subpopulation of VTA GABAergic neurons.

Long-term morphine use is associated with serious side effects, such as morphine-induced hyperalgesia and analgesic tolerance. Previous investigations have documented the association between dopamine (DA) neurons in the ventral tegmental area (VTA) and pain. However, whether VTA DA neurons are implicated in morphine-induced hyperalgesia and analgesic tolerance remains elusive. Initially, we observed behavioural effects of lidocaine administration into VTA or ablation of VTA DA neurons on morphine-induced hyperalgesia and anti-nociceptive tolerance. Subsequently, c-Fos expression in nucleus accumbens (NAc) shell-projecting and medial prefrontal cortex (mPFC)-projecting VTA DA neurons after chronic morphine treatment was respectively investigated. Afterwards, the effects of chemogenetic manipulation of NAc shell-projecting or mPFC-projecting DA neurons on morphine-induced hyperalgesia and anti-nociceptive tolerance were observed. Additionally, effects of chemogenetic manipulation of VTA GABA neurons on c-Fos expression in VTA DA neurons were investigated. Lidocaine injection into VTA relieved established hyperalgesia and anti-nociceptive tolerance whereas ablation of VTA DA neurons prevented the development of morphine-induced hyperalgesia and anti-nociceptive tolerance. Chronic morphine treatment increased c-Fos expression in NAc shell-projecting DA neurons, rather than in mPFC-projecting DA neurons. Chemogenetic manipulation of NAc shell-projecting DA neurons had influence on morphine-induced hyperalgesia and tolerance. However, chemogenetic manipulation of mPFC-projecting DA neurons had no significant effects on morphine-induced hyperalgesia and anti-nociceptive tolerance. Chemogenetic manipulation of VTA GABA neurons affected the c-Fos expression in VTA DA neurons. These findings revealed the involvement of NAc shell-projecting VTA DA neurons in morphine-induced hyperalgesia and anti-nociceptive tolerance, and may shed new light on the clinical management of morphine-induced hyperalgesia and analgesic tolerance. This study demonstrated that NAc shell-projecting DA neurons rather than mPFC-projecting DA neurons in the VTA were implicated in morphine-induced hyperalgesia and anti-nociceptive tolerance. Our findings may pave the way for the discovery of novel therapies for morphine-induced hyperalgesia and analgesic tolerance.

Dopamine (DA) neurons are thought to encode reward prediction error (RPE), in addition to other signals, such as salience. While RPE is known to support learning, the role of salience in learning remains less clear. To address this, we recorded and manipulated VTA DA neurons in mice during fear extinction. We applied deep learning to classify mouse freezing behavior, eliminating the need for human scoring. Our fiber photometry recordings showed DA neurons in medial and lateral VTA have distinct activity profiles during fear extinction: medial VTA activity more closely reflected RPE, while lateral VTA activity more closely reflected a salience-like signal. Optogenetic inhibition of DA neurons in either region slowed fear extinction, with the relevant time period for inhibition differing across regions. Our results indicate salience-like signals can have similar downstream consequences to RPE-like signals, although with different temporal dependencies.

How does the activity of midbrain dopamine (DA) neurons reinforce actions? A prominent hypothesis is that the activity of ventral tegmental area (VTA) DA neurons instructs representations of predicted reward, or value, in downstream neurons1. To directly test this model, we performed comprehensive striatal recordings in mice engaged in a trial-and-error probabilistic learning task where they continuously adapted their choices to obtain a reward of optogenetic stimulation of VTA DA neurons (paired with an auditory cue). We then assessed neural representations of action values (estimated from a behavioral model), revealing for the first time that VTA DA stimulation is sufficient to generate downstream neural correlates of action value. Surprisingly, these value correlates were strongest in the intermediate caudoputamen (CP) and weakest in the nucleus accumbens (NAc), despite NAc being the major projection target of VTA DA neurons2,3. This was true not only for the value of each choice, but also for state value (reward expectation) and relative value (the decision variable). However, value representations were differentially organized within the intermediate CP, with ventromedial domains (which receive inputs from orbitofrontal cortex) preferentially encoding state value and dorsolateral domains (which receive inputs from motor cortex) preferentially encoding relative value. A difference in learning rate for the value computation between NAc and CP did not explain the relatively weak value correlates in NAc. Instead, we found that VTA DA stimulation was sufficient to produce learned neural responses to the stimulation-paired auditory cue throughout the striatum, including in the NAc, and that animals work for this cue rather than for VTA DA stimulation itself. Overall, this suggests that VTA DA neurons support trial-and-error learning indirectly, by making stimuli valuable (“conditioned reinforcers”), which in turn support the generation of action value representations in the CP.

This paper revisits the temporal difference (TD) learning algorithm for the policy evaluation tasks in reinforcement learning. Typically, the performance of TD(0) and TD( λ) is very sensitive to the choice of stepsizes. Oftentimes, TD(0) suffers from slow convergence. Motivated by the tight link between the TD(0) learning algorithm and the stochastic gradient methods, we develop a provably convergent adaptive projected variant of the TD(0) learning algorithm with linear function approximation that we term AdaTD(0). In contrast to the TD(0), AdaTD(0) is robust or less sensitive to the choice of stepsizes. Analytically, we establish that to reach an ϵ accuracy, the number of iterations needed is [Formula: see text] in the general case, where ρ represents the speed of the underlying Markov chain converges to the stationary distribution. This implies that the iteration complexity of AdaTD(0) is no worse than that of TD(0) in the worst case. When the stochastic semi-gradients are sparse, we provide theoretical acceleration of AdaTD(0). Going beyond TD(0), we develop an adaptive variant of TD( λ), which is referred to as AdaTD( λ). Empirically, we evaluate the performance of AdaTD(0) and AdaTD( λ) on several standard reinforcement learning tasks, which demonstrate the effectiveness of our new approaches.

Connexin-36 (Cx36) gap junctions (GJs) appear to be involved in the synchronization of GABA interneurons in many brain areas. We have previously identified a population of Cx36-connected ventral tegmental area (VTA) GABA neurons that may regulate mesolimbic dopamine (DA) neurotransmission, a system implicated in reward from both natural behaviors and drugs of abuse. The aim of this study was to determine the effect mefloquine (MFQ) has on midbrain DA and GABA neuron inhibition, and the role Cx36 GJs play in regulating midbrain VTA DA neuron activity in mice. In brain slices from adolescent wild-type (WT) mice the Cx36-selective GJ blocker mefloquine (MFQ, 25 μM) increased VTA DA neuron sIPSC frequency sixfold, and mIPSC frequency threefold. However, in Cx36 KO mice, MFQ only increased sIPSC and mIPSC frequency threefold. The nonselective GJ blocker carbenoxolone (CBX, 100 μM) increased DA neuron sIPSC frequency twofold in WT mice, did not affect Cx36 KO mouse sIPSCs, and did not affect mIPSCs in WT or Cx36 KO mice. Interestingly, MFQ had no effect on VTA GABA neuron sIPSC frequency. We also examined MFQ effects on VTA DA neuron firing rate and current-evoked spiking in WT and Cx36 KO mice, and found that MFQ decreased WT DA neuron firing rate and current-evoked spiking, but did not alter these measures in Cx36 KO mice. Taken together these findings suggest that blocking Cx36 GJs increases VTA DA neuron inhibition, and that GJs play in key role in regulating inhibition of VTA DA neurons. Synapse, 2011. © 2011 Wiley-Liss, Inc.

A comparison of the effects of nicotine on dopamine and non-dopamine neurons in the rat ventral tegmental area: an in vitro electrophysiological study

The circuitry mediating top-down control of dopamine (DA) neurons in the ventral tegmental area (VTA) is exceedingly complex. Characterizing these networks will be critical to our understanding of fundamental behaviors, such as motivation and reward processing, as well as several disease states. Previous work suggests that the medial prefrontal cortex (mPFC) exerts a profound influence on VTA DA neuron firing. Recently, our group reported that inhibition of the infralimbic subdivision of the medial prefrontal cortex (ilPFC) increases the proportion of VTA DA neurons that are spontaneously active (i.e., "population activity") and that this effect depends on activity in the ventral subiculum of the hippocampus (vSub). However, there is no direct projection from the mPFC to the vSub. Anatomical evidence suggests that communication between the two structures is mediated by the nucleus reuniens of the midline thalamus (RE). Here, we used in vivo electrophysiological and behavioral approaches in rats to explore the role of the RE in the circuitry governing VTA DA neuron firing. We show that pharmacological stimulation of the RE enhances VTA DA neuron population activity and amphetamine-induced hyperlocomotion, a behavioral indicator of an over-responsive DA system. Furthermore, the effect of RE stimulation on population activity is prevented if vSub is also inhibited. Finally, pharmacological inhibition of ilPFC enhances VTA DA neuron population activity, but this effect does not occur if RE is also inhibited. These findings suggest that disruption of ilPFC-RE-vSub communication could lead to a dysregulated, hyperdopaminergic state, and may play a role in psychiatric disorders. Dopamine (DA) neurons in the ventral tegmental area (VTA) are involved in a variety of fundamental brain functions. To understand the neurobiological basis for these functions it is essential to identify regions controlling DA neuron activity. The medial prefrontal cortex (mPFC) is emerging as a key regulator of DA neuron activity, but the circuitry by which it exerts its influence remains poorly described. Here, we show that the nucleus reuniens of the midline thalamus gates mPFC control of VTA DA neuron firing by the hippocampus. These data identify a unique role for this corticothalamic-hippocampal circuit, and suggest that dysfunction in these regions likely influences the pathophysiology of psychiatric disorders.

Dopamine (DA) neurons of the VTA track cues and rewards to generate a reward prediction error signal during Pavlovian conditioning. Here we explored how these neurons respond to a self-paced, operant task in freely moving mice. The animal could trigger a reward-predicting cue by remaining in a specific location of an operant box for a brief time before moving to a spout for reward collection. VTA DA neurons were identified using DAT-Cre male mice that carried an optrode with minimal impact on the behavioral task. In vivo single-unit recordings revealed transient fast spiking responses to the cue and reward in correct trials, while for incorrect ones the activity paused, reflecting positive and negative error signals of a reward prediction. In parallel, a majority of VTA DA neurons simultaneously encoded multiple actions (e.g., movement velocity, acceleration, distance to goal, and licking) in sustained slow firing modulation. Applying a GLM, we show that such multiplexed encoding of rewarding and motor variables by individual DA neurons was only apparent while the mouse was engaged in the task. Downstream targets may exploit such goal-directed multiplexing of VTA DA neurons to adjust actions to optimize the task's outcome.SIGNIFICANCE STATEMENT VTA DA neurons code for multiple functions, including the reward prediction error but also motivation and locomotion. Here we show that about half of the recorded VTA DA neurons perform multiplexing: they exploit the phasic and tonic activity modes to encode, respectively, the cue/reward responses and motor parameters, most prominently when the mouse engages in a self-paced operand task. VTA non-DA neurons, by contrast, encode motor parameters regardless of task engagement.