Human Neural Learning Depends on Reward Prediction Errors in the Blocking Paradigm

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus–reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.

In order to survive, most animals including humans need to be able to learn and adapt flexibly their behavior so that optimal choices can be made in an uncertain environment. This chapter reviews functional neuroimaging and neuropsychological studies on the nature of the orbitofrontal cortices (OFC) contribution to adaptive and flexible behavior in humans. These studies indicate that the OFC encodes the reward and punishment value of stimuli, maintains flexible representations of predicted reward and punishment value (using both stimulus substitution and CS-specific coding mechanisms), encodes errors in reward prediction, and signals future behavioral choice. The OFC shows heterogeneous response profiles with distinct regions mediating each of these functions. The relationship of the OFC to other brains regions processing reward is also discussed.

Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies

In numerous brain areas, neuronal activity varies according to reward predictability. In many of these areas this activity is thought to represent errors in reward prediction, as has been described for dopamine neurons; however, it might alternatively be related to the animal's enhanced behavioural state, which may include surprise or changes in arousal or attention. Unfortunately, few studies have examined firing in these areas in the context of the same behavioural task, making it difficult to dissociate different types of encoding. This chapter compares neural correlates in these areas to that of dopamine neurons in the same behavioural task. It shows that while activity in dopamine neurons appears to signal prediction errors, similar activity in the orbitofrontal cortex, basolateral amygdala, and ventral striatum does not. Instead, increased firing in basolateral amygdala to unexpected outcomes likely reflects attention, whereas activity in the orbitofrontal cortex and ventral striatum is unaffected by prior expectations and may provide information on outcome expectancy. These results have important implications for how these areas interact to facilitate learning and guide behaviour.

Imaging of Eating Disorders: Multiple Techniques to Demonstrate the Dynamic Brain

The question of how the brain represents reward prediction errors is central to reinforcement learning and adaptive, goal-directed behavior. Previous studies have revealed prediction error representations in multiple electrophysiological signatures, but it remains elusive whether these electrophysiological correlates underlying prediction errors are sensitive to valence (in a signed form) or to salience (in an unsigned form). One possible reason concerns the loose correspondence between objective probability and subjective prediction resulting from the optimistic bias, that is, the tendency to overestimate the likelihood of encountering positive future events. In the present electroencephalography (EEG) study, we approached this question by directly measuring participants' idiosyncratic, trial-to-trial prediction errors elicited by subjective and objective probabilities across two experiments. We adopted monetary gain and loss feedback in Experiment 1 and positive and negative feedback as communicated by the same zero-value feedback in Experiment 2. We provided electrophysiological evidence in time and time-frequency domains supporting both reward and salience prediction error signals. Moreover, we showed that these electrophysiological signatures were highly flexible and sensitive to an optimistic bias and various forms of salience. Our findings shed new light on multiple presentations of prediction error in the human brain, which differ in format and functional role.

Background and HypothesisNegative symptoms of schizophrenia (SCZ), particularly amotivation, are prominent across both SCZ and bipolar disorder (BD). While orbitofrontal cortex (OFC) alterations have been implicated in the development of negative symptoms, their contributions across disorders remain to be established. Here, we examined how OFC thickness and network associations relate to amotivation compared to diminished expression across the BD-SCZ spectrum.Study DesignWe included 50 individuals with SCZ, 49 with BD, and 122 controls. We assessed amotivation and diminished expression and estimated thickness in the medial and lateral OFC as regions of interest as well as 64 other cortical regions.Study ResultsAcross BD and SCZ, reduced right lateral and bilateral medial OFC thickness were specifically associated with amotivation, but not diminished expression or other clinical factors. We then generated intra-individual OFC structural covariance networks to evaluate how the system-level embedding of the OFC would link to brain-wide cortical maps of negative symptoms. We found that medial OFC covariance networks spatially correlated with the brain-wide cortical alterations of both negative symptom dimensions. Further analyses in independent SCZ data from the ENIGMA consortium (n = 4474) revealed associations with lateral OFC covariance networks. Finally, the brain-wide cortical alterations of amotivation were significantly correlated with normative functional and structural white-matter connectivity profiles of the right medial and left lateral OFC as well as adjacent prefrontal and limbic regions.ConclusionsOur work identifies OFC alterations as a possible transdiagnostic signature of amotivation and provides insights into network associations underlying the system-wide cortical alterations of negative symptoms across SCZ and BD.

Rapid homotopic communication between human orbitofrontal subregions.

The orbitofrontal cortex is critical for goal-directed behavior. Recent work in macaques has suggested the lateral orbitofrontal cortex (lOFC) is relatively more concerned with assignment of credit for rewards to particular choices during value-guided learning, whereas the medial orbitofrontal cortex (often referred to as ventromedial prefrontal cortex in humans; vmPFC/mOFC) is involved in constraining the decision to the relevant options. We examined whether people with damage restricted to subregions of prefrontal cortex showed the patterns of impairment observed in prior investigations of the effects of lesions to homologous regions in macaques. Groups of patients with either lOFC (predominantly right hemisphere), mOFC/vmPFC, or dorsomedial prefrontal (DMF), and a comparison group of healthy age- and education-matched controls performed a probabilistic 3-choice decision-making task. We report anatomically specific patterns of impairment. We found that credit assignment, as indexed by the normal influence of contingent relationships between choice and reward, is reduced in lOFC patients compared with Controls and mOFC/vmPFC patients. Moreover, the effects of reward contingency on choice were similar for patients with lesions in DMF or mOFC/vmPFC, compared with Controls. By contrast, mOFC/vmPFC-lesioned patients made more stochastic choices than Controls when the decision was framed by valuable distracting alternatives, suggesting that value comparisons were no longer independent of irrelevant options. Once again, there was evidence of regional specialization: patients with lOFC lesions were unimpaired relative to Controls. As in macaques, human lOFC and mOFC/vmPFC are necessary for contingent learning and value-guided decision-making, respectively.SIGNIFICANCE STATEMENT The lateral and medial regions of the orbitofrontal cortex are cytoarchitectonically distinct and have different anatomical connections. Previous investigations in macaques have shown these anatomical differences are accompanied by functional specialization for learning and decision-making. Here, for the first time, we test the predictions made by macaque studies in an experiment with humans with frontal lobe lesions, asking whether behavioral impairments can be linked to lateral or medial orbitofrontal cortex. Using equivalent tasks and computational analyses, our findings broadly replicate the pattern reported after selective lesions in monkeys. Patients with lateral orbitofrontal damage had impaired credit assignment, whereas damage to medial orbitofrontal cortex meant that patients were more likely to be distracted by irrelevant options.

Model-based analysis of fMRI data is an important tool for investigating the computational role of different brain regions. With this method, theoretical models of behavior can be leveraged to find the brain structures underlying variables from specific algorithms, such as prediction errors in reinforcement learning. One potential weakness with this approach is that models often have free parameters and thus the results of the analysis may depend on how these free parameters are set. In this work we asked whether this hypothetical weakness is a problem in practice. We first developed general closed-form expressions for the relationship between results of fMRI analyses using different regressors, e.g., one corresponding to the true process underlying the measured data and one a model-derived approximation of the true generative regressor. Then, as a specific test case, we examined the sensitivity of model-based fMRI to the learning rate parameter in reinforcement learning, both in theory and in two previously-published datasets. We found that even gross errors in the learning rate lead to only minute changes in the neural results. Our findings thus suggest that precise model fitting is not always necessary for model-based fMRI. They also highlight the difficulty in using fMRI data for arbitrating between different models or model parameters. While these specific results pertain only to the effect of learning rate in simple reinforcement learning models, we provide a template for testing for effects of different parameters in other models.

Prediction error signals have been reported in human imaging studies in target areas of dopamine neurons such as ventral and dorsal striatum during learning with many different types of reinforcers. However, a key question that has yet to be addressed is whether prediction error signals recruit distinct or overlapping regions of striatum and elsewhere during learning with different types of reward. To address this, we scanned 17 healthy subjects with functional magnetic resonance imaging while they chose actions to obtain either a pleasant juice reward (1 ml apple juice), or a monetary gain (5 cents) and applied a computational reinforcement learning model to subjects' behavioral and imaging data. Evidence for an overlapping prediction error signal during learning with juice and money rewards was found in a region of dorsal striatum (caudate nucleus), while prediction error signals in a subregion of ventral striatum were significantly stronger during learning with money but not juice reward. These results provide evidence for partially overlapping reward prediction signals for different types of appetitive reinforcers within the striatum, a finding with important implications for understanding the nature of associative encoding in the striatum as a function of reinforcer type.

22 stable, medicated SZ and 19 matched HC completed a probabilistic monetary reward task, modeled after Rolls, McCabe, Redoute4. In this study we performed a magnetic resonance spectroscopy (MRS) study of the SN and a functional MRI (fMRI) study during performance of a PE task in patients with schizophrenia (SZ) and controls (HC). We hypothesized that in SZ (1) an abnormal relationship exists between blood-oxygen-level dependent (BOLD) signal during PE in the SN and ventral striatum, (2) glutamate + glutamine (Glx) levels are elevated in the SN, and (3) the relationship between PE and Glx is different between SZ and HC. It has been proposed that in schizophrenia hypofunction of N-methyl-D-aspartate (NMDA) receptors on γ-aminobutyric acid (GABA) inhibitory neurons could lead to excess glutamate (Glu) release. Because the substantia nigra (SN) receives Glu projections from cortical areas, increased cortical Glu transmission could therefore affect the SN and lead to altered dopaminergic (DA) transmission – a phenomenon well documented in schizophrenia1. Reward learning tasks are known to modulate the DA system2. Reward prediction errors (PE) occur when reward outcomes differ from what is predicted and DA neurons change their firing rates in response to reward PE3. INTRODUCTION

A leading theory of the negative symptoms of schizophrenia is that they reflect reduced responsiveness to rewarding stimuli. This proposal has been linked to abnormal (reduced) dopamine function in the disorder, because phasic release of dopamine is known to code for reward prediction error (RPE). Nevertheless, few functional imaging studies have examined if patients with negative symptoms show reduced RPE-associated activations. Matched groups of DSM-5 schizophrenia patients with high negative symptom scores (HNS, N = 27) or absent negative symptoms (ANS, N = 27) and healthy controls (HC, N = 30) underwent fMRI scanning while they performed a probabilistic monetary reward task designed to generate a measure of RPE. In the HC, whole-brain analysis revealed that RPE was positively associated with activation in the ventral striatum, the putamen, and areas of the lateral prefrontal cortex and orbitofrontal cortex, among other regions. Group comparison revealed no activation differences between the healthy controls and the ANS patients. However, compared to the ANS patients, the HNS patients showed regions of significantly reduced activation in the left ventrolateral and dorsolateral prefrontal cortex, and in the right lingual and fusiform gyrus. HNS and ANS patients showed no activation differences in ventral striatal or midbrain regions-of-interest (ROIs), but the HNS patients showed reduced activation in a left orbitofrontal cortex ROI. The findings do not suggest that a generalized reduction of RPE signalling underlies negative symptoms. Instead, they point to a more circumscribed dysfunction in the lateral frontal and possibly the orbitofrontal cortex.

Situations where rewards are unexpectedly obtained or withheld represent opportunities for new learning. Often, this learning includes identifying cues that predict reward availability. Unexpected rewards strongly activate midbrain dopamine neurons. This phasic signal is proposed to support learning about antecedent cues by signaling discrepancies between actual and expected outcomes, termed a reward prediction error. However, it is unknown whether dopamine neuron prediction error signaling and cue-reward learning are causally linked. To test this hypothesis, we manipulated dopamine neuron activity in rats in two behavioral procedures, associative blocking and extinction, that illustrate the essential function of prediction errors in learning. We observed that optogenetic activation of dopamine neurons concurrent with reward delivery, mimicking a prediction error, was sufficient to cause long-lasting increases in cue-elicited reward-seeking behavior. Our findings establish a causal role for temporally-precise dopamine neuron signaling in cue-reward learning, bridging a critical gap between experimental evidence and influential theoretical frameworks.

There is general consensus that dopaminergic midbrain neurons signal reward prediction errors, computed as the difference between expected and received reward value. However, recent work in rodents shows that these neurons also respond to errors related to inferred value and sensory features, indicating an expanded role for dopamine beyond learning cached values. Here we utilize a transreinforcer reversal learning task and functional magnetic resonance imaging (fMRI) to test whether prediction error signals in the human midbrain are evoked when the expected identity of an appetitive food odor reward is violated, while leaving value matched. We found that midbrain fMRI responses to identity and value errors are correlated, suggesting a common neural origin for these error signals. Moreover, changes in reward-identity expectations, encoded in the orbitofrontal cortex (OFC), are directly related to midbrain activity, demonstrating that identity-based error signals in the midbrain support the formation of outcome identity expectations in OFC.