Prediction error in reinforcement learning: A meta-analysis of neuroimaging studies

Learning occurs when an outcome deviates from expectation (prediction error). According to formal learning theory, the defining paradigm demonstrating the role of prediction errors in learning is the blocking test. Here, a novel stimulus is blocked from learning when it is associated with a fully predicted outcome, presumably because the occurrence of the outcome fails to produce a prediction error. We investigated the role of prediction errors in human reward-directed learning using a blocking paradigm and measured brain activation with functional magnetic resonance imaging. Participants showed blocking of behavioral learning with juice rewards as predicted by learning theory. The medial orbitofrontal cortex and the ventral putamen showed significantly lower responses to blocked, compared with nonblocked, reward-predicting stimuli. In reward-predicting control situations, deactivations in orbitofrontal cortex and ventral putamen occurred at the time of unpredicted reward omissions. Responses in discrete parts of orbitofrontal cortex correlated with the degree of behavioral learning during, and after, the learning phase. These data suggest that learning in primary reward structures in the human brain correlates with prediction errors in a manner that complies with principles of formal learning theory.

Remembering to execute pre-defined intentions at the appropriate time in the future is typically referred to as Prospective Memory (PM). Studies of PM showed that distinct cognitive processes underlie the execution of delayed intentions depending on whether the cue associated with such intentions is focal to ongoing activity processing or not (i.e., cue focality). The present activation likelihood estimation (ALE) meta-analysis revealed several differences in brain activity as a function of focality of the PM cue. The retrieval of intention is supported mainly by left anterior prefrontal cortex (Brodmann Area, BA 10) in nonfocal tasks, and by cerebellum and ventral parietal regions in focal tasks. Furthermore, the precuneus showed increased activation during the maintenance phase of intentions compared to the retrieval phase in nonfocal tasks, whereas the inferior parietal lobule showed increased activation during the retrieval of intention compared to maintenance phase in the focal tasks. Finally, the retrieval of intention relies more on the activity in anterior cingulate cortex for nonfocal tasks, and on posterior cingulate cortex for focal tasks. Such focality-related pattern of activations suggests that prospective remembering is mediated mainly by top-down and stimulus-independent processes in nonfocal tasks, whereas by more automatic, bottom-up, processes in focal tasks.

Functional neuroimaging of the encephalon of humans with chronic musculoskeletal pain (CMP) has consistently demonstrated functional alterations in the neurophysiological properties of cortical and subcortical circuits. Nevertheless, the current knowledge on specific neural circuitries that may occur in different CMP subgroups is limited, which in turn limits the understanding of the encephalon mechanisms associated with persistent pain and clinical heterogeneity. This systematic review (CRD42022382309) and activation likelihood estimation (ALE) meta-analysis of observational functional magnetic resonance imaging (fMRI) studies on human encephalon aimed to characterize specific patterns of connectomic reorganization in different CMP subgroups. PubMed, Web of Science, and Scopus databases were searched. Two independent reviewers read titles and abstracts, full texts, assessed methodological quality using the Newcastle-Ottawa scale, and extracted X, Y, and Z coordinates. The data analyses were conducted using the GingerALE 3.0.2 software and complemented by frequency analyses. All neuroanatomical coordinates used in the ALE meta-analyses were standardized to the Montreal Neurological Institute 152 space. A 95% confidence interval (CI) for the family-wise error (FWE) rate was applied using an initial uncorrected voxel-level threshold of p < 0.001, together with 1000 permutation tests and a minimum cluster volume of ≥200 mm3. In total, 43 studies out of 5543 records met the inclusion criteria and none presented a very high risk of bias. The seven encephalon regions comprising the basic neural circuitry of CMP (somatosensory cortex, motor cortex, anterior cingulate cortex, insula, prefrontal cortex, thalamus, and cerebellum) correspond to an integrated connectomic hub that coordinates sensory-motor discriminative, affective-emotional, and cognitive-motivational processing. Additionally, 13 other regions are specifically recruited in distinct conditions of CMP. These parallel connections expand the processing network and add dimensions related to reward, memory, descending pain modulation, and attention. This dynamic reorganization reflects a pattern of cross-modal plasticity with maladaptive mechanisms.

The development and maturation of multisensory processing and integration depend on the brain’s ability to detect patterns and adjust behaviour based on feedback. However, the developmental dynamics of these mechanisms are not yet fully understood. This study examines neural and behavioural differences in multisensory statistical and reinforcement learning processes by comparing adults and children, and further investigates these processes across two learning tasks of varying difficulty in children. Using a discriminative choice and a match recognition multisensory associative learning task with embedded statistical stimulus regularities, 28 adults (19.0–30.9 years) and 2 groups of children (N = 28 and N = 29, 8.5–12.8 years) learned novel, non-linguistic associations of audio–visual or tactile–visual stimuli. Behaviourally, accuracy increased and reaction time decreased over time in all groups, and these learning effects were stronger in adults than in children and for audio–visual than for tactile–visual learning. Computational modelling revealed that adults employed more advanced learning strategies, with greater sensitivity to value differences, and more deterministic choice patterns. Behaviourally, reaction times were slower for more surprising trials across all groups. Neuroimaging showed that statistical surprise modulated activity in the lateral prefrontal cortex, intraparietal sulcus, and anterior cingulate cortex, while reward prediction errors were linked to striatal, medial frontal, and hippocampal regions. Reward prediction errors were consistent between adults and children and between task variants, whereas neural surprise processing emerged only in the match recognition task and showed developmental differences, indicating immaturity in children’s ability to detect statistical regularities. These findings highlight the importance of task design when comparing learning differences across developmental groups and showcase how reinforcement and statistical learning can be studied in parallel.

Key brain networks underlying cognition and emotion are modulated by major neurotransmitter systems through reward/punishment circuits. Notably, the basal ganglia are thought to funnel emotional information towards motor outcomes. However, the mechanisms that would allow emotional stimuli to guide action selection have yet to be identified. Computational models may contribute to this goal. Here, by using a computer simulation of the guidedpropagation deterministic model, we show that emotional channels can quickly and selectively modulate actionoriented channels, by instantly retrieving all the emotional stimuli paired in the past with a current cue. In agreement with animal based data, the transient modulation signals that implement emotional anticipation appear to be more useful when targeting either newly formed or remote memory traces. The timing and evolution of these signals both suggest a new interpretation of the dopaminergic neuron activity in the basal ganglia during conditioning, usually regarded as coding for a ‘reward prediction error’ in the frame of reinforcement learning. After additional computer trainings involving ‘emotions’ of extreme values, the diversity of actions selected under the influence of a conditioned cue is shown to decrease through either compulsive or avoidance behaviors. Indeed, in the proposed functional model, similar modulation mechanisms account for the development of either drug addiction or posttraumatic stress disorder. Furthermore, spontaneous relapse into these dysfunctions is attributed here to local modulating deficits. The latter can partly be overcome by selectively shifting one of the few control parameters of the model, akin to neuromodulators.

Model-based analysis of fMRI data is an important tool for investigating the computational role of different brain regions. With this method, theoretical models of behavior can be leveraged to find the brain structures underlying variables from specific algorithms, such as prediction errors in reinforcement learning. One potential weakness with this approach is that models often have free parameters and thus the results of the analysis may depend on how these free parameters are set. In this work we asked whether this hypothetical weakness is a problem in practice. We first developed general closed-form expressions for the relationship between results of fMRI analyses using different regressors, e.g., one corresponding to the true process underlying the measured data and one a model-derived approximation of the true generative regressor. Then, as a specific test case, we examined the sensitivity of model-based fMRI to the learning rate parameter in reinforcement learning, both in theory and in two previously-published datasets. We found that even gross errors in the learning rate lead to only minute changes in the neural results. Our findings thus suggest that precise model fitting is not always necessary for model-based fMRI. They also highlight the difficulty in using fMRI data for arbitrating between different models or model parameters. While these specific results pertain only to the effect of learning rate in simple reinforcement learning models, we provide a template for testing for effects of different parameters in other models.

Motor adaptation can be achieved through error-based learning, driven by sensory prediction errors, or reinforcement learning, driven by reward prediction errors. Recent work on visuomotor adaptation has shown that reinforcement learning leads to more persistent adaptation when visual feedback is removed, compared to error-based learning in which continuous visual feedback of the movement is provided. However, there is evidence that error-based learning with terminal visual feedback of the movement (provided at the end of movement) may be driven by both sensory and reward prediction errors. Here we examined the influence of feedback on learning using a visuomotor adaptation task in which participants moved a cursor to a single target while the gain between hand and cursor movement displacement was gradually altered. Different groups received either continuous error feedback (EC), terminal error feedback (ET), or binary reinforcement feedback (success/fail) at the end of the movement (R). Following adaptation we tested generalization to targets located in different directions and found that generalization in the ET group was intermediate between the EC and R groups. We then examined the persistence of adaptation in the EC and ET groups when the cursor was extinguished and only binary reward feedback was provided. Whereas performance was maintained in the ET group, it quickly deteriorated in the EC group. These results suggest that terminal error feedback leads to a more robust form of learning than continuous error feedback. In addition our findings are consistent with the view that error-based learning with terminal feedback involves both error-based and reinforcement learning.

Background:Several studies using functional magnetic resonance imaging (fMRI) and positron emission tomography (PET) have indicated that cognitive remediation therapy (CRT) might improve cognitive function by changing brain activations in patients with schizophrenia. However, the results were not consistent in these changed brain areas in different studies. The present activation likelihood estimation (ALE) meta-analysis was conducted to investigate whether cognitive function change was accompanied by the brain activation changes, and where the main areas most related to these changes were in schizophrenia patients after CRT. Analyses of whole-brain studies and whole-brain + region of interest (ROI) studies were compared to explore the effect of the different methodologies on the results.Methods:A computerized systematic search was conducted to collect fMRI and PET studies on brain activation changes in schizophrenia patients from pre- to post-CRT. Nine studies using fMRI techniques were included in the meta-analysis. Ginger ALE 2.3.1 was used to perform meta-analysis across these imaging studies.Results:The main areas with increased brain activation were in frontal and parietal lobe, including left medial frontal gyrus, left inferior frontal gyrus, right middle frontal gyrus, right postcentral gyrus, and inferior parietal lobule in patients after CRT, yet no decreased brain activation was found. Although similar increased activation brain areas were identified in ALE with or without ROI studies, analysis including ROI studies had a higher ALE value.Conclusions:The current findings suggest that CRT might improve the cognition of schizophrenia patients by increasing activations of the frontal and parietal lobe. In addition, it might provide more evidence to confirm results by including ROI studies in ALE meta-analysis.

Reward expectation and reward prediction errors are thought to be critical for dynamic adjustments in decision-making and reward-seeking behavior, but little is known about their representation in the brain during uncertainty and risk-taking. Furthermore, little is known about what role individual differences might play in such reinforcement processes. In this study, it is shown behavioral and neural responses during a decision-making task can be characterized by a computational reinforcement learning model and that individual differences in learning parameters in the model are critical for elucidating these processes. In the fMRI experiment, subjects chose between high- and low-risk rewards. A computational reinforcement learning model computed expected values and prediction errors that each subject might experience on each trial. These outputs predicted subjects' trial-to-trial choice strategies and neural activity in several limbic and prefrontal regions during the task. Individual differences in estimated reinforcement learning parameters proved critical for characterizing these processes, because models that incorporated individual learning parameters explained significantly more variance in the fMRI data than did a model using fixed learning parameters. These findings suggest that the brain engages a reinforcement learning process during risk-taking and that individual differences play a crucial role in modeling this process.

The sensation of dyspnea is related to various cardiopulmonary and neuromuscular diseases and is characterized by its sensory and affective qualities. Although there is a vast number of studies investigating its pathophysiology, less is known about the neuroanatomy of dyspnea perception. An activation likelihood estimation (ALE) meta-analysis of 13 studies investigating different breathing challenges using either PET or fMRI was performed to demonstrate the neuroanatomical correlates of dyspnea perception. The ALE meta-analysis was performed using the GingerAle software 3.0.2 and was displayed with the Mango software 4.1. Synthesizing the results of all included studies, clusters involving the insula and cingulated cortex in both hemispheres were observed. Subgroup analysis for the restrained breathing condition revealed activation involving the right and left cingulate cortex and left anterior cingulate cortex. For the loaded breathing condition, statistically significant activation was found for the postcentral gyrus, the superior temporal gyrus, and the right thalamus. The combined ALE map for both conditions showed activity patterns in the right cingulate cortex, the right insula, and the right thalamus. This ALE meta-analysis demonstrates that two separate neuronal pathways related to either the affective or intensity domain are involved in the central processing of dyspnea perception.

Attributing outcomes to your own actions or to external causes is essential for appropriately learning which actions lead to reward and which actions do not. Our previous work showed that this type of credit assignment is best explained by a Bayesian reinforcement learning model which posits that beliefs about the causal structure of the environment modulate reward prediction errors (RPEs) during action value updating. In this study, we investigated the brain networks underlying reinforcement learning that are influenced by causal beliefs using functional magnetic resonance imaging while human participants (n = 31; 13 males, 18 females) completed a behavioral task that manipulated beliefs about causal structure. We found evidence that RPEs modulated by causal beliefs are represented in dorsal striatum, while standard (unmodulated) RPEs are represented in ventral striatum. Further analyses revealed that beliefs about causal structure are represented in anterior insula and inferior frontal gyrus. Finally, structural equation modeling revealed effective connectivity from anterior insula to dorsal striatum. Together, these results are consistent with a possible neural architecture in which causal beliefs in anterior insula are integrated with prediction error signals in dorsal striatum to update action values.SIGNIFICANCE STATEMENT Learning which actions lead to reward-a process known as reinforcement learning-is essential for survival. Inferring the causes of observed outcomes-a process known as causal inference-is crucial for appropriately assigning credit to one's own actions and restricting learning to effective action-outcome contingencies. Previous studies have linked reinforcement learning to the striatum, and causal inference to prefrontal regions, yet how these neural processes interact to guide adaptive behavior remains poorly understood. Here, we found evidence that causal beliefs represented in the prefrontal cortex modulate action value updating in posterior striatum, separately from the unmodulated action value update in ventral striatum posited by standard reinforcement learning models.

BackgroundStructural neuroimaging findings in Subthreshold depression (StD) patients at different ages are highly heterogeneous. This study aims to investigate the pathophysiology of StD across different ages.MethodsLiterature searches for MRI studies of StD were conducted in 11 databases, including PubMed and Embase, from database inception to June 18, 2024. An activation likelihood estimation (ALE) meta-analysis was performed on the studies across different ages.ResultsA total of 24 studies were included. The results revealed that the significant convergent brain regions in StD patients across different ages were primarily located within the frontostriatal circuit. Age-related differences were observed. For adolescent patients, the significant convergent brain regions were the caudate, putamen, anterior cingulate cortex (ACC), and medial frontal gyrus (MFG). For young adult patients, the significant convergent brain regions were the inferior frontal gyrus, parahippocampal gyrus, insula, putamen, claustrum, and medial globus pallidus. For middle-aged and older patients, the significant convergent brain regions were the ACC, the MFG, and the superior frontal gyrus.ConclusionsThis study revealed that abnormalities in the frontostriatal circuit were neuroimaging features common in StD patients across different ages. Additionally, unique different brain regions were identified between age groups. These findings elucidated the mechanisms of StD and provided a theoretical basis for its prevention and treatment.

The present systematic review and meta-analysis provide an updated assessment of how eye movement desensitization and reprocessing (EMDR) therapy affects the human brain during treatment for posttraumatic stress disorder (PTSD). Brain region activity was assessed using resting-state functional magnetic resonance imaging, and trauma symptom severity was measured using the PTSD Checklist for the DSM-5 (PCL-5). We used PubMed, EMBASE, and ScienceDirect as our primary databases. Selected articles were subjected to seven eligibility criteria, and 22 articles remained for statistical analysis. Article quality was measured using the QUADAS-2 instrument, which showed that our review sample comprised high-quality articles. Statistical analyses implicated a forest plot meta-analysis of PCL-5 scores pretreatment and posttreatment, an activation likelihood estimation (ALE) meta-analysis of brain region activation after treatment, and a linear regression analysis that served as our method of statistical synthesis between forest plot and ALE results. Our forest plot showed that EMDR had a robust effect on decreasing PCL-5 scores during treatment (g= −2.92). Our ALE meta-analysis indicated the activation of different regions in the default mode and salience, limbic, and ventral attention networks. Statistical synthesis showed a robust correlation between PCL-5 score decrease and increased brain function (β = −.94). We also measured publication bias in our review sample with a funnel plot and Egger’s test of random effects (p= .280).

This paper presents a Basal Ganglia–inspired reinforcement learning (RL) model that unifies biological learning mechanisms with computational intelligence. The proposed framework emulates the functional dynamics of the Basal Ganglia through a three-layer neural architecture comprising cortical, striatal, and action-selection modules. Learning is governed by a dopamine-modulated synaptic plasticity rule, where the Reward Prediction Error (RPE) serves as the core teaching signal to adapt corticostriatal synaptic weights. A synthetic environment was programmatically constructed to generate discrete state–action–reward sequences, allowing the agent to learn through iterative trial-based interactions. The model employs soft-max based probabilistic action selection and updates synaptic weights according to the magnitude and polarity of the RPE signal, effectively replicating long-term potentiation (LTP) and long-term depression (LTD) observed in biological neurons. Experimental results demonstrate progressive improvement in cumulative rewards, stable RPE convergence, and consistent policy refinement across training episodes. The proposed model bridges the gap between computational reinforcement learning and neural decision-making, offering a biologically interpretable and efficient framework for adaptive behaviour modelling in brain-inspired artificial intelligence systems.

The neural basis of gambling disorder: An activation likelihood estimation meta-analysis