A Bayesian framework for simultaneously modeling neural and behavioral data

Model-based cognitive neuroscience

Introduction: Obsessive-compulsive disorder (OCD) and treatment-resistant bipolar depression (TRBD) constitute some of the most challenging psychiatric disorders to treat, as a significant proportion of patients demonstrate poor response to pharmacological strategies alone or in combination with psychoterapeutic approaches.1 Deep brain stimulation (DBS) is a potential therapeutic approach for select patients, although clinical outcomes remain inconsistent. To improve treatment efficacy, clinicians need to gain a deeper understanding of the neural activity and associated behavioral outcomes that correspond to distinct symptom profiles.2 One of the major hurdles in any effort to explore psychiatric neuromodulation is linking neural activity with behavior that we can observe. Examining correlated neural features with behaviorial information may provide biomarkers to inform DBS interventions.3-5 The present study introduces a platform that integrates video, audio, and neural data, including theta-band activity and spectrograms, all collected during real-world observations. This multimodal data approach enables clinicians to examine brain-behavior interactions with greater precision, facilitating the development of more individualized and effective treatment strategies. Objective: to validate the feasibility of frame-level synchronization of video, audio, and intracranial signals during real-world clinical sessions, and to demonstrate its potential to support behavioral–neural biomarker discovery for DBS in OCD and TRBD.Method: We implemented a hardware-based synchronization system designed for ecological clinical environments, that interfaces continuous high-definition video (FLIR Blackfly), high-fidelity audio, and direct neural recordings from implanted DBS systems. As shown in Figure 1, an Arduino-based controller generates a unique frame ID that increases sequentially, embedding it into one audio channel. This ID serves as the common temporal reference to align video and audio streams, which are later synchronized with the neural recordings. This platform aligns: Patient video recordings capturing spontaneous behavior and facial expressions; multichannel audio, including verbal responses and environmental context; intracranial electrophysiology from DBS electrodes. By extracting the frame ID from the audio channel, all modalities can be aligned with sub-frame accuracy. This approach remains robust to environmental noise and brief recording outages, which is crucial for long term clinical monitoring. The resulting output is processed and merged into a single synchronized outputs for subsequent behavioral annotation and biomarker analysis.Results: In recent recordings, the system was able to constantly align continuous video, audio, and neural signals over sessions lasting up to one hour. The system was validated across five patient sessions, consistently achieving stable synchronization throughout. Each dataset contained thousands of precisely matched frame identifiers, ensuring accurate temporal alignment across all modalities. The resulting synchronized outputs (Figure 2) allowed frame-by-frame correlation of patient behavior, speech patterns, and neural activity without perceptible desynchronization. Across recordings, alignment error remained below one frame (<33 ms), ensuring temporal precision sufficient for clinical and research analysis.Discussion: Synchronizing neural, behavioral, and contextual data enables real-time insight into brain–behavior relationships, helping identify biomarkers that can inform and refine DBS treatment. In the future, synchronized datasets could be integrated into clinical routines by allowing physicians to review neural activity alongside patient behavior and speech, providing a more objective complement to standard psychiatric assessments. One limitation is that the system has so far been validated in sessions of about one hour, which is well suited for proof-of-concept but more challenging to scale to recordings lasting several hours.Conclusions: This platform demonstrates the feasibility of precise multimodal synchronization in DBS patients, providing clinicians with richer, time-aligned datasets to link behavior and neural activity. These results lay the groundwork for future clinical studies aimed at validating multimodal biomarkers and testing whether such data can inform adaptive DBS programming.

Intracortical brain-computer interfaces (iBCIs) can restore movement and communication abilities to individuals with paralysis by decoding their intended behavior from neural activity recorded with an implanted device. While this activity yields high-performance decoding over short timescales, neural data are often nonstationary, which can lead to decoder failure if not accounted for. To maintain performance, users must frequently recalibrate decoders, which requires the arduous collection of new neural and behavioral data. Aiming to reduce this burden, several approaches have been developed that either limit recalibration data requirements (few-shot approaches) or eliminate explicit recalibration entirely (zero-shot approaches). However, progress is limited by a lack of standardized datasets and comparison metrics, causing methods to be compared in an ad hoc manner. Here we introduce the FALCON benchmark suite (Few-shot Algorithms for COnsistent Neural decoding) to standardize evaluation of iBCI robustness. FALCON curates five datasets of neural and behavioral data that span movement and communication tasks to focus on behaviors of interest to modern-day iBCIs. Each dataset includes calibration data, optional few-shot recalibration data, and private evaluation data. We implement a flexible evaluation platform which only requires user-submitted code to return behavioral predictions on unseen data. We also seed the benchmark by applying baseline methods spanning several classes of possible approaches. FALCON aims to provide rigorous selection criteria for robust iBCI decoders, easing their translation to real-world devices.

Decision Making in Recurrent Neuronal Circuits

Emerging neuroscience research shows that environmental-related activity such as wayfinding, perception, cognition, and their behavioral consequences—anxiety, stress, happiness, and arousal—are both reflected in our brains' neural structures and electro-chemical processes (Amor, Oboyle, Pati, Pham, & Jou, 2014; Amor, Pati & OBoyle, 2013; Pati, Amor, & OBoyle, 2012; Eberhard, 2007; Mallgrave, 2011; Swanson, 2011; Zeisel, 2006). Behavioral evidences indicate that fluorescent lighting, among the indoor environmental variables, plays a critical role in facilitating or hindering daily activities for a neuro-typical population (Rashid & Zimiring, 2008). For a neuro-diverse population (ADHD), this becomes more complex. While there is a growing body of debatable environment behavior literature relative to the impact of fluorescent lighting on cognitive, behavioral, and psychosocial outcomes, little is known about the correlation between neural activity and fluorescent lighting. The purpose of this research is to 1) explore and compare behavioral and neural responses of twenty ADHD subjects, when exposed to 3 types of fluorescent lighting color temperature: a) Warm White WW with a 2800 CCT, b) Cool White CW with a 4100 CCT, and c) Daylight DX with 6000 CCT when presented in three different settings—commercial, educational, and healthcare (see figures 1, 2, and 3), and 2) explore the impact of different color temperatures on the activation of the prefrontal and parietal areas, brain regions associated with cognition that experience minimal neural activity for people with ADHD (see figure 4). A purposive sampling strategy was used to generate twenty ADHD subjects. This permitted the selection of subjects including gender, age, ethnicity, brain lateralization, as well as the exclusion of color blindness. The participants underwent 1) an anatomical scan T1 (5 minutes) and 2) an Eco-Planar Imaging (EPI) functional scan (17 minutes), while a random sequence of 72 digitally generated high resolution illustrations from the aforementioned settings were projected by a computer controlled visual presentation system (E-Prime). Concomitantly, the participants were asked to respond to each image by fiber optics Button Response Unit (BRU)rating each image on a seven-point Likert satisfaction scale of 1=very dissatisfied and 7=very satisfied. Behavioral data was analyzed using t-test factor analysis and one-way analysis of variance, while the neural data maps were analyzed using Statistical Parametric Mapping (SPM8). The comparison of neural and behavioral data indicated that the least satisfying color is the White Warm Color Temperature 2800K. The Cool White 4100K and Full Spectrum 6000K correlated color temperatures generated better levels of satisfaction. Important to note that under the Cool White and Full Spectrum, the activation of the Cerebellum, the Superior Temporal Gyrus, the Middle Frontal Gyrus, and the Angular Gyrus respectively responsible for critical structure of social interaction, analytical tasks, and memory retrieval are very suggestive for the activation of the neural cognitive processes. This study is provides fresh neural benchmark data for designers, architects, planners, decision makers, academicians, and industry professionals to assess, design, and produce built environments using neural evidence-based data.

In a rapidly changing society, knowledge and education are key factors for indivdual well-being and social development. To create ideal learning environments and achieve both, it is important to know how our mind develops over the life span. This is also highly relevant for mental health and especially the prevention of psychological stress. On the other hand, the physiological stress reaction is considered to play a vital role in psychobiological development. There is a long tradition to model psychobiological development based on behavioral data (e.g. cognitive perfomance, IQ, personality factors, social skills) and biographical data (e.g. family history, educational status, traumatic life events, psychiatric symptoms). Recent technological innovations in the life sciences (gene sequencing methods, imaging techniques) made it possible to also assess molecular (genetic and epigenetic) and neural correlates of psychobiological development. This is most obvious for the example of stress and the question how we relate psychophysiological and molecular (hormonal, epigenetic and genetic) measures to the experienced psychological stress. Especially, epigenetic mechanisms seem to contribute significantly to developmental processes in general and to the development of the physiological stress reaction in particular. But concepts that integrate newly genetic, epigenetic, neural and neuro-cognitive findings in an overall theory of psychobiological development have yet to be developed. Gilbert Gottlieb's probabilistic epigenesis and his scheme of psychobiological development provide an ideal starting point for this effort. Based on a modified version of Gottlieb's scheme of psychobiological development, this study aims at integrating different levels of empirical data collection relevant for psychobiological development. Possibilities and challenges of the model are discussed using the physiological stress reaction and the psychological stress concept as an example. The goal is to provide detailed hypotheses of inter-level interactions which can be tested in future empirical studies. Accordingly, epigenetic mechanisms are modeled as molecular underpinnings mediating interactions between neural and genetic activity levels. Three different functional contexts of epigenetic mechanisms in neuronal cells are identified: genomic, developmental, and synaptic. Furthermore, the distinction between structural and functional data usually used to interpret the difference between neural data and neuro-cognitive data is questioned. As shown for the example of stress, the model overall provides a new framework to interpret molecular and neural data in relation to behavioral and biographical data usually used in clinical practice.

The roles of prefrontal and posterior parietal cortex in algebra problem solving: A case of using cognitive modeling to inform neuroimaging data

Precise and synchronized multimodal data capture in neurosurgical environments is essential for further understanding brain function and will be crucial to advancing the development of brain-computer interface technology. We have developed an open-source software platform named Thalamus, for multimodal data capture integrated with existing sensors and hardware commonly utilized in the operating room and other clinical environments such as pulse oximeters, inertial sensors, electromyography and neural electrophysiology. Thalamus facilitates synchronous recording of neural and behavioral data, enabling real-time computation for closed-loop experiments and detailed analysis of complex motor functions and neural activity. Thalamus uses a modular, configurable node-based pipeline with a tiered Python and C + + architecture. These design elements allow Thalamus to support a wide range of high-resolution sensors for diverse behavioral data types and enable robust closed-loop synchronization of various data streams. Validation experiments demonstrate that Thalamus is capable of data integration and concurrent analysis with up to sub-millisecond precision, offering great potential for enhancing neurosurgical research and clinical applications. By leveraging conventional sensors and hardware already in use, Thalamus supports adoption into the clinical environment, paving the way for more comprehensive, data-driven approaches to neurological care and improving the personalization and rigor of treatment strategies.

Understanding the relationship between neural activity in motor cortex and muscle activity during movements is important both for basic science and for the design of neural prostheses. While there has been significant work in decoding muscle EMG from neural data, decoders often require many parameters which make the analysis susceptible to overfitting, which reduces generalizability and makes the results difficult to interpret. To address this issue, we recorded simultaneous neural activity from the motor cortices (M1/PMd) of rhesus monkeys performing an arm-reaching task while recording EMG from arm muscles. In this work, we focused on relating the mean neural activity (averaged across reach trials to one target) to the corresponding mean EMG. We reduced the dimensionality of the neural data and found that the curvature of the low-dimensional (low-D) neural activity could be used as a signature of muscle activity. Using this signature, and without directly fitting EMG data to the neural activity, we derived neural axes based on reaches to only one reach target (< 5% of the data) that could explain EMG for reaches across multiple targets (average R(2) = 0.65). Our results suggest that cortical population activity is tightly related to muscle EMG measurements, predicting a lag between cortical activity and movement generation of 47.5 ms. Furthermore, our ability to predict EMG features across different movements suggests that there are fundamental axes or directions in the low-D neural space along which the neural population activity moves to activate particular muscles.

The amygdala (AMY) is widely implicated in fear learning and fear behaviour, but it remains unclear how the many biological components present within AMY interact to achieve these abilities. Building on previous work, we hypothesize that individual AMY nuclei represent different quantities and that fear conditioning arises from error-driven learning on the synapses between AMY nuclei. We present a computational model of AMY that (a) recreates the divisions and connections between AMY nuclei and their constituent pyramidal and inhibitory neurons; (b) accommodates scalable high-dimensional representations of external stimuli; (c) learns to associate complex stimuli with the presence (or absence) of an aversive stimulus; (d) preserves feature information when mapping inputs to salience estimates, such that these estimates generalize to similar stimuli; and (e) induces a diverse profile of neural responses within each nucleus. Our model predicts (1) defensive responses and neural activities in several experimental conditions, (2) the consequence of artificially ablating particular nuclei and (3) the tendency to generalize defensive responses to novel stimuli. We test these predictions by comparing model outputs to neural and behavioural data from animals and humans. Despite the relative simplicity of our model, we find significant overlap between simulated and empirical data, which supports our claim that the model captures many of the neural mechanisms that support fear conditioning. We conclude by comparing our model to other computational models and by characterizing the theoretical relationship between pattern separation and fear generalization in healthy versus anxious individuals.

Factor analysis linking functions for simultaneously modeling neural and behavioral data

Understanding how collective neuronal activity in the brain orchestrates behavior is a central question in integrative neuroscience. Addressing this question requires models that can offer a unified interpretation of multimodal data. In this study, we jointly examine video-recordings of zebrafish larvae freely exploring their environment and calcium imaging of the Anterior Rhombencephalic Turning Region (ARTR) circuit, which is known to control swimming orientation, recorded in vivo under tethered conditions. We show that both behavioral and neural data can be accurately modeled using a Hidden Markov Model (HMM) with three hidden states. In the context of behavior, the hidden states correspond to leftward, rightward, and forward swimming. The HMM robustly captures the key statistical features of the swimming motion, including bout-type persistence and its dependence on bath temperature, while also revealing inter-individual phenotypic variability. For neural data, the three states are found to correspond to left- and right-lateral activation of the ARTR circuit, known to govern the selection of left vs. right reorientation, and a balanced state, which likely corresponds to the behavioral forward state. To further unify the two analyses, we exploit the generative nature of the HMM, using neural sequences to generate synthetic swimming trajectories, whose statistical properties are similar to the behavioral data. Overall, this work demonstrates how state-space models can be used to link neuronal and behavioral data, providing insights into the mechanisms of self-generated action.

Relationship between brainstem, cortical and behavioral measures relevant to pitch salience in humans

Dimension reduction on neural activity paves a way for unsupervised neural decoding by dissociating the measurement of internal neural state repetition from the measurement of external variable tuning. With assumptions only on the smoothness of latent dynamics and of internal tuning curves, the Poisson Gaussian-process latent variable model (P-GPLVM) (Wu et al., 2017) is a powerful tool to discover the low-dimensional latent structure for high-dimensional spike trains. However, when given novel neural data, the original model lacks a method to infer their latent trajectories in the learned latent space, limiting its ability for estimating the internal state repetition. Here, we extend the P-GPLVM to enable the latent variable inference of new data constrained by previously learned smoothness and mapping information. We also describe a principled approach for the constrained latent variable inference for temporally-compressed patterns of activity, such as those found in population burst events (PBEs) during hippocampal sharp-wave ripples, as well as metrics for assessing whether the inferred new latent variables are congruent with a previously learned manifold in the latent space. Applying these approaches to hippocampal ensemble recordings during active maze exploration, we replicate the result that P-GPLVM learns a latent space encoding the animal’s position. We further demonstrate that this latent space can differentiate one maze context from another. By inferring the latent variables of new neural data during running, certain internal neural states are observed to repeat, which is in accordance with the similarity of experiences encoded by its nearby neural trajectories in the training data manifold. Finally, repetition of internal neural states can be estimated for neural activity during PBEs as well, allowing the identification for replay events of versatile behaviors and more general experiences. Thus, our extension of the P-GPLVM framework for unsupervised analysis of neural activity can be used to answer critical questions related to scientific discovery.

Deep neural networks (DNNs) are promising models of the cortical computations supporting human object recognition. However, despite their ability to explain a significant portion of variance in neural data, the agreement between models and brain representational dynamics is far from perfect. We address this issue by asking which representational features are currently unaccounted for in neural time series data, estimated for multiple areas of the ventral stream via source-reconstructed magnetoencephalography data acquired in human participants (nine females, six males) during object viewing. We focus on the ability of visuo-semantic models, consisting of human-generated labels of object features and categories, to explain variance beyond the explanatory power of DNNs alone. We report a gradual reversal in the relative importance of DNN versus visuo-semantic features as ventral-stream object representations unfold over space and time. Although lower-level visual areas are better explained by DNN features starting early in time (at 66 ms after stimulus onset), higher-level cortical dynamics are best accounted for by visuo-semantic features starting later in time (at 146 ms after stimulus onset). Among the visuo-semantic features, object parts and basic categories drive the advantage over DNNs. These results show that a significant component of the variance unexplained by DNNs in higher-level cortical dynamics is structured and can be explained by readily nameable aspects of the objects. We conclude that current DNNs fail to fully capture dynamic representations in higher-level human visual cortex and suggest a path toward more accurate models of ventral-stream computations.SIGNIFICANCE STATEMENT When we view objects such as faces and cars in our visual environment, their neural representations dynamically unfold over time at a millisecond scale. These dynamics reflect the cortical computations that support fast and robust object recognition. DNNs have emerged as a promising framework for modeling these computations but cannot yet fully account for the neural dynamics. Using magnetoencephalography data acquired in human observers during object viewing, we show that readily nameable aspects of objects, such as 'eye', 'wheel', and 'face', can account for variance in the neural dynamics over and above DNNs. These findings suggest that DNNs and humans may in part rely on different object features for visual recognition and provide guidelines for model improvement.