Decoding Model Research Articles (Page 5)

Reconstruction of natural visual scenes from neural spikes with deep neural networks

Objective. Partial Least Squares (PLS) regression is a suitable linear decoder model for correlated and high dimensional neural data. This algorithm has been widely used in the application of brain–computer interface (BCI) for the decoding of motor parameters. PLS does not consider nonlinear relations between brain signal features. The nonlinear version of PLS that considers a nonlinear relationship between the latent variables has not been proposed for the decoding of intracranial data. This nonlinear model may cause overfitting in some cases due to a larger number of free parameters. In this paper, we develop a new version of nonlinear PLS, namely nonlinear sparse PLS (NLS PLS) and test it in BCI applications. Approach. In motor related BCI systems, improving the decoding accuracy of both kinetic and kinematic parameters of movement is crucial. To do this, two BCI datasets were chosen to decode the force amplitude and position of hand trajectory using the nonlinear and sparse versions of PLS algorithm. In our new NLS PLS method, we considered a polynomial relationship between the latent variables and used the lasso penalization in the latent space to avoid overfitting and to improve the decoding accuracy. Main results. Some linear and nonlinear based PLS models and our new proposed method, NLS PLS, were applied to the two datasets. According to our results, significant improvement from the NLS PLS method is confirmed over other methods. Our results show that nonlinear PLS outperforms generic PLS in the force decoding but it has lower accuracy in the hand trajectory decoding because of high dimensional feature space. By using lasso penalization, we presented a sparse nonlinear PLS-based model that outperforms generic PLS in both datasets and improves the coefficient of determination, 34% in the force decoding and 10% in the hand trajectory decoding. Significance. We constructed a simple PLS-based model that considers a nonlinear relationship between features and it is also robust to overfitting because of using the lasso penalty in the latent space. This model is suitable for a high dimensional and correlated datasets, like intracranial data and can improve the accuracy of estimation.

Sequence generation tasks, such as neural machine translation (NMT) and abstractive summarization, usually suffer from exposure bias as well as the error propagation problem due to the autoregressive training and generation. Many previous works have discussed the relationship between error propagation and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">accuracy drop</i> problem (i.e., the right part of the generated sentence is often worse than its left part in left-to-right decoding models). In this paper, taking NMT as a typical sequence generation task, we measure the accuracy of the generated sentence with various metrics and conduct a series of analyses to deeply understand the accuracy drop problem. We obtain several interesting findings. First, The role of error propagation on accuracy drop is overstated in the literature, although it is indeed a cause to the accuracy drop problem. Second, Characteristics of a language play a more important role in causing the accuracy drop problem: the left part of the generated sentence in a right-branching language (e.g., English) is more likely to be more accurate than its right part, while the right part is more accurate for a left-branching language (e.g., Japanese). Our discoveries are also confirmed on other generation tasks (e.g., image captioning, abstractive summarization and language modeling) with multiple left/right-branching languages, as well as in various model structures.

Development of noninvasive brain-machine interface (BMI) systems based on electroencephalography (EEG), driven by spontaneous movement intentions, is a useful tool for controlling external devices or supporting a neuro- rehabilitation. In this study, we present the possibility of brain-controlled robot arm system using arm trajectory decoding. To do that, we first constructed the experimental system that can acquire the EEG data for not only movement execution (ME) task but also movement imagery (MI) tasks. Five subjects participated in our experiments and performed four directional reaching tasks (Left, right, forward, and backward) in the 3D plane. For robust arm trajectory decoding, we propose a subject-dependent deep neural network (DNN) architecture. The decoding model applies the principle of bi-directional long short-term memory (LSTM) network. As a result, we confirmed the decoding performance (r-value: >0.8) for all X-, Y-, and Z-axis across all subjects in the MI as well as ME tasks. These results show the feasibility of the EEG-based intuitive robot arm control system for high-level tasks (e.g., drink water or moving some objects). Also, we confirm that the proposed method has no much decoding performance variations between ME and MI tasks for the offline analysis. Hence, we will demonstrate that the decoding model is capable of robust trajectory decoding even in a real-time environment.

Trial-by-trial surprise-decoding model for visual and auditory binary oddball tasks

Brain monitoring combined with automatic analysis of EEGs provides a clinical decision support tool that can reduce time to diagnosis and assist clinicians in real-time monitoring applications (e.g., neurological intensive care units). Clinicians have indicated that a sensitivity of 95% with specificity below 5% was the minimum requirement for clinical acceptance. In this study, a high-performance automated EEG analysis system based on principles of machine learning and big data is proposed. This hybrid architecture integrates hidden Markov models (HMMs) for sequential decoding of EEG events with deep learning-based post-processing that incorporates temporal and spatial context. These algorithms are trained and evaluated using the Temple University Hospital EEG, which is the largest publicly available corpus of clinical EEG recordings in the world. This system automatically processes EEG records and classifies three patterns of clinical interest in brain activity that might be useful in diagnosing brain disorders: (1) spike and/or sharp waves, (2) generalized periodic epileptiform discharges, (3) periodic lateralized epileptiform discharges. It also classifies three patterns used to model the background EEG activity: (1) eye movement, (2) artifacts, and (3) background. Our approach delivers a sensitivity above 90% while maintaining a specificity below 5%. We also demonstrate that this system delivers a low false alarm rate, which is critical for any spike detection application.

ContourGAN: Image contour detection with generative adversarial network

GraphVar 2.0: A user-friendly toolbox for machine learning on functional connectivity measures

Parallelizing and optimizing neural Encoder–Decoder models without padding on multi-core architecture

John Keats, a main figure in the second generation of Romantic poets, was not generally well received by his contemporary critics, though during the course of time, he has become one of the most beloved poets. Stuart Hall proposes an analytical model of communication, namely the encoding/decoding model, which assumes a complex structure of relations to be produced and sustained through linked but distinctive moments which are termed as production, circulation, distribution/ consumption, and reproduction. This paper employs Hall’s encoding/decoding communication model as a yardstick to move beyond his approach, which mainly addresses modern mass media and communication system, and relate the distinctive moments playing integrally in encoding and decoding to Keats’s Ode on a Grecian Urn (1819). Furthermore, there is an attempt to turn the spotlight on the ode’s durability after the French Revolution passions abate and the poem starts to gain its thingness.

Objective: Recently, several studies have documented the presence of a bimodal distribution of spike waveform widths in primary motor cortex. Although narrow and wide spiking neurons, corresponding to the two modes of the distribution, exhibit different response properties, it remains unknown if these differences give rise to differential decoding performance between these two classes of cells. Approach: We used a Gaussian mixture model to classify neurons into narrow and wide physiological classes. Using similar-size, random samples of neurons from these two physiological classes, we trained offline decoding models to predict a variety of movement features. We compared offline decoding performance between these two physiologically defined populations of cells. Main results: We found that narrow spiking neural ensembles decode motor parameters better than wide spiking neural ensembles including kinematics, kinetics, and muscle activity. Significance: These findings suggest that the utility of neural ensembles in brain machine interfaces may be predicted from their spike waveform widths.

Heading estimation involves both inertial and visual cues. Inertial motion is sensed by the labyrinth, somatic sensation by the body, and optic flow by the retina. Because the eye and head are mobile these stimuli are sensed relative to different reference frames and it remains unclear if a perception occurs in a common reference frame. Recent neurophysiologic evidence has suggested the reference frames remain separate even at higher levels of processing but has not addressed the resulting perception. Seven human subjects experienced a 2s, 16 cm/s translation and/or a visual stimulus corresponding with this translation. For each condition 72 stimuli (360° in 5° increments) were delivered in random order. After each stimulus the subject identified the perceived heading using a mechanical dial. Some trial blocks included interleaved conditions in which the influence of ±28° of gaze and/or head position were examined. The observations were fit using a two degree-of-freedom population vector decoder (PVD) model which considered the relative sensitivity to lateral motion and coordinate system offset. For visual stimuli gaze shifts caused shifts in perceived head estimates in the direction opposite the gaze shift in all subjects. These perceptual shifts averaged 13 ± 2° for eye only gaze shifts and 17 ± 2° for eye-head gaze shifts. This finding indicates visual headings are biased towards retina coordinates. Similar gaze and head direction shifts prior to inertial headings had no significant influence on heading direction. Thus inertial headings are perceived in body-centered coordinates. Combined visual and inertial stimuli yielded intermediate results.

The temporal coherence of amplitude fluctuations is a critical cue for segmentation of complex auditory scenes. The auditory system must accurately demarcate the onsets and offsets of acoustic signals. We explored how and how well the timing of onsets and offsets of gated tones are encoded by auditory cortical neurons in awake rhesus macaques. Temporal features of this representation were isolated by presenting otherwise identical pure tones of differing durations. Cortical response patterns were diverse, including selective encoding of onset and offset transients, tonic firing, and sustained suppression. Spike train classification methods revealed that many neurons robustly encoded tone duration despite substantial diversity in the encoding process. Excellent discrimination performance was achieved by neurons whose responses were primarily phasic at tone offset and by those that responded robustly while the tone persisted. Although diverse cortical response patterns converged on effective duration discrimination, this diversity significantly constrained the utility of decoding models referenced to a spiking pattern averaged across all responses or averaged within the same response category. Using maximum likelihood-based decoding models, we demonstrated that the spike train recorded in a single trial could support direct estimation of stimulus onset and offset. Comparisons between different decoding models established the substantial contribution of bursts of activity at sound onset and offset to demarcating the temporal boundaries of gated tones. Our results indicate that relatively few neurons suffice to provide temporally precise estimates of such auditory "edges," particularly for models that assume and exploit the heterogeneity of neural responses in awake cortex.

Bayesian neural decoding models aim at estimating an extrinsic stimulus, such as speech, from a neural population. They have been used to study several physiological systems such as the motor or auditory systems [1]. Such decoding models require the use of an encoding model, i.e. which aim at estimating the neural spikes from the stimulus [2]. One encoding model that has been used extensively in the past models the instantaneous spiking rate of a neuron using a generalized linear model (GLM). When applied to the auditory system, this GLM has three parameters that account respectively for 1) the spontaneous firing rate of the decoded neuron, 2) the spectro-temporal receptive field of the neuron and 3) the intrinsic dynamics of the neuron, such as refractory periods, bursting and network dynamics [3]. In a decoding framework, these parameters are estimated by fitting the encoding model to spike trains measured when presenting a specific stimulus (the training set) and used afterwards to perform the decoding of spike trains obtained when presenting a different stimulus (the test set). While it could be assumed that the parameters associated with the spontaneous rate and receptive fields won’t change significantly for different stimuli for a given neuron, the intrinsic dynamics will most likely change. Here we wish to investigate how trained GLM parameters, i.e. fitted for some specific stimulus, of a neural encoding model for the auditory system can accurately represent the spikes measured for a different stimulus. To do so we use a goodness of fit metric, the normalized Kolmogorov-Smirnov (KS) statistic; a normalized KS < 1 indicating an excellent fit. Table ​Table11 presents the normalized KS statistic obtained when evaluating the goodness of fit of the encoding model using trained parameters on measured spikes from the test set with 2-fold cross-validation. The recordings are from 54 auditory nerve neurons (20 trials each) in anesthetized cats. We observe that the normalized KS values are < 1 for only 13% (7/54) of the neurons suggesting an extremely poor fit of the encoding model. Using such encoding model parameters in a decoding framework will therefore introduce a strong bias in the decoded stimulus that is not currently taken into account in neural decoding models. Table 1 Number of neurons for which the maximum normalized KS statistics using 2-fold cross validation is within a given range for both training and test sets. A KS statistic < 1 indicates an accurate fit of the model. (n=54, 20 trials each, recordings ...

Trial by trial covariations between neural activity and perceptual decisions (quantified by choice Probability, CP) have been used to probe the contribution of sensory neurons to perceptual decisions. CPs are thought to be determined by both selective decoding of neural activity and by the structure of correlated noise among neurons, but the respective roles of these factors in creating CPs have been controversial. We used biologically-constrained simulations to explore this issue, taking advantage of a peculiar pattern of CPs exhibited by multisensory neurons in area MSTd that represent self-motion. Although models that relied on correlated noise or selective decoding could both account for the peculiar pattern of CPs, predictions of the selective decoding model were substantially more consistent with various features of the neural and behavioral data. While correlated noise is essential to observe CPs, our findings suggest that selective decoding of neuronal signals also plays important roles.

Multiclass fMRI data decoding and visualization using supervised self-organizing maps

Heading direction is determined from visual and vestibular cues. Both sensory modalities have been shown to have better direction discrimination for headings near straight ahead. Previous studies of visual heading estimation have not used the full range of stimuli, and vestibular heading estimation has not previously been reported. The current experiments measure human heading estimation in the horizontal plane to vestibular, visual, and spoken stimuli. The vestibular and visual tasks involved 16 cm of platform or visual motion. The spoken stimulus was a voice command speaking a heading angle. All conditions demonstrated direction dependent biases in perceived headings such that biases increased with headings further from the fore-aft axis. The bias was larger with the visual stimulus when compared with the vestibular stimulus in all 10 subjects. For the visual and vestibular tasks precision was best for headings near fore-aft. The spoken headings had the least bias, and the variation in precision was less dependent on direction. In a separate experiment when headings were limited to ±45°, the biases were much less, demonstrating the range of headings influences perception. There was a strong and highly significant correlation between the bias curves for visual and spoken stimuli in every subject. The correlation between visual-vestibular and vestibular-spoken biases were weaker but remained significant. The observed biases in both visual and vestibular heading perception qualitatively resembled predictions of a recent population vector decoder model (Gu et al., 2010) based on the known distribution of neuronal sensitivities.

In this paper, a time-variant decoding model of a convolutional network code (CNC) is proposed. New necessary and sufficient conditions are established for the decodability of a CNC at a node r with delay L. They only involve the first L+1 terms in the power series expansion of the global encoding kernel matrix at r. Concomitantly, a time-variant decoding algorithm is proposed with a decoding matrix over the base symbol field. The present time-variant decoding model only deals with partial information of the global encoding kernel matrix, and hence potentially makes CNCs applicable in a decentralized manner.

In this paper we analyze the two major transport protocols that are currently under discussion for IPTV and propose extensions to support convergence. On the one hand, the MPEG-2 transport stream (MPEG2-TS) is well established in the broadcast world, whereas the real-time transport protocol (RTP) is widely adopted in the Internet and mobile world. Our analysis focuses on the timing aspects such as synchronization and buffer management. It is shown that the system target decoder (STD) defined in MPEG2-TS provides a well defined decoder model that allows minimum buffering delay and predictable behavior. In contrast, an RTP receiver is intentionally defined less accurately to allow flexibility for multiple applications. However, this flexibility results in less clear receiver behavior which is a problem for the design and integration of IPTV systems. By comparing the main differences in the timing models of the two protocols we derive additional constraints and definitions for RTP that minimize the difference to MPEG2-TS. It is proposed to specify these changes as part of an RTP/IPTV profile which also defines an RTP system target decoder (RTP-STD). An example for the application of these extensions is a protocol gateway supporting the convergence of broadcast and the Internet world in IPTV.

Event Abstract Back to Event Bayesian reconstruction of perceptual experiences from human brain activity Sensory systems transform stimulus energy measured at the peripheral transducers into explicit representations of various abstract features. These transformations can be viewed as (nonlinear) mappings between the stimuli and brain activity measurements [1]. Most work in sensory neuroscience has focused on development of quantitative encoding models posed in terms of nonlinear mechanisms that filter the stimuli in order to produce appropriate responses. Recent interest in brain-machine interfaces has pushed development of decoding models that aim to classify, identify or reconstruct the stimulus directly from measured brain activity. These models work in the opposite direction, transforming responses into stimuli features. Most decoding models use non-parametric algorithms such as SVM, and are not based on explicit encoding models. We have pioneered an alternative approach in which the decoding algorithm is inferred from one or more explicit encoding models. In a study published last year [2] we showed that this approach can be used to extract far more information from functional MRI measurements than was generally believed possible. In more recent work along these lines, we have developed a new Bayesian decoding model that can reconstruct natural images seen by an observer from measured brain activity. The decoder combines three elements: a structural encoding model that characterizes signals from early visual areas; a semantic encoding model that characterizes signals from higher visual areas; and appropriate priors that incorporate statistical information about the structure and semantics of natural scenes. By combining all these elements the decoder produces reconstructions that accurately reflect the distribution, structure and semantic category of the objects contained in the original image. These results help clarify how distinct representations in different parts of the brain can be combined to provided a coherent reconstruction of the visual world; they also highlight a potentially important role for prior knowledge in visual perception. Our Bayesian decoding model can be generalized directly to permit reconstruction of other perceptual dimensions, such as color and motion. It might also be possible to use this framework to reconstruct subjective perceptual processes such as visual imagery and dreaming. More generally, these results suggest that Bayesian decoding algorithms could form the basis of powerful new brain-reading technologies and brain-computer interfaces. Acknowledgements. The functional MRI scans were conducted with the support of the staff at the Brain Imaging Center at UC Berkeley and at the Veterans Administration in Martinez, CA. This research was supported by the National Eye Institute and by UC Berkeley intramural funds.