decoding models Research Articles (Page 4)

Monotonic Gaussian regularization of attention for robust automatic speech recognition

Functional magnetic resonance imaging (fMRI) is one of the best techniques for precise localization of abnormal brain activity non-invasively. Machine-learning approaches have been widely used in neuroimaging studies; however, few studies have investigated the single-voxel modeling of fMRI data under cognitive tasks. We proposed a hybrid one-dimensional (1D) convolutional neural network (1D-CNN) based on the temporal dynamics of single-voxel fMRI time-series and successfully differentiated two continuous task states, namely, self-initiated (SI) and visually guided (VG) motor tasks. First, 25 activation peaks were identified from the contrast maps of SI and VG tasks in a blocked design. Then, the fMRI time-series of each peak voxel was transformed into a temporal-frequency domain by using continuous wavelet transform across a broader frequency range (0.003–0.313 Hz, with a step of 0.01 Hz). The transformed time-series was inputted into a 1D-CNN model for the binary classification of SI and VG continuous tasks. Compared with the univariate analysis, e.g., amplitude of low-frequency fluctuation (ALFF) at each frequency band, including, wavelet-ALFF, the 1D-CNN model highly outperformed wavelet-ALFF, with more efficient decoding models [46% of 800 models showing area under the curve (AUC) > 0.61] and higher decoding accuracies (94% of the efficient models), especially on the high-frequency bands (>0.1 Hz). Moreover, our results also demonstrated the advantages of wavelet decompositions over the original fMRI series by showing higher decoding performance on all peak voxels. Overall, this study suggests a great potential of single-voxel analysis using 1D-CNN and wavelet transformation of fMRI series with continuous, naturalistic, steady-state task design or resting-state design. It opens new avenues to precise localization of abnormal brain activity and fMRI-guided precision brain stimulation therapy.

ObjectiveDecoding arm movement kinematics from motor imagery can provide more natural control of the Brain-Computer Interface (BCI) system. This work aims to decode two-directional (left versus right direction) motor imagery hand movement using Electroencephalogram (EEG)-based BCI. Direction decoding from EEG is challenging due to the low signal-to-noise ratio and poor spatial resolution. To improve the direction decoding accuracy during an online session, this work proposes an additional corrective step using Error-Related Potential (ErrP). MethodsIn this work, a direction decoding model and an ErrP detection model are trained on calibration session data. During the online session, the trained direction decoding model uses segmented MI trials as input, and the decoded outcome is shown in real-time as feedback. The brain response to feedback shown is used to detect the presence of ErrP using a trained ErrP detection model. If the ErrP is detected, the outcome of the direction decoding model will be automatically corrected, which offers an improvement in the performance of the BCI system. ResultsThe online session is conducted on 14 healthy subjects’ data. The average online direction decoding accuracy of the proposed direction decoding model without the ErrP-based corrective step is 54.9%. The average sensitivity and specificity of the proposed ErrP detection model during the online session are 65.3% and 67.5%, respectively. Further, the proposed scheme of cascaded direction decoding model and ErrP detection model achieves an average online direction decoding accuracy of 64.9%, which is an improvement of 10% compared to the scheme without the ErrP detection model. SignificanceThis paper proposes the efficient use of the brain’s ErrP response to the visual neurofeedback presented to the user, in an online hand motor imagery direction decoding experiment paradigm. The significant improvement of the direction decoding accuracy achieved is a promising result that makes the existing BCI systems’ usability a step closer to practical real-world applications.

Neural machine translation, which has an encoder-decoder framework, is considered to be a feasible way for future machine translation. Nevertheless, with the fusion of multiple languages and the continuous emergence of new words, most current neural machine translation systems based on von Neumann’s architecture have seen a substantial increase in the number of devices for the decoder, resulting in high-energy consumption rate. Here, a multilevel photosensitive blending semiconductor optoelectronic synaptic transistor (MOST) with two different trapping mechanisms is firstly demonstrated, which exhibits 8 stable and well distinguishable states and synaptic behaviors such as excitatory postsynaptic current, short-term memory, and long-term memory are successfully mimicked under illumination in the wavelength range of 480–800 nm. More importantly, an optical decoder model based on MOST is successfully fabricated, which is the first application of neuromorphic device in the field of neural machine translation, significantly simplifying the structure of traditional neural machine translation system. Moreover, as a multi-level synaptic device, MOST can further reduce the number of components and simplify the structure of the codec model under light illumination. This work first applies the neuromorphic device to neural machine translation, and proposes a multi-level synaptic transistor as the based cell of decoding module, which would lay the foundation for breaking the bottleneck of machine translation.

Image Captioning is the process of generating textual descriptions of an image, which need to be syntactically and semantically correct. This paper extensively surveys very early literature that includes the advent of Artificial Intelligence, the Machine Learning pathway, the early Deep Learning and the current Deep Learning methodology for Image Captioning. This survey paper aims to develop a system to predict captions for the given images with a higher accuracy by combining the results of different Deep Learning Techniques. This model based on a neural network consists of a vision CNN followed by the language generator RNN. It generates complete sentence in natural language from an input image. The state of art is achieved by comparing three different encoder –decoder models. By comparing three models, the blue score of CNN-LSTM Model with Flikr 8k dataset is 0.44, CNN-LSTM Word Embedding with Flikr 8k dataset is 0.68 and CNN –GRU model Visual Attention with MSCOCO Dataset is 0.86.

Data transformation is a laborious and time-consuming task for analysts. Programming by example (PBE) is a technique that can simplify this difficult task for data analysts by automatically generating programs for data transformation. Most of the previously proposed PBE methods are based on search algorithms, but recent improvements in machine learning (ML) have led to its application in PBE research. For example, RobustFill was proposed as an ML-based PBE method for string transformation by using long short-term memory (LSTM) as the sequential encoder–decoder model. However, an ML-based PBE method has not been developed for tabular transformations, which are used frequently in data analysis. Thus, in the present study, we propose an ML-based PBE method for tabular transformations. First, we consider the features of tabular transformations, which are more complex and data intensive than string transformations, and propose a new ML-based PBE method using the state-of-the-art Transformer sequential encoder–decoder model. To our knowledge, this is the first ML-based PBE method for tabular transformations. We also propose two decoding methods comprising multistep beam search and program validation-beam search, which are optimized for program generation, and thus generate correct programs with higher accuracy. Our evaluation results demonstrated that the Transformer-based PBE model performed much better than LSTM-based PBE when applied to tabular transformations. Furthermore, the Transformer-based model with the proposed decoding method performed better than the conventional PBE model using the search-based method.

The commonly used fixed discrete Kalman filters (DKF) in neural decoders do not generalize well to the actual relationship between neuronal firing rates and movement intention. This is due to the underlying assumption that the neural activity is linearly related to the output state. They also face the issues of requiring large amount of training datasets to achieve a robust model and a degradation of decoding performance over time. In this paper, an adaptive adjustment is made to the conventional unscented Kalman filter (UKF) via intention estimation. This is done by incorporating a history of newly collected state parameters to develop a new set of model parameters. At each time point, a comparative weighted sum of old and new model parameters using matrix squared sums is used to update the neural decoding model parameters. The effectiveness of the resulting adaptive unscented Kalman filter (AUKF) is compared against the discrete Kalman filter and unscented Kalman filter-based algorithms. The results show that the proposed new algorithm provides higher decoding accuracy and stability while requiring less training data.

With the advantages of comfortable wearing and outdoor usage, the myoelectric gesture recognition techniques have gained much attention in the field of human-machine interaction (HMI). The purpose of this study is to optimize model structure and transfer generalized features to improve the robustness of myoelectric hand motion decoding. We derived the hand motion recognition framework from the muscle synergy theory, which is formulated as a temporal convolutional (TC) model of array sEMG signals, then a hierarchical myoelectric decoding model was proposed to predict simultaneous and continuous hand motion. The model was trained by the methods of unsupervised low-level feature learning and automated data labeling to minimize training supervision. Extensive experiments on the public sEMG database (17 subjects in Biopatrec) show that the TC model can extract muscle synergy features with higher fidelity ( R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> = 0.85±0.23) than the traditional instantaneous mixture model, the results of online test demonstrate robust myoelectric decoding on multiple simultaneous and continuous hand motions. More importantly, the analysis of weights visualization shows that the low-level feature representation layer of TC model can be migrated across the individuals, which provides a transferrable feature extraction layer for generalized hand motion decoding.

Interpretable deep learning model for building energy consumption prediction based on attention mechanism

Generating audio captions is a new research area that combines audio and natural language processing to create meaningful textual descriptions for audio clips. To address this problem, previous studies mostly use the encoder–decoder-based models without considering semantic information. To fill this gap, we present a novel encoder–decoder architecture using bi-directional Gated Recurrent Units (BiGRU) with audio and semantic embeddings. We extract semantic embedding by obtaining subjects and verbs from the audio clip captions and combine these embedding with audio embedding to feed the BiGRU-based encoder–decoder model. To enable semantic embeddings for the test audios, we introduce a Multilayer Perceptron classifier to predict the semantic embeddings of those clips. We also present exhaustive experiments to show the efficiency of different features and datasets for our proposed model the audio captioning task. To extract audio features, we use the log Mel energy features, VGGish embeddings, and a pretrained audio neural network (PANN) embeddings. Extensive experiments on two audio captioning datasets Clotho and AudioCaps show that our proposed model outperforms state-of-the-art audio captioning models across different evaluation metrics and using the semantic information improves the captioning performance.

Code generation from a graphical user interface via attention-based encoder–decoder model

Image captioning in Hindi language using transformer networks

BackgroundHigh frequency thalamic stimulation is an effective therapy for essential tremor, which mainly affects voluntary movements and/or sustained postures. However, continuous stimulation may deliver unnecessary current to the brain due to the intermittent nature of the tremor.ObjectiveHere we propose to close the loop of thalamic stimulation by detecting tremor provoking movement states using local field potentials recorded from the same electrodes implanted for stimulation, so that the stimulation is only delivered when necessary.MethodsEight patients with essential tremor participated in this study. Patient-specific support vector machine classifiers were first trained using data recorded while the patient performed tremor provoking movements. Then, the trained models were applied in real-time to detect these movements and triggered the delivery of stimulation.ResultsUsing the proposed method, stimulation was switched on for 80.37 ± 7.06% of time when tremor evoking movements were present. In comparison, the stimulation was switched on for 12.71 ± 7.06% of time when the patients were at rest and tremor-free. Compared with continuous stimulation, a similar amount of tremor suppression was achieved while only delivering 36.62 ± 13.49% of the energy used in continuous stimulation.ConclusionsThe results suggest that responsive thalamic stimulation for essential tremor based on tremor-provoking movement detection can be achieved without any requirement for external sensors or additional electrocorticography strips. Further research is required to investigate whether the decoding model is stable across time and generalizable to the variety of activities patients may engage in everyday life.

Seq2Emoji: A hybrid sequence generation model for short text emoji prediction

Decoding a person’s cognitive contents from evoked brain activity is becoming important in the field of brain-computer interaction. Previous studies have decoded a perceived image from functional magnetic resonance imaging (fMRI) activity by constructing brain decoding models that were trained with a single subject’s fMRI data. However, accurate decoding is still challenging since fMRI data acquired from only a single subject have several disadvantageous characteristics such as small sample size, noisy nature, and high dimensionality. In this article, we propose a method to decode categories of perceived images from fMRI activity using shared information of multi-subject fMRI data. Specifically, by aggregating fMRI data of multiple subjects that contain a large number of samples, we extract a low-dimensional latent representation shared by multi-subject fMRI data. Then the latent representation is nonlinearly transformed into visual features and semantic features of the perceived images to identify categories from various candidate categories. Our approach leverages rich information obtained from multi-subject fMRI data and improves the decoding performance. Experimental results obtained by using two public fMRI datasets showed that the proposed method can more accurately decode categories of perceived images from fMRI activity than previous approaches using a single subject’s fMRI data.

In this paper, we propose a method to estimate ink layer layout used as an input for 3D printer. This method makes it possible to reproduce a 3D printed patch that gives a desired translucency, which is represented as Line Spread Function (LSF) in this study. Deep neural networks of encoder decoder model is used for the estimation. In a previous research, it is reported that machine learning method is effective to formulate the complex relationship between the optical property such as LSF and the ink layer layout in 3D printer. However, it may be difficult to collect data large enough to train a neural network sufficiently. Especially, although 3D printer is getting more and more widespread, the printing process is still time consuming. Therefore, in this research, we prepare the training data, which is the correspondence between LSF and ink layer layout in 3D printer, by simulating it on a computer. MCML was used to perform the simulation. MCML is a method to simulate subsurface scattering of light for multi-layered media. Deep neural network was trained with the simulated data, and evaluated using a CG skin object. The result shows that our proposed method can estimate an appropriate ink layer layout which reproduce the appearance close to the target color and translucency.

The classic Bose – Chaudhuri – Hocquenghem (BCH) codes is famous and well-studied part in the theory of error-correcting codes. Generalization of BCH codes allows us to expand the range of activities in the practical correction of errors. Some generic BCH codes are able to correct more errors than classic BCH code in one message block. So it is important to provide appropriate method of error correction. After our investigation it was found that polynomial-norm method is most convenient and effective for that task. The result of the study was a model of a polynomial-norm decoder for a generic BCH code at length 65.

Visual perceptual decoding is one of the important and challenging topics in cognitive neuroscience. Building a mapping model between visual response signals and visual contents is the key point of decoding. Most previous studies used peak response signals to decode object categories. However, brain activities measured by functional magnetic resonance imaging are a dynamic process with time dependence, so peak signals cannot fully represent the whole process, which may affect the performance of decoding. Here, we propose a decoding model based on long short‐term memory (LSTM) network to decode five object categories from multitime response signals evoked by natural images. Experimental results show that the average decoding accuracy using the multitime (2–6 s) response signals is 0.540 from the five subjects, which is significantly higher than that using the peak ones (6 s; accuracy: 0.492; p < .05). In addition, from the perspective of different durations, methods and visual areas, the decoding performances of the five object categories are deeply and comprehensively explored. The analysis of different durations and decoding methods reveals that the LSTM‐based decoding model with sequence simulation ability can fit the time dependence of the multitime visual response signals to achieve higher decoding performance. The comparative analysis of different visual areas demonstrates that the higher visual cortex (VC) contains more semantic category information needed for visual perceptual decoding than lower VC.

Most sequence-to-sequence abstractive summarization models generate the summaries based on the source article and the generated words, but they often neglect the future information implied in the un-generated words, which means that they lack the ability of “lookahead.” In this paper, we present a novel summarization model with “lookahead” ability to fully employ the implied future information. Our model takes two steps: (1) in the first step, an asynchronous decoder model with a no ground truth guiding backward decoder that explicitly produces and exploits the future information is trained. (2) in the inference process, in addition to the joint probability of the generated sequence, an enriched-information decoding method is proposed to further take future ROUGE reward of the un-generated words into account. Furthermore, the future ROUGE reward is predicted by a novel reward-predict model, and it takes the hidden states of the pre-trained asynchronous decoder model as input. Experimental results show that our two-step summarization model achieves new state-of-the-art results on CNN/Daily Mail dataset and the generalization of our model on test-only DUC-2002 datasets achieves higher scores than the state-of-the-art model.

In recent years, teaching machines to ask meaningful and coherent questions has attracted considerable attention in natural language processing. Question generation has found wide applications in areas such as education (testing knowledge) and chatbots (enhancing interaction). Following previous studies on conversational question generation, we propose a pretrained, encoder–decoder model that can incorporate the semantic information from both passage and hidden conversation representations. We adopt BERT as the encoder to combine external text and dialogue history, and we design a multi-head attention-based decoder to incorporate the semantic information from both text and hidden dialogue representations into the decoding process, thereby generating coherent questions. Experiments with conversational question generation and document-grounded dialogue response generation tasks indicate that the proposed model is superior to baseline models in terms of both standard metrics and human evaluations.