The encoding / decoding model: criticisms and redevelopments for research on decoding

Encoding and decoding models

Image Captioning is the process of generating textual descriptions of an image, which need to be syntactically and semantically correct. This paper extensively surveys very early literature that includes the advent of Artificial Intelligence, the Machine Learning pathway, the early Deep Learning and the current Deep Learning methodology for Image Captioning. This survey paper aims to develop a system to predict captions for the given images with a higher accuracy by combining the results of different Deep Learning Techniques. This model based on a neural network consists of a vision CNN followed by the language generator RNN. It generates complete sentence in natural language from an input image. The state of art is achieved by comparing three different encoder –decoder models. By comparing three models, the blue score of CNN-LSTM Model with Flikr 8k dataset is 0.44, CNN-LSTM Word Embedding with Flikr 8k dataset is 0.68 and CNN –GRU model Visual Attention with MSCOCO Dataset is 0.86.

Using a brain-machine interface (BMI), a non-human primate (NHP) was trained to control a mobile robotic platform in real time using spike activity from the motor cortex, enabling self-motion through brain-control. The decoding model was initially trained using neural signals recorded when the NHP controlled the platform using a joystick. Using this decoding model, we compared the performance of the BMI during brain control with and without the use of a dummy joystick, and found that the success ratio dropped by 40% and time taken increased by 45% when the dummy joystick was removed. Performance during full brain control was only restored after a recalibration of the decoding model. We aimed to understand the differences in the underlying neural representations of movement intentions with and without the use of a dummy joystick, and showed that there were significant changes in both directional tuning, as well as global firing rates. These results indicate that the strategies used by the NHP for self-motion were different depending on whether a dummy joystick was present. We propose that a recalibration of the decoding model is an important step during the implementation of a BMI system for self-motion.

Data transformation is a laborious and time-consuming task for analysts. Programming by example (PBE) is a technique that can simplify this difficult task for data analysts by automatically generating programs for data transformation. Most of the previously proposed PBE methods are based on search algorithms, but recent improvements in machine learning (ML) have led to its application in PBE research. For example, RobustFill was proposed as an ML-based PBE method for string transformation by using long short-term memory (LSTM) as the sequential encoder–decoder model. However, an ML-based PBE method has not been developed for tabular transformations, which are used frequently in data analysis. Thus, in the present study, we propose an ML-based PBE method for tabular transformations. First, we consider the features of tabular transformations, which are more complex and data intensive than string transformations, and propose a new ML-based PBE method using the state-of-the-art Transformer sequential encoder–decoder model. To our knowledge, this is the first ML-based PBE method for tabular transformations. We also propose two decoding methods comprising multistep beam search and program validation-beam search, which are optimized for program generation, and thus generate correct programs with higher accuracy. Our evaluation results demonstrated that the Transformer-based PBE model performed much better than LSTM-based PBE when applied to tabular transformations. Furthermore, the Transformer-based model with the proposed decoding method performed better than the conventional PBE model using the search-based method.

Summary form only given. Recent interest in brain-computer interfaces has pushed development of decoding models that aim to classify, identify or reconstruct visual stimuli directly from measured brain activity. Most decoding models are based on non-parametric algorithms such as SVM and do not exploit current computational models of visual processing. We have pioneered an alternative approach in which the decoding algorithm is inferred from one or more explicit visual processing (nonlinear filtering) models. In previous work we showed that our approach extracts far more information from functional MRI measurements than was generally believed possible. In this task I will describe a new Bayesian decoding model that can actually reconstruct natural images that were seen by an observer from brain activity measured using fMRI. The decoder combines three elements: (1) a structural encoding model that characterizes signals from early visual areas; (2) a semantic encoding model that characterizes signals from higher visual areas; and (3) appropriate priors that incorporate statistical information about the structure and semantics of natural scenes. By combining all these elements the decoder produces reconstructions that accurately reflect the distribution, structure and semantic category of the objects contained in the original image. These results help clarify how distinct representations in different parts of the brain can be combined to provided a coherent reconstruction of the visual world; they also highlight a potentially important role for prior knowledge in visual perception. Our Bayesian decoding framework can be generalized directly to permit reconstruction of other perceptual dimensions, and might facilitate reconstruction of subjective perceptual processes such as visual imagery and dreaming. In the future Bayesian decoding algorithms might form the basis of powerful new brain-reading technologies and brain-computer interfaces.

Alterations of mismatch responses (ie, neural activity evoked by unexpected stimuli) are often considered a potential biomarker of schizophrenia. Going beyond establishing the type of observed alterations found in diagnosed patients and related cohorts, computational methods can yield valuable insights into the underlying disruptions of neural mechanisms and cognitive function. Here, we adopt a typology of model-based approaches from computational cognitive neuroscience, providing an overview of the study of mismatch responses and their alterations in schizophrenia from four complementary perspectives: (a) connectivity models, (b) decoding models, (c) neural network models, and (d) cognitive models. Connectivity models aim at inferring the effective connectivity patterns between brain regions that may underlie mismatch responses measured at the sensor level. Decoding models use multivariate spatiotemporal mismatch response patterns to infer the type of sensory violations or to classify participants based on their diagnosis. Neural network models such as deep convolutional neural networks can be used for improved classification performance as well as for a systematic study of various aspects of empirical data. Finally, cognitive models quantify mismatch responses in terms of signaling and updating perceptual predictions over time. In addition to describing the available methodology and reviewing the results of recent computational psychiatry studies, we offer suggestions for future work applying model-based techniques to advance the study of mismatch responses in schizophrenia.

Event Abstract Back to Event Bayesian reconstruction of perceptual experiences from human brain activity Sensory systems transform stimulus energy measured at the peripheral transducers into explicit representations of various abstract features. These transformations can be viewed as (nonlinear) mappings between the stimuli and brain activity measurements [1]. Most work in sensory neuroscience has focused on development of quantitative encoding models posed in terms of nonlinear mechanisms that filter the stimuli in order to produce appropriate responses. Recent interest in brain-machine interfaces has pushed development of decoding models that aim to classify, identify or reconstruct the stimulus directly from measured brain activity. These models work in the opposite direction, transforming responses into stimuli features. Most decoding models use non-parametric algorithms such as SVM, and are not based on explicit encoding models. We have pioneered an alternative approach in which the decoding algorithm is inferred from one or more explicit encoding models. In a study published last year [2] we showed that this approach can be used to extract far more information from functional MRI measurements than was generally believed possible. In more recent work along these lines, we have developed a new Bayesian decoding model that can reconstruct natural images seen by an observer from measured brain activity. The decoder combines three elements: a structural encoding model that characterizes signals from early visual areas; a semantic encoding model that characterizes signals from higher visual areas; and appropriate priors that incorporate statistical information about the structure and semantics of natural scenes. By combining all these elements the decoder produces reconstructions that accurately reflect the distribution, structure and semantic category of the objects contained in the original image. These results help clarify how distinct representations in different parts of the brain can be combined to provided a coherent reconstruction of the visual world; they also highlight a potentially important role for prior knowledge in visual perception. Our Bayesian decoding model can be generalized directly to permit reconstruction of other perceptual dimensions, such as color and motion. It might also be possible to use this framework to reconstruct subjective perceptual processes such as visual imagery and dreaming. More generally, these results suggest that Bayesian decoding algorithms could form the basis of powerful new brain-reading technologies and brain-computer interfaces. Acknowledgements. The functional MRI scans were conducted with the support of the staff at the Brain Imaging Center at UC Berkeley and at the Veterans Administration in Martinez, CA. This research was supported by the National Eye Institute and by UC Berkeley intramural funds.

There is no consensus on the state-of-the-art approach to historical text normalization. Many techniques have been proposed, including rule-based methods, distance metrics, character-based statistical machine translation, and neural encoder–decoder models, but studies have used different datasets, different evaluation methods, and have come to different conclusions. This paper presents the largest study of historical text normalization done so far. We critically survey the existing literature and report experiments on eight languages, comparing systems spanning all categories of proposed normalization techniques, analysing the effect of training data quantity, and using different evaluation methods. The datasets and scripts are made publicly available.

The temporal coherence of amplitude fluctuations is a critical cue for segmentation of complex auditory scenes. The auditory system must accurately demarcate the onsets and offsets of acoustic signals. We explored how and how well the timing of onsets and offsets of gated tones are encoded by auditory cortical neurons in awake rhesus macaques. Temporal features of this representation were isolated by presenting otherwise identical pure tones of differing durations. Cortical response patterns were diverse, including selective encoding of onset and offset transients, tonic firing, and sustained suppression. Spike train classification methods revealed that many neurons robustly encoded tone duration despite substantial diversity in the encoding process. Excellent discrimination performance was achieved by neurons whose responses were primarily phasic at tone offset and by those that responded robustly while the tone persisted. Although diverse cortical response patterns converged on effective duration discrimination, this diversity significantly constrained the utility of decoding models referenced to a spiking pattern averaged across all responses or averaged within the same response category. Using maximum likelihood-based decoding models, we demonstrated that the spike train recorded in a single trial could support direct estimation of stimulus onset and offset. Comparisons between different decoding models established the substantial contribution of bursts of activity at sound onset and offset to demarcating the temporal boundaries of gated tones. Our results indicate that relatively few neurons suffice to provide temporally precise estimates of such auditory "edges," particularly for models that assume and exploit the heterogeneity of neural responses in awake cortex.

Brain-computer interfaces (BCIs) enable direct communication with computers by encoding and decoding brain electrical signals to generate control signals and interact directly with external devices, thereby assisting patients with damaged motor nerves. Research on brain electrical signals based on motor imagery has always received widespread attention. To effectively extract and classify complicated brain electrical signal features, a system architecture that can withstand a low ratio between signal and noise, instability, and physiological artifacts must be constructed. Advancements in motor imagery electroencephalography have been made possible by the rapid development of deep learning over the past few years. This paper aims to introduce and analyze three decoding models that use convolutional neural network architectures (CNN) and three combinations of CNN and Transformer architectures. Through similar comparisons and cross-comparisons, by analyzing the accuracy of each model on the current mainstream motor imagery-related datasets BCI IV 2A and 2B, the advantages and disadvantages of each model are explored. Through comparison, it can be seen that models based on the CNN architecture still occupy a dominant position due to their fewer parameters and better adaptability to various situations. However, the confusion decoding model CTNet performs very well on the BCI IV 2A and 2B, indicating that the decoding model with the Transformer architecture has higher performance development potential and is the main direction of future development.

In this paper, a synchronous method based on state graph is proposed to calculate the evaluation feature for automatic scoring in computer-assisted language learning (CALL). The posterior probabilities of states are selected as the main feature. The score of hypothesized phonemes and words are estimated using the information of corresponding states. Traditional systems use two passes and two different models for decoding and computing posterior probabilities respectively. In this new algorithm, the posterior probabilities are calculated during the decoding of the state graph constructed from grammar. And in this new algorithm, the same acoustics model is used during the process of decoding and posterior probabilities computing. The old and new computing algorithms are compared through experiments, and the result shows that performance of the new algorithm is effectively improved. The scoring accuracy of new synchronous algorithm is improved, while the computing complexity reduces 16%.

With the rapid development of smart cities and intelligent transportation, the level of informatization and intelligence in traffic systems is continuously improving. Studies have shown that the accident rate in foggy conditions is more than 30% higher than under normal weather. Fog significantly affects drivers' visibility, reducing safety and increasing driving risks. To address this issue, this paper proposes an improved Transformer-based model, Retinexformer, aimed at enhancing multi-object detection performance in foggy environments. First, to enhance the model's adaptability in complex environments, the Retinexformer model is optimized by incorporating a multi-branch Transformer module. This module enables the model to handle diverse input conditions and process information from different types of scenes more effectively. In addition, an adaptive parameter estimation module is designed to automatically adjust dehazing and image enhancement strategies based on the image’s brightness and contrast. By dynamically setting different dehazing parameters, this module significantly improves the robustness of Retinexformer under varying foggy conditions.In the model's Decoder section, a multi-level feature enhancement module is introduced. After the IGT (Image Gradient Transformation) output, this module fuses multi-scale features, with a particular focus on enhancing details and small object detection capabilities, ensuring accurate identification of targets such as pedestrians and vehicles.The experimental findings indicate that the improved Retinexformer model demonstrates strong robustness and superior detection accuracy in foggy multi-object detection tasks, significantly enhancing traffic safety in foggy environments.

Visual perceptual decoding is one of the important and challenging topics in cognitive neuroscience. Building a mapping model between visual response signals and visual contents is the key point of decoding. Most previous studies used peak response signals to decode object categories. However, brain activities measured by functional magnetic resonance imaging are a dynamic process with time dependence, so peak signals cannot fully represent the whole process, which may affect the performance of decoding. Here, we propose a decoding model based on long short‐term memory (LSTM) network to decode five object categories from multitime response signals evoked by natural images. Experimental results show that the average decoding accuracy using the multitime (2–6 s) response signals is 0.540 from the five subjects, which is significantly higher than that using the peak ones (6 s; accuracy: 0.492; p < .05). In addition, from the perspective of different durations, methods and visual areas, the decoding performances of the five object categories are deeply and comprehensively explored. The analysis of different durations and decoding methods reveals that the LSTM‐based decoding model with sequence simulation ability can fit the time dependence of the multitime visual response signals to achieve higher decoding performance. The comparative analysis of different visual areas demonstrates that the higher visual cortex (VC) contains more semantic category information needed for visual perceptual decoding than lower VC.

3D SOC-Net: Deep 3D reconstruction network based on self-organizing clustering mapping

With the great popularity of Transformers, there has been many works using Transformers to explore the temporal association properties of objects between different video frames. However, due to the large-scale variation of visual entities and the high resolution of pixels in images, the original Transformers take so long time for both training and inference. Based on Swin Transformer, we propose SwinTransTrack, a novel shift-window encoder and decoder model. Different from the original model, we fuse low-rank adaptation to achieve feature dimension enhancement and propose a new shifted-window decoder network to obtain accurate displacement to associate trajectories. Finally, We conducted extensive quantitative experiments on different MOT datasets, MOT17 and MOT20. The experimental results show that SwinTransTrack achieves 75.5 MOTA on MOT17 and 67.5 MOTA on MOT20, leading both MOT competitions.