Divisive Normalization and Neuronal Oscillations in a Single Hierarchical Framework of Selective Visual Attention

The selective attention mechanism is employed by humans and primates to realize a truly intelligent perception system, which has the cognitive capability of learning and thinking about how to perceive the environment autonomously. The attention mechanism involves the top-down and bottom-up ways that correspond to the goal-directed and automatic perceptual behaviors, respectively. Rather than considering the automatic perception, this paper presents an artificial system of the goal-directed visual perception by using the object-based top-down visual attention mechanism. This cognitive system can guide the perception to an object of interest according to the current task, context and learned knowledge. It consists of three successive stages: preattentive processing, top-down attentional selection and post-attentive perception. The preattentive processing stage divides the input scene into homogeneous proto-objects, one of which is then selected by the top-down attention and finally sent to the post-attentive perception stage for high-level analysis. Experimental results of target detection in the cluttered environments are shown to validate this system.

Our intuitive sense of number allows rapid estimation for the number of objects (numerosity) in a scene. How does the continuous nature of neural information processing create a discrete representation of number? A neurocomputational model with divisive normalization explains this process and existing data; however, a successful model should not only explain existing data but also generate novel predictions. Here, we experimentally test novel predictions of this model to evaluate its merit for explaining mechanisms of numerosity perception. We did so by consideration of the coherence illusion: the underestimation of number for arrays containing heterogeneous compared to homogeneous items. First, we established the existence of the coherence illusion for homogeneity manipulations of both area and orientation of items in an array. Second, despite the behavioral similarity, the divisive normalization model predicted that these two illusions should reflect activity in different stages of visual processing. Finally, visual evoked potentials from an electroencephalography experiment confirmed these predictions, showing that area and orientation coherence modulate brain responses at distinct latencies and topographies. These results demonstrate the utility of the divisive normalization model for explaining numerosity perception, according to which numerosity perception is a byproduct of canonical neurocomputations that exist throughout the visual pathway.

Recently, many scholars have used attention mechanisms to achieve excellent performance results on various neural network applications. However, the attention mechanism also has shortcomings. Firstly, the high computational and storage consumption makes the attention mechanism difficult to apply on long sequences. Second, all tokens are involved in the computation of the attention map, which may increase the influence of noisy tokens on the results and lead to poor training results. Due to these two shortcomings, attention models are usually strictly limited to sequence length. Further, attention models have difficulty in exploiting their excellent properties for modelling long sequences. To solve the above problems, an efficient sparse attention mechanism (SSA) is proposed in this paper. SSA is based on two separate layers: the local layer and the global layer. These two layers jointly encode local sequence information and global context. This new sparse-attention patterns is powerful in accelerating reasoning. The experiments in this paper validate the effectiveness of the SSA mechanism by replacing the self-attentive structure with an SSA structure in a variety of transformer models. The SSA attention mechanism has achieved state-of-the-art performance on several major benchmarks. SSA was validated on a variety of datasets and models encompassing language translation, language modelling and image recognition. With a small improvement in accuracy, the inference calculation speed was increased by 24%.KeywordsTransformersSequenceLocal attentionGlobal attentionSparsity

Neural mechanisms of selective auditory attention are enhanced by computerized training: Electrophysiological evidence from language-impaired and typically developing children

Reliable estimation of desired motion trajectories plays a crucial part in the continuous control of lower extremity assistance devices such as prostheses and orthoses. Moreover, reliable estimation methods are also required to predict hard-to-measure biomechanical quantities (e.g., joint contact moment/force) for use in sports injury science. Recognising that human locomotion is an inherently time-sequential and limb-synergetic behaviour, this study investigates models and learning algorithms for predicting the motion of a subject’s leg from the motion of complementary limbs. The novel deep learning model architectures proposed are based on the Long Short-Term Memory approach with the addition of an attention mechanism. A dataset comprising Inertial Measurement Unit signals from 21 subjects traversing varied terrains was used, including stair ascent/descent, ramp ascent/descent, stopped, level-ground walking and the transitions between these conditions. Fourier Analysis is deployed to evaluate the model robustness, in addition to assessing time-based prediction errors. The experiment on three unseen test participants suggests that the branched neural network structure is preferred to tackle the multioutput problem, and the inclusion of an attention mechanism demonstrates improved performance in terms of accuracy, robustness and network size. An experimental comparison found that 57% of the model parameters were not needed after adding attention layers meanwhile the prediction error is lower than the LSTM model without attention mechanism. The attention model has errors of 9.06% and 7.64% (normalised root mean square error) for ankle and hip acceleration prediction respectively. Also, less high-frequency noise is present in the attention model predictions. We conclude that the internal structure of the proposed deep learning model is justified, principally the benefit of using an attention mechanism. Experimental results for biomechanical motion estimation are obtained, showing greater accuracy than only with LSTM. The trained attention model can be used throughout despite transitioning between terrain types. Such a model will be useful in, for example, the control of lower-limb prostheses, instead of the need to identify and switch between different trajectory generators for different walking modes.

Due to the vast amount of reviews available on the Web, in the past decades, a growing share of work has focused on sentiment analysis. Aspect-based sentiment classification is the subtask that seeks to detect the sentiment expressed by the content creators towards a defined target within a sentence. This paper introduces three novel unsupervised attentional neural network models for aspect-based sentiment classification, and tests them on English restaurant reviews. The first model employs an autoencoder-like structure to learn a sentiment embedding matrix where each row of the matrix represents the embedding for one sentiment. To improve the model, a target-based attention mechanism is included that de-emphasizes irrelevant words. Last, a redundancy and a seed regularization term constrain the sentiment embedding matrix. The second model extends the first by including a Bi-LSTM layer in the attention mechanism to exploit contextual information. The third model further adapts a Left-Center-Right separated neural network with Rotatory attention structure from the supervised realm to an unsupervised setting. Although all three models construct meaningful sentiment embeddings, experimental results indicate that the inclusion of the Bi-LSTM in the attention mechanism leads to a more precise attention mechanism and, thus, better predictions. The best model, i.e., the second, outperforms all investigated unsupervised and weakly supervised algorithms for aspect-based sentiment classification from the literature.

Recent deep sequential recommendation models often struggle to effectively model key characteristics of user behaviors, particularly in handling sequence length variations and capturing diverse interaction patterns. We propose STAR-Rec, a novel architecture that synergistically combines preference-aware attention and state-space modeling through a sequence-level mixture-of-experts framework. STAR-Rec addresses these challenges by: (1) employing preference-aware attention to capture both inherently similar item relationships and diverse preferences (2) utilizing state-space modeling to efficiently process variable-length sequences with linear complexity, and (3) incorporating a mixture-of-experts component that adaptively routes different behavioral patterns to specialized experts, handling both focused category-specific browsing and diverse category exploration patterns. We theoretically demonstrate how the state space model and attention mechanisms can be naturally unified in recommendation scenarios, where SSM captures temporal dynamics through state compression while attention models both similar and diverse item relationships. Extensive experiments on four real-world datasets demonstrate that STAR-Rec consistently outperforms state-of-the-art sequential recommendation methods, particularly in scenarios involving diverse user behaviors and varying sequence lengths. The implementation code is available anonymously online for easy reproducibility.

The selective visual attention mechanism in human visual system helps human to act efficiently when dealing with massive visual information. Over the last two decades, biologically inspired attention model has drawn lots of research attention and many models have been proposed. However, the top-down cues in human brain are still not fully understood, which makes top-down models not biologically plausible. This paper proposes an attention model containing both the bottom-up stage and top-down stage for the target detection from SAR (Synthetic Aperture Radar) images. The bottom-up stage is based on the biologically-inspired Itti model and is modified by taking fully into account the characteristic of SAR images. The top-down stage contains a novel learning strategy to make the full use of prior information. It is an extension of the bottom-up process and more biologically plausible. The experiments in this research aim to detect vehicles in different scenes to validate the proposed model by comparing with the well-known CFAR (constant false alarm rate) algorithm.

Shared attentional control of smooth eye movement and perception.

Fast neuronal network oscillations in the gamma frequency band (30–100 Hz) occur in various cortex regions, require timed synaptic excitation and inhibition with glutamate and GABA, respectively, and are associated with higher brain functions such as sensory perception, attentional selection and memory formation. However, little is known about energy and ion homeostasis during the gamma oscillation. Recent studies addressed this topic in slices of the rodent hippocampus using cholinergic and glutamatergic receptor models of gamma oscillations (GAM). Methods with high spatial and temporal resolution were applied in vitro, such as electrophysiological recordings of local field potential (LFP) and extracellular potassium concentration ([K+]o), live-cell fluorescence imaging of nicotinamide adenine dinucleotide (phosphate) and flavin adenine dinucleotide [NAD(P)H and FAD, respectively] (cellular redox state), and monitoring of the interstitial partial oxygen pressure (pO2) in depth profiles with microsensor electrodes, including mathematical modeling. The main findings are: (i) GAM are associated with high oxygen consumption rate and significant changes in the cellular redox state, indicating rapid adaptations in glycolysis and oxidative phosphorylation; (ii) GAM are accompanied by fluctuating elevations in [K+]o of less than 0.5 mmol/L from baseline, likely reflecting effective K+-uptake mechanisms of neuron and astrocyte compartments; and (iii) GAM are exquisitely sensitive to metabolic stress induced by lowering oxygen availability or by pharmacological inhibition of the mitochondrial respiratory chain. These findings reflect precise cellular adaptations to maintain adenosine-5′-triphosphate (ATP), ion and neurotransmitter homeostasis and thus neural excitability and synaptic signaling during GAM. Conversely, the exquisite sensitivity of GAM to metabolic stress might significantly contribute the exceptional vulnerability of higher brain functions in brain disease.

Selective visual attention describes the tendency of visual processing to be confined largely to stimuli that are relevant to behavior. It is among the most fundamental of cognitive functions, particularly in humans and other primates for whom vision is the dominant sense. We review recent progress in identifying the neural mechanisms of selective visual attention. We discuss evidence from studies of different varieties of selective attention and examine how these varieties alter the processing of stimuli by neurons within the visual system, current knowledge of their causal basis, and methods for assessing attentional dysfunctions. In addition, we identify some key questions that remain in identifying the neural mechanisms that give rise to the selective processing of visual information.

Aspect level sentiment classification aims to recognize the sentiment polarity of each aspect term in a sentence. However, most of the existing methods usually applied the attention mechanism over position-weighted memory and did not consider inter-aspect information. To address these issues, we propose a novel framework for aspect level sentiment classification, Deep Selective Memory Network (DSMN), which selects the context memory dynamically for better guiding the multi-hop attention mechanism and integrates inter-aspect information with deep memory network. By designing a selective attention mechanism based on the distance information between an aspect and its context, DSMN focuses on different parts of the context memory in different memory network layers to capture abundant aspect-aware context information. Besides, to make full use of the inter-aspect information, we also design effective inter-aspect modeling modules to generate both semantic and relation information of the nearby aspects for the desired aspect. We evaluate the advantages of our framework on three benchmark datasets, and experiment results show that our framework achieves state-of-the-art performance.

The ability to focus on a task while disregarding irrelevant information is an example of selective attention. The perceptual-load hypothesis argues that the occurrence of early or late selection mechanisms is determined by task-relevant perceptual load. Additionally, evidence shows that pupil size serves as proxy of locus coeruleus-norepinephrine (LC-NE) activity, a system associated with cognitive and attentional mediation. Here, we assessed pupil baseline (and pupil dilation) as predictors of load-related early and late selection performance. Participants were asked to search for a target in conditions of high and low perceptual load, while ignoring irrelevant stimuli. The results showed that pupil baseline size, measured prior trial onset, significantly predicted the upcoming search efficiency only in low perceptual load, when—according to the perceptual-load hypothesis—all perceptual information receives attentional resources. In addition, pupil dilation was linked to the time course of perceptual processing and predicted response times in both perceptual load conditions, an association that was enhanced in high load. Thus, this study relates attentional selection mechanisms, as defined by the perceptual-load theory, with pupil-related LC-NE activity. Because pupil baseline predicted attentional performance in low load but not in high load, this suggests that different attentional mechanisms are involved, one in which the LC-NE system plays a key role (low load) and one in which it is less relevant (high load). This suggests that the degree with which LC-NE influences behavioral performance is related to the perceptual load of the task at hand.

In order to solve the problem of insufficient accuracy of the existing environmental noise classification models, an environmental noise classification network based on multi-channel discrimination and attention mechanism was proposed. First, multi-channel resolution features are used to more comprehensively represent and capture the diversity and complexity of audio features. Then, a second-order dense connection method was used to replace the dense connection in DenseNet to solve the overfitting problem caused by the high similarity and redundancy of the sound signal. Finally, a G-SE Layer module is used for acoustic signals, which enhances the model's attention to important features by efficiently fusing channel features. Experimental results show that compared with the original DenseNet network, the improved GS-2DenseNet has an improved recognition rate of environmental noise by 4.83%, and the accuracy and F1-score are higher than those of acoustic networks such as ResNet and EcapaTdnn.

Unmanned aerial vehicle–human collaboration route planning for intelligent infrastructure inspection