SSA: A Content-Based Sparse Attention Mechanism

Divisive normalization models of covert attention commonly use spike rate modulations as indicators of the effect of top-down attention. In addition, an increasing number of studies have shown that top-down attention increases the synchronization of neuronal oscillations as well, particularly in gamma-band frequencies (25–100 Hz). Although modulations of spike rate and synchronous oscillations are not mutually exclusive as mechanisms of attention, there has thus far been little effort to integrate these concepts into a single framework of attention. Here, we aim to provide such a unified framework by expanding the normalization model of attention with a multi-level hierarchical structure and a time dimension; allowing the simulation of a recently reported backward progression of attentional effects along the visual cortical hierarchy. A simple cascade of normalization models simulating different cortical areas is shown to cause signal degradation and a loss of stimulus discriminability over time. To negate this degradation and ensure stable neuronal stimulus representations, we incorporate a kind of oscillatory phase entrainment into our model that has previously been proposed as the “communication-through-coherence” (CTC) hypothesis. Our analysis shows that divisive normalization and oscillation models can complement each other in a unified account of the neural mechanisms of selective visual attention. The resulting hierarchical normalization and oscillation (HNO) model reproduces several additional spatial and temporal aspects of attentional modulation and predicts a latency effect on neuronal responses as a result of cued attention.

Image recognition based on unsupervised learning is a worthy direction of research. Inspired by ImageNet and other open source image sets, more and more unsupervised learning methods have been applied to image recognition and achieved good results. Based on the unsupervised learning model based on momentum contrast, the attention mechanism module is introduced, and a new parameter adjustment strategy for learning rate is proposed for further optimization of the algorithm. In the module of attentional mechanism, self-SE attentional mechanism model is proposed by integrating self-attention and SENet models. Batch-size Warmup parameter adjustment strategy is proposed in this paper to further improve the performance of the model. In this paper, the proposed improved module and adjustment strategy are verified by several groups of experiments with CIFAR-100 as data set. Compared with the original method, the accuracy of the unsupervised learning image recognition method with improved modules and adjusted strategies has been improved by about 1.2%.

Reliable estimation of desired motion trajectories plays a crucial part in the continuous control of lower extremity assistance devices such as prostheses and orthoses. Moreover, reliable estimation methods are also required to predict hard-to-measure biomechanical quantities (e.g., joint contact moment/force) for use in sports injury science. Recognising that human locomotion is an inherently time-sequential and limb-synergetic behaviour, this study investigates models and learning algorithms for predicting the motion of a subject’s leg from the motion of complementary limbs. The novel deep learning model architectures proposed are based on the Long Short-Term Memory approach with the addition of an attention mechanism. A dataset comprising Inertial Measurement Unit signals from 21 subjects traversing varied terrains was used, including stair ascent/descent, ramp ascent/descent, stopped, level-ground walking and the transitions between these conditions. Fourier Analysis is deployed to evaluate the model robustness, in addition to assessing time-based prediction errors. The experiment on three unseen test participants suggests that the branched neural network structure is preferred to tackle the multioutput problem, and the inclusion of an attention mechanism demonstrates improved performance in terms of accuracy, robustness and network size. An experimental comparison found that 57% of the model parameters were not needed after adding attention layers meanwhile the prediction error is lower than the LSTM model without attention mechanism. The attention model has errors of 9.06% and 7.64% (normalised root mean square error) for ankle and hip acceleration prediction respectively. Also, less high-frequency noise is present in the attention model predictions. We conclude that the internal structure of the proposed deep learning model is justified, principally the benefit of using an attention mechanism. Experimental results for biomechanical motion estimation are obtained, showing greater accuracy than only with LSTM. The trained attention model can be used throughout despite transitioning between terrain types. Such a model will be useful in, for example, the control of lower-limb prostheses, instead of the need to identify and switch between different trajectory generators for different walking modes.

Divergent-convergent attention for image captioning

Where is my attention? An explainable AI exploration in water detection from SAR imagery

Recent deep sequential recommendation models often struggle to effectively model key characteristics of user behaviors, particularly in handling sequence length variations and capturing diverse interaction patterns. We propose STAR-Rec, a novel architecture that synergistically combines preference-aware attention and state-space modeling through a sequence-level mixture-of-experts framework. STAR-Rec addresses these challenges by: (1) employing preference-aware attention to capture both inherently similar item relationships and diverse preferences (2) utilizing state-space modeling to efficiently process variable-length sequences with linear complexity, and (3) incorporating a mixture-of-experts component that adaptively routes different behavioral patterns to specialized experts, handling both focused category-specific browsing and diverse category exploration patterns. We theoretically demonstrate how the state space model and attention mechanisms can be naturally unified in recommendation scenarios, where SSM captures temporal dynamics through state compression while attention models both similar and diverse item relationships. Extensive experiments on four real-world datasets demonstrate that STAR-Rec consistently outperforms state-of-the-art sequential recommendation methods, particularly in scenarios involving diverse user behaviors and varying sequence lengths. The implementation code is available anonymously online for easy reproducibility.

Context-based attention models achieve state-of-the-art results on constructing document representations. Current methods build document representations based on a trainable global context vector for all documents, while local contexts capturing document-specific information are ignored. Representing document context as a vector also restricts the model's capacity to capture complete semantic components of documents. In this work, we address these limitations by proposing novel attention-based approaches. We first propose a local and global context attention (LGCA) model, which constructs the context representations of a document as a combination of global and local contexts. The local context is used to capture local document-specific information. Furthermore, a multi-context attention (MCA) model is proposed to capture multiple local contexts within a document, thereby fusing the overall semantic information of documents. We conduct experiments on document-level sentiment classification tasks where extra user and product information is shown to be important. Experimental results indicate that our methods significantly outperform previous models with and without user and product information. It is also demonstrated that our model has great potential of handling extra information to improve document classification performance.

This study aims to address the diagnostic challenges in distinguishing gastric polyps from protrusions, emphasizing the need for accurate and cost-effective diagnosis strategies. It explores the application of Convolutional Neural Networks (CNNs) to improve diagnostic accuracy. This research introduces MultiAttentiveScopeNet, a deep learning model that incorporates multi-layer feature ensemble and attention mechanisms to enhance gastroscopy image analysis accuracy. A weakly supervised labeling strategy was employed to construct a large multi-class gastroscopy image dataset for training and validation. MultiAttentiveScopeNet demonstrates significant improvements in prediction accuracy and interpretability. The integrated attention mechanism effectively identifies critical areas in images to aid clinical decisions. Its multi-layer feature ensemble enables robust analysis of complex gastroscopy images. Comparative testing against human experts shows exceptional diagnostic performance, with accuracy, micro and macro precision, micro and macro recall, and micro and macro AUC reaching 0.9308, 0.9312, 0.9325, 0.9283, 0.9308, 0.9847 and 0.9853 respectively. This highlights its potential as an effective tool for primary healthcare settings. This study provides a comprehensive solution to address diagnostic challenges differentiating gastric polyps and protrusions. MultiAttentiveScopeNet improves accuracy and interpretability, demonstrating the potential of deep learning for gastroscopy image analysis. The constructed dataset facilitates continued model optimization and validation. The model shows promise in enhancing diagnostic outcomes in primary care.

Due to the vast amount of reviews available on the Web, in the past decades, a growing share of work has focused on sentiment analysis. Aspect-based sentiment classification is the subtask that seeks to detect the sentiment expressed by the content creators towards a defined target within a sentence. This paper introduces three novel unsupervised attentional neural network models for aspect-based sentiment classification, and tests them on English restaurant reviews. The first model employs an autoencoder-like structure to learn a sentiment embedding matrix where each row of the matrix represents the embedding for one sentiment. To improve the model, a target-based attention mechanism is included that de-emphasizes irrelevant words. Last, a redundancy and a seed regularization term constrain the sentiment embedding matrix. The second model extends the first by including a Bi-LSTM layer in the attention mechanism to exploit contextual information. The third model further adapts a Left-Center-Right separated neural network with Rotatory attention structure from the supervised realm to an unsupervised setting. Although all three models construct meaningful sentiment embeddings, experimental results indicate that the inclusion of the Bi-LSTM in the attention mechanism leads to a more precise attention mechanism and, thus, better predictions. The best model, i.e., the second, outperforms all investigated unsupervised and weakly supervised algorithms for aspect-based sentiment classification from the literature.

Unmanned aerial vehicle–human collaboration route planning for intelligent infrastructure inspection

Recently, hyperspectral image (HSI) classification has attracted increasing attention in the remote sensing field. Plenty of CNN-based methods with diverse attention mechanisms (AMs) have been proposed for HSI classification due to AMs being able to improve the quality of feature representations. However, some of the previous AMs squeeze global spatial or channel information directly by pooling operations to yield feature descriptors, which inadequately utilize global contextual information. Besides, some AMs cannot exploit the interactions among channels or positions with the aid of nonlinear transformation well. In this article, a spectral-spatial network with channel and position global context (GC) attention (SSGCA) is proposed to capture discriminative spectral and spatial features. Firstly, a spectral-spatial network is designed to extract spectral and spatial features. Secondly, two novel GC attentions are proposed to optimize the spectral and spatial features respectively for feature enhancement. The channel GC attention is used to capture channel dependencies to emphasize informative features while the position GC attention focuses on position dependencies. Both GC attentions aggregate global contextual features of positions or channels adequately, following a nonlinear transformation. Experimental results on several public HSI datasets demonstrate that the spectral-spatial network with GC attentions outperforms other related methods.

ClathPLM: Deep multi-view feature extraction with CNN and attention enhances clathrin protein identification.

In order to solve the problem of insufficient accuracy of the existing environmental noise classification models, an environmental noise classification network based on multi-channel discrimination and attention mechanism was proposed. First, multi-channel resolution features are used to more comprehensively represent and capture the diversity and complexity of audio features. Then, a second-order dense connection method was used to replace the dense connection in DenseNet to solve the overfitting problem caused by the high similarity and redundancy of the sound signal. Finally, a G-SE Layer module is used for acoustic signals, which enhances the model's attention to important features by efficiently fusing channel features. Experimental results show that compared with the original DenseNet network, the improved GS-2DenseNet has an improved recognition rate of environmental noise by 4.83%, and the accuracy and F1-score are higher than those of acoustic networks such as ResNet and EcapaTdnn.

The recent advances in Vision Transformer (ViT) have demonstrated its impressive performance in image classification, which makes it a promising alternative to Convolutional Neural Network (CNN). Unlike CNNs, ViT represents an input image as a sequence of image patches. The patch-wise input image representation makes the following question interesting: How does ViT perform when individual input image patches are perturbed with natural corruptions or adversarial perturbations, compared to CNNs? In this work, we study the robustness of vision transformers to patch-wise perturbations. Surprisingly, we find that vision transformers are more robust to naturally corrupted patches than CNNs, whereas they are more vulnerable to adversarial patches. Furthermore, we conduct extensive qualitative and quantitative experiments to understand the robustness to patch perturbations. We have revealed that ViT's stronger robustness to natural corrupted patches and higher vulnerability against adversarial patches are both caused by the attention mechanism. Specifically, the attention model can help improve the robustness of vision transformers by effectively ignoring natural corrupted patches. However, when vision transformers are attacked by an adversary, the attention mechanism can be easily fooled to focus more on the adversarially perturbed patches and cause a mistake.

A text corpus typically contains two types of context information -- global context and local context. Global context carries topical information which can be utilized by topic models to discover topic structures from the text corpus, while local context can train word embeddings to capture semantic regularities reflected in the text corpus. This encourages us to exploit the useful information in both the global and the local context information. In this paper, we propose a unified language model based on matrix factorization techniques which 1) takes the complementary global and local context information into consideration simultaneously, and 2) models topics and learns word embeddings collaboratively. We empirically show that by incorporating both global and local context, this collaborative model can not only significantly improve the performance of topic discovery over the baseline topic models, but also learn better word embeddings than the baseline word embedding models. We also provide qualitative analysis that explains how the cooperation of global and local context information can result in better topic structures and word embeddings.