CLSTAN: ConvLSTM-Based Spatiotemporal Attention Network for Traffic Flow Forecasting

PSTFormer: A novel parallel spatial-temporal transformer for remaining useful life prediction of aeroengine

Radio-frequency (RF) based human sensing technologies, due to their great practical value in various applications and privacy-preserving nature, have gained tremendous attention in recent years. However, without fully exploiting the characteristics of radio signals, the performance of existing methods are still limited. First, RF features of the moving human body have different representations in dimensions such as channel and scale, which is challenging when performing feature fusion. Besides, the human body is specularly reflective with respect to the radar, which means the human body cannot be fully captured by a single RF snapshot. Therefore, the radar signal reflected by the human body is sparse and incomplete, which is difficult to extract high-quality features for 3D human pose estimation. In this paper, we present the RF-based Pose Machines (RPM), a novel framework which can generate 3D skeletons from RF signals. Considering the characteristics of RF signals, RPM includes several modules to overcome the challenges. Firstly, a Feature Fusion Network (FFN) is designed to effectively fuse radio signals from horizontal and vertical planes based on the channels' correlation and maintain high-quality feature via a multi-scale fusion block. A Spatio-Temporal Attention network is then designed to reconstruct 3D skeletons from the sparse and incomplete RF signals. Specifically, a spatial attention module is designed to model non-local relationships among joints and reconstruct body parts that a single RF snapshot cannot capture. Afterwards, a temporal attention module is proposed to refine 3D pose based on temporal coherency learned from frame queries. To evaluate the performance of our RPM framework, we construct a large-scale dataset of synchronized 3d skeletons and RF signals, RFSkeleton3D. Our experimental results show that RPM locates 3D key points of the human body with an average error of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$5.71 cm$</tex-math></inline-formula> and maintains its performance in new environments with occlusion or bad illumination. The dataset and codes will be made in public.

Case instance segmentation of small farmland based on Mask R-CNN of feature pyramid network with double attention mechanism in high resolution satellite images

Skeleton-based human action recognition has great potential for human behavior analysis owing to its simplicity and robustness in varying environments. This paper presents a spatial and temporal attention-enhanced graph convolution network (STAEGCN) for human action recognition. The spatial-temporal attention module in the network uses convolution embedding for positional information and adopts multi-head self-attention mechanism to extract spatial and temporal attention separately from the input series of the skeleton. The spatial and temporal attention are then concatenated into an entire attention map according to a specific ratio. The proposed spatial and temporal attention module was integrated with an adaptive graph convolution network to form the backbone of STAEGCN. Based on STAEGCN, a two-stream skeleton-based human action recognition model was trained and evaluated. The model performed better on both NTU RGB+D and Kinetics 400 than 2s-AGCN and its variants. It was proven that the strategy of decoupling spatial and temporal attention and combining them in a flexible way helps improve the performance of graph convolution networks in skeleton-based human action recognition.

Finger-vein biometrics has recently gained significant attention due to its robust privacy and high security features. Despite notable advancements, most existing methods focus on extracting features from a 2-dimensional (2D) image projected from 3D vein vessels with a single view. However, recognition based on a single view is prone to errors due to variations in finger positioning, especially those caused by finger roll movements, which can degrade recognition performance. To address this challenge, we propose ABLSTM-TSVT, an Attention Bidirectional LSTM-based Temporal-Spatial Vein Transformer for multi-view finger-vein recognition. First, we enhance LSTM with an attention mechanism to create an attention LSTM for extracting temporal features. We further improve this by introducing a local attention module, which learns temporal dependencies between a patch (token) and its adjacent patches across multiple views, integrating it with the attention LSTM to form a temporal attention module. Second, we develop a spatial attention module that captures the spatial dependencies of patches within an image. Finally, merging the temporal and the spatial attention modules, we create our temporal-spatial transformer model, which effectively represents features from multi-view images. Experimental results on two multi-view datasets demonstrate that our approach outperforms state-of-the-art approaches in enhancing identification accuracy and reducing verification errors in vein classifiers.

Cardiac anatomy segmentation is essential for clinical assessment of cardiac function and disease diagnosis to inform treatment and intervention. Deep learning (DL) has improved cardiac anatomy segmentation accuracy, especially when information on cardiac motion dynamics is integrated into the networks. Several methods for incorporating motion information have been proposed; however, existing methods are not yet optimal: adding the time dimension to input data causes high computational costs, and incorporating registration into the segmentation network remains computationally costly and can be affected by errors of registration, especially with non-DL registration. While attention-based motion modeling is promising, suboptimal design constrains its capacity to learn the complex and coherent temporal interactions inherent in cardiac image sequences. Here, we propose a novel approach to incorporating motion information in the DL segmentation networks: a computationally efficient yet robust Temporal Attention Module (TAM), modeled as a small, multi-headed, cross-temporal attention module, which can be plug-and-play inserted into a broad range of segmentation networks (CNN, transformer, or hybrid) without a drastic architecture modification. Extensive experiments on multiple cardiac imaging datasets, such as 2D echocardiography (CAMUS and EchoNet-Dynamic), 3D echocardiography (MITEA), and 3D cardiac MRI (ACDC), confirm that TAM consistently improves segmentation performance across datasets when added to a range of networks, including UNet, FCN8s, UNetR, SwinUNetR, and the recent I2UNet and DT-VNet. Integrating TAM into SAM yields a temporal SAM that reduces Hausdorff distance (HD) from 3.99 mm to 3.51 mm on the CAMUS dataset, while integrating TAM into a pre-trained MedSAM reduces HD from 3.04 to 2.06 pixels after fine-tuning on the EchoNet-Dynamic dataset. On the ACDC 3D dataset, our TAM-UNet and TAM-DT-VNet achieve substantial reductions in HD, from 7.97 mm to 4.23 mm and 6.87 mm to 4.74 mm, respectively. Additionally, TAM's training does not require segmentation of ground truths from all time frames and can be achieved with sparse temporal annotation. TAM is thus a robust, generalizable, and adaptable solution for motion-awareness enhancement that is easily scaled from 2D to 3D. The code is available at https://github.com/kamruleee51/TAM.

The segmentation and classification of different types of nuclei plays an important role in discriminating and diagnosing of the initiation, development, invasion, metastasis and therapeutic response of tumors of various organs. Recently, deep learning method based on attention mechanism has achieved good results in nuclei semantic segmentation. However, the design of attention module architecture relies heavily on the experience of researchers and a large number of experiments. Therefore, in order to avoid this manual design and achieve better performance, we propose a new Neural Architecture Search-based Spatial and Channel joint Attention Module (NAS-SCAM) to obtain better spatial and channel weighting effect. To the best of our knowledge, this is the first time to apply NAS to the attention mechanism. At the same time, we also use synchronous search strategy to search architectures independently for different attention modules in the same network structure. We verify the superiority of our methods over the state-of-the-art attention modules and networks in public dataset of MoNuSAC 2020. We make our code and model available at https://github.com/ZuhaoLiu/NAS-SCAM.

In recent years, attention mechanisms have shown great potential in various computer vision tasks. However, most existing methods focus on developing more complex attention modules for better performance, which inevitably increases the complexity of the model. To overcome performance and complexity tradeoffs, this paper proposes efficient spatial and channel attention (ESCA), a symmetric, comprehensive, and efficient attention module. By analyzing squeeze-and-excitation (SE), convolutional block attention module (CBAM), coordinate attention (CA), and efficient channel attention (ECA) modules, we abandon the dimension-reduction operation of SE module, verify the negative impact of global max pooling (GMP) on the model, and apply a local cross-channel interaction strategy without dimension reduction to learn attention. We not only care about the channel features of the image, we also care about the spatial location of the target on the image, and we take into account the effectiveness of channel attention, so we designed the symmetric ESCA module. The ESCA module is effective, as demonstrated by its application in the ResNet-50 classification benchmark. With 26.26 M parameters and 8.545 G FLOPs, it introduces a mere 0.14% increment in FLOPs while achieving over 6.33% improvement in Top-1 accuracy and exceeding 3.25% gain in Top-5 accuracy. We perform image classification and object detection tasks on ResNet, MobileNet, YOLO, and other architectures on popular datasets such as Mini ImageNet, CIFAR-10, and VOC 2007. Experiments show that ESCA can achieve great improvement in model accuracy at a very small cost, and it performs well among similar models.

FBSA-Net: A novel model based on attention mechanisms for emotion recognition in VR and 2D scenes

Non-mass enhancement (NME) in breast dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) often shows subtle and diffuse enhancement, which can make accurate delineation challenging for radiologists. In this study, we developed an automated segmentation method for NME in DCE-MRI using a three-dimensional (3D) TransUNet model with a temporal attention mechanism. The attention module uses images of the difference between the pre- and early post-contrast phases to highlight regions that exhibit enhancement-related temporal changes. Our database consisted of 151 DCE-MRI cases with NME, each including both pre- and early post-contrast phases. The proposed network comprises two parallel CNN encoders (one for early post-contrast images and one for early–pre subtraction images), a temporal attention module, a transformer encoder, and a CNN decoder. The two encoders independently extract features from their respective inputs. The temporal attention module uses the features from the difference branch to weight those of the early branch. The weighted features are then fused and passed to the transformer encoder, which captures spatial and contextual dependencies within the 3D volume. The CNN decoder reconstructs the NME segmentation map through transposed convolutions and upsampling layers. Using five-fold cross-validation, the proposed method achieved a Jaccard index of 0.724 and a Dice coefficient of 0.802. Our approach significantly outperformed the baseline 3D TransUNet (Jaccard: 0.585, p < .001; Dice: 0.690, p < .001). These results demonstrate the high segmentation performance of the proposed method for NME and indicate its potential to improve diagnostic assessment in breast DCE-MRI.

In recent years, the research of deep learning has received extensive attention, and many breakthroughs have been made in various fields. On this basis, a neural network with the attention mechanism has become a research hotspot. In this paper, we try to solve the image classification task by implementing channel and spatial attention mechanism which improve the expression ability of neural network model. Different from previous studies, we propose an attention module consisting of channel attention module (CAM) and spatial attention module (SAM). The proposed module derives attention graphs from channel dimension and spatial dimension respectively, then the input features are selectively learned according to the importance of the features. Besides, this module is lightweight and can be easily integrated into image classification algorithms. In the experiment, we combine the deep residual network model with the attention module and the experimental results show that the proposed method brings higher image classification accuracy. The channel attention module adds weight to the signals on different convolution channels to represent the correlation. For different channels, the higher the weight, the higher the correlation which required more attention. The main function of spatial attention is to capture the most informative part in the local feature graph, which is a supplement to channel attention. We evaluate our proposed module based on the ImageNet-1K and Cifar-100 respectively. Through a large number of comparative experiments, our proposed model achieved outstanding performance.

Siamese-based models achieves outstanding performance in visual tracking. However, as the template is invariant during tracking, results in failing to capture the appearance changes over time, leading to incredible results. To tackle this problem, an object tracking method with dynamic weighting Siamese network has been proposed. First, we design a temporal attention module to obtain the time-sensitive features, which is learned by adaptively weighting the consecutive historical reference frames. Then, a spatial attention module is adopted to enhance the spatial information of the target, and promote the discriminative ability of the network. Finally, the Focal Loss is utilized to train the network offline, to improve the ability of handling the hard negative samples. We conducted the experiment on OTB2015 benchmark to evaluate the performance of the proposed method and other state-of-the-arts, comparison results demonstrate the good performance of our model.

Sheep behavior recognition helps to monitor the health status of sheep and prevent the outbreak of infectious diseases. Aiming at the problems of low detection accuracy and slow speed due to the crowding of sheep in real farming scenarios, which can easily obscure each other, this study proposes a lightweight sheep behavior recognition model based on the YOLOv8n model. First, the Convolutional Block Attention Module (CBAM) is introduced and improved in the YOLOv8n model, and the channel attention module and spatial attention module are changed from serial to parallel to construct a novel attention mechanism, PCBAM, to enhance the network’s attention to the sheep and eliminate redundant background information; second, the ordinary convolution in the backbone network is replaced with depth-separable convolution, which effectively reduces the number of parameters in the model and reduces the computational complexity. The study takes the housed breeding sheep as the test object, installs a camera diagonally above the sheep pen to collect images and makes a data set for testing, and in order to verify the superiority of the PD-YOLO model, compares it with a variety of target detection models. The experimental results show that the mean average precision (mAP) of the model proposed in this paper are 95.8%, 98.9%, and 96.2% for the three postures of sheep lying, feeding, and standing, respectively, which are 8.5%, 0.8%, and 0.8% higher than those of YOLOv8n, respectively, and the size of the model has been reduced by 13.3% and the amount of computation has been reduced by 12.1%. The inference speed reaches 52.1 FPS per second, which is better than other models in meeting the real-time detection requirement. To verify the practicality of this research method, the PD-YOLO model was deployed on the RK3399Pro development board for testing, and a high inference speed was achieved. It can provide effective technical support for sheep smart farming.

The rapid advancement of sensor technologies and artificial intelligence are creating new opportunities for traffic safety enhancement. Dashboard cameras (dashcams) have been widely deployed on both human driving vehicles and automated driving vehicles. A computational intelligence model that can accurately and promptly predict accidents from the dashcam video will enhance the preparedness for accident prevention. The spatial-temporal interaction of traffic agents is complex. Visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, the early anticipation of traffic accidents remains a challenge. Inspired by the attention behavior of humans in visually perceiving accident risks, this paper proposes a Dynamic Spatial-Temporal Attention (DSTA) network for the early accident anticipation from dashcam videos. The DSTA-network learns to select discriminative temporal segments of a video sequence with a Dynamic Temporal Attention (DTA) module. It also learns to focus on the informative spatial regions of frames with a Dynamic Spatial Attention (DSA) module. A Gated Recurrent Unit (GRU) is trained jointly with the attention modules to predict the probability of a future accident. The evaluation of the DSTA-network on two benchmark datasets confirms that it has exceeded the state-of-the-art performance. A thorough ablation study that assesses the DSTA-network at the component level reveals how the network achieves such performance. Furthermore, this paper proposes a method to fuse the prediction scores from two complementary models and verifies its effectiveness in further boosting the performance of early accident anticipation.

Multivariate time series classification (MTSC) is a fundamental and essential research problem in the domain of time series data mining. Recently deep neural networks emerged as an end-to-end solution for MTSC and achieve state-of-the-art results on several public datasets. It is favored by its hierarchical feature extraction ability and most of the researches focus on designing a network architecture to ensure its performance on MTSC. Despite this, there are seldom investigations on the attention mechanism in MTSC, which has been demonstrated as an effective module to extract features in other domains. In this paper, we propose a residual channel and temporal attention (CT_CAM) module, which aims to refine the feature extracted from the convolutional neural network and thus improve the classification performance. Extensive experiments on 15 public MTSC datasets show that the proposed CT_CAM module achieves competitive performance compared with nine baseline methods and three other attention modules.