Skeleton-Based Human Action Recognition with Spatial and Temporal Attention-Enhanced Graph Convolution Networks

The main core purpose of artificial emotional intelligence is to recognize human emotions. Technologies such as facial, semantic, or brainwave recognition applications have been widely proposed. However, the abovementioned recognition techniques for emotional features require a large number of training samples to obtain high accuracy. Human behaviour pattern can be trained and recognized by the continuous movement of the Spatial Temporal Graph Convolution Network (ST-GCN). However, this technology does not distinguish between the speed of delicate emotions, and the speed of human behaviour and delicate changes of emotions cannot be effectively distinguished. This research paper proposes Spatial Temporal Variation Convolutional Network training for human emotion recognition, using skeleton detection technology to calculate the degree of skeleton point change between consecutive actions and using the nearest neighbour algorithm to classify speed levels and train the ST-GCN recognition model to obtain the emotional state. Application of the speed change recognition ability of the Spatial Temporal Variation Graph Convolution Network (STV-GCN) to artificial emotional intelligence calculation makes it possible to efficiently recognize the delicate actions of happy, sad, fear, and angry in human behaviour. The STV-GCN technology proposed in this paper is compared with ST-GCN and can effectively improve the recognition accuracy by more than 50%.

Graph convolutional networks (GCNs) achieved promising performance in skeleton-based human action recognition by modeling a sequence of skeletons as a spatio-temporal graph. Most of the recently proposed GCN-based methods improve the performance by learning the graph structure at each layer of the network using spatial attention applied on a predefined graph Adjacency matrix that is optimized jointly with model's parameters in an end-to-end manner. In this paper, we analyze the spatial attention used in spatio-temporal GCN layers and propose a symmetric spatial attention for better reflecting the symmetric property of the relative positions of the human body joints when executing actions. We also highlight the connection of spatio-temporal GCN layers employing additive spatial attention to bilinear layers, and we propose the spatiotemporal bilinear network (ST-BLN) which does not require the use of predefined Adjacency matrices and allows for more flexible design of the model. Experimental results show that the three models lead to effectively the same performance. Moreover, by exploiting the flexibility provided by the proposed ST-BLN, one can increase the efficiency of the model.

Human action abnormality detection has been attempted by various sensors for application domains like rehabilitation, healthcare, and assisted living. Since the release of motion sensors that ease the human body skeleton retrieval, skeleton-based human action recognition has recently been an active topic in the area of artificial intelligence. Unlike human action recognition, human action abnormality detection is an emerging field that aims to detect the incorrect action from the same action class. Graph convolutional network has been widely adopted for human action recognition. However, to the best of our knowledge, whether it could be effective for the task of human action abnormality detection has not been attempted. To advance prior work in the emerging field of human action abnormality detection, we propose a novel method that uses graph convolutional network to detect abnormal actions in skeleton data. To validate the effectiveness of our proposed method, we conduct extensive experiments on a public dataset called UI-PRMD. Based on the experimental results, our proposed method achieved superior action abnormality detection performance comparing with existing deep learning methods.

This paper presents a novel framework for skeleton-based Human Action Recognition (HAR) based on Graph Convolution Networks (GCNs). The proposed framework aims to increase human action recognition performance of GCN-based methods by incorporating a missing-joint-handling pre-processing step and a novel adjacency matrix construction method in a single human action recognition pipeline. The missing-joint-handling pre-processing step is utilized to infer missing data in the input sequence, which may occur due to imperfect skeleton extraction, based on imputation methods. The novel adjacency matrix construction method is executed offline to compute an improved weighted adjacency matrix specifically designed for HAR, which is utilized in every layer of the employed GCN. Moreover, both the pre-processing step and the adjacency construction method can be utilized along with any GCN architecture, allowing any GCN-based HAR method to be employed in the proposed framework. Experimental evaluation on two public datasets indicate favorable human action classification scores compared to the employed baseline and all competing methods both for 2D and 3D skeleton-based human action recognition, while using a GCN architecture with less learnable parameters.

PSTFormer: A novel parallel spatial-temporal transformer for remaining useful life prediction of aeroengine

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Human action recognition plays a vital role in video understanding. In recent years, skeleton-based action recognition approaches have gained widespread attention due to their robustness against environmental interferences. This paper compiles 102 skeleton-based human action recognition methods and comparatively analyzes their performance on nine public datasets. This paper introduces the manual feature and deep learning based methods according to learning paradigms. Specifically, the manual feature methods are divided into three categories, i.e., geometric, kinetic, and statistical representations, in the light of feature descriptor. Meanwhile, the deep learning based methods are classified into five subclasses by backbones, i.e., recurrent neural networks, convolutional neural networks, graph convolutional networks, Transformer, and hybrid networks. Through comprehensive analysis, we not only present the research status of skeleton-based action recognition but also conclude that the existing methods have limitations such as poor generalization ability and high computational cost. Finally, this paper looks into future research directions from the aspects of network structure design, similar action distinction, domain data set expansion, and multi-person interaction.

Vehicle trajectory prediction is an important research basis for the decision making and path planning of the intelligent and connected vehicle. In the connected vehicle environment, vehicles share information and drive cooperatively, and the intelligent and connected vehicles are able to obtain more accurate and rich perception information, which provides a data basis for accurate prediction of vehicle trajectories. However, attaining accurate and effective vehicle trajectory predictions poses technical challenges due to insufficient extraction of vehicular spatial–temporal interaction features. In this paper, we propose a vehicle trajectory prediction model based on graph convolutional neural network (GCN) in a connected vehicle environment. Specifically, using the driving scene information obtained by the intelligent and connected vehicle, the spatial graph and temporal graph are constructed based on the spatial interaction coefficient (SIC) and self-attention mechanism, respectively. Then, the graph data are entered into the interaction extraction module, and the spatial interaction features and temporal interaction features are extracted separately using the graph convolutional networks, which are fused to obtain the spatial–temporal interaction information. Finally, the interaction features are learned based on the convolutional neural networks to output the future trajectory information of all vehicles in the scene by one forward operation rather than a step-by-step process. The ablation experiment results show that the method proposed in this study to model the spatiotemporal interaction among vehicles based on SIC and self-attention mechanism reduces the prediction error by 5% and 12%, respectively. The results from the model comparison experiment show that the proposed method engenders an 8% improvement in prediction accuracy over the state-of-the-art solution, providing technical and theoretical support for trajectory prediction research of intelligent and connected vehicles.

We used ERPs to investigate whether temporal attention interacts with spatial attention and feature-based attention to enhance visual processing. We presented a visual search display containing one singleton stimulus among a set of homogenous distractors. Participants were asked to respond only to target singletons of a particular color and shape that were presented in an attended spatial position. We manipulated temporal attention by presenting a warning signal before each search display and varying the foreperiod (FP) between the warning signal and the search display in a blocked manner. We observed distinctive ERP effects of both spatial and temporal attention. The amplitudes for the N2pc, SPCN, and P3 were enhanced by spatial attention indicating a processing benefit of relevant stimulus features at the attended side. Temporal attention accelerated stimulus processing; this was indexed by an earlier onset of the N2pc component and a reduction in reaction times to targets. Most importantly, temporal attention did not interact with spatial attention or stimulus features to influence visual processing. Taken together, the results suggest that temporal attention fosters visual perceptual processing in a visual search task independently from spatial attention and feature-based attention; this provides support for the nonspecific enhancement hypothesis of temporal attention.

Traffic flow forecasting is the essential part of intelligent transportation sSystem (ITS), which can fully protect traffic safety and improve traffic system management capability. Nevertheless, it is still a challenging problem, which is influenced by many complex factors, including regional distribution and external factors (e.g., holidays and weather). To combine various factors to forecast traffic flow, we presented a novel neural network structure called ConvLSTM-based Spatiotemporal Attention Network (CLSTAN). Specifically, our proposed model is composed of four modules: a preliminary feature extraction module, a spatial attention module, a temporal attention module, and an information fusion module. The spatiotemporal attention module can efficiently learn the complex spatiotemporal patterns of traffic flow through the attention mechanism. The spatial attention module uses a series of initial traffic flow maps as input and obtains the weights of the various regions through a ConvLSTM. The temporal attention module uses the spatially weighted traffic flow map as input and acquires the complex spatiotemporal patterns of traffic flow by a ConvLSTM that introduces an attention mechanism. Finally, the information fusion module integrates spatiotemporal information from multiple time dimensions to forecast future traffic flow. Moreover, to confirm the validity of our method, our experiments were conducted extensively on the TaxiBJ and BikeNYC datasets, and ultimately, CLSTAN performed better than other baseline experiments.

The three projects documented in this thesis investigated the role of three forms of attention in visual processing. Spatial, feature-based and temporal attention. The spatial and feature-based attentional characteristics were studied by conducting extracellular recordings from neurons of area MT and MSTd of macaque visual cortex. While the role of temporal attention was studied in a human psychophysics project. In the physiology projects we first defined the tuning characteristics of neurons from area MT and MSTd for two stimulus dimensions, namely spiral motion stimuli (SMS) and linear motion stimuli (LMS) on the basis of directionality index, the relative frequency of tuned versus untuned neurons, and the individual cells responses to the two forms of stimuli. We found that MT and MSTd showed differential tuning to LMS and SMS. Area MSTd demonstrated better selectivity for SMS as compared to LMS. Conversely, area MT was better tuned to LMS than SMS. Further, we compared spatial and feature-based attentional modulation in area MT and MSTd across SMS and LMS. We found that spatial attention modulates the responses of neurons in both the areas irrespective of the feature dimension defining the attended stimuli. On the other hand feature-based attention was restricted to the stimulus dimension for which an area showed better selectivity, that is SMS for MSTd and LMS for MT. We also observed that the correlation of firing rates with reaction times was restricted to an area’s preferred stimulus dimension. The presence of a significant negative correlation of firing rate with reaction time in area MSTd when attention was directed to its preferred dimension (SMS), indicated that these neurons were involved in perceptual processing of SMS. Similarly, the presence of a significant negative correlation of firing rates with the reaction times in area MT when attention was directed to LMS, provides evidence that MT neurons were involved in perceptual processing of LMS. Taken together, these results show that neurons across the visual cortex are tuned to a range of stimulus dimensions. This multiple stimulus selectivity is thought to be utilized to develop a saliency map (by interaction of bottom-up and top-down processes), which marks the behaviorally relevant aspects of visual space. These behaviorally relevant aspects of the visual space are then preferentially processed throughout the visual hierarchy, and our results suggests that a given area contributes to visual perception by flexibly processing only the stimulus dimension for which it shows best selectivity as a function of stimulus relevance. Finally in human subjects we investigated the temporal characteristics of visual motion processing, using the ‘attentional blink’ paradigm. Attentional blink is a widely studied phenomenon, but almost all studies to the date have been are restricted to stationary stimuli. We here extended this paradigm to motion stimuli. We found the presence of an attentional blink in the motion domain, in that the detection of second of the two target (motion) stimuli was severely degraded when it was presented within 450ms of the first target (motion) stimulus. However, our results were slightly different from the experiments with the stationary stimuli. First, there was an absence of the lag-1 sparing effect typically found with stationary stimuli and second, the time interval over which the attentional blink was observed was about 100ms shorter than typically reported for stationary stimuli. Post-hoc analysis showed that these effects in the motion domain were dependent upon the direction of motion of the first target. Our data showed that when the first target moved in the opposite direction to the second target, there was no attentional blink, but when the first target moved in an orthogonal direction to the second target, there was an attentional blink. These results are in agreement to previous studies showing that the processing of motion stimuli is facilitated when it is preceded by a stimulus moving in opposite directions, while processing of motion stimuli is suppressed when it is preceded by a stimulus moving in an orthogonal direction. Taken together the results of this thesis provides an important contribution to our understanding of the role of attention in visual motion processing. It will be interesting to extend this study in other visual cortical areas and measure spatial and feature-based attention from same set of neurons across different stimulus dimension.

Local and global self-attention enhanced graph convolutional network for skeleton-based action recognition

Graph convolutional networks (GCNs) can effectively learn graph data features and are widely used to predict the bearing remaining useful life (RUL). However, most existing GCN-based methods are based on single-node relationships to construct the graph structure while ignoring potential complex relationships between nodes. To enrich graph feature information and improve bearing RUL prediction accuracy, a novel bearing RUL prediction method is proposed in this article, which use spatial–temporal information fusion graph convolution network. The proposed method mines implicit temporal dependence and spatial correlation in graph samples by constructing PathGraph and RadiusGraph, which effectively improves the feature diversity of graph samples. In addition, to efficiently extract the spatial–temporal features of graph samples and alleviate the over-smoothing problem in GCNs, a feature extraction module combining graph attention network (GAT) and graph convolution network–temporal convolution network (GCN-TCN) is designed, where GAT dynamically assigns edge weights to capture spatial dependencies, while GCN-TCN enhances global temporal dependency capture. The experimental validation results show that the proposed method effectively improves bearing RUL prediction accuracy.

Traffic forecasting plays an important role in intelligent transportation systems. However, the prediction task is highly challenging due to the mixture of global and local spatiotemporal dependencies involved in traffic data. Existing graph neural networks (GNNs) typically capture spatial dependencies with the predefined or learnable static graph structure, ignoring the hidden dynamic patterns in traffic networks. Meanwhile, most recurrent neural networks (RNNs) or convolutional neural networks (CNNs) cannot effectively capture temporal correlations, especially for long-term temporal dependencies. In this paper, we propose a spatial–temporal attention graph convolution network (STAGCN), which acquires a static graph and a dynamic graph from data without any prior knowledge. The static graph aims to model global space adaptability, and the dynamic graph is designed to capture local dynamics in the traffic network. A gated temporal attention module is further introduced for long-term temporal dependencies, where a causal-trend attention mechanism is proposed to increase the awareness of causality and local trends in time series. Extensive experiments on four real-world traffic flow datasets demonstrate that STAGCN achieves an outstanding prediction accuracy improvement over existing solutions.

Non-mass enhancement (NME) in breast dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) often shows subtle and diffuse enhancement, which can make accurate delineation challenging for radiologists. In this study, we developed an automated segmentation method for NME in DCE-MRI using a three-dimensional (3D) TransUNet model with a temporal attention mechanism. The attention module uses images of the difference between the pre- and early post-contrast phases to highlight regions that exhibit enhancement-related temporal changes. Our database consisted of 151 DCE-MRI cases with NME, each including both pre- and early post-contrast phases. The proposed network comprises two parallel CNN encoders (one for early post-contrast images and one for early–pre subtraction images), a temporal attention module, a transformer encoder, and a CNN decoder. The two encoders independently extract features from their respective inputs. The temporal attention module uses the features from the difference branch to weight those of the early branch. The weighted features are then fused and passed to the transformer encoder, which captures spatial and contextual dependencies within the 3D volume. The CNN decoder reconstructs the NME segmentation map through transposed convolutions and upsampling layers. Using five-fold cross-validation, the proposed method achieved a Jaccard index of 0.724 and a Dice coefficient of 0.802. Our approach significantly outperformed the baseline 3D TransUNet (Jaccard: 0.585, p < .001; Dice: 0.690, p < .001). These results demonstrate the high segmentation performance of the proposed method for NME and indicate its potential to improve diagnostic assessment in breast DCE-MRI.