Gradient-Weighted Class Activation Mapping for Spatio Temporal Graph Convolutional Network

In recent years, human motion recognition in computer vision has become a hot research direction in this field. Based on 2D human motion recognition technology, real-time extraction of motion features from 2D planes is used to recognize human movements. This method can only learn the position contour and color information of the image. It cannot directly reflect the motion situation, which results in low recognition accuracy and efficiency. In response to this issue, this study proposes a combination method of motion recognition and 3D pose estimation to recognize and classify basketball movements. First, the 2D skeleton model is obtained by extracting the feature information in the video action, which is converted into a 3D model, and the model is replaced by the time-space convolutional network to establish a human action recognition model. The experiment showed that when the number of iterations reached 6, the accuracy of the spatio-temporal graph convolutional network algorithm model reached 92%. Comparing the accuracy rates of different algorithm models, the average accuracy rates of convolutional neural network, long short memory network, graph convolution, long short model of action recognition and graph convolution model of action recognition were 61.6%, 65.4%, 72.5%, 76.8% and 90.3% respectively. The results show that the proposed 3D recognition algorithm can accurately recognize different basketball movements. This study can provide reference for basketball coaches and athletes in basketball training.

Spatio-temporal Graph Convolutional Networks (ST-GCNs) have shown great performance in the context of skeleton-based action recognition. Nevertheless, ST-GCNs use raw skeleton data as vertex features. Such features have low dimensionality and might not be optimal for action discrimination. Moreover, a single layer of temporal convolution is used to model short-term temporal dependencies but can be insufficient for capturing both long-term. In this paper, we extend the Spatio-Temporal Graph Convolutional Network for skeleton-based action recognition by introducing two novel modules, namely, the Graph Vertex Feature Encoder (GVFE) and the Dilated Hierarchical Temporal Convolutional Network (DH-TCN). On the one hand, the GVFE module learns appropriate vertex features for action recognition by encoding raw skeleton data into a new feature space. On the other hand, the DH-TCN module is capable of capturing both short-term and long-term temporal dependencies using a hierarchical dilated convolutional network. Experiments have been conducted on the challenging NTU RGB-D 60, NTU RGB-D 120 and Kinetics datasets. The obtained results show that our method competes with state-of-the-art approaches while using a smaller number of layers and parameters; thus reducing the required training time and memory.

Spatio-temporal Graph Convolutional Networks (ST-GCNs) have shown great performance in the context of skeleton-based action recognition. Nevertheless, ST-GCNs use raw skeleton data as vertex features. Such features have low dimensionality and might not be optimal for action discrimination. Moreover, a single layer of temporal convolution is used to model short-term temporal dependencies but can be insufficient for capturing both long-term. In this paper, we extend the Spatio-Temporal Graph Convolutional Network for skeleton-based action recognition by introducing two novel modules, namely, the Graph Vertex Feature Encoder (GVFE) and the Dilated Hierarchical Temporal Convolutional Network (DH-TCN). On the one hand, the GVFE module learns appropriate vertex features for action recognition by encoding raw skeleton data into a new feature space. On the other hand, the DH-TCN module is capable of capturing both short-term and long-term temporal dependencies using a hierarchical dilated convolutional network. Experiments have been conducted on the challenging NTU RGB-D 60, NTU RGB-D 120 and Kinetics datasets. The obtained results show that our method competes with state-of-the-art approaches while using a smaller number of layers and parameters; thus reducing the required training time and memory.

This study proposes a vessel position prediction method using attention spatiotemporal graph convolutional networks, which addresses the issue of low prediction accuracy due to less consideration of inter-feature dependencies in current vessel trajectory prediction methods. First, the method cleans the vessel trajectory data and uses the Time-ratio trajectory compression algorithm to compress the trajectory data, avoiding data redundancy and providing feature points for vessel trajectories. Second, the Spectral Temporal Graph Neural Network (StemGNN) extracts the correlation matrix that describes the relationship between multiple variables as a priori matrix input to the prediction model. Then the vessel trajectory prediction model is constructed, and the attention mechanism is added to the spatial and temporal dimensions of the trajectory data based on the spatio-temporal graph convolutional network at the same time as the above operations are performed on different time scales. Finally, the features extracted from different time scales are fused through the full connectivity layer to predict the future trajectories. Experimental results show that this method achieves higher accuracy and more stable prediction results in trajectory prediction. The attention-based spatio-temporal graph convolutional networks effectively capture the spatio-temporal correlations of the main features in vessel trajectories, and the spatio-temporal attention mechanism and graph convolution have certain interpretability for the prediction results.

Dynamic traffic correlations based spatio-temporal graph convolutional network for urban traffic prediction

The spatiotemporal dynamics of traffic forecasting make it a challenging task. In recent years, by adapting to the topology of traffic networks where road segments serve as nodes, graph convolutional networks (GCNs) have been able to capture spatiotemporal dependencies, thereby improving traffic forecasting performance. However, there are two shortcomings of GCN-based methods: 1) existing methods treat the delays between nodes in the traffic network as equally important and fail to extract critical information effectively, leading to information redundancy, the introduction of irrelevant noise, and increased computational costs and 2) most methods overlook the issue that spatiotemporal correlations between nodes are inconsistent across different timescales. This article designs a new dynamic delay-aware multiscal spatiotemporal graph convolutional network (DDAMGCN) for traffic forecasting. Specifically, a dynamic delay-aware module is designed to identify key nodes and model the important delays from key nodes, so that the model focuses on key information and reduces computational cost. Additionally, a novel multiscale spatiotemporal graph convolution module is designed to achieve fine-grained modeling of the spatiotemporal correlation of different nodes at different timescales. Experiments on eight real traffic datasets verify the superiority of the proposed method compared to several state-of-the-art baselines.

Graph convolutional networks (GCNs) have revealed great promise in skeleton-based gait recognition by modeling human joints as a spatiotemporal graph. However, most current spatiotemporal GCN (ST-GCN) methods rely on a fixed graph topology and a static partitioning scheme. This fixed approach limits the model’s ability to capture complex relationships between distant body joints. Besides, Temporal Convolutional Networks (TCNs) are commonly used to model temporal dynamics. However, they have shortcomings in capturing temporal patterns at multiple scales. To overcome these constraints, this paper introduces a new architecture, i.e., SATGCN, that integrates a graph attention combination module (GACM) with a multi-scale temporal convolutional network (MSTCN). In GACM, a fusion approach is used, in which the attention module dynamically learns joint relationships with GCN at each frame. It enables the network to capture local spatial topology and long-range dependencies between physically distant body parts. Following the spatial module, the multi-scale TCN analyzes joint features over time at multiple scales, capturing both short- and long-range motion patterns across the gait sequence. The model is evaluated on the CASIA-B gait recognition dataset under normal, bag-carrying, and clothing conditions, achieving accuracies of 96.97%, 93.10%, and 90.65%, respectively. The performance is about 0.97%, 1.80%, and 0.65% better than that of previous works in the three conditions. The results reveal the effectiveness of combining an attention-guided GCN with multi-scale temporal convolution for skeleton-based gait recognition.

In order to provide intelligent and efficient healthcare services in the Internet of Medical Things (IoMT), human action recognition (HAR) can play a crucial role. As a result of their stringent requirements, such as high computational complexity and memory efficiency, classical HAR techniques are not applicable to modern and intelligent healthcare services, e.g., IoMT. To address these issues, we present in this paper a novel HAR technique for healthcare services in IoMT. This model, referred to as the spatio-temporal graph convolutional network (STGCN), primarily aims at skeleton-based human–machine interfaces. By independently extracting spatial and temporal features, STGCN significantly reduces information loss. Spatio-temporal information is extracted independently of the exact spatial and temporal point, ensuring the extraction of useful features for HAR. Using only joint data and fewer parameters, we demonstrate that our proposed STGCN achieved 92.2% accuracy on the skeleton dataset. Unlike multi-channel methods, which use a combination of joint and bone data and have a large number of parameters, multi-channel methods use both joint and bone data. As a result, STGCN offers a good balance between accuracy, memory consumption, and processing time, making it suitable for detecting medical conditions.

Spatio-temporal graph signal analysis has a significant impact on a wide range of applications, including hand/body pose action recognition. To achieve effective analysis, spatio-temporal graph convolutional networks (ST-GCN) leverage the powerful learning ability to achieve great empirical successes; however, those methods need a huge amount of high-quality training data and lack theoretical interpretation. To address this issue, the spatio-temporal graph scattering transform (ST-GST) was proposed to put forth a theoretically interpretable framework; however, the empirical performance of this approach is constrained by the fully mathematical design. To benefit from both sides, this work proposes a novel complementary mechanism to organically combine the spatio-temporal graph scattering transform and neural networks, resulting in the proposed spatio-temporal graph complementary scattering networks (ST-GCSN). The essence is to leverage the mathematically designed graph wavelets with pruning techniques to cover major information and use trainable networks to capture complementary information. The empirical experiments on hand pose action recognition show that the proposed ST-GCSN outperforms both ST-GCN and ST-GST.

The reason why China’s table tennis can continue to this day is because it is an important heritage sports event in the country. In order to ensure the high-level performance of the Chinese table tennis team in the game, the identification and detection of its sports characteristics is a very meaningful work. With the development of computer technology, the use of multimedia intelligence technology to conduct research on competitive tactics has become a general consensus in the sports world. This paper explores the recognition and detection technology of the strength of table tennis hitting movements from the perspective of biomechanics, aiming to provide scientific basis and technical support for improving the performance of athletes. Based on the principles of biomechanics and combined with multimedia intelligent algorithms, this study developed a spatiotemporal graph convolutional network (STGCN) and a motion detection method based on Kinect technology to identify and quantify the hitting strength of table tennis players. The experimental results of this paper show that in the recognition of different types of movements based on the STGCN method, the correct recognition rate of 100 groups of movement strength is 84%, and the correct recognition rate of 500 groups of movement strength is 91.6%; in the recognition of different types of movement strength based on Kinect, the correct recognition rate of 100 groups of movement strength is 97%, and the correct recognition rate of 500 groups of movement strength is 99.6%; it can be seen that no matter how many groups of hitting movements are made, the correct recognition rate of strength based on Kinect is higher than that of STGCN.

IntroductionAccurately recognizing and understanding human motion actions presents a key challenge in the development of intelligent sports robots. Traditional methods often encounter significant drawbacks, such as high computational resource requirements and suboptimal real-time performance. To address these limitations, this study proposes a novel approach called Sports-ACtrans Net.MethodsIn this approach, the Swin Transformer processes visual data to extract spatial features, while the Spatio-Temporal Graph Convolutional Network (ST-GCN) models human motion as graphs to handle skeleton data. By combining these outputs, a comprehensive representation of motion actions is created. Reinforcement learning is employed to optimize the action recognition process, framing it as a sequential decision-making problem. Deep Q-learning is utilized to learn the optimal policy, thereby enhancing the robot's ability to accurately recognize and engage in motion.Results and discussionExperiments demonstrate significant improvements over state-of-the-art methods. This research advances the fields of neural computation, computer vision, and neuroscience, aiding in the development of intelligent robotic systems capable of understanding and participating in sports activities.

Spatio-temporal heterogeneous graph using multivariate earth observation time series: Application for drought forecasting

Traffic flow prediction is a pivotal task in intelligent transportation systems. The primary challenge lies in accurately modeling the dynamically evolving and directional spatio-temporal dependencies inherent in road networks. Existing graph neural network-based methods suffer from three main limitations: (1) symmetric adjacency matrices fail to capture the causal propagation of traffic flow from upstream to downstream; (2) the serial combination of graph and temporal convolutions lacks an explicit modeling of joint spatio-temporal state transition probabilities; (3) the inherent low-pass filtering property of temporal convolutional networks tends to smooth high-frequency abrupt signals, thereby weakening responsiveness to sudden events. To address these issues, this paper proposes a causal-enhanced spatio-temporal Markov graph convolutional network (CSHGCN). At the spatial modeling level, we construct an asymmetric causal adjacency matrix by decoupling source and target node embeddings to learn directional traffic flow influences. At the spatio-temporal joint modeling level, we design a spatio-temporal Markov transition module (STMTM) based on spatio-temporal Markov chain theory, which explicitly learns conditional transition patterns through temporal dependency encoders, spatial dependency encoders, and a joint transition network. At the temporal modeling level, we introduce differential feature enhancement and high-frequency residual compensation mechanisms to preserve key abrupt change information through frequency-domain complementarity. Experiments on four datasets—PEMS03, PEMS04, PEMS07, and PEMS08—demonstrate that CSHGCN outperforms existing baselines in terms of MAE, RMSE, and MAPE, with ablation studies validating the effectiveness of each module.

This study introduces the MSTA-GNet (Multi-Scale Spatiotemporal Attention Graph Network), a novel deep learning model which integrates spatiotemporal self-attention mechanisms to model heterogeneous dependencies in traffic networks. The primary objective of the study is to improve existing traffic flow prediction models to address the inadequacies of traditional models in complex big data environments. Key innovations of the MSTA-GNet model include positional encoding and global and local self-attention mechanisms to capture long-term and short-term dependencies. Using the PeMS (Performance Measurement System) dataset, the study conducted performance comparison experiments among various deep learning models, including LSTM (Long Short-Term Memory), GCN (Graph Convolutional Network), DCRNN (Diffusion Convolutional Recurrent Neural Network), STGCN (Spatiotemporal Graph Convolutional Network), STMetaNet (Spatiotemporal Meta Network), and MSTA-GNet. The results showed that MSTA-GNet significantly outperformed other models with improvements of 13.4%, 11.8%, and 9.7% in Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE) metrics, respectively. Ablation studies further validated the significance of attention mechanisms, feature extraction, convolutional layers, and graph networks, confirming the effectiveness and practical application of MSTA-GNet in traffic flow prediction. This research provides important insights for AI-based congestion management, support for low-carbon traffic networks, and optimization of local traffic operations, demonstrating its significant practical value in intelligent transportation systems.

A novel anomaly detection method for multivariate time series based on spatial-temporal graph learning