Spatial Temporal Variation Graph Convolutional Networks (STV-GCN) for Skeleton-Based Emotional Action Recognition

In autonomous driving scenarios, pedestrian trajectory prediction is an important research direction. Based on the spatio-temporal graph convolutional neural network, we propose a new pedestrian trajectory prediction algorithm. The new algorithm constructs a series of new models around pedestrian intention estimation. The construction of the estimation algorithm considers the following aspects: the contextual information of pedestrians and the surrounding environment, the “pedestrian ego-vehicle” interaction combined with the vehicle speed estimation, the pedestrian’s own skeletal structure information and body language estimation, which includes head joints and the relative structural relationship of the torso joints, including whether it is out of the same plane, is rotated, and so on. Skeleton information feature extraction and construction adopts the method of graph convolutional neural network to structure pedestrians into joints in the form of graphs in non-Euclidean space, and further adopts spatial temporal graph convolutional network for feature extraction and learning. The new method is named a “head-torso”-based spatial temporal graph convolutional network (HT-STGCN). On the dataset PID, the novel method achieves substantial improvements over mainstream methods. Experimental results show that combining HT-STGCN with observed action can improve trajectory prediction.

In recent years, the increasing labor burden in Japanese agriculture hs become a serious issue, driving the development of robots to assist in transporting harvested crops. This study proposes a method that recognizes the action transitions of agricultural workers and enables smooth transport assistance by utilizing three-dimensional skeletal information obtained from RGB-D images. Specifically, we employ Spatial Temporal Graph Convolutional Networks (ST-GCN) to detect the transition from "harvesting" to "loading." The recognition results are used to control the robot so that it approaches the worker before the loading action begins. The proposed method introduces a new labeling scheme tailored to harvesting and crop-loading motions, thereby improving recognition performance with ST-GCN. Evaluation experiments verified its generalization capability to different harvesting postures and workers, demonstrating an 18.7% improvement in action transition recognition accuracy compared with conventional methods. Furthermore, in robot-following experiments with the proposed method implemented, we confirmed that the system could both recognize action transitions and adjust the target following distance before the worker started loading. These results show that an ST-GCN specialized for agricultural tasks can effectively recognize harvesting action transitions, contributing to reducing the burden on workers during crop transportation.

Hand gesture recognition methods play an important role in human-computer interaction. Among these methods are skeleton-based recognition techniques that seem to be promising. In literature, several methods have been proposed to recognize hand gestures with skeletons. One problem with these methods is that they consider little the connectivity between the joints of a skeleton, constructing simple graphs for skeleton connectivity. Observing this, we built a new model of hand skeletons by adding three types of edges in the graph to finely describe the linkage action of joints. Then, an end-to-end deep neural network, hand gesture graph convolutional network, is presented in which the convolution is conducted only on linked skeleton joints. Since the training dataset is relatively small, this work proposes expanding the coordinate dimensionality so as to let models learn more semantic features. Furthermore, relative coordinates are employed to help hand gesture graph convolutional network learn the feature representation independent of the random starting positions of actions. The proposed method is validated on two challenging datasets, and the experimental results show that it outperforms the state-of-the-art methods. Furthermore, it is relatively lightweight in practice for hand skeleton-based gesture recognition.

Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

In this paper, we address the problem of cross-view action recognition from a monocular RGB camera. This topic has been considered extremely challenging due to the lack of 3D information in 2D images. Exploiting the advances in 3D pose estimation from a single RGB camera, we propose a new framework termed DeepVI, for cross-view action recognition without the need for pose alignment. Virtual viewpoints are used to augment the variability of training data along with the use of an end-to-end Deep Neural Network (DNN). The proposed network is composed of two modules. The first one, called SmoothNet, implicitly smooths skeleton joint trajectories using revisited temporal convolution in order to reduce the noise in the estimated 3D skeletons. The second module consists of a state-of-the-art approach designed for action recognition based on Spatial Temporal Graph Convolutional Networks (ST-GCN [40]). Experiments have been conducted in cross-view settings on two datasets, namely, NTU RGB-D and Northwestern-UCLA. The obtained results show the effectiveness of the proposed framework.

Most of the existing gait emotion recognition methods use spatial convolution or temporal convolution separately. Without integrating these two dimensions, it is easy to loss valid information for emotion analysis. In order to solve this problem, we propose a supervised learning method using Spatial Temporal Graph Convolutional Networks (ST-GCN) to classify emotions into positive, neutral and negative patterns. In addition, we made a new gait dataset called EmoGait3d including 742 real gait skeleton sequences extracted from videos, also annotated with emotion labels. According to the experiment results, our method achieves the accuracy of 92.2% which outperforms current state-of-art algorithms for gait emotion recognition.

Because of societal changes, human activity recognition, part of home care systems, has become increasingly important. Camera-based recognition is mainstream but has privacy concerns and is less accurate under dim lighting. In contrast, radar sensors do not record sensitive information, avoid the invasion of privacy, and work in poor lighting. However, the collected data are often sparse. To address this issue, we propose a novel Multimodal Two-stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment (MTGEA), which improves recognition accuracy through accurate skeletal features from Kinect models. We first collected two datasets using the mmWave radar and Kinect v4 sensors. Then, we used zero-padding, Gaussian Noise (GN), and Agglomerative Hierarchical Clustering (AHC) to increase the number of collected point clouds to 25 per frame to match the skeleton data. Second, we used Spatial Temporal Graph Convolutional Network (ST-GCN) architecture to acquire multimodal representations in the spatio-temporal domain focusing on skeletal features. Finally, we implemented an attention mechanism aligning the two multimodal features to capture the correlation between point clouds and skeleton data. The resulting model was evaluated empirically on human activity data and shown to improve human activity recognition with radar data only. All datasets and codes are available in our GitHub.

Motion correction model based on multitask spatial temporal graph convolutional network

Using complex networks and multiple artificial intelligence algorithms for table tennis match action recognition and technical-tactical analysis

This study develops an intelligent sports evaluation system based on deep learning, designed to accurately recognize and count physical exercise movements. It also investigates the impact of physical activity on college students’ mental health, with a focus on the mediating role of social support. The system consists of three core modules: posture recognition, movement counting, and psychological prediction. The posture recognition module employs the Spatial Temporal Graph Convolutional Network (ST-GCN) and its improved variants to analyze skeletal motion. The movement counting module evaluates the accuracy of counting various sports actions using a self-constructed dataset. The psychological prediction module examines how social support mediates the relationship between physical exercise and mental health through an experimental design. Results indicate that the ST-GCN model achieves 93.7% accuracy and a 93.5% F1-score in posture recognition, demonstrating an effective balance between performance and computational efficiency. In movement counting, both missed detection and false detection rates remain low, with recognition rates reaching 95.7% for standard actions and 87.3% for non-standard actions. The psychological prediction module shows that social support significantly mediates the effect of physical exercise on mental health, accounting for 47.4% of the total effect. Overall, the intelligent sports evaluation system performs strongly in both posture recognition and movement counting, providing precise support for physical education and training. The study also confirms the crucial mediating role of social support in the relationship between physical activity and mental health among college students. These findings offer a theoretical foundation for improving physical education and student health management in higher education and hold practical value for application in university physical education programs, athletic training, and mental health interventions.

The collaborative energy scheduling of source‐load‐energy storage has great potential to meet the active control requirements of power‐distribution networks. In this study, a federated deep reinforcement learning framework was developed to facilitate collaborative energy scheduling and maximize the total economic benefit in a distribution network. Then, considering the application of Markov decision processes for energy scheduling, a spatial temporal graph convolutional network transformer based power generation packaging model for renewable energy sources was presented, and a collaborative energy scheduling strategy based on a federated duelling deep Q‐network was designed. The simulation results indicate that the developed collaborative scheduling strategy can maximize the economic benefits of a power distribution network while ensuring data privacy.

We tackle the problem of tracking the human lower body as an initial step toward an automatic motion assessment system for clinical mobility evaluation, using a multimodal system that combines Inertial Measurement Unit (IMU) data, RGB images, and point cloud depth measurements. This system applies the factor graph representation to an optimization problem that provides 3-D skeleton joint estimations. In this paper, we focus on improving the temporal consistency of the estimated human trajectories to greatly extend the range of operability of the depth sensor. More specifically, we introduce a new factor graph factor based on Koopman theory that embeds the nonlinear dynamics of several lower-limb movement activities. This factor performs a two-step process: first, a custom activity recognition module based on spatial temporal graph convolutional networks recognizes the walking activity; then, a Koopman pose prediction of the subsequent skeleton is used as an a priori estimation to drive the optimization problem toward more consistent results. We tested the performance of this module a dataset composed of multiple clinical lower-limb mobility tests, and we show that our approach reduces outliers on the skeleton form by almost 1 m, while preserving natural walking trajectories at depths up to more than 10 m.

Skeleton-based human action recognition has great potential for human behavior analysis owing to its simplicity and robustness in varying environments. This paper presents a spatial and temporal attention-enhanced graph convolution network (STAEGCN) for human action recognition. The spatial-temporal attention module in the network uses convolution embedding for positional information and adopts multi-head self-attention mechanism to extract spatial and temporal attention separately from the input series of the skeleton. The spatial and temporal attention are then concatenated into an entire attention map according to a specific ratio. The proposed spatial and temporal attention module was integrated with an adaptive graph convolution network to form the backbone of STAEGCN. Based on STAEGCN, a two-stream skeleton-based human action recognition model was trained and evaluated. The model performed better on both NTU RGB+D and Kinetics 400 than 2s-AGCN and its variants. It was proven that the strategy of decoupling spatial and temporal attention and combining them in a flexible way helps improve the performance of graph convolution networks in skeleton-based human action recognition.

Traffic speed prediction is difficult because of the complicated dynamic spatiotemporal correlations. Recent studies in spatiotemporal models have achieved impressive outcomes for traffic speed prediction. But many studies use graphs in graph convolutional networks to learn spatial features that are often static. Additionally, effectively modeling long-range temporal features is crucial for prediction accuracy. In order to overcome these challenges, a Spatial–Temporal Adaptive Graph Convolutional Network with Transformer (ST_AGCNT) is designed in this paper. Specifically, an adaptive graph convolution network (AGCN) is designed to extract spatial dependency. An adaptive graph that fuses predefined matrices and learnable matrix is proposed to learn the correlations between nodes. The predefined matrices provide the model with richer prior information, while the learnable matrix can extract the dynamic nature of the nodes. And a temporal transformer (TT) is proposed to extract the long-range temporal dependency. In addition, to learn more information to achieve better results, different historical segments are modeled. Experiments conducted on a real-world traffic dataset confirm the effectiveness of the proposed model when compared to other baseline models. This model demonstrated excellent performance in prediction tasks across different time steps, effectively accomplishing traffic speed forecasting. It provides data support for improving traffic efficiency and reducing resource waste, contributing to the sustainable development of traffic management.

Accurately forecasting transportation demand is crucial for efficient urban traffic guidance, control and management. One solution to enhance the level of prediction accuracy is to leverage graph convolutional networks (GCN), a neural network based modelling approach with the ability to process data contained in graph based structures. As a powerful extension of GCN, a spatial-temporal graph convolutional network (ST-GCN) aims to capture the relationship of data contained in the graphical nodes across both spatial and temporal dimensions, which presents a novel deep learning paradigm for the analysis of complex time-series data that also involves spatial information as present in transportation use cases. In this paper, we present an Attention-based ST-GCN (AST-GCN) for predicting the number of available bikes in bike-sharing systems in cities, where the attention-based mechanism is introduced to further improve the performance of an ST-GCN. Furthermore, we also discuss the impacts of different modelling methods of adjacency matrices on the proposed architecture. Our experimental results are presented using two real-world datasets, Dublinbikes and NYC-Citi Bike, to illustrate the efficacy of our proposed model which outperforms the majority of existing approaches.