Skeleton-based Action Recognition with Attention and Temporal Graph Convolutional Network

This monograph presents studies concerning the challenging topic of Human Action Recognition, which has recently become of increasing importance to those working in the field of Computer Vision. It contains an overview of the research literature on tennis movements recognition based on video, images, motion capture data, and data registered using various sensors. In this monograph, three novel methods of tennis movements recognition are presented in detail with success, achieving high performance of the proposed classifiers. They all utilize Graph Convolutional Neural networks (GCN) solutions. Two methods are also applied using attention modules which improve feature extraction and thus make the classification better. The Spatial- -Temporal Graph Convolutional Network, the Attention-Temporal Graph Convolutional Network and the Dual Attention Temporal Graph Convolutional Network are applied for recognition of the most frequently used tennis moves, such as: forehand, backhand, volley forehand, and volley backhand. Moreover, the forehand and backhand are further divided into two phases: 1) the preparation, starting from the player’s positioning until the moment just before the ball makes contact with the racket, 2) the hit together with the racket swinging until the move is finished. These separations allow detailed analysis. For the purpose of this study, a 3DTennisDS dataset was created that contains motion capture data of the forehand, backhand, volley forehand, and volley backhand. They were registered via the Vicon motion capture system. Three proposed neural networks methods were verified based on the gathered dataset. In this monograph, the interpretability aspect of GCN was also presented. This issue is important and was examined by presenting feature maps and heatmaps obtained after selected convolutional operations. These results showed how individual filters in GCNs extract features that are relevant for recognizing tennis movements.

Forced oscillation location based on temporal graph convolutional network

Temporal graph convolutional network for multi-agent reinforcement learning of action detection

Traffic speed prediction is difficult because of the complicated dynamic spatiotemporal correlations. Recent studies in spatiotemporal models have achieved impressive outcomes for traffic speed prediction. But many studies use graphs in graph convolutional networks to learn spatial features that are often static. Additionally, effectively modeling long-range temporal features is crucial for prediction accuracy. In order to overcome these challenges, a Spatial–Temporal Adaptive Graph Convolutional Network with Transformer (ST_AGCNT) is designed in this paper. Specifically, an adaptive graph convolution network (AGCN) is designed to extract spatial dependency. An adaptive graph that fuses predefined matrices and learnable matrix is proposed to learn the correlations between nodes. The predefined matrices provide the model with richer prior information, while the learnable matrix can extract the dynamic nature of the nodes. And a temporal transformer (TT) is proposed to extract the long-range temporal dependency. In addition, to learn more information to achieve better results, different historical segments are modeled. Experiments conducted on a real-world traffic dataset confirm the effectiveness of the proposed model when compared to other baseline models. This model demonstrated excellent performance in prediction tasks across different time steps, effectively accomplishing traffic speed forecasting. It provides data support for improving traffic efficiency and reducing resource waste, contributing to the sustainable development of traffic management.

The main core purpose of artificial emotional intelligence is to recognize human emotions. Technologies such as facial, semantic, or brainwave recognition applications have been widely proposed. However, the abovementioned recognition techniques for emotional features require a large number of training samples to obtain high accuracy. Human behaviour pattern can be trained and recognized by the continuous movement of the Spatial Temporal Graph Convolution Network (ST-GCN). However, this technology does not distinguish between the speed of delicate emotions, and the speed of human behaviour and delicate changes of emotions cannot be effectively distinguished. This research paper proposes Spatial Temporal Variation Convolutional Network training for human emotion recognition, using skeleton detection technology to calculate the degree of skeleton point change between consecutive actions and using the nearest neighbour algorithm to classify speed levels and train the ST-GCN recognition model to obtain the emotional state. Application of the speed change recognition ability of the Spatial Temporal Variation Graph Convolution Network (STV-GCN) to artificial emotional intelligence calculation makes it possible to efficiently recognize the delicate actions of happy, sad, fear, and angry in human behaviour. The STV-GCN technology proposed in this paper is compared with ST-GCN and can effectively improve the recognition accuracy by more than 50%.

Temporal graph convolutional network soft sensor for molecular weight distribution prediction

Due to the high uncertainties in temperature changes, traditional regression analysis and time series prediction methods fail to provide accurate temperature forecasts to reduce the impact of extreme weather on human society. Considering the spatiotemporal features of temperature changes, this paper proposes a variable weight combination model based on a temporal graph convolutional network (T-GCN), Luong attention network (LUA) and gated recurrent unit (GRU) network, which fully utilizes spatiotemporal information to predict future temperature changes more accurately. The model uses the T-GCN model to capture spatiotemporal features while introducing Luong attention to weight the inputs at different time steps to improve the prediction accuracy and further reduce the prediction error by fusing the outputs of the T-GCN-Luong attention and GRU models through the variable weight combination method. The results revealed that (1) the inclusion of spatial information significantly improved the effectiveness of the temperature predictions. (2) The Luong attention mechanism weights different time steps and improves the prediction accuracy of the T-GCN model. (3) The TGLAG combination model constructed via the variable weight method exhibited good predictive performance at 15 sites. Compared with that of the simple GRU model, the accuracy of the proposed model is improved by approximately 31.949% in terms of the root mean square error (RMSE) and 26.913% in terms of the mean absolute error (MAE). Compared with the second-best model, T-GCN-Luong attention, the TGLAG model yields a 5.946% lower RMSE and 9.535% lower MAE, which indicates that TGLAG has good application prospects in the field of temperature prediction.

Accurate load forecasting is essential for ensuring safe, stable, and economical operation of energy internet. Temporal convolutional networks (TCNs) have demonstrated superior performance, when compared to recurrent neural network models, since their introduction in electrical load forecasting. However, the current TCN‐based models are unable to obtain a large receptive field and strong long‐time feature extraction capability owing to the specific kernel size of the 1D convolution structure. This paper proposes a temporal inception convolutional network based on multi‐head attention (TICN‐Att) for ultra‐short‐term load prediction. By introducing an inception structure into the TCN, the proposed model can extract multi‐dimensional information from the input features, through the multiple hidden convolutional kernels of different scales, without stacking layers depth‐wise. Simultaneously, by introducing a multi‐head attention mechanism, the TICN‐Att model has a long time‐dependent extraction capability, similar to that of the long short‐term memory network models. The generalization and validity of the model are tested using the global energy forecasting competition (GEFCOM2014) dataset, electrical load data of a city in Jiangsu (China), and PJM power system dataset. The experimental results demonstrated that the proposed model has the best prediction effect, compared to the other state‐of‐the‐art models.

Multi-view fusion neural network for traffic demand prediction

Cloud‐assisted physical education teaching is an important direction for the development of smart education. However, the data processing and transmission of videos severely restrict the real‐time performance of action analysis. To this end, this paper proposes an efficient edge cloud‐assisted student movement recognition framework based on the graph convolutional network and human skeleton data. On the edge server, we use the YOLO‐pose algorithm to generate robust human skeleton sequences and design an improved spatial–temporal dual stream graph convolutional neural network with an early fusion structure, which introduces the node weight module and the dynamic graph module to exploit long‐distance dependency relationships of nodes. In the cloud server, we use a federated learning framework based on the density clustering mechanism to collaboratively train and aggregate parameters of models scattered across edge nodes. The experimental results show that our proposed model achieves excellent recognition accuracy on the self‐built sports action dataset, providing an effective solution for intelligent and real‐time feedback in outdoor sports teaching.

ABSTRACTSkeleton‐based action recognition using Graph Convolutional Networks (GCNs) has achieved remarkable performance, but recognising ambiguous actions, such as ‘waving’ and ‘saluting’, remains a significant challenge. Existing methods typically rely on a serial combination of GCNs and Temporal Convolutional Networks (TCNs), where spatial and temporal features are extracted independently, leading to an unbalanced spatial‐temporal information, which hinders accurate action recognition. Moreover, existing methods for ambiguous actions often overemphasise local details, resulting in the loss of crucial global context, which further complicates the task of differentiating ambiguous actions. To address these challenges, the authors propose a lightweight plug‐and‐play module called Synchronised and Fine‐grained Head (SF‐Head), inserted between GCN and TCN layers. SF‐Head first conducts Synchronised Spatial‐Temporal Extraction (SSTE) with a Feature Redundancy Loss (F‐RL), ensuring a balanced interaction between the two types of features. It then performs Adaptive Cross‐dimensional Feature Aggregation (AC‐FA), with a Feature Consistency Loss (F‐CL), which aligns the aggregated feature with their original spatial‐temporal feature. This aggregation step effectively combines both global context and local details, enhancing the model's ability to classify ambiguous actions. Experimental results on NTU RGB + D 60, NTU RGB + D 120, NW‐UCLA and PKU‐MMD I datasets demonstrate significant improvements in distinguishing ambiguous actions. Our code will be made available at https://github.com/HaoHuang2003/SFHead.

With the continuous development of smart cities, intelligent transportation systems (ITSs) have ushered in many breakthroughs and upgrades. As a solid foundation for an ITS, traffic flow prediction effectively helps the city to better manage intricate traffic flow. However, existing traffic flow prediction methods such as temporal graph convolutional networks(T-GCNs) ignore the dissimilarities between lanes. Thus, they cannot provide more specific information regarding predictions such as dynamic changes in traffic flow direction and deeper lane relationships. With the upgrading of intersection sensors, more and more intersection lanes are equipped with intersection sensors to detect vehicle information all day long. These spatio-temporal data help researchers refine the focus of traffic prediction research down to the lane level. More accurate and detailed data mean that it is more difficult to mine the spatio-temporal correlations between data, and modeling heterogeneous data becomes more challenging. In order to deal with these problems, we propose a heterogeneous graph convolution model based on dynamic graph generation. The model consists of three components. The internal graph convolution network captures the real-time spatial dependency between lanes in terms of generated dynamic graphs. The external heterogeneous data fusion network comprehensively considers other parameters such as lane speed, lane occupancy, and weather conditions. The codec neural network utilizes a temporal attention mechanism to capture the deep temporal dependency. We test the performance of this model based on two real-world datasets, and extensive comparative experiments indicate that the proposed heterogeneous graph convolution model can improve the prediction accuracy.

Action recognition, a field within human-centered computing, is vital for identifying and understanding human actions, benefiting applications like surveillance, autonomous vehicles, and human-computer interaction. Artificial intelligence’s enduring goal is to develop robust models for perceiving and understanding the visual world around us. Deep neural networks have shown exceptional performance in various tasks, profoundly impacting real-world applications, including action prediction and recognition, which have advanced significantly in recent years. Most work in visual human action recognition focuses on single viewpoints or modalities from complete observation. Yet, the real significance lies in predicting future actions from incomplete observations to prevent real-world tragedies. With the availability of multiple cameras and data from multiple modalities (RGB, Depth, and Skeleton) available today, it becomes possible to model human action in multi-view and multi-modality context, minimizing the data loss due to occlusions and signal quality issues to improve recognition accuracy on the strength of state-of-the-art deep learning models. However, deep neural network models are susceptible to adversarial attacks, where imperceptible perturbations can compromise action recognition model performance. This thesis focuses on identifying latent vulnerabilities and proposing a defense mechanism against such threats in a multi-modality and multi-view setting. This work introduces an efficient and effective attack mechanism that perturbs skeleton data by targeting key joints and segments while employing a graph attention mechanism that learns the semantics to perturb other modalities. Additionally, an approach has been developed that not only adds noise but also alters the visual spatial structure of skeleton data through generative modeling. Furthermore, this dissertation introduces a defense mechanism known as the Collaborative Knowledge Distillation Network, which leverages graph attention and knowledge distillation techniques. This network leverages the knowledge from compromised multi-view data and integrates information from clean data to address incomplete observations and noisy action videos, enhancing the robustness of action recognition models for real-world applications.

Graph convolutional networks (GCNs) have been proven to be effective for processing structured data, so that it can effectively capture the features of related nodes and improve the performance of model. More attention is paid to employing GCN in Skeleton-Based action recognition. But there are some challenges with the existing methods based on GCNs. First, the consistency of temporal and spatial features is ignored due to extracting features node by node and frame by frame. We design a generic representation of skeleton sequences for action recognition and propose a novel model called Temporal Graph Networks (TGN), which can obtain spatiotemporal features simultaneously. Secondly, the adjacency matrix of graph describing the relation of joints are mostly depended on the physical connection between joints. We propose a multi-scale graph strategy to appropriately describe the relations between joints in skeleton graph, which adopts a full-scale graph, part-scale graph and core-scale graph to capture the local features of each joint and the contour features of important joints. Extensive experiments are conducted on two large datasets including NTU RGB+D and Kinetics Skeleton. And the experiments results show that TGN with our graph strategy outperforms other state-of-the-art methods.

Research on smartphone-based human activity recognition (HAR) is prevalent in the field of healthcare, especially for elderly activity monitoring. Researchers usually propose to use of accelerometers, gyroscopes or magnetometers that are equipped in smartphones as an individual sensing modality for human activity recognition. However, any of these alone is limited in capturing comprehensive movement information for accurate human activity analysis. Thus, we propose a smartphone-based HAR approach by leveraging the inertial signals captured by these three sensors to classify human activities. These heterogeneous sensors deliver information on various aspects of nature, motion and orientation, offering a richer set of features for more accurate representations of the activities. Hence, a deep learning approach that amalgamates long short-term memory (LSTM) in temporal convolutional network (TCN) is proposed. We use independent temporal convolutional networks, coined as temporal convolutional streams, to independently analyse the temporal data of each sensing modality. We name this architecture multi-stream TC-LSTM. The performance of multi-stream TC-LSTM is assessed on the self-collected elderly activity database. Empirical results exhibit that multi-stream TC-LSTM outperforms the existing machine learning and deep learning models, with an F1 score of 98.3 %