Two-stream Graph Attention Convolutional for Video Action Recognition

The recognition of sports action is an important research subject, which is conducive to the improvement of athletes’ own level. To improve the accuracy of multi-modal data action recognition, based on the Transformer module, this study introduces a multi-head attention mechanism, fuses multi-modal data, and constructs a multi-stream structured object relationship inference network. Based on PointNet++ network and combining five different data fusion frameworks, a motion recognition model that integrates RGB data and 3D skeleton point cloud is constructed. The results showed that the Top-1 accuracy of multi-stream structured object relationship inference network was 42.5% and 42.7%, respectively, which was better than other algorithms. The accuracy of the multi-modal fusion model was improved by 15.6% and 5.1% compared with the single mode, and by 5.4% and 2.6% compared with the dual mode, which showed its superiority in the action recognition task. This showed that the fusion of multi-modal data can provide more abundant information, so as to improve the accuracy of action recognition. The accuracy of the action recognition model combining RGB data and 3D skeleton point cloud was 84.3%, 87.5%, 90.2%, 90.6% and 91.2% after the combination of different strategies, which effectively compensated for the problem of missing information in 3D skeleton point cloud and significantly improved the accuracy of action recognition. With a small amount of data, the Top-1 accuracy of the multi-stream structured object relationship inference network in this study was superior to other algorithms, showing its advantages in dealing with complex action recognition tasks. In addition, the action recognition model that fuses RGB data and 3D skeleton point cloud also achieved higher accuracy, which is better than other algorithms. This study can meet the needs of motion recognition in different scenarios and has certain reference value.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Graph convolutional networks (GCN), which can model the human body skeletons as spatial and temporal graphs, have shown remarkable potential in skeleton-based action recognition. However, in the existing GCN-based methods, graph-structured representation of the human skeleton makes it difficult to be fused with other modalities, especially in the early stages. This may limit their scalability and performance in action recognition tasks. In addition, the pose information, which naturally contains informative and discriminative clues for action recognition, is rarely explored together with skeleton data in existing methods. In this work, we proposed pose-guided GCN (PG-GCN), a multi-modal framework for high-performance human action recognition. In particular, a multi-stream network is constructed to simultaneously explore the robust features from both the pose and skeleton data, while a dynamic attention module is designed for early-stage feature fusion. The core idea of this module is to utilize a trainable graph to aggregate features from the skeleton stream with that of the pose stream, which leads to a network with more robust feature representation ability. Extensive experiments show that the proposed PG-GCN can achieve state-of-the-art performance on the NTU RGB+D 60 and NTU RGB+D 120 datasets.

Action recognition makes significant contributions to sports video analysis, especially for athletes’ training evaluations. For sports video analysis, the action information is mainly conveyed by human body parts’ temporal movement, and each of the parts has a unique importance to the action representation. Aiming to involve this point in action recognition, we propose a part-attention spatio-temporal graph convolutional network (PSGCN) to exploit the dynamic spatio-temporal information in a sports video; it learns the importance of different parts to emphasize the contribution on the task of action recognition. Specifically, PSGCN first divides the human body into six parts and extracts their convolutional neural network (CNN) features, as well as concatenating the global feature of the whole frame; it then utilizes a cross-part and cross-frame graph building module to formulate the graph correlation of the parts from different frames. Inspired by the larger temporal variation of the same part containing more action information, we further propose a part-attention (PA) learning module to estimate the importance of each part, which can strengthen the graph correlation to support a PA graph. Finally, PSGCN conducts a graph convolutional network on the learned PA spatio-temporal graph with the learned part CNN features, which can obtain the action representation for the given sports video. In addition, the whole network is optimized by two losses of PA and action classification. To perform the superiority of PSGCN, we carry out extensive experiments of our model compared with several state-of-the-art methods on widely used action recognition datasets, especially for sports action. The results reflect the advantages of the proposed PSGCN on sports video analysis.

Representations that can compactly and effectively capture the temporal evolution of semantic content are important to computer vision and machine learning algorithms that operate on multi-variate time-series data. We investigate such representations motivated by the task of human action recognition. Here each data instance is encoded by a multivariate feature (such as via a deep CNN) where action dynamics are characterized by their variations in time. As these features are often non-linear, we propose a novel pooling method, kernelized rank pooling, that represents a given sequence compactly as the pre-image of the parameters of a hyperplane in a reproducing kernel Hilbert space, projections of data onto which captures their temporal order. We develop this idea further and show that such a pooling scheme can be cast as an order-constrained kernelized PCA objective. We then propose to use the parameters of a kernelized low-rank feature subspace as the representation of the sequences. We cast our formulation as an optimization problem on generalized Grassmann manifolds and then solve it efficiently using Riemannian optimization techniques. We present experiments on several action recognition datasets using diverse feature modalities and demonstrate state-of-the-art results.

ABSTRACTSkeleton‐based action recognition using Graph Convolutional Networks (GCNs) has achieved remarkable performance, but recognising ambiguous actions, such as ‘waving’ and ‘saluting’, remains a significant challenge. Existing methods typically rely on a serial combination of GCNs and Temporal Convolutional Networks (TCNs), where spatial and temporal features are extracted independently, leading to an unbalanced spatial‐temporal information, which hinders accurate action recognition. Moreover, existing methods for ambiguous actions often overemphasise local details, resulting in the loss of crucial global context, which further complicates the task of differentiating ambiguous actions. To address these challenges, the authors propose a lightweight plug‐and‐play module called Synchronised and Fine‐grained Head (SF‐Head), inserted between GCN and TCN layers. SF‐Head first conducts Synchronised Spatial‐Temporal Extraction (SSTE) with a Feature Redundancy Loss (F‐RL), ensuring a balanced interaction between the two types of features. It then performs Adaptive Cross‐dimensional Feature Aggregation (AC‐FA), with a Feature Consistency Loss (F‐CL), which aligns the aggregated feature with their original spatial‐temporal feature. This aggregation step effectively combines both global context and local details, enhancing the model's ability to classify ambiguous actions. Experimental results on NTU RGB + D 60, NTU RGB + D 120, NW‐UCLA and PKU‐MMD I datasets demonstrate significant improvements in distinguishing ambiguous actions. Our code will be made available at https://github.com/HaoHuang2003/SFHead.

Action recognition methods based on human skeleton can explicitly represent human actions, and have gradually become one of the important research directions in the field of computer vision. To address the problems that the skeleton graph in graph convolutional networks is fixed to represent only the physical structure of the human body and the lack of adaptive ability to the skeleton topology graph structure, this paper proposes a dual-stream non-local graph convolutional network based on the attention mechanism. First, the temporal convolution layer is extended to a parallel structure with multiple kernels, and different temporal convolution kernel modules are adaptively selected to collect features by channel weights; second, an attention model consisting of an attention pooling layer is proposed to capture the correlation and temporal continuity among joints; finally, the nonlocal graph convolution network is used as the basic framework with joint information, skeletal information and respective motion information A dual-stream fusion model is constructed. The proposed method is compared with the mainstream methods in recent years on the action recognition dataset NTU RGB+D, and the experimental results show that the proposed method achieves high accuracy in action recognition.

This paper presents a method of human action recognition based on the key points of skeleton, aiming to guide the robot to follow a leader in complex environments. We propose a two-stage human pose estimation model which combines the Single Shot Detector (SSD) algorithm based on ResNet with Convolutional Pose Machines (CPMs) to obtain the key points positions of the human skeleton in 2D images. Based on the position information, we construct structure vectors. Feature models consisting of eight angle features and four modulus ratio features are then extracted as the representation of actions. Finally, multi-classification SVM is used to classify the feature models for action recognition. The experimental results demonstrate the validity of the two-stage human pose estimation model to accomplish the task of human action recognition. Our method achieves 97% recognition accuracy on the self-collected dataset composed of six command actions.

Facing complex and diverse action recognition scenarios, artificially designed neural networks show poor generalization performance. Therefore, an automatic designing method of 3D convolutional neural networks based on neural architecture search is proposed. Firstly, a variety of human behaviors are extracted to construct training sets and validation sets. Additionally, the weights of neuron connections are updated using the loss on the training set, the discrete network architecture search space is continuous through continuous relaxation, and the search space is reduced by using the hierarchical idea. What’s more, the objective function is optimized by combining gradient descent, realizing fast search, and stacking the obtained computing units to form an overall network at the same time. Evaluations based on public data sets show that the designed neural network model achieves comparable performance to the artificially designed network model in the task of human action recognition, solving the difficulty of deep learning method migration between working at different scenarios by automatically customizing the network.

Graph convolutional network – Long short term memory neural network- multi layer perceptron- Gaussian progress regression model: A new deep learning model for predicting ozone concertation

Using skeleton data to classify human behavior has been received a lot of attention and various technologies have been developed because it can be utilized in various domains. Among them, Graph Convolutional Networks (GCN) shows high accuracy in action recognition through the effective extraction of features from graph data. GCN has been further developed and two stream adaptive graph convolution networks (2s-AGCN) are made. This model utilizes bone data as well as spatial and temporal information of joints to compensate for the problems of conventional GCN limited to local region. Although 2s-AGCN show the highest accuracy among existing technologies, new approaches using CNN, LSTM and auto encoder continue to emerge and present several methods. In this paper, we verify if 2s-AGCN show higher accuracy than recent CNN-LSTM networks and compare structures and features of their models.

This study aims to design an epidemic prevention and control bracelet system that integrates convolutional neural network (CNN). This system can collect and process the user’s physiological index data in real time, especially in the remote physical education scene, and provide learners with immediate physiological index feedback and personalized adaptive training suggestions through accurate human action recognition (HAR) technology. One-dimensional acceleration signal is converted into two-dimensional image, and CNN’s powerful feature extraction and classification ability is used to effectively solve the problem that manual feature extraction is complex and nonlinear features are difficult to capture. By considering the joint action trajectory in the time window, a dynamic Recurrence Plot (RP) is constructed to capture the dynamic changes among joints. To input recursive graph data into CNN, it needs to be converted into image form. In the task of HAR, CNN can automatically learn useful features from images without manually designing features. It can not only effectively extract features from images, but also be directly used in classification tasks. Experimental results show that compared with other algorithms, the proposed RP + CNN model has the best performance in action recognition, with an accuracy of 96.89% and a F1 value of 86.76%. RP captures the dynamic patterns and periodic behaviors in time series by visualizing the repeated appearance of system states over time. The RP + CNN model is used to extract and classify human action features, which significantly improves the accuracy and efficiency of HAR. This innovative method not only simplifies the complex process of traditional manual feature extraction, but also enhances the system’s ability to identify nonlinear and complex action patterns, which provides strong technical support for remote physical education.

Much of the existing work on action recognition combines simple features (e.g., joint angle trajectories, optical flow, spatio-temporal video features) with somewhat complex classifiers or dynamical models (e.g., kernel SVMs, HMMs, LDSs, deep belief networks). Although successful, these approaches represent an action with a set of parameters that usually do not have any physical meaning. As a consequence, such approaches do not provide any qualitative insight that relates an action to the actual motion of the body or its parts. For example, it is not necessarily the case that clapping can be correlated to hand motion or that walking can be correlated to a specific combination of motions from the feet, arms and body. In this paper, we propose a new representation of human actions called Sequence of the Most Informative Joints (SMIJ), which is extremely easy to interpret. At each time instant, we automatically select a few skeletal joints that are deemed to be the most informative for performing the current action. The selection of joints is based on highly interpretable measures such as the mean or variance of joint angles, maximum angular velocity of joints, etc. We then represent an action as a sequence of these most informative joints. Our experiments on multiple databases show that the proposed representation is very discriminative for the task of human action recognition and performs better than several state-of-the-art algorithms.

Recently, deep learning, which are able to extract automatically features from data, has achieved state-of-the-art performance across a variety of sensor based human activity recognition (HAR) tasks. However, the existing deep neural networks are usually trained with a global loss, and all hidden layer weights have to be always kept in memory before the forward and backward pass has completed. The backward locking phenomenon prevents the reuse of memory, which is a crucial limitation for wearable activity recognition. In the paper, we proposed a layer-wise convolutional neural networks (CNN) with local loss for the use of HAR task. To our knowledge, this paper is the first that uses local loss based CNN for HAR in ubiquitous and wearable computing arena. We performed experiments on five public HAR datasets including UCI HAR dataset, OPPOTUNITY dataset, UniMib-SHAR dataset, PAMAP dataset, and WISDM dataset. The results show that local loss works better than global loss for tested baseline architectures. At no extra cost, the local loss can approach the state-of-the-arts on a variety of HAR datasets, even though the number of parameters was smaller. We believe that the layer-wise CNN with local loss can be used to update the existing deep HAR methods.

In the field of human action recognition, a good action classification algorithm has always been the key to measure the accuracy of action recognition. Since the trajectory of human targets is often complex, the recognition of RGB color images is restricted by many influencing factors, such as light intensity, complex background and angle. In this paper, kinect equipment is used to collect depth image data, and the image is generated into bone data. The Alexnet model in convolutional neural network is used to train and classify the data, which effectively improves the accuracy of action recognition.