Algebra Based Human Skeleton Sequence Reduction and Action Recognition

Human action abnormality detection has been attempted by various sensors for application domains like rehabilitation, healthcare, and assisted living. Since the release of motion sensors that ease the human body skeleton retrieval, skeleton-based human action recognition has recently been an active topic in the area of artificial intelligence. Unlike human action recognition, human action abnormality detection is an emerging field that aims to detect the incorrect action from the same action class. Graph convolutional network has been widely adopted for human action recognition. However, to the best of our knowledge, whether it could be effective for the task of human action abnormality detection has not been attempted. To advance prior work in the emerging field of human action abnormality detection, we propose a novel method that uses graph convolutional network to detect abnormal actions in skeleton data. To validate the effectiveness of our proposed method, we conduct extensive experiments on a public dataset called UI-PRMD. Based on the experimental results, our proposed method achieved superior action abnormality detection performance comparing with existing deep learning methods.

Human action recognition plays a vital role in video understanding. In recent years, skeleton-based action recognition approaches have gained widespread attention due to their robustness against environmental interferences. This paper compiles 102 skeleton-based human action recognition methods and comparatively analyzes their performance on nine public datasets. This paper introduces the manual feature and deep learning based methods according to learning paradigms. Specifically, the manual feature methods are divided into three categories, i.e., geometric, kinetic, and statistical representations, in the light of feature descriptor. Meanwhile, the deep learning based methods are classified into five subclasses by backbones, i.e., recurrent neural networks, convolutional neural networks, graph convolutional networks, Transformer, and hybrid networks. Through comprehensive analysis, we not only present the research status of skeleton-based action recognition but also conclude that the existing methods have limitations such as poor generalization ability and high computational cost. Finally, this paper looks into future research directions from the aspects of network structure design, similar action distinction, domain data set expansion, and multi-person interaction.

Skeleton-based human action recognition has great potential for human behavior analysis owing to its simplicity and robustness in varying environments. This paper presents a spatial and temporal attention-enhanced graph convolution network (STAEGCN) for human action recognition. The spatial-temporal attention module in the network uses convolution embedding for positional information and adopts multi-head self-attention mechanism to extract spatial and temporal attention separately from the input series of the skeleton. The spatial and temporal attention are then concatenated into an entire attention map according to a specific ratio. The proposed spatial and temporal attention module was integrated with an adaptive graph convolution network to form the backbone of STAEGCN. Based on STAEGCN, a two-stream skeleton-based human action recognition model was trained and evaluated. The model performed better on both NTU RGB+D and Kinetics 400 than 2s-AGCN and its variants. It was proven that the strategy of decoupling spatial and temporal attention and combining them in a flexible way helps improve the performance of graph convolution networks in skeleton-based human action recognition.

This paper presents a novel framework for skeleton-based Human Action Recognition (HAR) based on Graph Convolution Networks (GCNs). The proposed framework aims to increase human action recognition performance of GCN-based methods by incorporating a missing-joint-handling pre-processing step and a novel adjacency matrix construction method in a single human action recognition pipeline. The missing-joint-handling pre-processing step is utilized to infer missing data in the input sequence, which may occur due to imperfect skeleton extraction, based on imputation methods. The novel adjacency matrix construction method is executed offline to compute an improved weighted adjacency matrix specifically designed for HAR, which is utilized in every layer of the employed GCN. Moreover, both the pre-processing step and the adjacency construction method can be utilized along with any GCN architecture, allowing any GCN-based HAR method to be employed in the proposed framework. Experimental evaluation on two public datasets indicate favorable human action classification scores compared to the employed baseline and all competing methods both for 2D and 3D skeleton-based human action recognition, while using a GCN architecture with less learnable parameters.

In recent years, many researchers have focused on using convolutional neural networks to perform human activity recognition as evidenced by the emergence of a number of convolutional neural network architectures such as LeNet-5, AlexNet and VGG16 and modern architectures such as ResNet, Inception V3, Inception-ResNet, MobileNet V2, NASNet and PNASNet. The main characteristic of a convolutional neural network (CNN) is its ability to extract features automatically from input images, which facilitates the processes of activity recognition and classification. Convolutional networks indeed derive more relevant and complex features with every additional layer. In addition, CNNs have achieved perfect classification on highly similar activities that were previously extremely difficult to classify. In this paper, we evaluate modern convolutional neural networks in terms of their human activity recognition accuracy, and we compare the results with the state of the art methods. In our research, we used two public data sets, HMDB (Shooting gun, kicking, falling to the floor, punching) and the Weizman dataset (walking, running, jumping, bending, one hand waving, two-hand waving, jumping in place, jumping jack, skipping). Our experimental results indicated that the CNN with NASNet architecture achieves the best performance of the six CNN architectures on both human activity data sets (HMDB and Weizman).

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Human Action Recognition (HAR) is the process of understanding human actions and behavior. HAR has a broad range of applications, and it has been focused on increasing the attention in various domain of computed vision. Abnormal detection from video stream is vigorous to guarantee the security in both outside spaces with the internal. Furthermore, the abnormal actions are really infrequent and rare, which makes the supervision process more challenging and difficult. In this research, skeleton graph-based Convolutional Neural Network (CNN) is devised for human abnormal activity detection. Here, the skeleton graph-based CNN (Skeleton graph_CNN) is devised based on the concept of classical convolution and skeleton graph generation. The human action recognition classifies the human actions into normal and abnormal class. The abnormal actions from the recognized outcome are detected with Skeleton graph_CNN, which provides the various actions of human as an output. The Skeleton graph_CNNgenerates the skeleton shaped human structure by connecting the joints within the frame to consecutive frames. Moreover, the HAR is carried out using IITB-Corridor Dataset based on metrics, such as testing accuracy of 0.961, sensitivity of 0.956 and specificity of 0.960, correspondingly.

Skeleton-based human action recognition has been a hot topic in recent years. Most existing studies are based on the skeleton data obtained from Kinect, which is noisy and unstable, in particular, in the case of occlusions. To cope with the noisy skeleton data and variation of viewpoints, this paper presents a view-invariant method for human action recognition by recovering the corrupted skeletons based on a 3D bio-constrained skeleton model and visualizing those body-level motion features obtained during the recovery process with images. The bio-constrained skeleton model is defined with two types of constraints: 1) constant bone lengths and 2) motion limits of joints. Based on the bio-constrained model, an effective method is proposed for skeleton recovery. Two types of new motion features, the Euclidean distance matrix between joints (JEDM), which contains the global structure information of the body, and the local dynamic variation of the joint Euler angles (JEAs) are used in describing human action. These two types of features are encoded into different motion images, which are fed into a two-stream convolutional neural network for learning different action patterns. The experiments on three benchmark datasets achieve better accuracy than the state-of-the-art approaches, which demonstrates the effectiveness of the proposed method.

Skeleton data from depth sensors, enriched with substantial information and compact in storage, provides a streamlined sequential representation. It is resilient to environmental variables such as background noise, attire, and illumination. Consequently, skeleton-based human action recognition emerges as an important topic in computer vision. Its key task is to extract distinctive features, specifically those critical for differentiating human actions. This work proposes an Ambiguous-feature-refinement and Orthogonal-basis-coordinate-transformation-based residual graph convolution neural network (AOnet) model for skeleton-based human action recognition. Firstly, multiple input branches are proposed and employed to enrich skeletal information features at the early data-input stage while maintaining a compact and effective representation of data. Secondly, since many actions exhibit highly similar joint-features, an ambiguous-feature-refinement model is introduced to further distinguish ambiguous features without considering external information interaction. This step benefits the extraction of distinctive spatial-temporal features, thereby improving the differentiation among action categories. Lastly, an orthogonal-basis-coordinate-transformation method is proposed to convert refined nonlinear spatial-temporal features into linear ones, and augment the model’s ability to capture higher-order dynamic information. The proposed method has been evaluated on two large-scale datasets. Experimental results confirm that AOnet demonstrates superior performance over the state of the art.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

Video and image analysis technologies have made human action recognition as an interesting field and used in many practical application systems such as visual surveillance systems, healthcare monitoring systems, 3D games and smart home. In this paper, we address the challenges of automatic tracking, detection and recognition of three-dimensional human pose-based actions from sequences of depth maps. Specifically, we have designed our proposed idea into a probabilistic framework: 1) dividing each action into meaningful ordered temporal segments, 2) using pixel-neighboring intensity difference approach to identify human shape region from the confused scenes, 3) introducing a robust spatiotemporal features to extract 3D joints information via local distance features and depth human shape information to get motion variation features and 4) finally these sets of features are merged together and their feature vectors are further reduced and discriminated to get optimal vectors for real-world applications. Next, the augmented features are enhanced and symbolized by Linde-Buzo-Gray clustering algorithm for better action recognition. With these symbols, each action hidden Markov model (HMM) is trained and tested for action recognition. The experimental results demonstrated our proposed approach on three challenging depth video datasets: IM-DepthActions, MSRAction3D and MSRDailyActivity3D, producing significantly improved results over the state-of-the-art methods: especially for those actions that are not easily discernible by the existing methods.

This study aims to design an epidemic prevention and control bracelet system that integrates convolutional neural network (CNN). This system can collect and process the user’s physiological index data in real time, especially in the remote physical education scene, and provide learners with immediate physiological index feedback and personalized adaptive training suggestions through accurate human action recognition (HAR) technology. One-dimensional acceleration signal is converted into two-dimensional image, and CNN’s powerful feature extraction and classification ability is used to effectively solve the problem that manual feature extraction is complex and nonlinear features are difficult to capture. By considering the joint action trajectory in the time window, a dynamic Recurrence Plot (RP) is constructed to capture the dynamic changes among joints. To input recursive graph data into CNN, it needs to be converted into image form. In the task of HAR, CNN can automatically learn useful features from images without manually designing features. It can not only effectively extract features from images, but also be directly used in classification tasks. Experimental results show that compared with other algorithms, the proposed RP + CNN model has the best performance in action recognition, with an accuracy of 96.89% and a F1 value of 86.76%. RP captures the dynamic patterns and periodic behaviors in time series by visualizing the repeated appearance of system states over time. The RP + CNN model is used to extract and classify human action features, which significantly improves the accuracy and efficiency of HAR. This innovative method not only simplifies the complex process of traditional manual feature extraction, but also enhances the system’s ability to identify nonlinear and complex action patterns, which provides strong technical support for remote physical education.

In order to support smart construction, digital twin has been a well‐recognized concept for virtually representing the physical facility. It is equally important to recognize human actions and the movement of construction equipment in virtual construction scenes. Compared to the extensive research on human action recognition (HAR) that can be applied to identify construction workers, research in the field of construction equipment action recognition (CEAR) is very limited, mainly due to the lack of available datasets with videos showing the actions of construction equipment. The contributions of this research are as follows: (1) the development of a comprehensive video dataset of 2,064 clips with five action types for excavators and dump trucks; (2) a new deep learning‐based CEAR approach (known as a simplified temporal convolutional network or STCN) that combines a convolutional neural network (CNN) with long short‐term memory (LSTM, an artificial recurrent neural network), where CNN is used to extract image features and LSTM is used to extract temporal features from video frame sequences; and (3) the comparison between this proposed new approach and a similar CEAR method and two of the best‐performing HAR approaches, namely, three‐dimensional (3D) convolutional networks (ConvNets) and two‐stream ConvNets, to evaluate the performance of STCN and investigate the possibility of directly transferring HAR approaches to the field of CEAR.