Temporal-Variation Skeleton Point Correction Algorithm for Improved Accuracy of Human Action Recognition

Human action recognition is a fundamental research problem in computer vision. The accuracy of human action recognition has important applications. In this book chapter, the authors use a YOLOv7-based model for human action recognition. To evaluate the performance of the model, the action recognition results of YOLOv7 were compared with those using CNN+LSTM, YOLOv5, and YOLOv4. Furthermore, a small human action dataset suitable for YOLO model training is designed. This data set is composed of images extracted from KTH, Weizmann, MSR data sets. In this book chapter, the authors make use of this data set to verify the experimental results. The final experimental results show that using the YOLOv7 model for human action recognition is very convenient and effective, compared with the previous YOLO model.

The recognition of sports action is an important research subject, which is conducive to the improvement of athletes’ own level. To improve the accuracy of multi-modal data action recognition, based on the Transformer module, this study introduces a multi-head attention mechanism, fuses multi-modal data, and constructs a multi-stream structured object relationship inference network. Based on PointNet++ network and combining five different data fusion frameworks, a motion recognition model that integrates RGB data and 3D skeleton point cloud is constructed. The results showed that the Top-1 accuracy of multi-stream structured object relationship inference network was 42.5% and 42.7%, respectively, which was better than other algorithms. The accuracy of the multi-modal fusion model was improved by 15.6% and 5.1% compared with the single mode, and by 5.4% and 2.6% compared with the dual mode, which showed its superiority in the action recognition task. This showed that the fusion of multi-modal data can provide more abundant information, so as to improve the accuracy of action recognition. The accuracy of the action recognition model combining RGB data and 3D skeleton point cloud was 84.3%, 87.5%, 90.2%, 90.6% and 91.2% after the combination of different strategies, which effectively compensated for the problem of missing information in 3D skeleton point cloud and significantly improved the accuracy of action recognition. With a small amount of data, the Top-1 accuracy of the multi-stream structured object relationship inference network in this study was superior to other algorithms, showing its advantages in dealing with complex action recognition tasks. In addition, the action recognition model that fuses RGB data and 3D skeleton point cloud also achieved higher accuracy, which is better than other algorithms. This study can meet the needs of motion recognition in different scenarios and has certain reference value.

The recent development of wireless wearable sensor networks offers a spectrum of new applications in fields of healthcare, medicine, activity monitoring, sport, safety, human-machine interfacing, and beyond. Successful use of this technology depends on lifetime of the battery-powered sensor nodes. This paper presents a new method for extending the lifetime of the wearable sensor networks by avoiding unnecessary data transmissions. The introduced method is based on embedded classifiers that allow sensor nodes to decide if current sensor readings have to be transmitted to cluster head or not. In order to train the classifiers, a procedure was elaborated, which takes into account the impact of data selection on accuracy of a recognition system. This approach was implemented in a prototype of wearable sensor network for human activity monitoring. Real-world experiments were conducted to evaluate the new method in terms of network lifetime, energy consumption, and accuracy of human activity recognition. Results of the experimental evaluation have confirmed that, the proposed method enables significant prolongation of the network lifetime, while preserving high accuracy of the activity recognition. The experiments have also revealed advantages of the method in comparison with state-of-the-art algorithms for data transmission reduction.

Two-stream Convolutional Neural Networks have shown excellent performance in video action recognition. Most existing works train each sampling group independently, or just fuse at the last level, which obviously ignore the continuity of action in temporal and the complementary information between action fragments. In this paper, a temporal segment connection network is proposed to overcome these limitations. On the one hand, the forget gate module of the long short-term memory (LSTM) network is used to establish feature-level connections between each sampling group. This not only strengthens the information transmission between the sampling groups to enhance the temporal connectivity, but also extracts the complementary information between the sampling groups to enhance the overall representation of the action. On the other hand, a bi-directional long short-term memory (Bi-LSTM) network is used to automatically evaluate the importance weights of each sampling group based on the deep feature sequence. The experimental results on UCF101 and HMDB51 datasets show that the proposed model can effectively improve the utilization rate of temporal information and the ability of overall action representation, thus significantly improves the accuracy of human action recognition.

In order to improve the accuracy of human action recognition in video and the computational efficiency of large data sets, an action recognition algorithm based on multiple features and modified deep learning model is proposed. First, the deep network pre-training process is used to learn and optimize the RBM parameters, and the deep belief nets (DBN) model is constructed through deep learning. Then, human 13 joint points and critical points of optical flow are automatically extracted by DBN model. Second, these more abstract and more effective human motion features are combined to represent human actions. Ultimately, the entire DBN network structure is fine-tuned by support vector machine (SVM) algorithm to classify human actions. We demonstrate that human 13 joint points and critical points of optical flow are two very effective human action characterizations, our proposed approach greatly reduces the required samples, and shortens the training time of the samples, can efficiently process large data sets and can effectively recognize novel actions. We performed experiments on the KTH data set, Weizmann data set, the ballet data set and UCF101 data set to evaluate the proposed method, the experiment results show that the average recognition accuracy is over 98%, which validates its effectiveness, and show that our results are stable, reliable, and significantly better than the results of two state-of-the-art approaches on four different data sets. So, it lays a good theoretical foundation for practical applications.

Accurate human activity recognition (HAR) is the key to enable emerging context-aware applications that require an understanding and identification of human behavior, e.g., monitoring disabled or elderly people who live alone. Traditionally, HAR has been implemented either through ambient sensors, e.g., cameras, or through wearable devices, e.g., a smartwatch, with an inertial measurement unit (IMU). The ambient sensing approach is typically more generalizable for different environments as this does not require every user to have a wearable device. However, utilizing a camera in privacy-sensitive areas such as a home may capture superfluous ambient information that a user may not feel comfortable sharing. Radars have been proposed as an alternative modality for coarse-grained activity recognition that captures a minimal subset of the ambient information using micro-Doppler spectrograms. However, training fine-grained, accurate activity classifiers is a challenge as low-cost millimeter-wave (mmWave) radar systems produce sparse and non-uniform point clouds. In this paper, we propose RadHAR, a framework that performs accurate HAR using sparse and non-uniform point clouds. RadHAR utilizes a sliding time window to accumulate point clouds from a mmWave radar and generate a voxelized representation that acts as input to our classifiers. We evaluate RadHAR using a low-cost, commercial, off-the-shelf radar to get sparse point clouds which are less visually compromising. We evaluate and demonstrate our system on a collected human activity dataset with 5 different activities. We compare the accuracy of various classifiers on the dataset and find that the best performing deep learning classifier achieves an accuracy of 90.47%. Our evaluation shows the efficacy of using mmWave radar for accurate HAR detection and we enumerate future research directions in this space.

EAAR: Efficient and Accurate Action Recognition model with enhanced spatio-temporal perception.

A micro-Doppler spectrogram denoising algorithm for radar human activity recognition

Targeting the problems of the insufficient utilization of temporal and spatial information in videos and a lower accuracy rate, this paper proposes a human action recognition method for dynamic videos of emergency rescue based on a spatial-temporal fusion network. A time domain segmentation strategy based on random sampling maintains the overall time domain structure of the video. Considering the spatial-temporal asynchronous relationship, multiple asynchronous motion sequences are increased as input of the temporal convolutional network. spatial-temporal features are fused in convolutional layers to reduce feature loss. Because time series information is crucial for human action recognition, the acquired mid-layer spatial-temporal fusion features are sent into Bidirectional Long Short-Term Memory (Bi-LSTM) to obtain the human movement features in the whole video temporal dimension. Experiment results show the proposed method fully fuses spatial and temporal dimension information and improves the accuracy of human action recognition in dynamic scenes. It is also faster than traditional methods.

In this paper, we present a novel method of 3-D XYT feature extraction for efficient human action recognition. We propose the spatial first 3D feature extraction method to reduce the computation time and data bandwidth compared with Harris 3D corner detection. We also present experimental results through the proposed spatial first feature extraction to reach 83.33% accuracy of human action recognition through KTH dataset. Finally, we demonstrate the 67.8% computation time and 66.7% bandwidth reduction for the efficient human action recognition system.

In order to improve the accuracy of human action recognition and accelerate the recognition speed, we propose a method for human action recognition by modeling the human primary visual cortex neurons. The method firstly extracted motion information by using 3DGabor spatial-temporal filters to model the classical receptive field (CRF) of simple cells in the primary visual cortex. Secondly, conductance-driven integrate and fire neuron model was used to simulate the primary visual cortex neuron, by which motion information was converted into spike train. Finally, the mean firing rate of spike train formed a feature vector that captures the characteristic of human actions in this video sequence. Using Support Vector Machine (SVM), the method is tested on the Weizmann action dataset. The obtained impressive results show that our method was more effective than model of Escobar in human action recognition.

In order to improve the accuracy of human action recognition, a Residual Generative Adversarial Network (Res-GAN) method is proposed by combining the characteristics of Generative Adversarial Networks and Residual Networks. The generator uses an improved residual block extraction feature to obtain deep texture information and enhance the regularization of the model; residual blocks construct the discriminator. On the one hand, the accuracy of image feature extraction is improved, and the convergence speed of the network is accelerated. On the other hand, the discriminator performs the classification pre-training of human behavior recognition while distinguishing true and false samples. In order to avoid the influence of residual generative adversarial network generator on the recognition accuracy due to the instability of the reconstructed samples, the trained residual generative adversarial network discriminator was constructed as a classifier for human behavior recognition. Experiments on the KTH dataset demonstrate the effectiveness of the proposed method.

Along with the development of the Industrial Internet of Things, human activity recognition (HAR) has received widespread attention in many fields. Support Vector Machine (SVM), is widely used by researchers for human activity recognition. However, the inherent difference of signal properties from different sensors and various orientations is potentially lost when using the vector-based SVM for human activity recognition. What's more, the outlier sensitivity problem of the standard SVM reduces the accuracy of human activity recognition. To tackle this problem, we present a tensor-based feature representation model and a weighted support tensor machine (WSTM) for human activity recognition. Specifically, tensor-based representations are first used to model features from different sensors and various orientations to retain the latent relationship. In addition, the weighted support tensor machine is proposed to classify the human activities in tensor space while avoiding the outlier sensitivity problem. Experimental results demonstrate the proposed WSTM algorithm.

Hyperparameter optimization poses a significant challenge when developing deep neural networks. Building a convolutional neural network (CNN) for implementation can be an arduous and time-intensive task. This work proposed an approach to optimize the hyperparameters of one dimensional (1D-CNN) to improve the accuracy of human activity recognition (HAR). The framework includes a parametric depiction of 1D-CNNs along with an optimization process for hyperparameters aimed at maximizing the model's performance. This work designed the method called OPTConvNet for hyperparameter optimization of 1D-CNN using Hierarchical Particle Swarm Optimization (H-PSO). The H-PSO algorithm is designed to optimize the architectural, layer and training parameters of 1D-CNN. The H-PSO optimizes the architecture of the 1D-CNN at initial level. Layer and training hyperparameters will be optimized at the next level. The proposed approach employs an exponential-like inertia weight to fine-tune the balance between exploration and exploitation of particles to prevent premature convergence to a local optimum solution in the PSO algorithm. The H-PSO- CNN is evaluated on publicly available sensor- human activity recognition (S-HAR) datasets namely, UCI-HAR, Daphnet Gait, Opportunity and PAMPA2 datasets.

Video action recognition technology provides technical support for automated monitoring and early event warning, liberating human resources, preventing anomalies in advance, and handling events more promptly. However, the issue of user privacy leakage is also a growing concern. This paper discusses how to conduct action recognition while protecting privacy, proposing a generative adversarial network architecture that combines a multi-scale feature fusion generator with a spatiotemporal consistency discriminator. The network continuously enhances its capabilities through adversarial training strategies to protect facial privacy in videos. This network extracts features at different levels in the video through a multi-scale feature fusion mechanism so that the video after facial privacy protection still maintains a high degree of realism at the same time, to ensure that the accuracy of action recognition is not compromised, the feature enhancement module is designed to enhance action features and inhibit the privacy features significantly C3D is used as the action recognition model to accurately recognize a variety of actions in the video, such as running, jumping, falling, realizing the action analysis of video content. In this paper, the proposed method is evaluated in terms of privacy protection level and action recognition performance. Experiments are conducted on three datasets, LFW, HMDB51, and Hollywood2, and the results show that this framework effectively protects personal information while maintaining high action recognition accuracy.