Human action recognition based on convolutional neural network

The recognition of sports action is an important research subject, which is conducive to the improvement of athletes’ own level. To improve the accuracy of multi-modal data action recognition, based on the Transformer module, this study introduces a multi-head attention mechanism, fuses multi-modal data, and constructs a multi-stream structured object relationship inference network. Based on PointNet++ network and combining five different data fusion frameworks, a motion recognition model that integrates RGB data and 3D skeleton point cloud is constructed. The results showed that the Top-1 accuracy of multi-stream structured object relationship inference network was 42.5% and 42.7%, respectively, which was better than other algorithms. The accuracy of the multi-modal fusion model was improved by 15.6% and 5.1% compared with the single mode, and by 5.4% and 2.6% compared with the dual mode, which showed its superiority in the action recognition task. This showed that the fusion of multi-modal data can provide more abundant information, so as to improve the accuracy of action recognition. The accuracy of the action recognition model combining RGB data and 3D skeleton point cloud was 84.3%, 87.5%, 90.2%, 90.6% and 91.2% after the combination of different strategies, which effectively compensated for the problem of missing information in 3D skeleton point cloud and significantly improved the accuracy of action recognition. With a small amount of data, the Top-1 accuracy of the multi-stream structured object relationship inference network in this study was superior to other algorithms, showing its advantages in dealing with complex action recognition tasks. In addition, the action recognition model that fuses RGB data and 3D skeleton point cloud also achieved higher accuracy, which is better than other algorithms. This study can meet the needs of motion recognition in different scenarios and has certain reference value.

In recent years, many researchers have focused on using convolutional neural networks to perform human activity recognition as evidenced by the emergence of a number of convolutional neural network architectures such as LeNet-5, AlexNet and VGG16 and modern architectures such as ResNet, Inception V3, Inception-ResNet, MobileNet V2, NASNet and PNASNet. The main characteristic of a convolutional neural network (CNN) is its ability to extract features automatically from input images, which facilitates the processes of activity recognition and classification. Convolutional networks indeed derive more relevant and complex features with every additional layer. In addition, CNNs have achieved perfect classification on highly similar activities that were previously extremely difficult to classify. In this paper, we evaluate modern convolutional neural networks in terms of their human activity recognition accuracy, and we compare the results with the state of the art methods. In our research, we used two public data sets, HMDB (Shooting gun, kicking, falling to the floor, punching) and the Weizman dataset (walking, running, jumping, bending, one hand waving, two-hand waving, jumping in place, jumping jack, skipping). Our experimental results indicated that the CNN with NASNet architecture achieves the best performance of the six CNN architectures on both human activity data sets (HMDB and Weizman).

Human activity recognition (HAR) is an important research topic in the field of human-computer interaction (HCI), and wearable HAR based on convolutional neural networks (CNNs) has achieved the most advanced performance. In this study, wearable plastic optical fiber (POF) sensors based on GAF-ConvNeXt algorithm are used for HAR for the first time. The GAF-ConvNeXt algorithm, combined with the Gramian Angular Field (GAF) and the latest CNN model ConvNeXt, can convert motion signal of a one-dimensional time series into a two-dimensional image, which can not only retain the complete information of the signal, but also maintain its dependence on time, thereby presenting more depth features and facilitating accurate recognition of human activities. Furthermore, transfer learning (TL) was introduced to further optimize the GAF-ConvNeXt model, which effectively reduces the number of training samples, improves the recognition accuracy and accelerates the convergence speed. The GAF-ConvNeXt algorithm based on TL can recognize seven motion modes with an accuracy of 99.75%, and has great application potential in the fields of human motion recognition and HCI.

Hyperparameter optimization poses a significant challenge when developing deep neural networks. Building a convolutional neural network (CNN) for implementation can be an arduous and time-intensive task. This work proposed an approach to optimize the hyperparameters of one dimensional (1D-CNN) to improve the accuracy of human activity recognition (HAR). The framework includes a parametric depiction of 1D-CNNs along with an optimization process for hyperparameters aimed at maximizing the model's performance. This work designed the method called OPTConvNet for hyperparameter optimization of 1D-CNN using Hierarchical Particle Swarm Optimization (H-PSO). The H-PSO algorithm is designed to optimize the architectural, layer and training parameters of 1D-CNN. The H-PSO optimizes the architecture of the 1D-CNN at initial level. Layer and training hyperparameters will be optimized at the next level. The proposed approach employs an exponential-like inertia weight to fine-tune the balance between exploration and exploitation of particles to prevent premature convergence to a local optimum solution in the PSO algorithm. The H-PSO- CNN is evaluated on publicly available sensor- human activity recognition (S-HAR) datasets namely, UCI-HAR, Daphnet Gait, Opportunity and PAMPA2 datasets.

The convolutional neural network (CNN) is a potent and popular neural network types and has been crucial to deep learning in recent years. A standard CNN which is known as 2-dimensions CNN was first proposed to solve image classification problems at first. With advancements in science and technology, as well as the growth of the internet, tasks involving video analysis have received increased attention in recent years. As a result, it was suggested that a 3-dimensions convolutional neural network (3D CNN) be applied to tackle 3D image processing or video analysis. 3D CNN is primarily used in the field of human activity recognition, video classification, medical imaging detection, and so on. This paper offers a succinct introduction to the general architecture and representative models of 3D CNN, contrasts the variations between 2D CNN and 3D CNN, explores the widely used models derived from 3D CNN, and displays the application outcomes of 3D CNN.

With the advancement and ubiquitousness of wearable devices, wearable sensor-based human activity recognition (HAR) has become a prominent research area in the healthcare domain and human-computer interaction. Inertial measurement unit (IMU) which can provide a wide range of information such as acceleration, angular velocity has become one of the most commonly used sensors in HAR. Recently, with the growing demand for soft and flexible wearable devices, mountable stretch sensors have become a new promising modality in wearable sensor-based HAR. In this paper, we propose a deep learning-based multi-modality HAR framework which consists of three IMUs and two fabric stretch sensors in order to evaluate the potential of stretch sensors independently and in combination with IMU sensors for the activity recognition task. Three different deep learning algorithms: long short-term memory (LSTM), convolutional neural network (CNN) and hybrid CNN-LSTM are deployed to the sensor data for automatically extracting deep features and performing activity classification. The impact of sensor type on recognition accuracy of different activities is also examined in this study. A dataset collected from the proposed framework, namely iSPL IMU-Stretch and a public dataset called w-HAR are used for experiments and performance evaluation.

SUMMARYRemote surveillance of large‐scale equipment such as power plants and building complexes is important to prevent serious attacks and troubles. Automatic human action recognition can reduce the burdens of the surveillance. Multiview video is useful for human action recognition, because it provides robustness to the changes of people's appearance by orientation and occlusion. One problem of conventional multiview action recognition is that it requires both detection and tracking before action recognition. Human pose and motion vary depending on the person's action, and such variances may cause detection and tracking error. To solve this problem, previous work has proposed simultaneous action recognition and location estimation for single‐view videos using Hough voting. In this paper, we extend the Hough voting approach to simultaneous multiview action recognition and location estimation. Our proposed method independently casts votes for the action labels and spatiotemporal reference positions of actions in each view and integrates them using homographical transformations in the multiview extension. We evaluated our method and confirmed that it achieved high accuracy in action recognition and location estimation. The contribution of this paper is that it enables multiview action recognition without prior human detection and tracking.

Human action identification has advanced significantly as a result of the development of deep learning algorithms. Convolutional Neural Networks (CNNs) are known for their adeptness at extracting crucial information within video frames, especially shapes and postures. However, understanding movements necessitates not just individual frames as well as the temporal flow of the movement; this is wherein LSTM (long short-term memory) variants of Recurrent Neural Networks (RNNs) come into play. By analyzing frame sequences, LSTMs are particularly effective in gathering temporal connections, recalling prior motions, and overcoming the gap amongst static features as well as dynamic actions. Even with CNNs and LSTMs' remarkable accuracy in human action recognition (HAR), challenges remain.

Graph convolution network can efficiently obtain the spatial features in the task of human action recognition based on skeleton information, and HSGAC network incorporating graph attention convolution mechanism could obtain more abstract spatial features. However, HSGAC does not consider the spatial dynamic importance of joints. In this paper, we propose a two-stream graph attention convolutional network(Two-Stream GAC), which aims to fuse the static and dynamic feature relationships between joints. Two-Stream GAC first preprocesses the original human pose information dynamically, and use two HSGAC models to obtain the static feature relationship of the original pose information and the dynamic feature relationship of the preprocessed pose information respectively. The two network models are fused by weighted sum. The experimental results on the RDT and NTU-RGB+D datasets show that the method achieves better accuracy of action recognition.

Deep learning (DL) methods receive increasing attention within the field of human activity recognition (HAR) due to their success in other machine learning domains. Nonetheless, a direct transfer of these methods is often not possible due to domain specific challenges (e.g. handling of multi-modal sensor data, lack of large labeled datasets). In this paper, we address three key aspects for the future development of robust DL methods for HAR: (1) Is it beneficial to apply data specific normalization? (2) How to optimally fuse multimodal sensor data? (3) How robust are these approaches with respect to available training data? We evaluate convolutional neuronal networks (CNNs) on a new large real-world multimodal dataset (RBK) as well as the PAMAP2 dataset. Our results indicate that sensor specific normalization techniques are required. We present a novel pressure specific normalization method which increases the F1-score by ∼ 4.5 percentage points (pp) on the RBK dataset. Further, we show that late- and hybrid fusion techniques are superior compared to early fusion techniques, increasing the F1-score by up to 3.5 pp (RBK dataset). Finally, our results reveal that in particular CNNs based on a shared filter approach have a smaller dependency on the amount of available training data compared to other fusion techniques.

In the field of human action recognition (HAR), two-stream models have been widely employed. In recent years, traditional two-stream network models have disregarded the interframe sequence characteristics of video, resulting in a decrease in model robustness when local sequence information and long-term motion information interact. In light of this, a novel three-stream neural network is proposed by combining the long-term and short-term characteristics of a frame sequence with spatio-temporal information. Initially, the optical flow sequence image frames and RGB image frames in the video are extracted, the optical flow motion information and image space information in the video is obtained, the corresponding time network and space network are entered, and the spatial information is entered into the sequence feature processing network; the three networks are then pretrained. At the conclusion of training, the operation of feature extraction is executed, the features are incorporated with the parallel fusion algorithm by adding weights, and the behavior categories are classified using Multi-Layer Perception. Experimental results on the UCF11, UCF50, and HMDB51 datasets demonstrate that our model effectively integrates the spatial-temporal and frame-sequence information of human actions, resulting in a significant improvement in recognition accuracy. Its classification accuracy on the three datasets was 99.17%, 97.40%, and 96.88%, respectively, significantly enhancing the generalization capability and validity of conventional two-stream or three-stream models.

TCN-Inception: Temporal Convolutional Network and Inception modules for sensor-based Human Activity Recognition

This paper brings to light one of the most prominent applications of human activity recognition which is the anomaly detection. Providing security to an individual is a major concern of any society today due to the constantly increasing actions causing threats, starting from deliberate violence to an injury caused by an accident. Mere installation of a traditional closed-circuit television is not enough as it requires a human being to constantly remain alert and monitor the cameras, which is quite inefficient. This calls for the need to develop an automated security system that identifies abnormal activities in real time and brings immediate help to the victims. But it is time-consuming to recognize activities from lengthy surveillance videos and hence a new implementation idea which adaptively compresses the videos before passing it through the activity recognition system has been proposed. Adaptive video compression is a unique compression technology that compresses only the least significant parts of the video, retaining the objects of interest. By combining adaptive video compression and the convolutional 3D network with contextual multiple scales based on the temporal features (Xu et al. in Learning deep representations of appearance and motion for anomalous event detection. In: Computer Vision and Pattern Recognition (CVPR), 2015), the paper aims to provide an accurate anomalous human activity recognition system that works in real time. The method is implemented on the UCF101 crime dataset with around 13 different anomalies (Learning Spatiotemporal Features with 3D Convolutional Networks. In: International Conference on Computer Vision (ICCV), IEEE, 2015). It resulted in improved recognition performance compared to state-of-the-art techniques.

Recent advancements in digital technologies have profoundly transformed healthcare delivery, sports analytics, and physical activity surveillance. The accelerated proliferation of wearable sensing devices and Internet of Things (IoT) infrastructures has facilitated unprecedented large-scale acquisition of multimodal physiological and kinematic data. However, accurate recognition of complex human activities remains a formidable challenge, primarily due to inadequate modeling of the spatiotemporal dependencies inherent in wearable-sensor signals. To address these fundamental limitations, this paper proposed a novel IoT-oriented activity recognition framework predicated on a Convolutional Recurrent Neural Network (CRNN) architecture, engineered to simultaneously model the spatial and temporal characteristics of multimodal wearable-sensor data streams. The proposed framework synergistically integrates Convolutional Neural Networks (CNNs) for hierarchical spatial feature extraction with Recurrent Neural Networks (RNNs) for temporal sequence modeling, thereby enabling more discriminative and robust activity classification. The methodological pipeline comprises several sequential stages: First, multidimensional wearable-sensor datasets are employed, encompassing physiological and inertial measurements including heart rate variability, triaxial accelerometer readings, gyroscopic angular velocity, magnetometer orientation data, and cutaneous temperature signals acquired from multiple anatomical locations. Second, raw sensor signals undergo preprocessing and temporal segmentation procedures to enhance data quality and optimize temporal representation. Third, spatiotemporal feature representations are learned autonomously within the hierarchical CRNN architecture. Finally, the proposed model is systematically evaluated through a comparative analysis with five representative baseline methods, encompassing both conventional machine learning algorithms and contemporary deep learning approaches. Experimental results demonstrate that the proposed CRNN framework achieves superior performance, achieving 98.2% classification accuracy, 97.2% sensitivity, 99.2% specificity, 97.4% recall, and 97.6% precision on the evaluated wearable-sensor datasets. Compared with existing methodologies, the proposed model consistently achieves higher recognition accuracy and greater generalization robustness, underscoring its efficacy for wearable-sensor-based activity recognition and its considerable potential for broader deployment in IoT-enabled monitoring paradigms and adaptable solutions in educational settings.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.