Research on Privacy-Preserving Action Recognition Method Based on Adversarial Learning and Feature Enhancement

The recognition of sports action is an important research subject, which is conducive to the improvement of athletes’ own level. To improve the accuracy of multi-modal data action recognition, based on the Transformer module, this study introduces a multi-head attention mechanism, fuses multi-modal data, and constructs a multi-stream structured object relationship inference network. Based on PointNet++ network and combining five different data fusion frameworks, a motion recognition model that integrates RGB data and 3D skeleton point cloud is constructed. The results showed that the Top-1 accuracy of multi-stream structured object relationship inference network was 42.5% and 42.7%, respectively, which was better than other algorithms. The accuracy of the multi-modal fusion model was improved by 15.6% and 5.1% compared with the single mode, and by 5.4% and 2.6% compared with the dual mode, which showed its superiority in the action recognition task. This showed that the fusion of multi-modal data can provide more abundant information, so as to improve the accuracy of action recognition. The accuracy of the action recognition model combining RGB data and 3D skeleton point cloud was 84.3%, 87.5%, 90.2%, 90.6% and 91.2% after the combination of different strategies, which effectively compensated for the problem of missing information in 3D skeleton point cloud and significantly improved the accuracy of action recognition. With a small amount of data, the Top-1 accuracy of the multi-stream structured object relationship inference network in this study was superior to other algorithms, showing its advantages in dealing with complex action recognition tasks. In addition, the action recognition model that fuses RGB data and 3D skeleton point cloud also achieved higher accuracy, which is better than other algorithms. This study can meet the needs of motion recognition in different scenarios and has certain reference value.

SUMMARYRemote surveillance of large‐scale equipment such as power plants and building complexes is important to prevent serious attacks and troubles. Automatic human action recognition can reduce the burdens of the surveillance. Multiview video is useful for human action recognition, because it provides robustness to the changes of people's appearance by orientation and occlusion. One problem of conventional multiview action recognition is that it requires both detection and tracking before action recognition. Human pose and motion vary depending on the person's action, and such variances may cause detection and tracking error. To solve this problem, previous work has proposed simultaneous action recognition and location estimation for single‐view videos using Hough voting. In this paper, we extend the Hough voting approach to simultaneous multiview action recognition and location estimation. Our proposed method independently casts votes for the action labels and spatiotemporal reference positions of actions in each view and integrates them using homographical transformations in the multiview extension. We evaluated our method and confirmed that it achieved high accuracy in action recognition and location estimation. The contribution of this paper is that it enables multiview action recognition without prior human detection and tracking.

Activity recognition is an important component of many pervasive computing applications. Device-free activity recognition has the advantage that it does not have the privacy concern of using cameras and the subjects do not have to carry a device on them. Recently, it has been shown that channel state information (CSI) can be used for activity recognition in a device-free setting. With the proliferation of wireless devices, it is important to understand how radio frequency interference (RFI) can impact on pervasive computing applications. In this paper, we investigate the impact of RFI on device-free CSI-based location-oriented activity recognition. We conduct experiments in environments without and with RFI. We present data to show that RFI can have a significant impact on the CSI vectors. In the absence of RFI, different activities give rise to different CSI vectors that can be differentiated visually. However, in the presence of RFI, the CSI vectors become much noisier and activity recognition also becomes harder. Our extensive experiments shows that the performance of state-of-the-art classification methods may degrade significantly with RFI. We then propose a number of counter measures to mitigate the impact of RFI and improve the location-oriented activity recognition performance. Our evaluation shows the proposed method can improve up to 10% true detection rate in the presence of RFI. We also study the impact of bandwidth on activity recognition performance. We show that with a channel bandwidth of 20 MHz (which is used by WiFi), it is possible to achieve a good activity recognition accuracy when RFI is present.

Action recognition methods based on human skeleton can explicitly represent human actions, and have gradually become one of the important research directions in the field of computer vision. To address the problems that the skeleton graph in graph convolutional networks is fixed to represent only the physical structure of the human body and the lack of adaptive ability to the skeleton topology graph structure, this paper proposes a dual-stream non-local graph convolutional network based on the attention mechanism. First, the temporal convolution layer is extended to a parallel structure with multiple kernels, and different temporal convolution kernel modules are adaptively selected to collect features by channel weights; second, an attention model consisting of an attention pooling layer is proposed to capture the correlation and temporal continuity among joints; finally, the nonlocal graph convolution network is used as the basic framework with joint information, skeletal information and respective motion information A dual-stream fusion model is constructed. The proposed method is compared with the mainstream methods in recent years on the action recognition dataset NTU RGB+D, and the experimental results show that the proposed method achieves high accuracy in action recognition.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Most of the semantic segmentation real‐time networks improve the segmentation speed by reducing the spatial resolution, leading to the accuracy being significantly reduced as a result. To solve this problem, we propose feature enhancement module (FEM), feature extraction and fusion module (FEFM). By extracting and enhancing the future map before the image down‐sample on the backbone and fusing the low‐level features with rich details and the high‐level features with more semantic information. Based on the FEM and FEFM, we introduce a real‐time semantic segmentation network feature extraction and enhancement network. In the experiment, using Cityscapes and CamVid datasets, the proposed network achieves a balance between computing speed and accuracy. Without additional processing and pretraining, it achieves 75.47% Mean IoU on the Cityscapes test dataset with only 29.96G Flops and a speed of 94 frames per second on a single RTX 2080Ti card. Code is available at https://github.com/favoMJ/FEENet.

Recent advancements in digital technologies have profoundly transformed healthcare delivery, sports analytics, and physical activity surveillance. The accelerated proliferation of wearable sensing devices and Internet of Things (IoT) infrastructures has facilitated unprecedented large-scale acquisition of multimodal physiological and kinematic data. However, accurate recognition of complex human activities remains a formidable challenge, primarily due to inadequate modeling of the spatiotemporal dependencies inherent in wearable-sensor signals. To address these fundamental limitations, this paper proposed a novel IoT-oriented activity recognition framework predicated on a Convolutional Recurrent Neural Network (CRNN) architecture, engineered to simultaneously model the spatial and temporal characteristics of multimodal wearable-sensor data streams. The proposed framework synergistically integrates Convolutional Neural Networks (CNNs) for hierarchical spatial feature extraction with Recurrent Neural Networks (RNNs) for temporal sequence modeling, thereby enabling more discriminative and robust activity classification. The methodological pipeline comprises several sequential stages: First, multidimensional wearable-sensor datasets are employed, encompassing physiological and inertial measurements including heart rate variability, triaxial accelerometer readings, gyroscopic angular velocity, magnetometer orientation data, and cutaneous temperature signals acquired from multiple anatomical locations. Second, raw sensor signals undergo preprocessing and temporal segmentation procedures to enhance data quality and optimize temporal representation. Third, spatiotemporal feature representations are learned autonomously within the hierarchical CRNN architecture. Finally, the proposed model is systematically evaluated through a comparative analysis with five representative baseline methods, encompassing both conventional machine learning algorithms and contemporary deep learning approaches. Experimental results demonstrate that the proposed CRNN framework achieves superior performance, achieving 98.2% classification accuracy, 97.2% sensitivity, 99.2% specificity, 97.4% recall, and 97.6% precision on the evaluated wearable-sensor datasets. Compared with existing methodologies, the proposed model consistently achieves higher recognition accuracy and greater generalization robustness, underscoring its efficacy for wearable-sensor-based activity recognition and its considerable potential for broader deployment in IoT-enabled monitoring paradigms and adaptable solutions in educational settings.

Human activity recognition is an important area of ubiquitous computing. Most current researches in activity recognition mainly focus on simple activities, e.g., sitting, running, walking, and standing. Compared with simple activities, complex activities are more complicated with high-level semantics, e.g., working, commuting, and having a meal. This paper presents a hierarchical model to recognize complex activities as mixtures of simple activities and multiple actions. We generate the components of complex activities using a clustering algorithm, represent and recognize complex activities by applying a topic model on these components. It is a data-driven method that can retain effective information for representing and recognizing complex activities. In addition, acceleration and physiological signals are fused in classifier level to ensure the overall performance of complex activity recognition. The results of experiments show that our method has ability to represent and recognize complex activities effectively.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Using skeleton data to classify human behavior has been received a lot of attention and various technologies have been developed because it can be utilized in various domains. Among them, Graph Convolutional Networks (GCN) shows high accuracy in action recognition through the effective extraction of features from graph data. GCN has been further developed and two stream adaptive graph convolution networks (2s-AGCN) are made. This model utilizes bone data as well as spatial and temporal information of joints to compensate for the problems of conventional GCN limited to local region. Although 2s-AGCN show the highest accuracy among existing technologies, new approaches using CNN, LSTM and auto encoder continue to emerge and present several methods. In this paper, we verify if 2s-AGCN show higher accuracy than recent CNN-LSTM networks and compare structures and features of their models.

Two-stream Convolutional Neural Networks have shown excellent performance in video action recognition. Most existing works train each sampling group independently, or just fuse at the last level, which obviously ignore the continuity of action in temporal and the complementary information between action fragments. In this paper, a temporal segment connection network is proposed to overcome these limitations. On the one hand, the forget gate module of the long short-term memory (LSTM) network is used to establish feature-level connections between each sampling group. This not only strengthens the information transmission between the sampling groups to enhance the temporal connectivity, but also extracts the complementary information between the sampling groups to enhance the overall representation of the action. On the other hand, a bi-directional long short-term memory (Bi-LSTM) network is used to automatically evaluate the importance weights of each sampling group based on the deep feature sequence. The experimental results on UCF101 and HMDB51 datasets show that the proposed model can effectively improve the utilization rate of temporal information and the ability of overall action representation, thus significantly improves the accuracy of human action recognition.

The recent development of wireless wearable sensor networks offers a spectrum of new applications in fields of healthcare, medicine, activity monitoring, sport, safety, human-machine interfacing, and beyond. Successful use of this technology depends on lifetime of the battery-powered sensor nodes. This paper presents a new method for extending the lifetime of the wearable sensor networks by avoiding unnecessary data transmissions. The introduced method is based on embedded classifiers that allow sensor nodes to decide if current sensor readings have to be transmitted to cluster head or not. In order to train the classifiers, a procedure was elaborated, which takes into account the impact of data selection on accuracy of a recognition system. This approach was implemented in a prototype of wearable sensor network for human activity monitoring. Real-world experiments were conducted to evaluate the new method in terms of network lifetime, energy consumption, and accuracy of human activity recognition. Results of the experimental evaluation have confirmed that, the proposed method enables significant prolongation of the network lifetime, while preserving high accuracy of the activity recognition. The experiments have also revealed advantages of the method in comparison with state-of-the-art algorithms for data transmission reduction.

Accurate recognition of patients' physical activities leads to correct diagnosis and treatments. However, currently deployed approaches are deficient in recognizing the activities requiring frequent interposture transitions, such as jogging, jumping, turning left, and going upstairs. The reason is that with the change in position and rotation, different activity signals are generated, which are difficult to distinguish from other activities and can therefore mislead the physicians. Therefore, we propose to employ a methodology that utilizes the energy expenditure for each activity and reduces the dimensions of the feature space to differentiate among the activities. In this regard, we employ a feature descriptor called local energy-based shape histogram to preserve the maximum information of local energy. Considering the high volumes of continuously generated data, our methodology integrates cloud computing services with the body area networks. We also investigate the effects of on-body sensors' location on the activity recognition accuracy and also identify the best sensor position for a certain activity with the maximum accuracy. We used the wearable action recognition database dataset to perform the experiments. Our analysis shows that for each activity to be recognized at a decent level, it is imperative to observe the activity recognition performance by simultaneously applying different combinations of sensors.

Action recognition in static image is challenging. The authors propose mutually incoherent pose bases which are implicit poselet co‐occurrences and are learned by dictionary training to describe body pose. Poselets in a pose basis are not constrained in space and quantity, thus pose basis can describe body pose more flexibly than k ‐poselet. In their method, body pose in an image is represented by a sparse linear combination of pose bases because pose in an action varies while each image only captures a snapshot from a single viewpoint. In dictionary training, the challenge is how to stabilise the sparse representation which is the input of Support Vector Machine (SVM) for action recognition, because the original pose signal is ambiguous while dictionary is an over complete matrix. Their solution is to add cumulative coherence as penalty in objective function and induce pose bases become mutually incoherent. They evaluate the method on two popular datasets and experiment results show the pose representation has encouraging performance in action recognition. Furthermore, they empirically exploit the complementary role of the local pose feature with deep convolutional neural network features from holistic image. Experiment results demonstrate aggressive performance improvement by concatenating the two features.

Existing studies in skeleton‐based action recognition mainly utilise skeletal data taken from a single camera. Since the quality of skeletal tracking of a single camera is noisy and unreliable, however, combining data from multiple cameras can improve the tracking quality and hence increase the recognition accuracy. In this study, the authors propose a method called weighted averaging fusion which merges skeletal data of two or more camera views. The method first evaluates the reliability of a set of corresponding joints based on their distances to the centroid, then computes the weighted average of selected joints, that is, each joint is weighted by the overall reliability of the camera reporting the joint. Such obtained, fused skeletal data are used as the input to the action recognition step. Experiments using various frame‐level features and testing schemes show that more than 10% improvement can be achieved in the action recognition accuracy using these fused skeletal data as compared with the single‐view case.