Hard Sample Mining and Learning for Skeleton-Based Human Action Recognition and Identification

Dynamic acoustic compensation and adaptive focal training for personalized speech enhancement

Human activity recognition is an important area of computer vision research. Its applications include surveillance systems, patient monitoring systems, and a variety of systems that involve interactions between persons and electronic devices such as human-computer interfaces. Most of these applications require an automated recognition of high-level activities, composed of multiple simple (or atomic) actions of persons. This article provides a detailed overview of various state-of-the-art research papers on human activity recognition. We discuss both the methodologies developed for simple human actions and those for high-level activities. An approach-based taxonomy is chosen that compares the advantages and limitations of each approach. Recognition methodologies for an analysis of the simple actions of a single person are first presented in the article. Space-time volume approaches and sequential approaches that represent and recognize activities directly from input images are discussed. Next, hierarchical recognition methodologies for high-level activities are presented and compared. Statistical approaches, syntactic approaches, and description-based approaches for hierarchical recognition are discussed in the article. In addition, we further discuss the papers on the recognition of human-object interactions and group activities. Public datasets designed for the evaluation of the recognition methodologies are illustrated in our article as well, comparing the methodologies' performances. This review will provide the impetus for future research in more productive areas.

In recent years, there has been a significant increase in research focused on Human Activity Analysis (HAA). This field has progressed from basic activity recognition tasks to addressing more challenging ones, such as predicting future human actions based on partially observed videos and even predicting actions before they happen. The evolution of HAA has been driven by recent advancements in attention-based models like Transformers, along with a wide range of applications from security surveillance to advanced monitoring systems, behaviour analysis, and more. A comprehensive review of HAA literature from 2017 to 2025, with a novel taxonomy emphasising activity recognition, prediction, and anticipation, is presented. We critically review and examine recognition methods from trimmed and untrimmed videos, context-aware and trajectory-based prediction, and short-term and long-term anticipation. Through a comprehensive analysis, we review and evaluate key aspects of this domain, including attention-based contextual comprehension, temporal dynamics modelling, and multi-model fusion methods. Furthermore, we critically examine and assess the public datasets utilised in driving this research forward, pinpointing limitations and primary challenges within this domain. Finally, the paper provides a summary of recent developments in HAA and suggests future directions, with the hope that it will serve as a valuable reference for researchers in the field.

Breast ultrasound image classification with hard sample generation and semi-supervised learning

In order to realize high-accuracy recognition of aerobics actions, a highly applicable deep learning model and faster data processing methods are required. Therefore, it is a major difficulty in the field of research on aerobics action recognition. Based on this, this paper studies the application of the convolution neural network (CNN) model combined with the pyramid algorithm in aerobics action recognition. Firstly, the basic architecture of the convolution neural network model based on the pyramid algorithm is proposed. Combined with the application strategy of the common recognition model in aerobics action recognition, the traditional aerobics action capture information is processed. Through the characteristics of different aerobics actions, different accurate recognition is realized, and then, the error of the recognition model is evaluated. Secondly, the composite recognition function of the convolution neural network model in this application is constructed, and the common data layer effect recognition method is used in the optimization recognition. Aiming at the shortcomings of the composite recognition function, the pyramid algorithm is used to improve the convolution neural network recognition model by deep learning optimization. Finally, through the effectiveness comparison experiment, the results show that the convolution neural network model based on the pyramid algorithm is more efficient than the conventional recognition method in aerobics action recognition.

With the rise of computer vision, it is becoming more and more important to accurately recognize and evaluate human actions. However, the complexity, intraclass differences, and viewing angle changes of human actions significantly impact the accuracy of identification human actions. This paper proposed an action recognition and evaluation method based on skeleton information extraction. Briefly, We use Lightweight OpenPose to extract the key points of human skeleton and perform processing work, including video data cutting, deleting some key points, supplementing missing key points, filtering processing, feature extraction, etc. Through an in-depth exploration of related theoretical technologies, we proposed a model for recognition and evaluation of human table tennis actions with an ordinary camera. The support vector machine algorithm (SVM) classification model is used to identify table tennis actions in real-time. Then the dynamic time regularization (DTW) algorithm calculates the similarity of each human skeleton key point in the action sequence. The low-scoring bone key points are marked to evaluate the human table tennis action in real time. The results show that a recognition rate of more than 95% is achieved in the test set, which proves the method's effectiveness. In addition, we compared the results with previous work using inertial sensors for action recognition, which shows our method can preserve the same accuracy with a much lower cost of implementation.

Action recognition, a field within human-centered computing, is vital for identifying and understanding human actions, benefiting applications like surveillance, autonomous vehicles, and human-computer interaction. Artificial intelligence’s enduring goal is to develop robust models for perceiving and understanding the visual world around us. Deep neural networks have shown exceptional performance in various tasks, profoundly impacting real-world applications, including action prediction and recognition, which have advanced significantly in recent years. Most work in visual human action recognition focuses on single viewpoints or modalities from complete observation. Yet, the real significance lies in predicting future actions from incomplete observations to prevent real-world tragedies. With the availability of multiple cameras and data from multiple modalities (RGB, Depth, and Skeleton) available today, it becomes possible to model human action in multi-view and multi-modality context, minimizing the data loss due to occlusions and signal quality issues to improve recognition accuracy on the strength of state-of-the-art deep learning models. However, deep neural network models are susceptible to adversarial attacks, where imperceptible perturbations can compromise action recognition model performance. This thesis focuses on identifying latent vulnerabilities and proposing a defense mechanism against such threats in a multi-modality and multi-view setting. This work introduces an efficient and effective attack mechanism that perturbs skeleton data by targeting key joints and segments while employing a graph attention mechanism that learns the semantics to perturb other modalities. Additionally, an approach has been developed that not only adds noise but also alters the visual spatial structure of skeleton data through generative modeling. Furthermore, this dissertation introduces a defense mechanism known as the Collaborative Knowledge Distillation Network, which leverages graph attention and knowledge distillation techniques. This network leverages the knowledge from compromised multi-view data and integrates information from clean data to address incomplete observations and noisy action videos, enhancing the robustness of action recognition models for real-world applications.

SUMMARYRemote surveillance of large‐scale equipment such as power plants and building complexes is important to prevent serious attacks and troubles. Automatic human action recognition can reduce the burdens of the surveillance. Multiview video is useful for human action recognition, because it provides robustness to the changes of people's appearance by orientation and occlusion. One problem of conventional multiview action recognition is that it requires both detection and tracking before action recognition. Human pose and motion vary depending on the person's action, and such variances may cause detection and tracking error. To solve this problem, previous work has proposed simultaneous action recognition and location estimation for single‐view videos using Hough voting. In this paper, we extend the Hough voting approach to simultaneous multiview action recognition and location estimation. Our proposed method independently casts votes for the action labels and spatiotemporal reference positions of actions in each view and integrates them using homographical transformations in the multiview extension. We evaluated our method and confirmed that it achieved high accuracy in action recognition and location estimation. The contribution of this paper is that it enables multiview action recognition without prior human detection and tracking.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Asymmetric 3D Convolutional Neural Networks for action recognition

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

The intricacies of human motion and action offer a rich vein of information crucial for enhancing the interaction between humans and machines. Human action understanding serves as the linchpin in this dynamic, bridging advanced computational methodologies with real-world applications across various sectors. This dissertation delves into the nuanced realm of action recognition and pose estimation, underscoring the transition from theoretical frameworks to practical applications. Central to the presented research is the development of the Graph Convolution Network (GCN)-based InfoGCN model for human action recognition, which introduces an informa tion bottleneck principle to optimize learning efficiency and accuracy. Additionally, the Pose Relation Transformer (PORT) is crafted to address challenges in hand and human pose esti mation, particularly under conditions of occlusion, by employing strategies akin to Masked Language Modeling in natural language processing. Further advancements are manifested in InfoGCN++, designed to facilitate real-time ac tion recognition, thereby catering to immediate analytical needs in dynamic environments. Concurrently, CARING-AI emerges as a novel framework for generating context-aware hu man motion, capturing both temporal and spatial dimensions to produce more realistic and accurate motion representations. Anticipating future needs, the dissertation proposes CARING-AI++, focusing on the enhancement of hand-object interaction within human motion generation. This progression signifies a deliberate move towards solving user-specific problems, illustrating a profound understanding of the necessary integration of technology and practicality. The collective work outlined in this dissertation not only propels the understanding of human action but also crystallizes its application in real-world scenarios, thereby bridging the gap between theoretical research and practical utility. With implications for healthcare, autonomous vehicles, and interactive technologies, the research underscores the potential of these advancements to revolutionize how we interact with and leverage technology for societal benefit, paving the way for future interdisciplinary collaboration and the ethical use of action recognition systems.

Most of the state-of-the-art action recognition methods focus on offline learning, where the samples of all types of actions need to be provided at once. Here, we address continual learning of action recognition, where various types of new actions are continuously learned over time. This task is quite challenging, owing to the catastrophic forgetting problem stemming from the discrepancies between the previously learned actions and current new actions to be learned. Therefore, we propose Else-Net, a novel Elastic Semantic Network with multiple learning blocks to learn diversified human actions over time. Specifically, our Else-Net is able to automatically search and update the most relevant learning blocks w.r.t. the current new action, or explore new blocks to store new knowledge, preserving the unmatched ones to retain the knowledge of previously learned actions and alleviates forgetting when learning new actions. Moreover, even though different human actions may vary to a large extent as a whole, their local body parts can still share many homogeneous features. Inspired by this, our proposed Else-Net mines the shared knowledge of the decomposed human body parts from different actions, which benefits continual learning of actions. Experiments show that the proposed approach enables effective continual action recognition and achieves promising performance on two large-scale action recognition datasets.

As increasing attention is paid to human action recognition from skeleton data, this paper focuses on such tasks by proposing a hierarchical model to discover the structure information of body-parts involved in actions for better analysis of human actions in the skeleton data. Considering human actions as simultaneous motions of body-parts of the human skeleton, we propose a hierarchical model to simultaneously apply discriminative body-parts selection at a same scale and group coupling of bundles of body-parts at different scales, while we decompose the human skeleton into a hierarchy of body-parts of varying scales. To represent such hierarchy of body-parts, we accordingly build a hierarchical rotation and relative velocity (HRRV) descriptor. The hierarchical representations encoded by Fisher vectors of the HRRV descriptors are properly formulated into the hierarchical model via the proposed mixed norm, to apply the sparse selection of body-parts and regularize the structure of such hierarchy of body-parts. The extensive evaluations on three challenging datasets demonstrate the effectiveness of our proposed approach, which achieves superior performance compared to the state-of-the-art algorithms on datasets with various sizes, showing it is more widely applicable than existing approaches.

Recognizing human actions is crucial for an effective and safe collaboration between humans and robots. For example, in a collaborative assembly task, human workers can use gestures to communicate with the robot, and the robot can use the recognized actions to anticipate the next steps in the assembly process, leading to improved safety and productivity. In this work, we propose a general framework for human action recognition based on 3D pose estimation and ensemble techniques, which allows to recognize both body actions and hand gestures. The framework relies on OpenPose and 2D to 3D lifting methods to estimate 3D joints for the human body and the hands, feeding then these joints into a set of graph convolutional networks based on the Shift-GCN architecture. The output scores of all networks are combined using an ensemble approach to predict the final human action. The proposed framework was evaluated on a custom dataset designed for human–robot collaboration tasks, named IAS-Lab Collaborative HAR dataset. The results showed that using an ensemble of action recognition models improves the accuracy and robustness of the overall system; moreover, the proposed framework can be easily specialized on different scenarios and achieve state-of-the-art results on the HRI30 dataset when coupled with an object detector or classifier.