A Single Framework for Action Recognition Based on Boosted Randomized Trees

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Various motion detection methods have been proposed in the past decade, but there are seldom attempts to investigate the advantages and disadvantages of different detection mechanisms so that they can complement each other to achieve a better performance. Toward such a demand, this paper proposes a human action detection and recognition framework to bridge the semantic gap between low-level pixel intensity change and the high-level understanding of the meaning of an action. To achieve a robust estimation of the region of action with the complexities of an uncontrolled background, we propose the combination of the optical flow field and Harris3D corner detector to obtain a new spatial-temporal estimation in the video sequences. The action detection method, considering the integrated motion information, works well with the dynamic background and camera motion, and demonstrates the advantage of the proposed method of integrating multiple spatial-temporal cues. Then the local features (SIFT and STIP) extracted from the estimated region of action are used to learn the Universal Background Model (UBM) for the action recognition task. The experimental results on KTH and UCF YouTube Action (UCF11) data sets show that the proposed action detection and recognition framework can not only better estimate the region of action but also achieve better recognition accuracy comparing with the peer work.

This paper presents a novel framework for skeleton-based Human Action Recognition (HAR) based on Graph Convolution Networks (GCNs). The proposed framework aims to increase human action recognition performance of GCN-based methods by incorporating a missing-joint-handling pre-processing step and a novel adjacency matrix construction method in a single human action recognition pipeline. The missing-joint-handling pre-processing step is utilized to infer missing data in the input sequence, which may occur due to imperfect skeleton extraction, based on imputation methods. The novel adjacency matrix construction method is executed offline to compute an improved weighted adjacency matrix specifically designed for HAR, which is utilized in every layer of the employed GCN. Moreover, both the pre-processing step and the adjacency construction method can be utilized along with any GCN architecture, allowing any GCN-based HAR method to be employed in the proposed framework. Experimental evaluation on two public datasets indicate favorable human action classification scores compared to the employed baseline and all competing methods both for 2D and 3D skeleton-based human action recognition, while using a GCN architecture with less learnable parameters.

Human action recognition and localization is a challenging vision task with promising applications. To tackle this problem, recently developed commodity depth sensor (e.g., Microsoft Kinect) has opened up new opportunities with several developed human motion features based on depth image for action representation. However, how depth information can be effectively adopted in the middle or high level representation in action detection, in particular, the depth induced three dimensional contextual information for modeling interactions between human-human, human-object and human-surroundings has yet been explored. In this paper, we propose a novel action recognition and localization framework which effectively fuses depth-induced contextual information from different levels of the processing pipeline for understanding various interactions. First, depth image is combined with grayscale image for more robust human subject and object detection. Second, three dimensional spatial and temporal relationship among human subjects or objects is represented based on the combination of grayscale and depth images. Third, depth information is further utilized to represent different types of indoor scenes. Finally, we fuse these multiple stage depth-induced contextual information to yield an unified action detection framework. Extensive experiments on a challenging grayscale + depth human action detection benchmark database demonstrate the effectiveness of the depth-induced contextual information and the high detection accuracy of the proposed framework.

Recognizing human actions is crucial for an effective and safe collaboration between humans and robots. For example, in a collaborative assembly task, human workers can use gestures to communicate with the robot, and the robot can use the recognized actions to anticipate the next steps in the assembly process, leading to improved safety and productivity. In this work, we propose a general framework for human action recognition based on 3D pose estimation and ensemble techniques, which allows to recognize both body actions and hand gestures. The framework relies on OpenPose and 2D to 3D lifting methods to estimate 3D joints for the human body and the hands, feeding then these joints into a set of graph convolutional networks based on the Shift-GCN architecture. The output scores of all networks are combined using an ensemble approach to predict the final human action. The proposed framework was evaluated on a custom dataset designed for human–robot collaboration tasks, named IAS-Lab Collaborative HAR dataset. The results showed that using an ensemble of action recognition models improves the accuracy and robustness of the overall system; moreover, the proposed framework can be easily specialized on different scenarios and achieve state-of-the-art results on the HRI30 dataset when coupled with an object detector or classifier.

Human Activity Recognition (HAR) is widely used in many applications and HAR using smartphone only has been proved to be effective, flexible and unobtrusive for activity recognition. In this paper, a two-layer and multi-strategy HAR framework is proposed to overcome the major challenge of HAR using smartphone only, i.e., the variation in orientation and position of the device. In the first layer, the activities are classified into different groups with high accuracy and for each group in the second layer, the appropriate strategy is designed according to the characteristics of the group to improve the recognition performance. For static activity group, the transitional activities are introduced to help classifying the activities indirectly. For dynamic activity group sensitive to the position variation of the smartphone, a position-assisted strategy is proposed to alleviate the influence of position variation. The simulation results demonstrate the effectiveness of the proposed two-layer multi-strategy HAR framework.

Conventional human activity recognition approaches are mainly based on machine learning methods, which are not working well for composite activity recognition due to the complexity and uncertainty of real scenarios. We propose in this paper an automated reasoning based hierarchical framework for human activity recognition. This approach constructs a hierarchical structure for representing the composite activity by a composition of lower-level actions and gestures according to its semantic meaning. This hierarchical structure is then transformed into logical formulas and rules, based on which the resolution based automated reasoning is applied to recognize the composite activity given the recognized lower-level actions by machine learning methods.

Real-time identification of human activities in urban environments is increasingly becoming important in the context of public safety and national security. Distributed camera networks that provide multiple views of a scene are ideally suited for real-time action recognition. However, deployments of multi-camera based real-time action recognition systems have thus far been inhibited because of several practical issues and restrictive assumptions that are typically made such as the knowledge of a subjects orientation with respect to the cameras, the duration of each action and the conformation of a network deployment during the testing phase to that of a training deployment. In reality, action recognition involves classification of continuously streaming data from multiple views which consists of an interleaved sequence of various human actions. While there has been extensive research on machine learning techniques for action recognition from a single view, the issues arising in the fusion of data from multiple views for reliable action recognition have not received as much attention. In this thesis, I have developed a fusion framework for human action recognition using a multi-camera network that addresses these practical issues of unknown subject orientation, unknown view configurations, action interleaving and variable duration actions.;The proposed framework consists of two components: (1) a score-fusion technique that utilizes underlying view-specific supervised learning classifiers to classify an action within a given set of frames and (2) a sliding window technique that is used to parse a sequence of frames into multiple actions. The use of a score-fusion technique as opposed to a feature-level fusion of data from multiple views allows us to robustly classify actions even when camera configurations are arbitrary and different from training phase and at the same time reduces the required network bandwidth for data transmission permitting wireless deployments. Moreover, the proposed framework is independent of the underlying classifier that is used to generate scores for each action snippet and thus offers more flexibility compared to sequential approaches like Hidden Markov Models. The amount of training and parameterization is also significantly lower compared to HMM-based approaches. This Real-Time recognition system has been tested on 4 classifiers which are Linear Discriminant Analysis, Multinomial Naive Bayes, Logistic Regression and Support Vector Machines. Over 90% accuracy has been achieved by this system in Real-Time recognizing variable duration actions performed by the subject. The performance of the system is also shown to be robust to camera failures.

Human Activity Recognition (HAR) is the automatic detection and understanding of human motion behavior based on data extracted from video camera, ambient sensors or wearable sensors which particularly has recently attracted increased attention from both researchers and industry. However, for running practical HAR systems on wearable devices, there are some requirements such as design and development of small, lightweight, powerful, and low-cost smart sensors. In this context, data must be continuously collected, and edge computing is a viable solution, which is an energy-efficient technique, offering real-time response and privacy requirements for HAR applications. Rather than sending data to the cloud, edge computing is a local process that minimizes the data transmission time and responds with low latency. Recently, HAR system designers have adopted deep learning techniques inspired by their outstanding performance in many application areas and achieved relevant gain in activity recognition performance, however, these techniques were not demonstrated suitable for running on resource constrained devices. Thus, designing energy-efficient deep learning models is critical for realizing efficient HAR for mobile applications. In this work, we present a lightweight framework for the deployment of low-power but accurate HAR systems for these devices. We also implement and run the system in a microcontroller and analyze computational cost and energy consumption, and how different system configurations and deep learning model complexity influence on that.

The paper presents a novel lightweight framework for real-time human detection, tracking, and action recognition in low-light environments, addressing critical challenges of poor visibility and noise in surveillance and autonomous systems. Key problems tackled are insufficient image contrast, real-time detection misses, high computational load, and class imbalance. We propose: Zero-Reference Deep Curve Estimation (Zero-DCE) for dark region contrast enhancement; an optimized YOLOv5 detector with cascaded 3×3 max-pooling stacks; a core Epsilon sampling strategy ensuring temporal diversity and computational efficiency by strategically selecting frames to avoid omitting semantically critical content; a ResNet-34-based R(2+1)D network combined with a Transformer-style BERT module for robust action recognition under occlusion/low contrast; and a “deep compression” pipeline. Focused loss and data augmentation mitigate class imbalance. Evaluated on low-light datasets (ARID, HMDB51, synthetic HMDB51-dark), our framework achieves state-of-the-art performance. It reduces model size by 50% and inference latency by 28% while maintaining near-original accuracy.

In order to study the action recognition, tracking, and optimization of the training process based on the support vector regression model, a method of human action recognition based on support vector machine optimization is proposed. This method uses the improved strategy of support vector machine to realize the action recognition through the human action recognition based on the optimization of the vector machine. During the recognition, the DAG SVM strategy is improved according to the recognition accuracy of the classifier, and when outputting the result, output the recognition result and the corresponding confidence level, and use the confidence level to process the recognition result. Finally, through the experimental results, it is realized that the recognition rate based on support vector optimization is 98.7%, indicating that this method is effective and can improve the accuracy and efficiency of human body action recognition.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.

The article considers the problem of undefined complex classes activity recognition. The human activity recognition relevance in the field of production and services is presented. The classification of physical activity types is described. The possibilities of using methods based on data and knowledge for human activity recognition are considered. An analysis of zero-shot learning is given. A functional model of the method for undefined complex human activity recognition is proposed. The datasets are described. The HH101 and HH105 datasets obtained from CASAS smart homes were used to conduct experimental studies. An experimental study of the developed method for human activity recognition is carried out.

A framework for human action modeling and recognition in continuous action sequences is proposed. A star figure enclosed by a bounding convex polygon is used to effectively represent the extremities of the silhouette of a human body. Thus, human actions are recorded as a sequence of the star figure's parameters, which is then used for action modeling. To model human actions in a compact manner while characterizing their spatio-temporal patterns, star figure parameters are represented by a 2-D feature map, which is used and regarded as a spatio-temporal template. Experiments to evaluate the performance of the proposed framework show that it can recognize human actions in an efficient and effective manner.