Action recognition by discriminative EdgeBoxes

Aiming at the personal safety problems in the complex working conditions and high-risk working environments such as high temperature and high radiation in the industrial field, a human-machine collaboration method based on the key nodes of human posture is proposed. First, compare the human pose detection performance of AlphaPose, OpenPose, and HRNet models in complex environments such as different lighting, different human poses, and occlusions, and select AlphaPose, which is suitable for real-time human-machine collaboration, as the basic model of human-machine collaboration; then, formulate standard human actions, establish the mapping relationship between human actions and robot actions; then, use the spatial position data of key human nodes to establish a human action recognition algorithm to calculate the similarity between the action to be detected and the standard human action to identify the human action; finally, real-time control according to the mapping relationship the movement of the robot realizes the collaboration between human and robot. Experimental results show that the average action recognition accuracy of the method proposed in this paper can reach 91.2%, the average action recognition speed can reach 67ms, the average action recognition accuracy under 0-60 Lux illumination intensity can reach 90.4%, and the average recognition accuracy under partial occlusion conditions up to 87.6%.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Human activity recognition is a very active research on pervasive computing and mobile health application. Many human activity systems based on inertial measurement unit (IMU) sensor data were proposed in the past few years. These systems mainly use IMU sensor placed on he torso and limbs to collect data and utilize supervised machine learning algorithms on sensor data. One main issue of these systems is that wearing multiple on-body IMU sensors may bring inconvenience to users' daily life. The other issue of these exiting methods is that an activity recognition model that is trained on a specific subject does not work well when being applied to predict another subject's activities since IMU activity data always carry information that is specific to the human subject who conducts the activities. In our work, inspired by the principle of domain adaption, we proposed a new deep-learning activity recognition model based on an adversarial network which can remove the subject-specific information within the IMU activity data and extract subject-independent features shared by the data collected on different subjects. We also for the first time use data collected from insole based IMU sensors on 8 participants for 5 common activities to build a new real world human activity dataset which can minimize the inconvenience for users to wear. We conducted experiments with our new real-world dataset. Results show that our subject independent activity recognition model outperforms state-of-art supervised learning techniques and eliminates the effects of individual differences between subjects successfully. The average recognition accuracy under the leave-one-out (L1O) condition achieves 99.0% which is higher than the performance of traditional human activity recognition system based on CNNs.

Action recognition with appearance–motion features and fast search trees

With the development of movement sensors, activity recognition becomes more and more popular. Compared with daily-life activity recognition, physical violence detection is more meaningful and valuable. This paper proposes a physical violence detecting method. Movement data of acceleration and gyro are gathered by role playing of physical violence and daily-life activities. Time domain features and frequency domain ones are extracted and filtered to discribe the differences between physical violence and daily-life activities. A specific BPNN trained with the L-M method works as the classifier. Altogether 9 kinds of activities are involved. For 9-class classification, the average recognition accuracy is 67.0%, whereas for 2-class classification, i.e. activities are classified as violence or daily-life activity, the average recognition accuracy reaches 83.7%.

In order to improve the accuracy of human action recognition in video and the computational efficiency of large data sets, an action recognition algorithm based on multiple features and modified deep learning model is proposed. First, the deep network pre-training process is used to learn and optimize the RBM parameters, and the deep belief nets (DBN) model is constructed through deep learning. Then, human 13 joint points and critical points of optical flow are automatically extracted by DBN model. Second, these more abstract and more effective human motion features are combined to represent human actions. Ultimately, the entire DBN network structure is fine-tuned by support vector machine (SVM) algorithm to classify human actions. We demonstrate that human 13 joint points and critical points of optical flow are two very effective human action characterizations, our proposed approach greatly reduces the required samples, and shortens the training time of the samples, can efficiently process large data sets and can effectively recognize novel actions. We performed experiments on the KTH data set, Weizmann data set, the ballet data set and UCF101 data set to evaluate the proposed method, the experiment results show that the average recognition accuracy is over 98%, which validates its effectiveness, and show that our results are stable, reliable, and significantly better than the results of two state-of-the-art approaches on four different data sets. So, it lays a good theoretical foundation for practical applications.

Recent human action recognition models mainly focus on exploiting human features, such as pose or skeleton features. However, most of these methods do not pay enough attention to action-related backgrounds. In this work we propose a novel multi-stream features fusion framework based on discriminative patch exploiting. Unlike existing part-based or attention-based multi-stream methods, our work improves the recognition accuracy by 1) Paying more attention to exploiting of global and local discriminative patches, which include not only the acting human but also the interactive scenes. 2) Proposing an effective multi-stream feature pooling and fusion mechanism: 2D and 3D features from RGB frames and discriminative patches are combined to enhance spatial-temporal feature representation ability. Our framework is evaluated on two widely used video action benchmarks, where it outperforms other state-of-the-art methods: the accuracy up to 87.8% at HMDB51, and 98.8% at UCF101.

<span lang="EN-US">In this paper, an experiment has been carried out based on a simple k-nearest neighbor (kNN) classifier to investigate the capabilities of three extracted facial features for the better recognition of facial emotions. The feature extraction techniques used are histogram of oriented gradient (HOG), Gabor, and local binary pattern (LBP). A comparison has been made using performance indices such as average recognition accuracy, overall recognition accuracy, precision, recall, kappa coefficient, and computation time. Two databases, i.e., Cohn-Kanade (CK+) and Japanese female facial expression (JAFFE) have been used here. Different training to testing data division ratios is explored to find out the best one from the performance point of view of the three extracted features, Gabor produced 94.8%, which is the best among all in terms of average accuracy though the computational time required is the highest. LBP showed 88.2% average accuracy with a computational time less than that of Gabor while HOG showed minimum average accuracy of 55.2% with the lowest computation time.</span>

Human Activity Recognition (HAR) holds significant importance in health and human-machine interaction. However, recognizing actions from 2D information faces challenges like occlusion, illumination variation, cluttered backgrounds, and view invariance. These hurdles are particularly pronounced in indoor patient monitoring settings due to fluctuating lighting conditions and cluttered backgrounds, which compromise the accuracy of activity recognition systems. A new architecture named illuminationRevive has been proposed to tackle this issue. Which utilizes an encoder-decoder convolutional neural network and image post-processing blocks to enhance the image's visual appearance. A new dataset comprising seven indoor physical activities has been proposed, created with contributions from thirty individuals aged 20 to 45. A hybrid fusion architecture is proposed to classify activities, integrating motion sequence information and body joint features. The proposed classification model incorporates generated Skeleton Motion History Images (SMHIs), collected human joint motion features from video frames, and novel kinematic and geometric features within window frames as inputs. The model can extract spatial and temporal feature vectors by integrating ResNet50-ViT (Residual Network-50 layers, Vision Transformer) and CNN-BiLSTM (Convolutional Neural Network-Bidirectional Long Short-Term Memory) layers. The suggested classification model was evaluated alongside state-of-the-art models using the LNMIIT-SMD (The LNM Institute of Information Technology-Skelton Motion Dataset) and established NTU-RGBD (Nanyang Technological University's Red Blue Green and Depth information) dataset. The evaluation aimed to assess the effectiveness of the proposed classification model architecture. Results demonstrate the superiority of the proposed model, achieving impressive accuracies of 98.21% on real-time data, 98.45% on the proposed dataset, and 97.12% on the NTU-RGBD dataset.

Action recognition on large categories of unconstrained videos taken from the web is a very challenging problem compared to datasets like KTH (6 actions), IXMAS (13 actions), and Weizmann (10 actions). Challenges like camera motion, different viewpoints, large interclass variations, cluttered background, occlusions, bad illumination conditions, and poor quality of web videos cause the majority of the state-of-the-art action recognition approaches to fail. Also, an increased number of categories and the inclusion of actions with high confusion add to the challenges. In this paper, we propose using the scene context information obtained from moving and stationary pixels in the key frames, in conjunction with motion features, to solve the action recognition problem on a large (50 actions) dataset with videos from the web. We perform a combination of early and late fusion on multiple features to handle the very large number of categories. We demonstrate that scene context is a very important feature to perform action recognition on very large datasets. The proposed method does not require any kind of video stabilization, person detection, or tracking and pruning of features. Our approach gives good performance on a large number of action categories; it has been tested on the UCF50 dataset with 50 action categories, which is an extension of the UCF YouTube Action (UCF11) dataset containing 11 action categories. We also tested our approach on the KTH and HMDB51 datasets for comparison.

The main focus of our work is to create a database for action recognition of unattended infants in uncontrolled environment with wide variations in surroundings, lighting, interactions with objects, camera motions, etc., Action recognition of infants is emerging as an important and technically challenging computer vision problem as compared with adult action recognition because of their physical appearance. Most of the previous action recognition techniques have focused on the recognition of action captured under a controlled environment in a standard laboratory setting. In this study a novel database is introduced that can be used as a benchmark for surveillance parenting. This database involves nine normal and nine abnormal actions classes which consist of actions and movements of infants occurring in fairly uncontrolled conditions. This database consists of realistic user-uploaded videos which are recorded in the clustered background and different camera motion. After the collection of all videos, they are manually trimmed to form a database. To further evaluate the performance of the database, HOG features were extracted from database set and trained by different Machine Learning classifiers like Multi-class Naive Bayes, Support Vector Machine, Ensemble classifier, Discriminant analysis and Decision tree classifiers. This experimentation shows that the database is complex and robust that can serve as a base for testing action recognition algorithms.

In reality, the recognition of Human Activity (HAR) is not an easy task due to the variation in the illumination, occlusions, background clutters and camera motion. The paper presents an advanced system of spatio-temporal fusion, that is, including the application of the EfficientNet-based spatial feature extraction and temporal motion data to enhance the recognition accuracy score in low-level and dynamic conditions. It is implemented in a PyQt5 desktop case, where real time classification of images, pre-recorded videos and webcam streams are possible. The architecture entails confidence thresholding, top-k class probability visualization, and prediction history log so as to make it easier to interpret and gain user confidence. It is preprocessed using the preprocess_input pipeline supported by EfficientNet and subsequent inference of the result (fast-optimised) with the assistance of TensorFlow 2.18.0. Experimental results on datasets in experiments show the accuracy and robustness of the method is higher than the methods of the baseline, and the interactive interface allows the practical application in surveillance, sports analytics, and assistive robotics.

Real-time action recognition on smartphones and wearable cameras is a challenging task. The effect of camera motion on action recognition algorithms is not trivial. To our advantage, smartphones and wearable devices are often augmented with depth, audio, geo-location, and inertial data, which can be incorporated into action recognition framework. Our study aims to quantify the effect of ego-centric motion on standard action recognition algorithms. As a part of the study, we collected multi-sensor video dataset with seven actions from the same subjects captured in three different camera motion scenarios—minimal, pure rotational and both rotational and translational. We plan to create a multi-sensor fusion based action recognition framework to improve the recognition accuracy on smartphones and wearable cameras, which are equipped with inertial sensors. We present preliminary experiment results on our multi-sensor video dataset.

Modeling Self-Occlusions/Disocclusions in Dynamic Shape and Appearance Tracking for Obtaining Precise Shape We present a method to determine the precise shape of a dynamic object from video. This problem is fundamental to computer vision, and has a number of applications, for example, 3D video/cinema post-production, activity recognition and augmented reality. Current tracking algorithms that determine precise shape can be roughly divided into two categories: 1) Global statistics partitioning methods, where the shape of the object is determined by discriminating global image statistics, and 2) Joint shape and appearance matching methods, where a template of the object from the previous frame is matched to the next image. The former is limited in cases of complex object appearance and cluttered background, where global statistics cannot distinguish between the object and background. The latter is able to cope with complex appearance and a cluttered background, but is limited in cases of camera viewpoint change and object articulation, which induce self-occlusions and self-disocclusions of the object of interest. The purpose of this thesis is to model self-occlusion/disocclusion phenomena in a joint shape and appearance tracking framework. We derive a non-linear dynamic model of the object shape and appearance taking into account occlusion phenomena, which is then used to infer self-occlusions/disocclusions, shape and appearance of the object in a variational optimization framework. To ensure robustness to other unmodeled phenomena 5 that are present in real-video sequences, the Kalman filter is used for appearance updating. Experiments show that our method, which incorporates the modeling of self-occlusion/disocclusion, increases the accuracy of shape estimation in situations of viewpoint change and articulation, and out-performs current state-of-the-art methods for shape tracking.

The research discussed in this paper is part of a project directed at increasing productivity in construction through mitigating the risk of Musculoskeletal Disorders (MSD). Postures and activities recognition through motion capturing techniques have shown promising potential for monitoring, assessing, and reducing such risks. Current motion sensing systems require a complex whole-body senor placement to capture and recognize construction activities, which limits the practicality and requires great computational effort. This challenge can be addressed through using a machine learning approach that recognizes specific activities from human motion data. The feasibility of reducing the computational effort through using fewer sensors rather than whole-body sensor placement was assessed through a case study. Five sensors were placed in targeted motion areas. The authors propose a novel automatic model configuration process to improve recognition performance under the selected sensor placement. It is based on designing optimal combination of data segmentation window size, feature sets, and classification algorithms for a specific set of injury-prone construction activities. The proposed approach achieved an average overall recognition accuracy of 0.81 and 0.74 for two sets of activities. The recognition model operation time is also reduced to less than 0.01 s under the proposed approach. In this initial case study, the model configuration process was developed iteratively based on the output from the test case. In subsequent efforts, the authors will develop a generic activity recognition model with predefined rules and criteria. This will further accelerate and automate the model configuration process.