A Vision-based Human Action Recognition System for Moving Cameras Through Deep Learning

This paper presents a vision-based human action recognition system to support human interaction for companion robots. The system is divided into three parts: motion map construction, feature extraction, and human action classification. First, the Kinect 2.0 captures depth images and color images simultaneously using its depth sensor and RGB camera. Second, the information in the depth images and the color images are respectively used to three depth motion maps and a color motion map. These are then combined into one image to calculate the corresponding histogram of oriented gradient (HOG) features. Finally, a support vector machine (SVM) recognizes these HOG features as human actions. The proposed system can recognize eight kinds of human actions, wave left hand, wave right hand, holding left hand, holding right hand, hugging, bowing, walking, and punching. Three databases were used to test the proposed system: Database1 includes videos of adult actions, Database2 includes videos of child actions, and Database3 includes human action videos taken with a moving camera. The recognition accuracy rates of the three tests were 88.7%, 74.37%, and 51.25%, respectively. The experimental results show that the proposed system is efficient and robust.

This research proposes a vision-based online human action recognition system. This system uses deep learning methods to recognise human action under moving camera circumstances. The proposed system consists of five stages: human detection, human tracking, feature extraction, action classification and fusion. The system uses three kinds of input information: colour intensity, short-term dynamic information and skeletal joints. In the human detection stage, a two-dimensional (2D) pose estimator method is used to detect a human. In the human tracking stage, a deep SORT tracking method is used to track the human. In the feature extraction stage, three kinds of features, spatial, temporal and structural, are extracted to analyse human actions. In the action classification stage, three kinds of features of human actions are respectively classified by three kinds of long short-term memory (LSTM) classifiers. In the fusion stage, a fusion method is used to leverage the three output results from the LSTM classifiers. This study constructs a computer vision and image understanding (CVIU) Moving Camera Human Action dataset (CVIU dataset), containing 3,646 human action sequences, including 11 types of single human actions and 5 types of interactive human actions. This dataset was used to train and evaluate the proposed system. Experimental results showed that the recognition rates of spatial features, temporal features and structural features were 96.64%, 81.87% and 68.10%, respectively. Finally, the fusion result of human action recognition for indoor smart mobile robots in this study was 96.84%.

Recognizing Human action in video is an very active research topic. There are a growing variety of human action datasets with different video length, different practitioners. Make human action recognition becomes a very difficult topic. A majority researchers solve the problem by extracting key frames from the videos. Most paper use feature Clustering methods to extract key frames in videos. On one hand, the large variety of visual content in videos make handcraft feature isn't effective enough, since there are no fixed descriptors can describe all video cases. On the other hand, traditional clustering algorithms are easily influenced by the choice of initial clustering centers. An Unsupervised feature learning and clustering method for key frame extraction is proposed in this paper, which can be used for human action recognition. Stacked auto-encoder(SAE) is trained using videos from 10 different human actions, SAE is used as a feature extractor to learn features representing human actions. Affinity Propagation Clustering algorithm is used to select key frames from video sequences. We use a variety of videos to do the experiments. Experiments demonstrate that our method can be effectively summarizing video shots considering different human actions.

Human Activity Recognition is one of the active research areas in computer vision for various contexts like security surveillance, healthcare and human computer interaction. In this paper, a total of thirty-two recent research papers on sensing technologies used in HAR are reviewed. The review covers three area of sensing technologies namely RGB cameras, depth sensors and wearable devices. It also discusses on the pros and cons of the mentioned sensing technologies. The findings showed that RGB cameras have lower popularity when compared to depth sensors and wearable devices in HAR research.

Discriminative two-level feature selection for realistic human action recognition

Human action recognition from 2D videos is a demanding area due to its broad applications. Many methods have been proposed by the researchers for recognizing human actions. The improved accuracy in identifying human actions is desirable. This paper presents an improved method of human action recognition using support vector machine (SVM) classifier. This paper proposes a novel feature descriptor constructed by fusing the various investigated features. The handcrafted features such as scale invariant feature transform (SIFT) features, speed up robust features (SURF), histogram of oriented gradient (HOG) features and local binary pattern (LBP) features are obtained on online 2D action videos. The proposed method is tested on different action datasets having both static and dynamically varying backgrounds. The proposed method achieves shows best recognition rates on both static and dynamically varying backgrounds. The datasets considered for the experimentation are KTH, Weizmann, UCF101, UCF sports actions, MSR action and HMDB51.The performance of the proposed feature fusion model with SVM classifier is compared with the individual features with SVM. The fusion method showed best results. The efficiency of the classifier is also tested by comparing with the other state of the art classifiers such as k-nearest neighbors (KNN), artificial neural network (ANN) and Adaboost classifier. The method achieved an average of 94.41% recognition rate.

With the rise of computer vision, it is becoming more and more important to accurately recognize and evaluate human actions. However, the complexity, intraclass differences, and viewing angle changes of human actions significantly impact the accuracy of identification human actions. This paper proposed an action recognition and evaluation method based on skeleton information extraction. Briefly, We use Lightweight OpenPose to extract the key points of human skeleton and perform processing work, including video data cutting, deleting some key points, supplementing missing key points, filtering processing, feature extraction, etc. Through an in-depth exploration of related theoretical technologies, we proposed a model for recognition and evaluation of human table tennis actions with an ordinary camera. The support vector machine algorithm (SVM) classification model is used to identify table tennis actions in real-time. Then the dynamic time regularization (DTW) algorithm calculates the similarity of each human skeleton key point in the action sequence. The low-scoring bone key points are marked to evaluate the human table tennis action in real time. The results show that a recognition rate of more than 95% is achieved in the test set, which proves the method's effectiveness. In addition, we compared the results with previous work using inertial sensors for action recognition, which shows our method can preserve the same accuracy with a much lower cost of implementation.

Recognizing different human actions from still images or videos is an important research area in the computer vision and artificial intelligence domains. It represents a key step for a wide range of applications including: human-computer interaction, ambient assisted living, intelligent driving and video surveillance. However, unless the many research works being involved, there are still many challenges ahead including: the high changes in human body shapes, clothing and viewpoint changes and the conditions of system acquisition (illumination variations, occlusions, etc.). With the emergence of new deep learning techniques, many approaches are recently proposed for Human Action Recognition (HAR). Compared with conventional machine learning methods, deep learning techniques have more powerful learning ability. The most wide-spread deep learning approach is the Convolutional Neural Network (CNN/ConvNets). It has shown remarkable achievements due to its precision and robustness. As a branch of neural network, 3D CNN is a relatively new technique in the field of deep learning. In this paper, we propose a HAR approach based on a 3D CNN model. We apply the developed model to recognize human actions of KTH and J-HMDB datasets, and we achieve state of the art performance in comparison to baseline methods.

In order to study the action recognition, tracking, and optimization of the training process based on the support vector regression model, a method of human action recognition based on support vector machine optimization is proposed. This method uses the improved strategy of support vector machine to realize the action recognition through the human action recognition based on the optimization of the vector machine. During the recognition, the DAG SVM strategy is improved according to the recognition accuracy of the classifier, and when outputting the result, output the recognition result and the corresponding confidence level, and use the confidence level to process the recognition result. Finally, through the experimental results, it is realized that the recognition rate based on support vector optimization is 98.7%, indicating that this method is effective and can improve the accuracy and efficiency of human body action recognition.

Deep learning techniques are being widely applied to Human Activity Recognition (HAR). This paper describes the implementation and evaluation of a HAR system for daily life activities using the accelerometer of an iPhone 6S. This system is based on a deep neural network including convolutional layers for feature extraction from accelerations and fully-connected layers for classification. Different transformations have been applied to the acceleration signals in order to find the appropriate input data to the deep neural network. This study has used acceleration recordings from the MotionSense dataset, where 24 subjects performed 6 activities: walking downstairs, walking upstairs, sitting, standing, walking and jogging. The evaluation has been performed using a subject-wise cross-validation: recordings from the same subject do not appear in training and testing sets at the same time. The proposed system has obtained a 9% improvement in accuracy compared to the baseline system based on Support Vector Machines. The best results have been obtained using raw data as input to a deep neural network composed of two convolutional and two max-pooling layers with decreasing kernel sizes. Results suggest that using the module of the Fourier transform as inputs provides better results when classifying only between dynamic activities.

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

The recent evolution in sensor-based depth information has been developed a sounding scope to work for human activity recognition using depth image sequences. Activity recognition is having a key importance due to its advantages in several domain like surveillance systems at airport, patient monitoring system, care of elderly people, etc. This work proposed the objective and evaluations to recognise daily life human activities by spatiotemporal depth information. For this purpose, Kinect sensor is used to collect the data pertaining to a single activity performed by multiple person at a time. The spatiotemporal depth features are computed for activity recognition and support vector machine is used in classification phase. We have nine class of human actions in the database for RGB-D human activity recognition. This dataset is reconfigured from Cornell human activity and Berkeley multimodal human action databases. For multiple human action recognition, 91.38% accuracy is achieved on the synthetic dataset.

Recognizing human actions accurately from video data is a difficult task because of the variability in human motion, appearance, and context. A new approach to human action recognition is proposed in this study, which uses a combination of Variational Autoencoder (VAE) and Transformer models. The VAE with IWGECO optimization is utilized for acquiring a compact and discriminative representation of the input 2D skeletons from video frames along with its spatial relations. This encoded representation of 2D poses is then fed into a Transformer model for sequence modeling and classification, resulting in effective recognition of human actions. The proposed approach is evaluated on the MPOSE2021 benchmark dataset, outperforming existing methods (90.6%) with an average classification accuracy of 92.0%. The model performs well on real-life data too showcasing its generalizing potential. The study also addresses the domain shift problem occurring in real-world situations by using a pre-trained VAE and subsequently training the Transformer on the MPOSE2021 dataset. The results demonstrate the effectiveness of combining VAE and Transformer models for human action recognition.

In order to improve the accuracy of human action recognition in video and the computational efficiency of large data sets, an action recognition algorithm based on multiple features and modified deep learning model is proposed. First, the deep network pre-training process is used to learn and optimize the RBM parameters, and the deep belief nets (DBN) model is constructed through deep learning. Then, human 13 joint points and critical points of optical flow are automatically extracted by DBN model. Second, these more abstract and more effective human motion features are combined to represent human actions. Ultimately, the entire DBN network structure is fine-tuned by support vector machine (SVM) algorithm to classify human actions. We demonstrate that human 13 joint points and critical points of optical flow are two very effective human action characterizations, our proposed approach greatly reduces the required samples, and shortens the training time of the samples, can efficiently process large data sets and can effectively recognize novel actions. We performed experiments on the KTH data set, Weizmann data set, the ballet data set and UCF101 data set to evaluate the proposed method, the experiment results show that the average recognition accuracy is over 98%, which validates its effectiveness, and show that our results are stable, reliable, and significantly better than the results of two state-of-the-art approaches on four different data sets. So, it lays a good theoretical foundation for practical applications.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.