Deep Learning for Video Based Human Activity Recognition: Review and Recent Developments

Human activity recognition (HAR) identifies people’s motions and actions in daily life. HAR research has grown with the popularity of internet-connected, wearable sensors that capture human movement data to detect activities. Recent deep learning advances have enabled more HAR research and applications using data from wearable devices. However, prior HAR research often focused on a few sensor locations on the body. Recognizing real-world activities poses challenges when device positioning is uncontrolled or initial user training data are unavailable. This research analyzes the feasibility of deep learning models for both position-dependent and position-independent HAR. We introduce an advanced residual deep learning model called Att-ResBiGRU, which excels at accurate position-dependent HAR and delivers excellent performance for position-independent HAR. We evaluate this model using three public HAR datasets: Opportunity, PAMAP2, and REALWORLD16. Comparisons are made to previously published deep learning architectures for addressing HAR challenges. The proposed Att-ResBiGRU model outperforms existing techniques in accuracy, cross-entropy loss, and F1-score across all three datasets. We assess the model using k-fold cross-validation. The Att-ResBiGRU achieves F1-scores of 86.69%, 96.23%, and 96.44% on the PAMAP2, REALWORLD16, and Opportunity datasets, surpassing state-of-the-art models across all datasets. Our experiments and analysis demonstrate the exceptional performance of the Att-ResBiGRU model for HAR applications.

A Lightweight Deep Learning Model for Human Activity Recognition on Edge Devices

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Recognizing human activity is a challenging task in many applications like visual surveillance, Human–computer interaction, etc. Numerous machine learning (ML) and deep learning (DL) approaches are used to emulate the human behavior of everyday life, as they provide better recognition by learning complex features. This paper proposes an innovative human activity recognition framework with the following steps. Initially, the given video is converted into a set of frames and then applies median filtering to remove the noise for efficient recognition. Subsequently, the pixel-relatedness retrieval-fuzzy clustering means algorithm (PRR-FCM) approach is proposed for image segmentation, in which a pixel-relatedness retrieval strategy is used that considers only the proximity-related pixel for efficient segmentation. The retrieval of features is an important aspect, thereby this paper goes with the extraction of texture features, relative standard deviation induced in multi-texon (RSIMT), in which relative standard deviation is induced for interchanging the center pixel value. Also, the other features like the hierarchy of the skeleton and local binary pattern (LBP) are extracted from the segmented image. Further, the retrieved features are fed to the hybrid model that combines models like modified deep Maxout (MDM) and convolution neural network (CNN) models to recognize human activity efficiently. The modified exponential linear unit (ELU) activation function and Log loss function are the enhancing formulation of the MDM model to determine accurate recognition results.

Unsupervised deep domain adaptation algorithm for video based human activity recognition via recurrent neural networks

Recently, a significant amount of literature concerning machine learning techniques has focused on automatic recognition of activities performed by people. The main reason for this considerable interest is the increasing availability of devices able to acquire signals which, if properly processed, can provide information about human activities of daily living (ADL). The recognition of human activities is generally performed by machine learning techniques that process signals from wearable sensors and/or cameras appropriately arranged in the environment. Whatever the type of sensor, activities performed by human beings have a strong subjective characteristic that is related to different factors, such as age, gender, weight, height, physical abilities, and lifestyle. Personalization models have been studied to take into account these subjective factors and it has been demonstrated that using these models, the accuracy of machine learning algorithms can be improved. In this work we focus on the recognition of human activities using signals acquired by the accelerometer embedded in a smartphone. The contributions of this research are mainly three. A first contribution is the definition of a clear validation model that takes into account the problem of personalization and which thus makes it possible to objectively evaluate the performances of machine learning algorithms. A second contribution is the evaluation, on three different public datasets, of a personalization model which considers two aspects: the similarity between people related to physical aspects (age, weight, and height) and similarity related to intrinsic characteristics of the signals produced by these people when performing activities. A third and last contribution is the development of a personalization model that considers both the physical and signal similarities. The experiments show that the employment of personalization models improves, on average, the accuracy, thus confirming the soundness of the approach and paving the way for future investigations on this topic.

Artificial intelligence (AI) technology has become integral in the realm of medicine and healthcare, particularly in human activity recognition (HAR) applications such as fitness and rehabilitation tracking. This study introd... | Find, read and cite all the research you need on Tech Science Press

Human Activity Recognition (HAR) by employing smart home sensors is a topic that is undergoing intense research in the area of ambient-supported living and the basis of ubiquitous computing in smart environments. Recently, HAR majorly utilizes Deep Learning (DL) method since they employ representation learning methods that could automatically produce maximum factors from raw input datasets resulting from the sensor in the absence of human interference and could detect Hidden Layer (HL) patterns in a dataset. The study presents an Improved Manta-Ray Foraging Optimization with Deep Autoencoder (IMRFO-DAE) model for HAR. The presented algorithm majorly recognizes the diverse types of human activities. The presented IMRFO-DAE model pre-processes the human activity data via a standardization approach to obtain this. Next, the IMRFO-DAE method uses the DAE model to perform the Activity Recognition (AR) process. To advance the AR accomplishment of the DAE model, the IMRFO method is applied as a hyperparameter optimizer. Moreover, the IMRFO model is derived by modifying the initialization of the MRFO model by implementing the chaotic concept. An extensive range of simulations was performed to depict the enhanced efficiency of the IMRFO-DAE approach. The simulation results assured the improved outcomes of the IMRFO-DAE approach compared to existing approaches.

Human activity recognition (HAR) is one of the leading research fields in ubiquitous computing working to integrate seamless technologies in our daily lives. The researches in this field focus on the technological advancement of fine activity recognition through using minimal technological deployments along with consideration of human factors. The domain specific knowledge of human activity recognition has shed significant importance on the activity data processing, due to the innate random nature of the activity data. The necessity for improved prediction accuracy of activities within limited infrastructural support and cost is necessary for the research works to be conducted at large. The data set utilized in this study is modified through tree-based feature-engineering method, which has significantly impacted the intrinsic time-series pattern associated with the original data set acquired from UCI machine learning repository. In this paper, the primary focus lies on improving classification accuracy of the human activities included in the data set through utilizing deep learning classifier models. The particular modified pattern of the data devoid of time-series pattern to major extent has resulted in lower classification accuracy by benchmark time-series classification models namely Keras-LSTM and RNN-LSTM. CNN based networks have demonstrated consisted improved training performance on the data set. The state-of-the-art classifier ResNet introduced by Microsoft Research in 2015 has achieved 99.9% classification accuracy on the data set of this study. Another important aim of this study is to infer the risk attached with adversarial attack on machine learning models. The high vulnerability of machine learning models in face of adversarial attacks has emerged as security concern of the deployment pipeline of machine learning models. To the author’s best knowledge, there has been no prior study to find out the high insecurity associated with the human activity recognition data set and concerned benchmark models. In this study, two prevalent adversarial approaches namely Fast Gradient Sign Method (FGSM) and Basic Iterative Method (BIM) have been employed to carry out non-targeted attack on the pre-trained ResNet model. The accuracy loss by the adversarial attack over a small range of added perturbation is tremendous, nearly 96% accuracy is diminished in the process. Finally, we have demonstrated the comparative performance by FGSM and BIM on reducing the classification accuracy of pre-trained model. The findings shed light on the nature of the data influencing the classification accuracy and the resiliency of state-of-the-art classifier model on facing adversarial attacks.

Recognizing human activity from highly sparse body sensor data is becoming an important problem in Internet of Medical Things (IoMT) industry. As IoMT currently uses batteryless or passive wearable body sensors for activity recognition, the data signals of these sensors have a high sparsity, which means that the time intervals of sensors’ readings are irregular and the number of sensors’ readings in time unit is frequently limited. Therefore, learning activity recognition models from temporally sparse data is challenging. Traditional machine learning techniques cannot be applicable in this scenario due to their focus on a single sensing modality requiring regular sampling rate. In this work, we propose an effective end-to-end deep neural network (DNN) model to recognize human activities from temporally sparse data signals of passive wearable sensors that improve the accuracy rate of human activity recognition. A dropout technique is used in the developed model to deal with sparsity problem and avoiding overfitting problem. In addition, optimization of the proposed DNN model was performed by evaluating a different number of hidden layers. Various experiments were conducted on a public clinical room dataset of sparse data signals to compare the performance of the proposed DNN model with the conventional and other deep learning approaches. The experimental results demonstrate that the proposed DNN model outperforms the existing state-of-the-art methods in terms of lower inference delay and activity recognition accuracy.

Human activity recognition (HAR) is one of the important research areas in pervasive computing. Among HAR, sensor-based activity recognition refers to acquiring a high-level knowledge about human activities from readings of many low-level sensor. In recent years, although traditional methods of deep learning (DL) have been widely used for sensor-based HAR with some good performance, they still face such challenges as feature extraction and characterization, continuous action segmentation in dealing with time series problems. In this study, a multichannel fusion model is proposed by the idea of dividing. In this proposed architecture, a multichannel convolutional neural network (CNN) is used to enhance the ability to extract features at different scales, and then the fused features are fed into gated recurrent unit (GRU) for feature labeling and enhanced feature representation, through the learning of temporal relationships. Finally, the multichannel CNN-GRU model is designed using global average pooling (GAP) to connect the feature maps with the final classification. The model performance was conducted on three benchmark datasets of WISDM, UCI-HAR, and PAMAP2 with the accuracy of 96.41%, 96.67%, and 96.25% respectively. The results show that the proposed model demonstrates better activity detection capability than some of the reported results.

With the rapid development of the computer and sensor field, inertial sensor data have been widely used in human activity recognition. At present, most relevant studies divide human activities into basic actions and transitional actions, in which basic actions are classified by unified features, while transitional actions usually use context information to determine the category. For the existing single method that cannot well realize human activity recognition, this paper proposes a human activity classification and recognition model based on smartphone inertial sensor data. The model fully considers the feature differences of different properties of actions, uses a fixed sliding window to segment the human activity data of inertial sensors with different attributes and, finally, extracts the features and recognizes them on different classifiers. The experimental results show that dynamic and transitional actions could obtain the best recognition performance on support vector machines, while static actions could obtain better classification effects on ensemble classifiers; as for feature selection, the frequency-domain feature used in dynamic action had a high recognition rate, up to 99.35%. When time-domain features were used for static and transitional actions, higher recognition rates were obtained, 98.40% and 91.98%, respectively.

Deep Learning methods have become very attractive in the wider, wearables-based human activity recognition (HAR) research community. The majority of models are based on either convolutional or explicitly temporal models, or combinations of both. In this paper we introduce attention models into HAR research as a data driven approach for exploring relevant temporal context. Attention models learn a set of weights over input data, which we leverage to weight the temporal context being considered to model each sensor reading. We construct attention models for HAR by adding attention layers to a state-of-the-art deep learning HAR model (DeepConvLSTM) and evaluate our approach on benchmark datasets achieving significant increase in performance. Finally, we visualize the learned weights to better understand what constitutes relevant temporal context.

Human activity recognition (HAR) is a main research field of context-aware computing; the performance of HAR mainly depends on the feature extraction method and classification algorithm. Extreme learning machine (ELM) is a single hidden layer neural network, which has better classification and generalization ability. However, ELM is not suitable for feature extraction. Deep learning is a hot research field as it can automatically extract significant features from raw data. In this paper, we propose an approach: an ELM-based deep model, which combined convolutional neural network (CNN), multilayer ELM (ML-ELM) as feature extractor, and used kernel ELM (KELM) as classifier. We used CNN and ML-ELM to extract significant features, and used KELM to achieve stable performance. The performance of proposed approach is validated on two public HAR datasets, and the experimental results show that the proposed approach is effective.

Smartphone sensor-based Human Activity Recognition (HAR) is a significant facilitator for many real-world applications such as smart homes, personal healthcare, and illness detection because it has the potential to discover activity patterns in daily life. While solutions for detecting sequential activities like walking or running are already matured, however recognizing additional forms of complex, concurrent, interleaved, and heterogeneous human activities remains a research challenge. Furthermore, several solutions have been developed for specific types of activity complexity. According to the best of our knowledge, there is no common solution for detecting all types of action-based activities. In this paper, we address the problem of recognizing all types of action-based human activities i.e. sequential, complex, concurrent, interleaved, and heterogeneous activities. The intention behind this paper is to present a generic methodology for HAR by leveraging transfer learning and evaluating the performance on a plethora of HAR datasets. In this regard, we have proposed a model called RecurrentHAR, which stands for Recurrent layers for HAR, a novel adversarial knowledge transfer approach that uses the Gated Recurrent Units (GRUs) architecture for smartphone sensor-based HAR. The performance of the proposed model is evaluated through extensive experiments using three public datasets namely WISDM (sequential activities), PAMAP2 (complex, concurrent, and interleaved activities), and KU-HAR (Heterogeneous activities), and the RecurrentHAR model has outperformed compared with other state-of-the-art approaches. The proposed model achieved 96.26%, 94.77%, and 98.98% of F1-score for WISDM, PAMAP2, and KU-HAR datasets respectively. The experimental results provide insight into the applicability of the proposed model and future research possibilities.