Deep Learning and Attention-Based Methods for Human Activity Recognition and Anticipation: A Comprehensive Review

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

The article considers the problem of undefined complex classes activity recognition. The human activity recognition relevance in the field of production and services is presented. The classification of physical activity types is described. The possibilities of using methods based on data and knowledge for human activity recognition are considered. An analysis of zero-shot learning is given. A functional model of the method for undefined complex human activity recognition is proposed. The datasets are described. The HH101 and HH105 datasets obtained from CASAS smart homes were used to conduct experimental studies. An experimental study of the developed method for human activity recognition is carried out.

Human activity recognition (HAR) has been increasingly used in medical care, behavior analysis, and entertainment industry to improve the experience of users. Most of the existing works use fixed models to identify various activities. However, they do not adapt well to the dynamic nature of human activities. We investigated the activity recognition with postural transition awareness. The inertial sensor data was processed by filters and we used both time domain and frequency domain of the signals to extract the feature set. For the corresponding posture classification, three feature selection algorithms were considered to select 585 features to obtain the optimal feature subset for the posture classification. And We adopted three classifiers (support vector machine, decision tree, and random forest) for comparative analysis. After experiments, the support vector machine gave better classification results than other two methods. By using the support vector machine, we could achieve up to 98% accuracy in the Multi-class classification. Finally, the results were verified by probability estimation.

Batteryless or so called passive wearables are providing new and innovative methods for human activity recognition (HAR), especially in healthcare applications for older people. Passive sensors are low cost, lightweight, unobtrusive and desirably disposable; attractive attributes for healthcare applications in hospitals and nursing homes. Despite the compelling propositions for sensing applications, the data streams from these sensors are characterised by high sparsity---the time intervals between sensor readings are irregular while the number of readings per unit time are often limited. In this paper, we rigorously explore the problem of learning activity recognition models from temporally sparse data. We describe how to learn directly from sparse data using a deep learning paradigm in an end-to-end manner. We demonstrate significant classification performance improvements on real-world passive sensor datasets from older people over the state-of-the-art deep learning human activity recognition models. Further, we provide insights into the model's behaviour through complementary experiments on a benchmark dataset and visualisation of the learned activity feature spaces.

Unsupervised learning for human activity recognition using smartphone sensors

This paper presents some more recent developments in the research area of human activity recognition. In many cases, human activity recognition can be achieved with the support of body sensors that deliver information about the motion of a person or about acceleration and position. Such data can contribute substantially to the accuracy of the results, however wearing extra equipment is not a very popular practice for most human users and therefore camera-based methods are mostly preferred in today’s systems, which is also a result of the tremendous progress achieved in recent years in pattern recognition and machine learning. Therefore, also this paper concentrates on camera-based methods for human activity recognition. Activity recognition has a large variety of sub-research areas and has many exciting application areas, such as e.g. interaction in smart spaces, surveillance, human-robot interaction or automatic analysis of interest, emotions and human traits.

In order to study the action recognition, tracking, and optimization of the training process based on the support vector regression model, a method of human action recognition based on support vector machine optimization is proposed. This method uses the improved strategy of support vector machine to realize the action recognition through the human action recognition based on the optimization of the vector machine. During the recognition, the DAG SVM strategy is improved according to the recognition accuracy of the classifier, and when outputting the result, output the recognition result and the corresponding confidence level, and use the confidence level to process the recognition result. Finally, through the experimental results, it is realized that the recognition rate based on support vector optimization is 98.7%, indicating that this method is effective and can improve the accuracy and efficiency of human body action recognition.

Exercise is an indispensable part of people’s lives and is closely related to their health. Human Activity Recognition (HAR), which involves detects and analyzes human body activity, has become the focus of current research. Photoplethysmography (PPG) has advantages such as convenience for detection and low cost, and is widely used in wearable devices becoming an ideal choice for HAR. In this study, we used wavelet scattering transform (WST) to extract features from PPG and then performed activity recognition on it. We achieved excellent classification accuracy of 92.54% and 97.76% respectively in the experiments of three-class and four-class exercise detection. The results showed this method based on wavelet scattering transform and PPG can accurately detect exercise types and provide effective support for HAR.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

Asymmetric 3D Convolutional Neural Networks for action recognition

Human Activity Recognition (HAR), alongside Ambient Assisted Living (AAL), are integral components of smart homes, sports, surveillance, and investigation activities. To recognize daily activities, researchers are focusing on lightweight, cost-effective, wearable sensor-based technologies as traditional vision-based technologies lack elderly privacy, a fundamental right of every human. However, it is challenging to extract potential features from 1D multi-sensor data. Thus, this research focuses on extracting distinguishable patterns and deep features from spectral images by time-frequency-domain analysis of 1D multi-sensor data. Wearable sensor data, particularly accelerator and gyroscope data, act as input signals of different daily activities, and provide potential information using time-frequency analysis. This potential time series information is mapped into spectral images through a process called use of 'scalograms', derived from the continuous wavelet transform. The deep activity features are extracted from the activity image using deep learning models such as CNN, MobileNetV3, ResNet, and GoogleNet and subsequently classified using a conventional classifier. To validate the proposed model, SisFall and PAMAP2 benchmark datasets are used. Based on the experimental results, this proposed model shows the optimal performance for activity recognition obtaining an accuracy of 98.4% for SisFall and 98.1% for PAMAP2, using Morlet as the mother wavelet with ResNet-101 and a softmax classifier, and outperforms state-of-the-art algorithms.

In this paper, we make full use of the complementarity between low-rank feature and part-based feature and present a novel method extracting discriminative parts with flexible number from low-rank features for action recognition. The proposed method avoids some intermediate processing steps (e.g., actor segmentation, body tracking) required by many traditional methods and can greatly avoid memorizing background information suffered by traditional part-based methods. In addition, traditional part-based methods usually set a fixed and identical number of discriminative parts for all action categories neglecting the differences of recognizing complexity among different action categories. On the contrary, we automatically extract discriminative parts with flexible number for each action category by introducing group sparse regularizer into our model, which is more reasonable and effective. In our method, we first extract low-rank features of all action sequences and transform them into corresponding low-rank images. Then, we densely sample each low-rank image into a large number of parts in multi-scale and represent each part into a feature vector. Afterward, our model automatically learn a set of discriminative part detectors with flexible number for each action category. We further define new similarity constraints to force the responses of detected parts from the same class more similar and consistent and that from different class more different. Finally, we define a corresponding recognition criterion to perform final action recognition. The efficacy of the proposed method is verified on three public datasets, and experimental results have shown the promising results of our method for human action recognition.

This paper proposes a human activity recognition (HAR) method for frequency-modulated continuous wave (FMCW) radar sensors. The method utilizes a multi-domain feature attention fusion network (MFAFN) model that addresses the limitation of relying on a single range or velocity feature to describe human activity. Specifically, the network fuses time-Doppler (TD) and time-range (TR) maps of human activities, resulting in a more comprehensive representation of the activities being performed. In the feature fusion phase, the multi-feature attention fusion module (MAFM) combines features of different depth levels by introducing a channel attention mechanism. Additionally, a multi-classification focus loss (MFL) function is applied to classify confusable samples. The experimental results demonstrate that the proposed method achieves 97.58% recognition accuracy on the dataset provided by the University of Glasgow, UK. Compared to existing HAR methods for the same dataset, the proposed method showed an improvement of about 0.9–5.5%, especially in the classification of confusable activities, showing an improvement of up to 18.33%.

We propose a hierarchical method for human detection and activity recognition in MPEG sequences. The algorithm consists of three stages at different resolution levels. The first step is based on the principal component analysis of MPEG motion vectors of macroblocks grouped according to velocity, distance and human body proportions. This step reduces the complexity and amount of processing data. The DC DCT components of luminance and chrominance are the input for the second step, to be matched to activity templates and a human skin template. A more detailed analysis of the uncompressed regions extracted in previous steps is done at the last step via model-based segmentation and graph matching. This hierarchical scheme enables working at different levels, from low complexity to low false rates. It is important and interesting to realize that significant information can be obtained from the compressed domain in order to connect to high level semantics.

Action recognition with motion map 3D network