On attention models for human activity recognition

Human action recognition is a fundamental research problem in computer vision. The accuracy of human action recognition has important applications. In this book chapter, the authors use a YOLOv7-based model for human action recognition. To evaluate the performance of the model, the action recognition results of YOLOv7 were compared with those using CNN+LSTM, YOLOv5, and YOLOv4. Furthermore, a small human action dataset suitable for YOLO model training is designed. This data set is composed of images extracted from KTH, Weizmann, MSR data sets. In this book chapter, the authors make use of this data set to verify the experimental results. The final experimental results show that using the YOLOv7 model for human action recognition is very convenient and effective, compared with the previous YOLO model.

Human activity recognition (HAR) is a main research field of context-aware computing; the performance of HAR mainly depends on the feature extraction method and classification algorithm. Extreme learning machine (ELM) is a single hidden layer neural network, which has better classification and generalization ability. However, ELM is not suitable for feature extraction. Deep learning is a hot research field as it can automatically extract significant features from raw data. In this paper, we propose an approach: an ELM-based deep model, which combined convolutional neural network (CNN), multilayer ELM (ML-ELM) as feature extractor, and used kernel ELM (KELM) as classifier. We used CNN and ML-ELM to extract significant features, and used KELM to achieve stable performance. The performance of proposed approach is validated on two public HAR datasets, and the experimental results show that the proposed approach is effective.

Human activity recognition (HAR) is one of the leading research fields in ubiquitous computing working to integrate seamless technologies in our daily lives. The researches in this field focus on the technological advancement of fine activity recognition through using minimal technological deployments along with consideration of human factors. The domain specific knowledge of human activity recognition has shed significant importance on the activity data processing, due to the innate random nature of the activity data. The necessity for improved prediction accuracy of activities within limited infrastructural support and cost is necessary for the research works to be conducted at large. The data set utilized in this study is modified through tree-based feature-engineering method, which has significantly impacted the intrinsic time-series pattern associated with the original data set acquired from UCI machine learning repository. In this paper, the primary focus lies on improving classification accuracy of the human activities included in the data set through utilizing deep learning classifier models. The particular modified pattern of the data devoid of time-series pattern to major extent has resulted in lower classification accuracy by benchmark time-series classification models namely Keras-LSTM and RNN-LSTM. CNN based networks have demonstrated consisted improved training performance on the data set. The state-of-the-art classifier ResNet introduced by Microsoft Research in 2015 has achieved 99.9% classification accuracy on the data set of this study. Another important aim of this study is to infer the risk attached with adversarial attack on machine learning models. The high vulnerability of machine learning models in face of adversarial attacks has emerged as security concern of the deployment pipeline of machine learning models. To the author’s best knowledge, there has been no prior study to find out the high insecurity associated with the human activity recognition data set and concerned benchmark models. In this study, two prevalent adversarial approaches namely Fast Gradient Sign Method (FGSM) and Basic Iterative Method (BIM) have been employed to carry out non-targeted attack on the pre-trained ResNet model. The accuracy loss by the adversarial attack over a small range of added perturbation is tremendous, nearly 96% accuracy is diminished in the process. Finally, we have demonstrated the comparative performance by FGSM and BIM on reducing the classification accuracy of pre-trained model. The findings shed light on the nature of the data influencing the classification accuracy and the resiliency of state-of-the-art classifier model on facing adversarial attacks.

Clinicians lack objective means for monitoring if their knee osteoarthritis patients are improving outside of the clinic (e.g., at home). Previous human activity recognition (HAR) models using wearable sensor data have only used data from healthy people and such models are typically imprecise for people who have medical conditions affecting movement. HAR models designed for people with knee osteoarthritis have classified rehabilitation exercises but not the clinically relevant activities of transitioning from a chair, negotiating stairs and walking, which are commonly monitored for improvement during therapy for this condition. Therefore, it is unknown if a HAR model trained on data from people who have knee osteoarthritis can be accurate in classifying these three clinically relevant activities. Therefore, we collected inertial measurement unit (IMU) data from 18 participants with knee osteoarthritis and trained convolutional neural network models to identify chair, stairs and walking activities, and phases. The model accuracy was 85% at the first level of classification (activity), 89–97% at the second (direction of movement) and 60–67% at the third level (phase). This study is the first proof-of-concept that an accurate HAR system can be developed using IMU data from people with knee osteoarthritis to classify activities and phases of activities.

Human activity recognition (HAR) is one of the important research areas in pervasive computing. Among HAR, sensor-based activity recognition refers to acquiring a high-level knowledge about human activities from readings of many low-level sensor. In recent years, although traditional methods of deep learning (DL) have been widely used for sensor-based HAR with some good performance, they still face such challenges as feature extraction and characterization, continuous action segmentation in dealing with time series problems. In this study, a multichannel fusion model is proposed by the idea of dividing. In this proposed architecture, a multichannel convolutional neural network (CNN) is used to enhance the ability to extract features at different scales, and then the fused features are fed into gated recurrent unit (GRU) for feature labeling and enhanced feature representation, through the learning of temporal relationships. Finally, the multichannel CNN-GRU model is designed using global average pooling (GAP) to connect the feature maps with the final classification. The model performance was conducted on three benchmark datasets of WISDM, UCI-HAR, and PAMAP2 with the accuracy of 96.41%, 96.67%, and 96.25% respectively. The results show that the proposed model demonstrates better activity detection capability than some of the reported results.

Human activity recognition is an important and challenging topic for computer vision research community. Action representation using deep learning models are currently the dominant technique compared with other methods. However, supervised convolutional neural networks require large computational and memory resources to optimize their parameters. Recently, a simple unsupervised deep learning architecture Principal Component Analysis Network (PCANet) has emerged as an alternative of Convolutional Neural Networks (CNNs) and has significant accomplishments in various vision applications. Meanwhile, encoding and representation techniques using Bag of Words (BoW) and Vector of Locally Aggregated Descriptors (VLAD) have demonstrated great success for several visual tasks specifically in activity recognition. This work presents a novel human activity recognition technique by combining global and local features of PCANet with BoW and VLAD encoding schemes. Both global and local features are learned by PCANet utilizing selected frames from each action video. After that the dimensionality of these features is decreased via Whitening PCA (WPCA). Then encoding schemes are applied on both features to represent the final descriptors for each action. Ultimately, Support Vector Machines classifier (SVM) is trained for recognition process. Several experiments are conducted on UCF sports dataset to evaluate our method. All experimental results utilizing leave-one-out-cross validation (LOOCV) strategy are satisfactorv and comparable.

A Lightweight Deep Learning Model for Human Activity Recognition on Edge Devices

Deep learning models for human activity recognition (HAR) based on sensor data have been heavily studied recently. However, the generalization ability of deep models on complex real-world HAR data is limited by the availability of high-quality labeled activity data, which are hard to obtain. In this article, we design a similarity embedding neural network that maps input sensor signals onto real vectors through carefully designed convolutional and Long Short-Term Memory (LSTM) layers. The embedding network is trained with a pairwise similarity loss, encouraging the clustering of samples from the same class in the embedded real space, and can be effectively trained on a small dataset and even on a noisy dataset with mislabeled samples. Based on the learned embeddings, we further propose both nonparametric and parametric approaches for activity recognition. Extensive evaluation based on two public datasets has shown that the proposed similarity embedding network significantly outperforms state-of-the-art deep models on HAR classification tasks, is robust to mislabeled samples in the training set, and can also be used to effectively denoise a noisy dataset.

Scene image and human skeleton-based dual-stream human action recognition

The enormous applicability of advanced human-computer interaction technology in daily life has facilitated the interest of the researchers toward the development of more intelligent autonomous systems. These human-computer interaction systems can be successful in real-life by addressing the gap among the existing techniques. This research work focuses on one of the prominent human-computer interaction applications of human activity recognition. Human activity recognition (HAR) is the process to detect human gestures, actions, and different types of interactions. The HAR process requires competent knowledge about the day to day human activities and advanced technology to recognize their activities. The conventional pattern recognition techniques are significant to recognize the human activities using the machine learning techniques but only within the controlled environment for the recognition of limited actions. In recent years, deep learning techniques are developed that can learn the deep attributes of the problem application and determine the outcomes with promising performance. The present paper has presented a systematic review of the deep learning models for video-based human activity recognition. The work describes the recent developments in the field for the analysis of different models. The paper also discusses the process of human activity recognition and the eminent datasets available for experimentation. The summarization of the work is illustrated with the future directions in the field.KeywordsDeep learningHuman computer interactionHuman activity recognitionVision based human activity recognitionVideo surveillanceVideo dataset

Human activity recognition (HAR) identifies people’s motions and actions in daily life. HAR research has grown with the popularity of internet-connected, wearable sensors that capture human movement data to detect activities. Recent deep learning advances have enabled more HAR research and applications using data from wearable devices. However, prior HAR research often focused on a few sensor locations on the body. Recognizing real-world activities poses challenges when device positioning is uncontrolled or initial user training data are unavailable. This research analyzes the feasibility of deep learning models for both position-dependent and position-independent HAR. We introduce an advanced residual deep learning model called Att-ResBiGRU, which excels at accurate position-dependent HAR and delivers excellent performance for position-independent HAR. We evaluate this model using three public HAR datasets: Opportunity, PAMAP2, and REALWORLD16. Comparisons are made to previously published deep learning architectures for addressing HAR challenges. The proposed Att-ResBiGRU model outperforms existing techniques in accuracy, cross-entropy loss, and F1-score across all three datasets. We assess the model using k-fold cross-validation. The Att-ResBiGRU achieves F1-scores of 86.69%, 96.23%, and 96.44% on the PAMAP2, REALWORLD16, and Opportunity datasets, surpassing state-of-the-art models across all datasets. Our experiments and analysis demonstrate the exceptional performance of the Att-ResBiGRU model for HAR applications.

Human Activity Recognition (HAR) by employing smart home sensors is a topic that is undergoing intense research in the area of ambient-supported living and the basis of ubiquitous computing in smart environments. Recently, HAR majorly utilizes Deep Learning (DL) method since they employ representation learning methods that could automatically produce maximum factors from raw input datasets resulting from the sensor in the absence of human interference and could detect Hidden Layer (HL) patterns in a dataset. The study presents an Improved Manta-Ray Foraging Optimization with Deep Autoencoder (IMRFO-DAE) model for HAR. The presented algorithm majorly recognizes the diverse types of human activities. The presented IMRFO-DAE model pre-processes the human activity data via a standardization approach to obtain this. Next, the IMRFO-DAE method uses the DAE model to perform the Activity Recognition (AR) process. To advance the AR accomplishment of the DAE model, the IMRFO method is applied as a hyperparameter optimizer. Moreover, the IMRFO model is derived by modifying the initialization of the MRFO model by implementing the chaotic concept. An extensive range of simulations was performed to depict the enhanced efficiency of the IMRFO-DAE approach. The simulation results assured the improved outcomes of the IMRFO-DAE approach compared to existing approaches.

Recently, a significant amount of literature concerning machine learning techniques has focused on automatic recognition of activities performed by people. The main reason for this considerable interest is the increasing availability of devices able to acquire signals which, if properly processed, can provide information about human activities of daily living (ADL). The recognition of human activities is generally performed by machine learning techniques that process signals from wearable sensors and/or cameras appropriately arranged in the environment. Whatever the type of sensor, activities performed by human beings have a strong subjective characteristic that is related to different factors, such as age, gender, weight, height, physical abilities, and lifestyle. Personalization models have been studied to take into account these subjective factors and it has been demonstrated that using these models, the accuracy of machine learning algorithms can be improved. In this work we focus on the recognition of human activities using signals acquired by the accelerometer embedded in a smartphone. The contributions of this research are mainly three. A first contribution is the definition of a clear validation model that takes into account the problem of personalization and which thus makes it possible to objectively evaluate the performances of machine learning algorithms. A second contribution is the evaluation, on three different public datasets, of a personalization model which considers two aspects: the similarity between people related to physical aspects (age, weight, and height) and similarity related to intrinsic characteristics of the signals produced by these people when performing activities. A third and last contribution is the development of a personalization model that considers both the physical and signal similarities. The experiments show that the employment of personalization models improves, on average, the accuracy, thus confirming the soundness of the approach and paving the way for future investigations on this topic.

Human Activity Recognition (HAR) systems are devised for continuously observing human behavior - primarily in the fields of environmental compatibility, sports injury detection, senior care, rehabilitation, entertainment, and the surveillance in intelligent home settings. Inertial sensors, e.g., accelerometers, linear acceleration, and gyroscopes are frequently employed for this purpose, which are now compacted into smart devices, e.g., smartphones. Since the use of smartphones is so widespread now-a-days, activity data acquisition for the HAR systems is a pressing need. In this article, we have conducted the smartphone sensor-based raw data collection, namely H-Activity, using an Android-OS-based application for accelerometer, gyroscope, and linear acceleration. Furthermore, a hybrid deep learning model is proposed, coupling convolutional neural network and long-short term memory network (CNN-LSTM), empowered by the self-attention algorithm to enhance the predictive capabilities of the system. In addition to our collected dataset (H-Activity), the model has been evaluated with some benchmark datasets, e.g., MHEALTH, and UCI-HAR to demonstrate the comparative performance of our model. When compared to other models, the proposed model has an accuracy of 99.93% using our collected H-Activity data, and 98.76% and 93.11% using data from MHEALTH and UCI-HAR databases respectively, indicating its efficacy in recognizing human activity recognition. We hope that our developed model could be applicable in the clinical settings and collected data could be useful for further research.

Recognizing human activity is a challenging task in many applications like visual surveillance, Human–computer interaction, etc. Numerous machine learning (ML) and deep learning (DL) approaches are used to emulate the human behavior of everyday life, as they provide better recognition by learning complex features. This paper proposes an innovative human activity recognition framework with the following steps. Initially, the given video is converted into a set of frames and then applies median filtering to remove the noise for efficient recognition. Subsequently, the pixel-relatedness retrieval-fuzzy clustering means algorithm (PRR-FCM) approach is proposed for image segmentation, in which a pixel-relatedness retrieval strategy is used that considers only the proximity-related pixel for efficient segmentation. The retrieval of features is an important aspect, thereby this paper goes with the extraction of texture features, relative standard deviation induced in multi-texon (RSIMT), in which relative standard deviation is induced for interchanging the center pixel value. Also, the other features like the hierarchy of the skeleton and local binary pattern (LBP) are extracted from the segmented image. Further, the retrieved features are fed to the hybrid model that combines models like modified deep Maxout (MDM) and convolution neural network (CNN) models to recognize human activity efficiently. The modified exponential linear unit (ELU) activation function and Log loss function are the enhancing formulation of the MDM model to determine accurate recognition results.