Wearable Sensor-Based Human Activity Recognition with Transformer Model.

Bidirectional Long Short-Term Memory with Temporal Dense Sampling for human action recognition

Cross-subject transfer learning in human activity recognition systems using generative adversarial networks

Clinicians lack objective means for monitoring if their knee osteoarthritis patients are improving outside of the clinic (e.g., at home). Previous human activity recognition (HAR) models using wearable sensor data have only used data from healthy people and such models are typically imprecise for people who have medical conditions affecting movement. HAR models designed for people with knee osteoarthritis have classified rehabilitation exercises but not the clinically relevant activities of transitioning from a chair, negotiating stairs and walking, which are commonly monitored for improvement during therapy for this condition. Therefore, it is unknown if a HAR model trained on data from people who have knee osteoarthritis can be accurate in classifying these three clinically relevant activities. Therefore, we collected inertial measurement unit (IMU) data from 18 participants with knee osteoarthritis and trained convolutional neural network models to identify chair, stairs and walking activities, and phases. The model accuracy was 85% at the first level of classification (activity), 89–97% at the second (direction of movement) and 60–67% at the third level (phase). This study is the first proof-of-concept that an accurate HAR system can be developed using IMU data from people with knee osteoarthritis to classify activities and phases of activities.

With the advancement and ubiquitousness of wearable devices, wearable sensor-based human activity recognition (HAR) has become a prominent research area in the healthcare domain and human-computer interaction. Inertial measurement unit (IMU) which can provide a wide range of information such as acceleration, angular velocity has become one of the most commonly used sensors in HAR. Recently, with the growing demand for soft and flexible wearable devices, mountable stretch sensors have become a new promising modality in wearable sensor-based HAR. In this paper, we propose a deep learning-based multi-modality HAR framework which consists of three IMUs and two fabric stretch sensors in order to evaluate the potential of stretch sensors independently and in combination with IMU sensors for the activity recognition task. Three different deep learning algorithms: long short-term memory (LSTM), convolutional neural network (CNN) and hybrid CNN-LSTM are deployed to the sensor data for automatically extracting deep features and performing activity classification. The impact of sensor type on recognition accuracy of different activities is also examined in this study. A dataset collected from the proposed framework, namely iSPL IMU-Stretch and a public dataset called w-HAR are used for experiments and performance evaluation.

Recognizing human activity is a challenging task in many applications like visual surveillance, Human–computer interaction, etc. Numerous machine learning (ML) and deep learning (DL) approaches are used to emulate the human behavior of everyday life, as they provide better recognition by learning complex features. This paper proposes an innovative human activity recognition framework with the following steps. Initially, the given video is converted into a set of frames and then applies median filtering to remove the noise for efficient recognition. Subsequently, the pixel-relatedness retrieval-fuzzy clustering means algorithm (PRR-FCM) approach is proposed for image segmentation, in which a pixel-relatedness retrieval strategy is used that considers only the proximity-related pixel for efficient segmentation. The retrieval of features is an important aspect, thereby this paper goes with the extraction of texture features, relative standard deviation induced in multi-texon (RSIMT), in which relative standard deviation is induced for interchanging the center pixel value. Also, the other features like the hierarchy of the skeleton and local binary pattern (LBP) are extracted from the segmented image. Further, the retrieved features are fed to the hybrid model that combines models like modified deep Maxout (MDM) and convolution neural network (CNN) models to recognize human activity efficiently. The modified exponential linear unit (ELU) activation function and Log loss function are the enhancing formulation of the MDM model to determine accurate recognition results.

Due to the increasing number of mobile robots including domestic robots for cleaning and maintenance in developed countries, human activity recognition is inevitable for congruent human-robot interaction. Needless to say that this is indeed a challenging task for robots, it is expedient to learn human activities for autonomous mobile robots (AMR) for navigating in an uncontrolled environment without any guidance. Building a correct classifier for complex human action is non-trivial since simple actions can be combined to recognize a complex human activity. In this paper, we trained a model for human activity recognition using convolutional neural network. We trained and validated our model using the Vicon physical action dataset and also tested the model on our generated dataset (VMCUHK). Our experiment shows that our method performs with high accuracy, human activity recognition task both on the Vicon physical action dataset and VMCUHK dataset.

Human action recognition is a fundamental research problem in computer vision. The accuracy of human action recognition has important applications. In this book chapter, the authors use a YOLOv7-based model for human action recognition. To evaluate the performance of the model, the action recognition results of YOLOv7 were compared with those using CNN+LSTM, YOLOv5, and YOLOv4. Furthermore, a small human action dataset suitable for YOLO model training is designed. This data set is composed of images extracted from KTH, Weizmann, MSR data sets. In this book chapter, the authors make use of this data set to verify the experimental results. The final experimental results show that using the YOLOv7 model for human action recognition is very convenient and effective, compared with the previous YOLO model.

Human activity recognition is an important and challenging topic for computer vision research community. Action representation using deep learning models are currently the dominant technique compared with other methods. However, supervised convolutional neural networks require large computational and memory resources to optimize their parameters. Recently, a simple unsupervised deep learning architecture Principal Component Analysis Network (PCANet) has emerged as an alternative of Convolutional Neural Networks (CNNs) and has significant accomplishments in various vision applications. Meanwhile, encoding and representation techniques using Bag of Words (BoW) and Vector of Locally Aggregated Descriptors (VLAD) have demonstrated great success for several visual tasks specifically in activity recognition. This work presents a novel human activity recognition technique by combining global and local features of PCANet with BoW and VLAD encoding schemes. Both global and local features are learned by PCANet utilizing selected frames from each action video. After that the dimensionality of these features is decreased via Whitening PCA (WPCA). Then encoding schemes are applied on both features to represent the final descriptors for each action. Ultimately, Support Vector Machines classifier (SVM) is trained for recognition process. Several experiments are conducted on UCF sports dataset to evaluate our method. All experimental results utilizing leave-one-out-cross validation (LOOCV) strategy are satisfactorv and comparable.

Human activity recognition (HAR) is one of the important research areas in pervasive computing. Among HAR, sensor-based activity recognition refers to acquiring a high-level knowledge about human activities from readings of many low-level sensor. In recent years, although traditional methods of deep learning (DL) have been widely used for sensor-based HAR with some good performance, they still face such challenges as feature extraction and characterization, continuous action segmentation in dealing with time series problems. In this study, a multichannel fusion model is proposed by the idea of dividing. In this proposed architecture, a multichannel convolutional neural network (CNN) is used to enhance the ability to extract features at different scales, and then the fused features are fed into gated recurrent unit (GRU) for feature labeling and enhanced feature representation, through the learning of temporal relationships. Finally, the multichannel CNN-GRU model is designed using global average pooling (GAP) to connect the feature maps with the final classification. The model performance was conducted on three benchmark datasets of WISDM, UCI-HAR, and PAMAP2 with the accuracy of 96.41%, 96.67%, and 96.25% respectively. The results show that the proposed model demonstrates better activity detection capability than some of the reported results.

Convolutional neural networks (CNN) have shown remarkable performance in enormous computer vision applications over the years, and many works have been done for human activity recognition (HAR) using CNN. However, most of deep learning-based architectures require large training data and plenty of computational resources. Therefore, we proposed a simple and efficient deep learning model for human activity recognition, which works on complex visual data and require less computational resources. In the proposed work, we designed a novel CNN architecture by stacking the repeated components called micro-networks to incorporate multiscale processing in the network layers. We have used a feature fusion strategy to pass the previous layer’s abstract complementary information to the next adjacent layer, believing that each layer encapsulates specific feature maps. Therefore, the hidden complementary information can potentially enhance the feature discrimination capacity of the network and help in learning the network. The proposed architecture is fully trained from scratch with stochastic gradient descent (SGD) optimizer at 0.05 initial learning rate and a softmax classifier is used for activity recognition. The merit of the proposed method over standard deep learning models is its computational efficiency in terms of learnable parameters and computational resources. The proposed model gives good performance on small as well as large size datasets. For authentication of the proposed method several extensive experiments are conducted on publically available datasets, namely UCF-101, HMDB-51, YouTube, and IXMAS datasets. The results have shown the outperformance of the proposed method over the existing state-of-the-art methods.

Recognizing human actions accurately from video data is a difficult task because of the variability in human motion, appearance, and context. A new approach to human action recognition is proposed in this study, which uses a combination of Variational Autoencoder (VAE) and Transformer models. The VAE with IWGECO optimization is utilized for acquiring a compact and discriminative representation of the input 2D skeletons from video frames along with its spatial relations. This encoded representation of 2D poses is then fed into a Transformer model for sequence modeling and classification, resulting in effective recognition of human actions. The proposed approach is evaluated on the MPOSE2021 benchmark dataset, outperforming existing methods (90.6%) with an average classification accuracy of 92.0%. The model performs well on real-life data too showcasing its generalizing potential. The study also addresses the domain shift problem occurring in real-world situations by using a pre-trained VAE and subsequently training the Transformer on the MPOSE2021 dataset. The results demonstrate the effectiveness of combining VAE and Transformer models for human action recognition.

Biomedical images contain a huge number of sensor measurements that can provide disease characteristics. Computer-assisted analysis of such parameters aids in the early detection of disease, and as a result aids medical professionals in quickly selecting appropriate medications. Human Activity Recognition, abbreviated as ‘HAR’, is the prediction of common human measurements, which consist of movements such as walking, running, drinking, cooking, etc. It is extremely advantageous for services in the sphere of medical care, such as fitness trackers, senior care, and archiving patient information for future use. The two types of data that can be fed to the HAR system as input are, first, video sequences or images of human activities, and second, time-series data of physical movements during different activities recorded through sensors such as accelerometers, gyroscopes, etc., that are present in smart gadgets. In this paper, we have decided to work with time-series kind of data as the input. Here, we propose an ensemble of four deep learning-based classification models, namely, ‘CNN-net’, ‘CNNLSTM-net’, ‘ConvLSTM-net’, and ‘StackedLSTM-net’, which is termed as ‘Ensem-HAR’. Each of the classification models used in the ensemble is based on a typical 1D Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) network; however, they differ in terms of their architectural variations. Prediction through the proposed Ensem-HAR is carried out by stacking predictions from each of the four mentioned classification models, then training a Blender or Meta-learner on the stacked prediction, which provides the final prediction on test data. Our proposed model was evaluated over three benchmark datasets, WISDM, PAMAP2, and UCI-HAR; the proposed Ensem-HAR model for biomedical measurement achieved 98.70%, 97.45%, and 95.05% accuracy, respectively, on the mentioned datasets. The results from the experiments reveal that the suggested model performs better than the other multiple generated measurements to which it was compared.

Deep learning models for human activity recognition (HAR) based on sensor data have been heavily studied recently. However, the generalization ability of deep models on complex real-world HAR data is limited by the availability of high-quality labeled activity data, which are hard to obtain. In this article, we design a similarity embedding neural network that maps input sensor signals onto real vectors through carefully designed convolutional and Long Short-Term Memory (LSTM) layers. The embedding network is trained with a pairwise similarity loss, encouraging the clustering of samples from the same class in the embedded real space, and can be effectively trained on a small dataset and even on a noisy dataset with mislabeled samples. Based on the learned embeddings, we further propose both nonparametric and parametric approaches for activity recognition. Extensive evaluation based on two public datasets has shown that the proposed similarity embedding network significantly outperforms state-of-the-art deep models on HAR classification tasks, is robust to mislabeled samples in the training set, and can also be used to effectively denoise a noisy dataset.

Recognizing human activity from highly sparse body sensor data is becoming an important problem in Internet of Medical Things (IoMT) industry. As IoMT currently uses batteryless or passive wearable body sensors for activity recognition, the data signals of these sensors have a high sparsity, which means that the time intervals of sensors’ readings are irregular and the number of sensors’ readings in time unit is frequently limited. Therefore, learning activity recognition models from temporally sparse data is challenging. Traditional machine learning techniques cannot be applicable in this scenario due to their focus on a single sensing modality requiring regular sampling rate. In this work, we propose an effective end-to-end deep neural network (DNN) model to recognize human activities from temporally sparse data signals of passive wearable sensors that improve the accuracy rate of human activity recognition. A dropout technique is used in the developed model to deal with sparsity problem and avoiding overfitting problem. In addition, optimization of the proposed DNN model was performed by evaluating a different number of hidden layers. Various experiments were conducted on a public clinical room dataset of sparse data signals to compare the performance of the proposed DNN model with the conventional and other deep learning approaches. The experimental results demonstrate that the proposed DNN model outperforms the existing state-of-the-art methods in terms of lower inference delay and activity recognition accuracy.

Multiple cameras are used to resolve occlusion problem that often occur in single-view human activity recognition. Based on the success of learning representation with deep neural networks (DNNs), recent works have proposed DNNs models to estimate human activity from multi-view inputs. However, currently available datasets are inadequate in training DNNs model to obtain high accuracy rate. Against such an issue, this study presents a DNNs model, trained by employing transfer learning and shared-weight techniques, to classify human activity from multiple cameras. The model comprised pre-trained convolutional neural networks (CNNs), attention layers, long short-term memory networks with residual learning (LSTMRes), and Softmax layers. The experimental results suggested that the proposed model could achieve a promising performance on challenging MVHAR datasets: IXMAS (97.27%) and i3DPost (96.87%). A competitive recognition rate was also observed in online classification.