Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learning

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

In recent years, Deep Learning (DL) models, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory (LSTM), have been widely used for Human Activities Recognition (HAR). They have achieved considerable performance improvements over classical Machine Learning (ML) approaches due to their excellent feature representation capabilities. Simple human activities are performed in sequential order with no overlaps or concurrent actions. The paucity of labeled training activity samples, high computational cost, and system resource requirements of deep learning architectures compared to lightweight models are some of the research challenges confronting the HAR community. We framed the lightweight DL-based CNNs, RNNs, and LSTM model for HAR to tackle these research challenges. The RNNs and LSTM have single layers with softmax activation functions, whereas the one-dimensional CNNs have two convolutional and single max-pooling layers. To evaluate the performance of our models, we used the publicly available benchmark WISDM experimental dataset, which includes the reading of six activities (walking, jogging, upstairs, downstairs, sitting, and standing) performed by 36 participants using a single accelerometer sensor. The experimental result illustrates the efficacy of our models in activity recognition, demonstrating that they attain higher accuracy while being computationally efficient.

The recent development of machines exhibiting intelligent characteristics involves numerous techniques including computer hardware and software architecture development. Many different hardware devices, wearable sensors, machine learning, and deep learning model implementations are being applied in human activity recognition (HAR) applications in recent times. However, to develop high accuracy classification systems for activity recognition using ilow-cost hardware technology is of significant importance. To achieve this goal this study uses sensor data from two low-cost sensors, gyroscope and accelerometer along with the implementation of an Artificial Neural Network (ANN) based deep learning model for HAR. In particular, Deep Stacked Multilayered Perceptron (DS-MLP) has been proposed. In the implementation of DS-MLP, an ANN model has been used as a meta-learner while five MLP models have been used as base-learners. In this study, these base-learners and meta-learner have been combined using a stack ensemble technique. The performance evaluations have been done first on the applicability of individual base-models followed by the application of DS-MLP, the results prove the high accuracy of 97.3% and 99.4% for heterogeneous datasets used for testing. The performance of the proposed DS-MLP models has been compared to some existing machine learning classifiers and several state-of-the-art activity recognition systems. The comparative result analysis also proves that the proposed system performed better than these classification approaches in terms of important performance metrics such as accuracy, precision, recall, F-score, Cohen's Kappa, and Mathew correlation coefficient.

Human Activity Recognition (HAR) systems are designed to read sensor data and analyse it to classify any detected movement and respond accordingly. However, there is a need for more responsive and near real-time systems to distinguish between false and true alarms. To accurately determine alarm triggers, the motion pattern of legitimate users need to be stored over a certain period and used to train the system to recognise features associated with their movements. This training process is followed by a testing cycle that uses actual data of different patterns of activity that are either similar or different to the training data set. This paper evaluates the use of a combined Convolutional Neural Network (CNN) and Naive Bayes for accuracy and robustness to correctly identify true alarm triggers in the form of a buzzer sound for example. It shows that pattern recognition can be achieved using either of the two approaches, even when a partial motion pattern is derived as a subset out of a full-motion path.

Human Activity Recognition (HAR) deals with the automatic recognition of physical activities and plays a crucial role in healthcare and sports where wearable sensors and intelligent computational techniques are used. We propose a HAR algorithm that uses the smartphones accelerometer data for human activity recognition. In particular, we present a recurrent convolutional neural network-based HAR algorithm that combines a Convolutional Neural Network (CNN) to extract temporal features from the sensor data, a Fuzzy C-Means (FCM) clustering algorithm to cluster the features extracted by the CNN, and a Long Short-Term Memory (LSTM) network to learn the temporal dependencies between the features. We evaluate the proposed methodology on two distinct datasets: the MotionSense dataset and the WISDM dataset. We evaluate the proposed CNN-FCM-LSTM model on the publicly available MotionSense dataset to classify ten activity types: 1) walking upstairs, 2) walking downstairs, 3) jogging, 4) sitting, 5) standing, 6) level ground walking, 7) jumping jacks, 8) brushing teeth, 9) writing, and 10) eating. Next, we evaluate the model’s performance on the WISDM dataset to assess its ability to generalize to unseen data. On the MotionSense test dataset, CNN-FCM-LSTM achieves a classification accuracy of 99.69%, a sensitivity of 99.62%, a specificity of 99.63%, and a false positive rate per hour (FPR/h) of 0.37%. Meanwhile, it achieves a classification accuracy of 97.27% on the WISDM dataset. The CNN-FCM-LSTM model’s capability to classify a diverse range of activities within a single architecture is noteworthy. The results suggest that the proposed CNN-FCM-LSTM model using smartphone inputs is more accurate, reliable, and robust in detecting and classifying activities than the state-of-the-art models. It should be noted that activity recognition technology has the potential to aid in studying the underpinnings of physical activity, designing more effective training regimens, and simulating the rigors of competition in sports.

Activity recognition has received considerable attention in many research fields, such as industrial and healthcare fields. However, many researches about activity recognition have focused on static activities and dynamic activities in current literature, while, the transitional activities, such as stand-to-sit and sit-to-stand, are more difficult to recognize than both of them. Consider that it may be important in real applications. Thus, a novel framework is proposed in this paper to recognize static activities, dynamic activities, and transitional activities by utilizing stacked denoising autoencoders (SDAE), which is able to extract features automatically as a deep learning model rather than utilize manual features extracted by conventional machine learning methods. Moreover, the resampling technique (random oversampling) is used to improve problem of unbalanced samples due to relatively short duration characteristic of transitional activity. The experiment protocol is designed to collect twelve daily activities (three types) by using wearable sensors from 10 adults in smart lab of Ulster University, the experiment results show the significant performance on transitional activity recognition and achieve the overall accuracy of 94.88% on three types of activities. The results obtained by comparing with other methods and performances on other three public datasets verify the feasibility and priority of our framework. This paper also explores the effect of multiple sensors (accelerometer and gyroscope) to determine the optimal combination for activity recognition.

Human Activity Recognition (HAR) based on data from wearable sensors has become an attractive research topic thanks to its applications in different fields such as healthcare and smart environments. Recently, the advancement of deep learning with capability to perform automatically high-level feature extraction has achieved promising results. However, the performance of the deep learning models depends deeply on the characteristics of the datasets such as the number of classes, the inter-similarity and intra-variation. Therefore, directly comparing these models has become difficult since a wide variety of experimental protocols, evaluation metrics, and datasets are employed. In this paper, for the first time, a systematic evaluation of several deep learning models for HAR from wearable sensors is provided. In particular, three models named Convolutional Neural Network (CNN) [1], DeepConvLSTM - a combination of CNN and Long Short Term Memory (LSTM) [2], and SensCapsNet - a Capsule Neural Network for wearable sensor-based HAR [3] were implemented and evaluated on three benchmark datasets that are 19NonSens, CMDFall, and UCI-HAR dataset. Moreover, to have an intuitive explanation of deep learning models, a visualization of features learnt from these models is given. The evaluation codebase and results will be made publicly available for community use.

The synergy of communication, computation and sensing capabilities in mobile systems-on-chip artifacts such as smartphones has made possible the development of wearable smart sensor systems for user activity monitoring and recognition. Assessing physical activity is useful to enhance people health as experts have evidenced a clear correlation between physical activity and overweight, obesity and metabolism-related syndromes. Due to user acceptability and the convenient nonintrusive manner for measuring data, smartphones have the advantage of taking proprioceptive motion measurements outside of a controlled environment for rather long periods of time using embedded sensors such as the accelerometer, however, activity recognition poses several challenges. Particularly, though work has been reported for accelerometer-based activity recognition using smartphones, the portability of the device to a single fixed tight position has been a major constraint to easy the interpretation of the collected data on resource-limited devices. In this chapter, a human activity hierarchical recognition system based on neural networks without the need of the smartphone to be constrained to a single fixed position is presented. Yet it is used as a representative example to show the challenges, the role and some of the main potential impacts that smartphones have and will have in mHealth. Experimental results on Android-capable smartphones on four on-body locations show that the recognition system achieves high classification rates, above 92%, for five activities including static, walking, running, and up-down stairs walking, which outperforms other proposals. The system is fully implemented in a smartphone running continuously in near real-time with reduced power consumption in a proof-of-concept client-server application for mHealth.

Human action identification has advanced significantly as a result of the development of deep learning algorithms. Convolutional Neural Networks (CNNs) are known for their adeptness at extracting crucial information within video frames, especially shapes and postures. However, understanding movements necessitates not just individual frames as well as the temporal flow of the movement; this is wherein LSTM (long short-term memory) variants of Recurrent Neural Networks (RNNs) come into play. By analyzing frame sequences, LSTMs are particularly effective in gathering temporal connections, recalling prior motions, and overcoming the gap amongst static features as well as dynamic actions. Even with CNNs and LSTMs' remarkable accuracy in human action recognition (HAR), challenges remain.

IoT-FAR: A multi-sensor fusion approach for IoT-based firefighting activity recognition

Many human activities consist of physical gestures that tend to be performed in certain sequences. Wearable inertial sensor data have as a consequence been employed to automatically detect human activities, lately predominantly with deep learning methods. This article focuses on the necessity of recurrent layers—more specifically Long Short-Term Memory (LSTM) layers—in common Deep Learning architectures for Human Activity Recognition (HAR). Our experimental pipeline investigates the effects of employing none, one, or two LSTM layers, as well as different layers' sizes, within the popular DeepConvLSTM architecture. We evaluate the architecture's performance on five well-known activity recognition datasets and provide an in-depth analysis of the per-class results, showing trends which type of activities or datasets profit the most from the removal of LSTM layers. For 4 out of 5 datasets, an altered architecture with one LSTM layer produces the best prediction results. In our previous work we already investigated the impact of a 2-layered LSTM when dealing with sequential activity data. Extending upon this, we now propose a metric, rGP, which aims to measure the effectiveness of learned temporal patterns for a dataset and can be used as a decision metric whether to include recurrent layers into a network at all. Even for datasets including activities without explicit temporal processes, the rGP can be high, suggesting that temporal patterns were learned, and consequently convolutional networks are being outperformed by networks including recurrent layers. We conclude this article by putting forward the question to what degree popular HAR datasets contain unwanted temporal dependencies, which if not taken care of, can benefit networks in achieving high benchmark scores and give a false sense of overall generability to a real-world setting.

Adaptive Bayesian inference system for recognition of walking activities and prediction of gait events using wearable sensors

Nowadays, Human Activity Recognition (HAR) is a key research area with many ubiquitous innovative solutions, where both accelerometer and gyroscope data provide information about an observed person’s physical activity. HAR offers a diverse variety of important applications, including healthcare, burglary detection, workplace monitoring, and emergency identification. Traditional recognition approaches rely on extracting handmade features from the obtained data to identify the typeof human action. Additionally, the efficacy of these works is dependent upon the specific customized features that are chosen. One potential approach to tackle this issue is to utilize Convolutional Neural Networks (CNN) to automatically learn the relevant features. In this paper, we propose a deep learning model, WISNet, a custom 1D-CNN approach to recognize six complex human activities: Jogging, Walking Downstairs, Sitting, Standing, Walking and Climbing Upstairs. The model includes a Convolved Normalized Pooled (CNPM) Block to generate significant features from the initial layers. An Identity and Basic (IDBN) Block is incorporated to extract residual progressive features for capturing complex sequential data dependencies. Channel and Spatial attention (CASb) Block is integrated with the network to prioritize or minimize essential features based on relative weights. The proposed WISNet model achieved an enhanced accuracy and F1-score of 96.41 % and 0.95 for the HAR dataset by surpassing the existing transfer learning architectures such as Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and Recurrent Neural Network (SimpleRNN). By strategically integrating the CNPM, IDBN, and CASb blocks, this study aims to tackle distinct challenges encountered in the classification process by enhancing the discernment of features essential for the precise identification of multi-class human activity recognition. The seamless integration of these blocks within the model plays a pivotal role in elevating the overall performance of the WISNet architecture. The work also validates WISNet with similar open-source datasets (UCI-HAR and KU-HAR) and dissimilar open-source datasets (Sleep state detection, Fall detection, and ECG Heartbeat).

Activity recognition plays a key role in providing activity assistance and care for users in smart homes. In this work, we present an activity recognition system that classifies in the near real-time a set of common daily activities exploiting both the data sampled by sensors embedded in a smartphone carried out by the user and the reciprocal Received Signal Strength (RSS) values coming from worn wireless sensor devices and from sensors deployed in the environment. In order to achieve an effective and responsive classification, a decision tree based on multisensor data-stream is applied fusing data coming from embedded sensors on the smartphone and environmental sensors before processing the RSS stream. To this end, we model the RSS stream, obtained from a Wireless Sensor Network (WSN), using Recurrent Neural Networks (RNNs) implemented as efficient Echo State Networks (ESNs), within the Reservoir Computing (RC) paradigm. We targeted the system for the EvAAL scenario, an international competition that aims at establishing benchmarks and evaluation metrics for comparing Ambient Assisted Living (AAL) solutions. In this paper, the performance of the proposed activity recognition system is assessed on a purposely collected real-world dataset, taking also into account a competitive neural network approach for performance comparison. Our results show that, with an appropriate configuration of the information fusion chain, the proposed system reaches a very good accuracy with a low deployment cost.