Sensor-Based Human Activity Recognition Using Deep Stacked Multilayered Perceptron Model

Due to the breadth of its application domains, Hu-man Activity Recognition (HAR) is a problematic area of human-computer interaction. HAR can be used in remote monitoring of senior healthcare and concern situations in intelligent man-ufacturing, among other applications. HAR based on wearable inertial sensors has been researched identification efficiency in various kinds of human actions considerably more than vision-based HAR. The sensor-based HAR is generally applicable to indoor and outdoor locations without privacy considerations of implementation. In this research, we explore the recognition performance of multiple deep learning (DL) models to recognize everyday living human activities. We developed a deep residual neural network that employed aggregated multi-branch transformation to boost identification performance. The proposed model is called the ResNeXt model. To evaluate its performance, three standard DL models (CNN, LSTM, and CNN-LSTM) are investigated and compared to our proposed model using a standard HAR dataset called Daily Living Activity dataset. These datasets gathered mobility signal data from multimodal sensors (accelerometer, gyroscope, and magnetometer) in three distinct body areas (wrist, hip, and ankle). The experimental findings reveal that the proposed model surpasses other benchmark DL models with maximum accuracy and F1-scores. Furthermore, the findings show that the ResNeXt model is more resistant than other models with fewer training parameters.

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

Human activity recognition (HAR) identifies people’s motions and actions in daily life. HAR research has grown with the popularity of internet-connected, wearable sensors that capture human movement data to detect activities. Recent deep learning advances have enabled more HAR research and applications using data from wearable devices. However, prior HAR research often focused on a few sensor locations on the body. Recognizing real-world activities poses challenges when device positioning is uncontrolled or initial user training data are unavailable. This research analyzes the feasibility of deep learning models for both position-dependent and position-independent HAR. We introduce an advanced residual deep learning model called Att-ResBiGRU, which excels at accurate position-dependent HAR and delivers excellent performance for position-independent HAR. We evaluate this model using three public HAR datasets: Opportunity, PAMAP2, and REALWORLD16. Comparisons are made to previously published deep learning architectures for addressing HAR challenges. The proposed Att-ResBiGRU model outperforms existing techniques in accuracy, cross-entropy loss, and F1-score across all three datasets. We assess the model using k-fold cross-validation. The Att-ResBiGRU achieves F1-scores of 86.69%, 96.23%, and 96.44% on the PAMAP2, REALWORLD16, and Opportunity datasets, surpassing state-of-the-art models across all datasets. Our experiments and analysis demonstrate the exceptional performance of the Att-ResBiGRU model for HAR applications.

Human Activity Recognition (HAR) is the automatic detection and understanding of human motion behavior based on data extracted from video camera, ambient sensors or wearable sensors which particularly has recently attracted increased attention from both researchers and industry. However, for running practical HAR systems on wearable devices, there are some requirements such as design and development of small, lightweight, powerful, and low-cost smart sensors. In this context, data must be continuously collected, and edge computing is a viable solution, which is an energy-efficient technique, offering real-time response and privacy requirements for HAR applications. Rather than sending data to the cloud, edge computing is a local process that minimizes the data transmission time and responds with low latency. Recently, HAR system designers have adopted deep learning techniques inspired by their outstanding performance in many application areas and achieved relevant gain in activity recognition performance, however, these techniques were not demonstrated suitable for running on resource constrained devices. Thus, designing energy-efficient deep learning models is critical for realizing efficient HAR for mobile applications. In this work, we present a lightweight framework for the deployment of low-power but accurate HAR systems for these devices. We also implement and run the system in a microcontroller and analyze computational cost and energy consumption, and how different system configurations and deep learning model complexity influence on that.

Healthcare using body sensor data has been getting huge research attentions by a wide range of researchers because of its good practical applications such as smart health care systems. For instance, smart wearable sensor-based behavior recognition system can observe elderly people in a smart eldercare environment to improve their lifestyle and can also help them by warning about forthcoming unprecedented events such as falls or other health risk, to prolong their independent life. Although there are many ways of using distinguished sensors to observe behavior of people, wearable sensors mostly provide reliable data in this regard to monitor the individual’s functionality and lifestyle. In this paper, we propose a body sensor-based activity modeling and recognition system using time-sequential information-based deep Neural Structured Learning (NSL), a promising deep learning algorithm. First, we obtain data from multiple wearable sensors while the subjects conduct several daily activities. Once the data is collected, the time-sequential information then go through some statistical feature processing. Furthermore, kernel-based discriminant analysis (KDA) is applied to see the better clustering of the features from different activity classes by minimizing inner-class scatterings while maximizing inter-class scatterings of the samples. The robust time-sequential features are then applied with Neural Structured Learning (NSL) based on Long Short-Term Memory (LSTM), for activity modeling. The proposed approach achieved around 99% recall rate on a public dataset. It is also compared to existing different conventional machine learning methods such as typical Deep Belief Network (DBN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) where they yielded the maximum recall rate of 94%. Furthermore, a fast and efficient explainable Artificial Intelligence (XAI) algorithm, Local Interpretable Model-Agnostic Explanations (LIME) is used to explain and check the machine learning decisions. The robust activity recognition system can be adopted for understanding peoples' behavior in their daily life in different environments such as homes, clinics, and offices.

With the rise of artificial intelligence, sensor-based human activity recognition (S-HAR) is increasingly being employed in healthcare monitoring for the elderly, fitness tracking, and patient rehabilitation using smart devices. Inertial sensors have been commonly used for S-HAR, but wearable devices have been demanding more comfort and flexibility in recent years. Consequently, there has been an effort to incorporate stretch sensors into S-HAR with the advancement of flexible electronics technology. This paper presents a deep learning network model, utilizing aggregation residual transformation, that can efficiently extract spatial–temporal features and perform activity classification. The efficacy of the suggested model was assessed using the w-HAR dataset, which included both inertial and stretch sensor data. This dataset was used to train and test five fundamental deep learning models (CNN, LSTM, BiLSTM, GRU, and BiGRU), along with the proposed model. The primary objective of the w-HAR investigations was to determine the feasibility of utilizing stretch sensors for recognizing human actions. Additionally, this study aimed to explore the effectiveness of combining data from both inertial and stretch sensors in S-HAR. The results clearly demonstrate the effectiveness of the proposed approach in enhancing HAR using inertial and stretch sensors. The deep learning model we presented achieved an impressive accuracy of 97.68%. Notably, our method outperformed existing approaches and demonstrated excellent generalization capabilities.

A survey on wearable sensor modality centred human activity recognition in health care

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

With the proliferation of miniaturized movement sensing capabilities, human activity recognition using wearables and other forms of pervasive sensing (HAR) has seen an enormous boost in both research and practical deployments. Prominent application domains are health assessments, wellbeing monitoring, and novel interaction paradigms. Arguably, and especially when compared to video based activity recognition, the use of movement sensors such as inertial measurement units has many practical advantages including mobility, more direct movement capturing, and less privacy invasive sensing. However, those advantages come with a cost with regards to the design of machine learning based pipelines for the analysis of the sensor data streams, i.e., multi-variate time series data. Probably the biggest challenge lies in the absence of (very) large, labeled datasets that can readily be used for developing and training complex analysis models. In this paper I explore the challenges for deriving machine learning based systems for sensor-based human activity recognition. My focus is on: i) the role of representation learning in modern HAR systems that overcome the need for explicit feature engineering; ii) self-supervised methods for most effective use of small, labeled datasets; and iii) cross-modality transfer. I conclude with an outlook of next steps in the field that promise to lead to increased robustness and more flexible applicability of contemporary, machine-learnlng based HAR systems.

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

In this work, we propose a Human Activity Prediction (HAP) system using activity sequence spanning trees constructed from a life-log created by a video sensor-based daily Human Activity Recognition (HAR) system using time-sequential Independent Component (IC)-based depth silhouette features with Hidden Markov Models (HMMs). In the daily HAR system, the IC features are extracted from the collection of the depth silhouettes containing various daily human activities such as walking, sitting, lying, cooking, eating etc. Using these features, HMMs are used to model the time sequential features and recognize various human activities. The depth silhouette-based human activity recognition system is used to recognize daily human activities automatically in real time, which creates a life-log of daily activity events. In this work, we propose a method for human activity prediction using fixed-length activity sequence spanning trees based on the life-log. Utilizing the consecutive activities recorded in an activity sequence database (i.e. life-log) for a specific period of time of each day over a period such as a month, the fixed-length spanning trees can be constructed for the sequences starting with each activity where the leaf nodes contain the frequency of the fixed-length consecutive activity sequences. Once the trees are constructed, to predict an activity after a sequence of activities, we traverse the spanning trees until a path up to the previous node of the leaf nodes is matched with the testing pattern. Finally, we can predict the next activity based on the highest frequency of the leaf nodes along the matched path. The prediction experiments over the computer simulated data which is based on the daily logs show satisfactory results. Our video sensor-based human activity recognition and prediction systems can be utilized for practical applications such as smart and proactive healthcare.

This paper brings to light one of the most prominent applications of human activity recognition which is the anomaly detection. Providing security to an individual is a major concern of any society today due to the constantly increasing actions causing threats, starting from deliberate violence to an injury caused by an accident. Mere installation of a traditional closed-circuit television is not enough as it requires a human being to constantly remain alert and monitor the cameras, which is quite inefficient. This calls for the need to develop an automated security system that identifies abnormal activities in real time and brings immediate help to the victims. But it is time-consuming to recognize activities from lengthy surveillance videos and hence a new implementation idea which adaptively compresses the videos before passing it through the activity recognition system has been proposed. Adaptive video compression is a unique compression technology that compresses only the least significant parts of the video, retaining the objects of interest. By combining adaptive video compression and the convolutional 3D network with contextual multiple scales based on the temporal features (Xu et al. in Learning deep representations of appearance and motion for anomalous event detection. In: Computer Vision and Pattern Recognition (CVPR), 2015), the paper aims to provide an accurate anomalous human activity recognition system that works in real time. The method is implemented on the UCF101 crime dataset with around 13 different anomalies (Learning Spatiotemporal Features with 3D Convolutional Networks. In: International Conference on Computer Vision (ICCV), IEEE, 2015). It resulted in improved recognition performance compared to state-of-the-art techniques.

Recently, deep learning, which are able to extract automatically features from data, has achieved state-of-the-art performance across a variety of sensor based human activity recognition (HAR) tasks. However, the existing deep neural networks are usually trained with a global loss, and all hidden layer weights have to be always kept in memory before the forward and backward pass has completed. The backward locking phenomenon prevents the reuse of memory, which is a crucial limitation for wearable activity recognition. In the paper, we proposed a layer-wise convolutional neural networks (CNN) with local loss for the use of HAR task. To our knowledge, this paper is the first that uses local loss based CNN for HAR in ubiquitous and wearable computing arena. We performed experiments on five public HAR datasets including UCI HAR dataset, OPPOTUNITY dataset, UniMib-SHAR dataset, PAMAP dataset, and WISDM dataset. The results show that local loss works better than global loss for tested baseline architectures. At no extra cost, the local loss can approach the state-of-the-arts on a variety of HAR datasets, even though the number of parameters was smaller. We believe that the layer-wise CNN with local loss can be used to update the existing deep HAR methods.

Sensor-based Human Activity Recognition (HAR) is a research field that has been widely studied in Pervasive Computing. Due to its many applications from healthcare to well-being, sensor-based HAR has been proposed to recognize high-level activities considering smart-homes environmental sensors as well as low-level activities using wearable sensors. The most effective solutions presented so far rely on supervised learning approaches, that have the potential of reaching high recognition rates. However, such approaches assume that large amounts of labeled data are available. Annotating human activities is costly, time-consuming, intrusive, and often prohibitive. Considering the high variability of activity execution among different subjects, methods that are capable of personalizing to each subject with limited labeled data are needed. In the last few years, several research groups are indeed working hard trying to mitigate this problem. However, data scarcity still remains an open challenge in sensor-based HAR. This talk presents latest research efforts on these topics. In particular, the talk will cover: a) purely knowledge-based approaches enhanced with active learning for smart-home sensor data, b) hybrid data-driven and knowledge-based approaches relying on context data to mitigate data scarcity in mobile settings, and c) semi-supervised federated learning approaches. We will discuss the above-mentioned challenges, report our experience and identify critical aspects which still need to be investigated.

A review of multimodal human activity recognition with special emphasis on classification, applications, challenges and future directions