A Spanning Tree-Based Human Activity Prediction System Using Life Logs from Depth Silhouette-Based Human Activity Recognition

When developing a fully automatic system for evaluating motor activities performed by a person, it is necessary to segment and recognize the different activities in order to focus the analysis. This process must be carried out by a Human Activity Recognition (HAR) system. This paper proposes a user adaptation technique for improving a HAR system based on Hidden Markov Models (HMMs). This system segments and recognizes six different physical activities (walking, walking upstairs, walking downstairs, sitting, standing and lying down) using inertial signals from a smartphone. The system is composed of a feature extractor for obtaining the most relevant characteristics from the inertial signals, a module for training the six HMMs (one per activity), and the last module for segmenting new activity sequences using these models. The user adaptation technique consists of a Maximum A Posteriori (MAP) approach that adapts the activity HMMs to the user, using some activity examples from this specific user. The main results on a public dataset have reported a significant relative error rate reduction of more than 30%. In conclusion, adapting a HAR system to the user who is performing the physical activities provides significant improvement in the system’s performance.

Modern monostatic radar-based human activity recognition (HAR) systems perform very well as long as the direction of human activities is either towards or away from the radar. The monostatic single-input single-output (SISO) and monostatic multiple-input multiple-output (MIMO) radar systems cannot detect motion of an object that moves perpendicularly to the radar’s boresight axis. Due to this physical layer limitation, today’s radar-based HAR systems fail to classify multi-directional human activities. In this paper, we resolve this typical but critical physical layer problem of contemporary HAR systems. We propose a HAR system underlying a distributed MIMO radar configuration, where multiple antennas of a millimeter wave MIMO radar system (Ancortek SDR-KIT 2400T2R4) are distributed in an indoor environment. In our proposed HAR system, we have two independent and identical monostatic radar subsystems that irradiate and capture the multi-directional human movement from two perspectives, which allows to compute two distinct time-variant radial velocity distributions. A feature extraction network extracts numerous features from the measured time-variant radial velocity distributions, which are then fused by a multiclass classifier to detect five types of human activities. The proposed multi-perspective MIMO-radar-based HAR system achieves a classification accuracy of 98.52%, which surpasses the accuracy of SISO radar-based HAR system by more than 9%. Our approach resolves the physical layer limitations of modern HAR systems that are based on either monostatic SISO or monostatic MIMO radar systems.

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

Human activity recognition (HAR) based on ambient sensors aims to recognize a conducted activity. A large number of deep learning models (DLMs) for HAR have been proposed. In contrast, human activity prediction (HAP) aims to early predict an activity. Compared to HAR, the advantage of HAP is to prevent a person from being exposed to unexpected cases by early predicting the activity. However, few DLMs for HAP have been proposed. They predict the next activity via a non-end-to-end fashion, e.g., they take a sequence of consecutive activities where the activities were classified from the sensor information. Thus, the information that which sensors are activated is not used in the prediction. In this study, we propose an end-to-end HAP model to predict the next activity from a sequence of consecutive events. The model has an encoder, a classifier, and a regressor. The encoder gives an encoded vector by encoding events. The regressor learns temporal dependency from a sequence of encoded vectors to predict the next encoded vector. The classifier predicts the next activity using the next encoded vector. We use the Milan and Aruba datasets to study a prediction accuracy of the model. We compare our model with a non-end-to-end model based on long-term memory, taking a sequence of past activities. We show that our model achieves the better prediction accuracy than the non-end-to-end model by up to 4.73% and 7.39% for Milan and Aruba, respectively, meaning that the information related to events can be used in the prediction.

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

Sensor-based Human Activity Recognition (HAR) has been applied across various domains, including healthcare monitoring, fitness tracking, and smart home systems. These applications require the ability to accurately detect and respond to a wide range of human activities, each with varying distributions, which imposes a significant challenge. The task-incremental learning paradigm can address this problem by enabling HAR systems to adapt to changes in distribution and learn new activities over time without forgetting the previously learned ones. Continual adaptation is essential for maintaining high performance, as it allows the system to effectively respond to these changes. Although several strategies in the continual learning literature have been evaluated for task-incremental scenarios in HAR, there is still room for improvement, as the results are not as good as those achieved with conventional approaches. This work proposes two new neuroplasticity-inspired techniques that can be integrated with any learning strategy. Inspired by the brain’s ability to reorganize and strengthen connections over time, these methods focus on enhancing the model’s flexibility and long-term knowledge retention. The proposed techniques were evaluated alongside the WA-ADB and WA-MDF strategies on well-known HAR datasets. Experimental results demonstrated that the new techniques significantly enhanced the models’ ability to retain knowledge, which holds significant potential for improving the robustness and longevity of HAR systems in real-world applications.

Recent advancements in depth video sensors technologies have made human activity recognition (HAR) realizable for elderly monitoring applications. Although conventional HAR utilizes RGB video sensors, HAR could be greatly improved with depth video sensors which produce depth or distance information. In this paper, a depth-based life logging HAR system is designed to recognize the daily activities of elderly people and turn these environments into an intelligent living space. Initially, a depth imaging sensor is used to capture depth silhouettes. Based on these silhouettes, human skeletons with joint information are produced which are further used for activity recognition and generating their life logs. The life-logging system is divided into two processes. Firstly, the training system includes data collection using a depth camera, feature extraction and training for each activity via Hidden Markov Models. Secondly, after training, the recognition engine starts to recognize the learned activities and produces life logs. The system was evaluated using life logging features against principal component and independent component features and achieved satisfactory recognition rates against the conventional approaches. Experiments conducted on the smart indoor activity datasets and the MSRDailyActivity3D dataset show promising results. The proposed system is directly applicable to any elderly monitoring system, such as monitoring healthcare problems for elderly people, or examining the indoor activities of people at home, office or hospital.

Segmenting human activities based on HMMs using smartphone inertial sensors

The conventional Human Activity Recognition (HAR) systems use smartwatches, smartphones, and other wearable devices to autonomously derive consumer physical activities. However, implementing real-time HAR in resource-constrained applications is challenging. Computational offloading approaches must address network reliance, latency, and data privacy, which affect system performance. The present study proposes a low-cost, wearable HAR device that collects and transmits the Inertial Measurement Unit (IMU) telemetry signals to smartphones, enabling real-time on-device activity detection. Human actions are subjective and affected by physiological, environmental, and behavioural factors, complicating HAR systems' performance in personalized contexts. Generalized methods for existing systems reduce performance for new consumers or due to alterations in consumer locomotive signatures. Thus, our second goal is to enable the on-device personalization of wearable HAR models with minimum customer calibration. The HAR device, which is in the form of a smart-band, transfers data to and fro smartphones through Bluetooth Low Energy (BLE). A lightweight 1D convolutional neural network (CNN) is built and, using transfer learning, the network model is fine-tuned from real-time sensed data. On-device HAR inferences ensure real-time processing on smartphones without computational offloading. Smartphone-based on-device training for fine-tuning the model allows the HAR system to get customized without compromising privacy or computational costs. The experimental results show that the personalized classification achieves on average 98% accuracy when assessed using four benchmark datasets.

Human activity recognition (HAR) is an important technology for a wide range of applications including elderly people monitoring, ambient assisted living, sport and fitness activities. The aim of this paper is to address the HAR task directly on a wearable device, implementing a recurrent neural network (RNN) on a low cost, low power microcontroller, ensuring the required performance in terms of accuracy and low complexity. To reach this goal we first develop a lightweight RNN on the Human Activity Recognition Using Smartphones dataset in order to accurately detect human activity and then we port the RNN to the embedded device Cloud-JAM L4, based on an STM32 microcontroller. Experimental results show that this HAR RNN-based detector can be effectively implemented on the chosen constrained-resource system, achieving an accuracy of about 90.50% with a very low memory cost (40.883 KB) and inference time (67.131 ms), allowing the design of a wearable embedded system for human activity recognition.

Human activity recognition (HAR) is an interesting research area in machine learning. The purpose of human activity recognition study is to automatically detect human activities from the information acquired from different sensors, and to analyze this information using statistical techniques. Big data still a challenging problem in human activity prediction. Several approaches have recently been developed to find practical ways to solve high dimensionality of data problems. The mean objective of this work is to deal with HAR big data modeling in order to maximize the classification accuracy while minimizing the number of factors. The proposed framework has been tested on a publicly HAR available dataset and the results have been interpreted and discussed.

International audience

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

With the proliferation of miniaturized movement sensing capabilities, human activity recognition using wearables and other forms of pervasive sensing (HAR) has seen an enormous boost in both research and practical deployments. Prominent application domains are health assessments, wellbeing monitoring, and novel interaction paradigms. Arguably, and especially when compared to video based activity recognition, the use of movement sensors such as inertial measurement units has many practical advantages including mobility, more direct movement capturing, and less privacy invasive sensing. However, those advantages come with a cost with regards to the design of machine learning based pipelines for the analysis of the sensor data streams, i.e., multi-variate time series data. Probably the biggest challenge lies in the absence of (very) large, labeled datasets that can readily be used for developing and training complex analysis models. In this paper I explore the challenges for deriving machine learning based systems for sensor-based human activity recognition. My focus is on: i) the role of representation learning in modern HAR systems that overcome the need for explicit feature engineering; ii) self-supervised methods for most effective use of small, labeled datasets; and iii) cross-modality transfer. I conclude with an outlook of next steps in the field that promise to lead to increased robustness and more flexible applicability of contemporary, machine-learnlng based HAR systems.

Since human bodies are good reflectors of wireless signals, human activities can be recognized by monitoring changes in WiFi signals. However, existing WiFi-based human activity recognition systems do not build models that can quantify the correlation between WiFi signal dynamics and human activities. In this paper, we propose a Channel State Information (CSI)-based human Activity Recognition and Monitoring system (CARM). CARM is based on two theoretical models. First, we propose a CSI-speed model that quantifies the relation between CSI dynamics and human movement speeds. Second, we propose a CSI-activity model that quantifies the relation between human movement speeds and human activities. Based on these two models, we implemented the CARM on commercial WiFi devices. Our experimental results show that the CARM achieves recognition accuracy of 96% and is robust to environmental changes.