Personalized Human Activity Recognition: Real-Time On-Device Training and Inference

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

Modern monostatic radar-based human activity recognition (HAR) systems perform very well as long as the direction of human activities is either towards or away from the radar. The monostatic single-input single-output (SISO) and monostatic multiple-input multiple-output (MIMO) radar systems cannot detect motion of an object that moves perpendicularly to the radar’s boresight axis. Due to this physical layer limitation, today’s radar-based HAR systems fail to classify multi-directional human activities. In this paper, we resolve this typical but critical physical layer problem of contemporary HAR systems. We propose a HAR system underlying a distributed MIMO radar configuration, where multiple antennas of a millimeter wave MIMO radar system (Ancortek SDR-KIT 2400T2R4) are distributed in an indoor environment. In our proposed HAR system, we have two independent and identical monostatic radar subsystems that irradiate and capture the multi-directional human movement from two perspectives, which allows to compute two distinct time-variant radial velocity distributions. A feature extraction network extracts numerous features from the measured time-variant radial velocity distributions, which are then fused by a multiclass classifier to detect five types of human activities. The proposed multi-perspective MIMO-radar-based HAR system achieves a classification accuracy of 98.52%, which surpasses the accuracy of SISO radar-based HAR system by more than 9%. Our approach resolves the physical layer limitations of modern HAR systems that are based on either monostatic SISO or monostatic MIMO radar systems.

PurposeThe emergence of smart wearables using clothing as a technology platform is a significant milestone with considerable implications for industrial convergence, creating new value for fashion. This paper aimed to present a premeditated prototype to integrate a human activity recognition (HAR) system into outdoor clothing.Design/methodology/approachFor the development of wearable HAR (WHAR) clothing, this paper explored three subject areas: fashion design related to the structural feature of the clothing platform, electronics related to wearable circuits and modules design and graphic user interface design related to smartphone application development.FindingsFor WHAR functions in outdoor terrains, the coexistence of accelerometer–gyroscope sensing and distance-sensing could be practical to surpass the technological limitation of activity and posture recognition with gyro sensors highly depending on the changes of acceleration and angles.Research limitations/implicationsThrough the vital sign check and physical activity–change recognition function, this study's WHAR system allows users to check their health by themselves and avoid overwork. A quick rescue is possible manually and automatically in a dangerous situation by notifying others. Thus, it can help protect users' health and safety (life).Originality/valueThis study designed the modularization of HAR functions generally installed in indoor medical spaces. Through the approach, smart clothing–embracing WHAR systems optimized for health and safety care for outdoor environments was pursued to diversify expensive roles of clothing for technological applications.

Wireless body area network (WBAN) is a novel technology with the incorporation of numerous types of devices, which is also employed in health monitoring applications. Human activity recognition (HAR) receives more interest in recent times along with wearable sensors. HAR system provides information about a person's identity, personality, and psychological state. In the present scenario, it becomes important to model the active learning paradigms with the help of wearable sensors for analysing human activities. Although various deep learning models and existing algorithms secure better outcomes through the sensor data analysis regarding HAR, the decision‐making evaluation seems to be a complex one. The main intent of this paper is to implement the WBAN‐based HAR system using the improved deep learning model. By connecting the wearable sensors, the signals are gathered regarding human activity from diverse benchmark sources. After collecting the required signals, the pre‐processing of the input signals is done using artefact removal and median filtering. Further, the feature extraction is performed, which intends to extract a set of features by utilizing short‐time Fourier transform (STFT) and statistical features. For reducing the feature‐length, a multi‐objective‐based optimal feature selection is adopted. Human activities such as ‘walking, walking upstairs, walking downstairs, sitting, standing, lying, and jogging’ are recognized with help of selected optimal features. The optimized probabilistic neural network (PNN) and convolutional neural network (CNN) are combined and named as adaptive probabilistic‐based CNN (AP‐CNN). The effective performance of optimal feature selection and recognition is accomplished by incorporating the developed backward updating position‐based sea lion optimization algorithm (BU‐SLnO). Finally, the performance of the BU‐SLnO‐AP‐CNN‐based suggested model is analysed, which shows 1.001%, 1.237%, 0.811%, and 0.859% advanced than SLnO‐AP‐CNN, feed‐forward‐AP‐CNN (FF‐AP‐CNN), grey wolf optimization‐AP‐CNN (GWO‐AP‐CNN), particle swarm optimization‐AP‐CNN (PSO‐AP‐CNN) when observing the dataset 1. The experimental outcomes from comparison with various classification techniques demonstrate the efficiency of the developed technique.

Advances in deep learning, especially Convolutional Neural Networks (CNNs) have revolutionized intelligent frame-works such as Human Activity Recognition (HAR) systems by effectively and efficiently inferring human activity from various modalities of data. However, the training and inference of CNNs are often resource-intensive. Recent research developments are focused on bringing the effectiveness of CNNs in resource con-strained edge devices through Tiny Machine Learning (TinyML). However, this is extremely hard to achieve due to the limitations in memory, compute power, and energy of resource constrained edge devices. This paper provides a benchmark to understand these trade-offs among variations of CNN network architectures, different training methodologies, and different modalities of data in the context of HAR, TinyML, and edge devices. We tested and reported the performance of CNN and Depthwise Separable CNN (DSCNN) models as well as two training methodologies: Quantization Aware Training (QAT) and Post-training Quantization (PTQ) on five commonly used benchmark datasets containing image and time-series data: UP-Fall, Fall Detection Dataset (FDD), PAMAP2, UCI-HAR, and WISDM. We also deployed and tested the performance of the model-based standalone applications on multiple commonly available resource constrained edge devices in terms of inference time and power consumption. The experimental results demonstrate the effectiveness and feasibility of Tiny ML for HAR in edge devices.

Human Activity Recognition (HAR) systems are devised for continuously observing human behavior - primarily in the fields of environmental compatibility, sports injury detection, senior care, rehabilitation, entertainment, and the surveillance in intelligent home settings. Inertial sensors, e.g., accelerometers, linear acceleration, and gyroscopes are frequently employed for this purpose, which are now compacted into smart devices, e.g., smartphones. Since the use of smartphones is so widespread now-a-days, activity data acquisition for the HAR systems is a pressing need. In this article, we have conducted the smartphone sensor-based raw data collection, namely H-Activity, using an Android-OS-based application for accelerometer, gyroscope, and linear acceleration. Furthermore, a hybrid deep learning model is proposed, coupling convolutional neural network and long-short term memory network (CNN-LSTM), empowered by the self-attention algorithm to enhance the predictive capabilities of the system. In addition to our collected dataset (H-Activity), the model has been evaluated with some benchmark datasets, e.g., MHEALTH, and UCI-HAR to demonstrate the comparative performance of our model. When compared to other models, the proposed model has an accuracy of 99.93% using our collected H-Activity data, and 98.76% and 93.11% using data from MHEALTH and UCI-HAR databases respectively, indicating its efficacy in recognizing human activity recognition. We hope that our developed model could be applicable in the clinical settings and collected data could be useful for further research.

When developing a fully automatic system for evaluating motor activities performed by a person, it is necessary to segment and recognize the different activities in order to focus the analysis. This process must be carried out by a Human Activity Recognition (HAR) system. This paper proposes a user adaptation technique for improving a HAR system based on Hidden Markov Models (HMMs). This system segments and recognizes six different physical activities (walking, walking upstairs, walking downstairs, sitting, standing and lying down) using inertial signals from a smartphone. The system is composed of a feature extractor for obtaining the most relevant characteristics from the inertial signals, a module for training the six HMMs (one per activity), and the last module for segmenting new activity sequences using these models. The user adaptation technique consists of a Maximum A Posteriori (MAP) approach that adapts the activity HMMs to the user, using some activity examples from this specific user. The main results on a public dataset have reported a significant relative error rate reduction of more than 30%. In conclusion, adapting a HAR system to the user who is performing the physical activities provides significant improvement in the system’s performance.

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

Human activity recognition (HAR) is an important technology for a wide range of applications including elderly people monitoring, ambient assisted living, sport and fitness activities. The aim of this paper is to address the HAR task directly on a wearable device, implementing a recurrent neural network (RNN) on a low cost, low power microcontroller, ensuring the required performance in terms of accuracy and low complexity. To reach this goal we first develop a lightweight RNN on the Human Activity Recognition Using Smartphones dataset in order to accurately detect human activity and then we port the RNN to the embedded device Cloud-JAM L4, based on an STM32 microcontroller. Experimental results show that this HAR RNN-based detector can be effectively implemented on the chosen constrained-resource system, achieving an accuracy of about 90.50% with a very low memory cost (40.883 KB) and inference time (67.131 ms), allowing the design of a wearable embedded system for human activity recognition.

Human action recognition using genetic algorithms and convolutional neural networks

Human Activity Recognition (HAR) is the automatic detection and understanding of human motion behavior based on data extracted from video camera, ambient sensors or wearable sensors which particularly has recently attracted increased attention from both researchers and industry. However, for running practical HAR systems on wearable devices, there are some requirements such as design and development of small, lightweight, powerful, and low-cost smart sensors. In this context, data must be continuously collected, and edge computing is a viable solution, which is an energy-efficient technique, offering real-time response and privacy requirements for HAR applications. Rather than sending data to the cloud, edge computing is a local process that minimizes the data transmission time and responds with low latency. Recently, HAR system designers have adopted deep learning techniques inspired by their outstanding performance in many application areas and achieved relevant gain in activity recognition performance, however, these techniques were not demonstrated suitable for running on resource constrained devices. Thus, designing energy-efficient deep learning models is critical for realizing efficient HAR for mobile applications. In this work, we present a lightweight framework for the deployment of low-power but accurate HAR systems for these devices. We also implement and run the system in a microcontroller and analyze computational cost and energy consumption, and how different system configurations and deep learning model complexity influence on that.

Human Activity Recognition (HAR) systems are designed to read sensor data and analyse it to classify any detected movement and respond accordingly. However, there is a need for more responsive and near real-time systems to distinguish between false and true alarms. To accurately determine alarm triggers, the motion pattern of legitimate users need to be stored over a certain period and used to train the system to recognise features associated with their movements. This training process is followed by a testing cycle that uses actual data of different patterns of activity that are either similar or different to the training data set. This paper evaluates the use of a combined Convolutional Neural Network (CNN) and Naive Bayes for accuracy and robustness to correctly identify true alarm triggers in the form of a buzzer sound for example. It shows that pattern recognition can be achieved using either of the two approaches, even when a partial motion pattern is derived as a subset out of a full-motion path.

PurposeTo find a successful human action recognition system (HAR) for the unmanned environments.Design/methodology/approachThis paper describes the key technology of an efficient HAR system. In this paper, the advancements for three key steps of the HAR system are presented to improve the accuracy of the existing HAR systems. The key steps are feature extraction, feature descriptor and action classification, which are implemented and analyzed. The usage of the implemented HAR system in the self-driving car is summarized. Finally, the results of the HAR system and other existing action recognition systems are compared.FindingsThis paper exhibits the proposed modification and improvements in the HAR system, namely the skeleton-based spatiotemporal interest points (STIP) feature and the improved discriminative sparse descriptor for the identified feature and the linear action classification.Research limitations/implicationsThe experiments are carried out on captured benchmark data sets and need to be analyzed in a real-time environment.Practical implicationsThe middleware support between the proposed HAR system and the self-driven car system provides several other challenging opportunities in research.Social implicationsThe authors’ work provides the way to go a step ahead in machine vision especially in self-driving cars.Originality/valueThe method for extracting the new feature and constructing an improved discriminative sparse feature descriptor has been introduced.

Virtual Inertial Measurement Units (IMUs) offer a promising approach to generating synthetic motion data for training and evaluating human activity recognition (HAR) systems. However, existing virtual IMU workflows remain fragmented and technically demanding, requiring users to switch between 3D editors, scripting tools, and offline signal processing pipelines. These limitations hinder usability and iterative refinement for researchers and developers who design HAR systems. We present Vsens , an XR-based system that unifies virtual IMU configuration, visualization, and data synthesis within an immersive workspace. Vsens allows developers to directly manipulate sensor placements on digital avatars and observe synthesized IMU signals in real time. To understand how developers with different expertise levels benefit from such an environment, we conducted a user study with 20 HAR developers covering a broad range of experience levels. Results show that Vsens improves configuration efficiency and fosters deeper spatial understanding of sensor behavior across the participants. Beyond usability, our findings reveal how embodied interaction and real-time feedback inform the design of practical virtual data collection systems that foster more effective and human-centered workflows.

Cross-subject transfer learning in human activity recognition systems using generative adversarial networks