TinyHAR: Benchmarking Human Activity Recognition Systems in Resource Constrained Devices

A multi-resolution fusion approach for human activity recognition from video data in tiny edge devices

Convolutional neural networks (CNNs) have gained a huge attention for real-world artificial intelligence (AI) applications such as image classification and object detection. On the other hand, for better accuracy, the size of the CNNs’ parameters (weights) has been increasing, which in turn makes it difficult to enable on-device CNN inferences in resource-constrained edge devices. Though weight pruning and 5-bit quantization methods have shown promising results, it is still challenging to deploy large CNN models in edge devices. In this paper, we propose an encoding and hardware-based decoding technique which can be applied to 5-bit quantized weight data for on-device CNN inferences in resource-constrained edge devices. Given 5-bit quantized weight data, we employ arithmetic coding with range scaling for lossless weight compression, which is performed offline. When executing on-device inferences with underlying CNN accelerators, our hardware decoder enables a fast in-situ weight decompression with small latency overhead. According to our evaluation results with five widely used CNN models, our arithmetic coding-based encoding method applied to 5-bit quantized weights shows a better compression ratio by 9.6× while also reducing the memory data transfer energy consumption by 89.2%, on average, as compared to the case of uncompressed 32-bit floating-point weights. When applying our technique to pruned weights, we obtain better compression ratios by 57.5×–112.2× while reducing energy consumption by 98.3%–99.1% as compared to the case of 32-bit floating-point weights. In addition, by pipelining the weight decoding and transfer with the CNN execution, the latency overhead of our weight decoding with 16 decoding unit (DU) hardware is only 0.16%–5.48% and 0.16%–0.91% for non-pruned and pruned weights, respectively. Moreover, our proposed technique with 4-DU decoder hardware reduces system-level energy consumption by 1.1%–9.3%.

The conventional Human Activity Recognition (HAR) systems use smartwatches, smartphones, and other wearable devices to autonomously derive consumer physical activities. However, implementing real-time HAR in resource-constrained applications is challenging. Computational offloading approaches must address network reliance, latency, and data privacy, which affect system performance. The present study proposes a low-cost, wearable HAR device that collects and transmits the Inertial Measurement Unit (IMU) telemetry signals to smartphones, enabling real-time on-device activity detection. Human actions are subjective and affected by physiological, environmental, and behavioural factors, complicating HAR systems' performance in personalized contexts. Generalized methods for existing systems reduce performance for new consumers or due to alterations in consumer locomotive signatures. Thus, our second goal is to enable the on-device personalization of wearable HAR models with minimum customer calibration. The HAR device, which is in the form of a smart-band, transfers data to and fro smartphones through Bluetooth Low Energy (BLE). A lightweight 1D convolutional neural network (CNN) is built and, using transfer learning, the network model is fine-tuned from real-time sensed data. On-device HAR inferences ensure real-time processing on smartphones without computational offloading. Smartphone-based on-device training for fine-tuning the model allows the HAR system to get customized without compromising privacy or computational costs. The experimental results show that the personalized classification achieves on average 98% accuracy when assessed using four benchmark datasets.

For model inference of convolutional neural networks (CNNs), we nowadays witness a shift from the Cloud to the Edge. Unfortunately, deploying and inferring large, compute- and memory-intensive CNNs on Internet of Things devices at the Edge is challenging as they typically have limited resources. One approach to address this challenge is to leverage all available resources across multiple edge devices to execute a large CNN by properly partitioning it and running each CNN partition on a separate edge device. However, there currently does not exist a design and programming framework that takes a trained CNN model as input and subsequently allows for efficiently exploring and automatically implementing a range of different CNN partitions on multiple edge devices to facilitate distributed CNN inference. Therefore, in this article, we propose a novel framework that automates the splitting of a CNN model into a set of submodels as well as the code generation needed for the distributed and collaborative execution of these submodels on multiple, possibly heterogeneous, edge devices, while supporting the exploitation of parallelism among and within the edge devices. In addition, since the number of different CNN mapping possibilities on multiple edge devices is vast, our framework also features a multistage and hierarchical design space exploration methodology to efficiently search for (near-)optimal distributed CNN inference implementations. Our experimental results demonstrate that our work allows for rapidly finding and realizing distributed CNN inference implementations with reduced energy consumption and memory usage per edge device, and under certain conditions, with improved system throughput as well.

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

Modern monostatic radar-based human activity recognition (HAR) systems perform very well as long as the direction of human activities is either towards or away from the radar. The monostatic single-input single-output (SISO) and monostatic multiple-input multiple-output (MIMO) radar systems cannot detect motion of an object that moves perpendicularly to the radar’s boresight axis. Due to this physical layer limitation, today’s radar-based HAR systems fail to classify multi-directional human activities. In this paper, we resolve this typical but critical physical layer problem of contemporary HAR systems. We propose a HAR system underlying a distributed MIMO radar configuration, where multiple antennas of a millimeter wave MIMO radar system (Ancortek SDR-KIT 2400T2R4) are distributed in an indoor environment. In our proposed HAR system, we have two independent and identical monostatic radar subsystems that irradiate and capture the multi-directional human movement from two perspectives, which allows to compute two distinct time-variant radial velocity distributions. A feature extraction network extracts numerous features from the measured time-variant radial velocity distributions, which are then fused by a multiclass classifier to detect five types of human activities. The proposed multi-perspective MIMO-radar-based HAR system achieves a classification accuracy of 98.52%, which surpasses the accuracy of SISO radar-based HAR system by more than 9%. Our approach resolves the physical layer limitations of modern HAR systems that are based on either monostatic SISO or monostatic MIMO radar systems.

Human Activity Recognition (HAR) systems are designed to read sensor data and analyse it to classify any detected movement and respond accordingly. However, there is a need for more responsive and near real-time systems to distinguish between false and true alarms. To accurately determine alarm triggers, the motion pattern of legitimate users need to be stored over a certain period and used to train the system to recognise features associated with their movements. This training process is followed by a testing cycle that uses actual data of different patterns of activity that are either similar or different to the training data set. This paper evaluates the use of a combined Convolutional Neural Network (CNN) and Naive Bayes for accuracy and robustness to correctly identify true alarm triggers in the form of a buzzer sound for example. It shows that pattern recognition can be achieved using either of the two approaches, even when a partial motion pattern is derived as a subset out of a full-motion path.

Human action recognition using genetic algorithms and convolutional neural networks

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

Wireless body area network (WBAN) is a novel technology with the incorporation of numerous types of devices, which is also employed in health monitoring applications. Human activity recognition (HAR) receives more interest in recent times along with wearable sensors. HAR system provides information about a person's identity, personality, and psychological state. In the present scenario, it becomes important to model the active learning paradigms with the help of wearable sensors for analysing human activities. Although various deep learning models and existing algorithms secure better outcomes through the sensor data analysis regarding HAR, the decision‐making evaluation seems to be a complex one. The main intent of this paper is to implement the WBAN‐based HAR system using the improved deep learning model. By connecting the wearable sensors, the signals are gathered regarding human activity from diverse benchmark sources. After collecting the required signals, the pre‐processing of the input signals is done using artefact removal and median filtering. Further, the feature extraction is performed, which intends to extract a set of features by utilizing short‐time Fourier transform (STFT) and statistical features. For reducing the feature‐length, a multi‐objective‐based optimal feature selection is adopted. Human activities such as ‘walking, walking upstairs, walking downstairs, sitting, standing, lying, and jogging’ are recognized with help of selected optimal features. The optimized probabilistic neural network (PNN) and convolutional neural network (CNN) are combined and named as adaptive probabilistic‐based CNN (AP‐CNN). The effective performance of optimal feature selection and recognition is accomplished by incorporating the developed backward updating position‐based sea lion optimization algorithm (BU‐SLnO). Finally, the performance of the BU‐SLnO‐AP‐CNN‐based suggested model is analysed, which shows 1.001%, 1.237%, 0.811%, and 0.859% advanced than SLnO‐AP‐CNN, feed‐forward‐AP‐CNN (FF‐AP‐CNN), grey wolf optimization‐AP‐CNN (GWO‐AP‐CNN), particle swarm optimization‐AP‐CNN (PSO‐AP‐CNN) when observing the dataset 1. The experimental outcomes from comparison with various classification techniques demonstrate the efficiency of the developed technique.

When developing a fully automatic system for evaluating motor activities performed by a person, it is necessary to segment and recognize the different activities in order to focus the analysis. This process must be carried out by a Human Activity Recognition (HAR) system. This paper proposes a user adaptation technique for improving a HAR system based on Hidden Markov Models (HMMs). This system segments and recognizes six different physical activities (walking, walking upstairs, walking downstairs, sitting, standing and lying down) using inertial signals from a smartphone. The system is composed of a feature extractor for obtaining the most relevant characteristics from the inertial signals, a module for training the six HMMs (one per activity), and the last module for segmenting new activity sequences using these models. The user adaptation technique consists of a Maximum A Posteriori (MAP) approach that adapts the activity HMMs to the user, using some activity examples from this specific user. The main results on a public dataset have reported a significant relative error rate reduction of more than 30%. In conclusion, adapting a HAR system to the user who is performing the physical activities provides significant improvement in the system’s performance.

In this paper, we analyze the suitability of convolutional neural network (CNN) inference workloads on a phase-change memory (PCM) platform. CNN inference has an average of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$14\times$</tex> more read requests than write requests (i.e., read dominant) and a significantly low last-level cache misses per kilo instructions (LLC MPKI) of 2 on average (i.e., computation intensive). In addition, to compare the latency and energy of PCM and DRAM systems, we evaluate CNN inference workloads on two memory systems through a memory simulator. As a result, compared to DRAM, PCM can save total energy by 54% on average, but instruction per cycle (IPC) of PCM is reduced by an average of 28%. In conclusion, CNN inference is a workload suitable for PCM in terms of energy efficiency, but it must be accompanied by a scheme to improve IPC for practical use.

Human activity recognition (HAR) is an important technology for a wide range of applications including elderly people monitoring, ambient assisted living, sport and fitness activities. The aim of this paper is to address the HAR task directly on a wearable device, implementing a recurrent neural network (RNN) on a low cost, low power microcontroller, ensuring the required performance in terms of accuracy and low complexity. To reach this goal we first develop a lightweight RNN on the Human Activity Recognition Using Smartphones dataset in order to accurately detect human activity and then we port the RNN to the embedded device Cloud-JAM L4, based on an STM32 microcontroller. Experimental results show that this HAR RNN-based detector can be effectively implemented on the chosen constrained-resource system, achieving an accuracy of about 90.50% with a very low memory cost (40.883 KB) and inference time (67.131 ms), allowing the design of a wearable embedded system for human activity recognition.

With the proliferation of IoT technologies and the development of edge computing, achieving efficient deep learning inference on resource-constrained edge devices has become a critical challenge. This paper proposes a model compression framework for IoT edge devices based on knowledge distillation (PKD), aimed at effectively compressing models while enhancing their robustness and privacy protection on edge devices. Specifically, the proposed PKD framework introduces three key technical components: (1) a partitioned knowledge distillation strategy that enables efficient deployment of teacher models in distributed edge environments, (2) a Gradient-Weighted Feature Attention Loss (GW-FAL) mechanism to improve feature-level knowledge transfer, and (3) a Targeted Fast Gradient Perturbation (TFGP) method to enhance model robustness against adversarial disturbances. By partitioning the teacher model deployment, applying GW-FAL to improve knowledge transfer, and introducing adversarial perturbations to enhance model robustness, PKD achieves a high compression rate without significantly sacrificing accuracy. Through extensive experiments, the PKD framework demonstrates superior noise robustness and defense against adversarial attacks while improving model compression efficiency. The experimental results show that the framework provides an ideal balance between accuracy and speed on resource-constrained devices, making it suitable for practical applications such as IoT.

Human Activity Recognition (HAR) based on data from wearable sensors has become an attractive research topic thanks to its applications in different fields such as healthcare and smart environments. Recently, the advancement of deep learning with capability to perform automatically high-level feature extraction has achieved promising results. However, the performance of the deep learning models depends deeply on the characteristics of the datasets such as the number of classes, the inter-similarity and intra-variation. Therefore, directly comparing these models has become difficult since a wide variety of experimental protocols, evaluation metrics, and datasets are employed. In this paper, for the first time, a systematic evaluation of several deep learning models for HAR from wearable sensors is provided. In particular, three models named Convolutional Neural Network (CNN) [1], DeepConvLSTM - a combination of CNN and Long Short Term Memory (LSTM) [2], and SensCapsNet - a Capsule Neural Network for wearable sensor-based HAR [3] were implemented and evaluated on three benchmark datasets that are 19NonSens, CMDFall, and UCI-HAR dataset. Moreover, to have an intuitive explanation of deep learning models, a visualization of features learnt from these models is given. The evaluation codebase and results will be made publicly available for community use.