Real-time human activity recognition on edge devices using quantized deep neural networks

<title>Abstract</title> The advent of wearable sensor technologies has transformed sports science and personal health monitoring, but real-time Human Activity Recognition (HAR) remains a challenge, especially in dynamic environments. Existing approaches often struggle to adapt to rapid activity transitions, handle multimodal sensor noise, and optimize for computational efficiency on resource-constrained wearable devices. In this work, we introduce an innovative system that overcomes these limitations through a novel integration of deep learning models and sensor fusion techniques. At its core, our framework features the Attention-Gated Spatiotemporal Fusion Network (AG-SFN), a deep neural network that combines Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks with a unique attention mechanism to dynamically prioritize the most relevant features from multimodal time-series data for accurate activity classification. To address sensor noise and improve data integrity, we propose the Adaptive Multimodal Kalman Fusion (AMKF) algorithm, which adapts the filtering process based on the motion intensity and sensor variance in real time. To ensure the efficient deployment of this model on wearable devices, we introduce a Dynamic Channel Pruning with Knowledge Distillation (DCP-KD) method, which significantly reduces the model size and computational load by pruning less relevant features and distilling knowledge from a large teacher model to a smaller student model. Our system was rigorously evaluated on a custom dataset of 40 participants performing six distinct sport-related activities, achieving a state-of-the-art activity recognition accuracy of 98.6%, outperforming conventional models by a significant margin. The system also demonstrated exceptional real-time performance with heart rate monitoring error reduced to 0.85 beats per minute (bpm), latency under 90 ms, and a 35% reduction in energy consumption. This research establishes a new benchmark for the development of efficient and adaptive wearable systems for real-time sports health monitoring, offering a robust solution for next-generation wearable technologies with practical applications in sports, fitness, and healthcare.

Real-time human activity recognition is essential for human-robot interactions for assisted healthy independent living. Most previous work in this area is performed on traditional two-dimensional (2D) videos and both global and local methods have been used. Since 2D videos are sensitive to changes of lighting condition, view angle, and scale, researchers begun to explore applications of 3D information in human activity understanding in recently years. Unfortunately, features that work well on 2D videos usually don't perform well on 3D videos and there is no consensus on what 3D features should be used. Here we propose a model of human activity recognition based on 3D movements of body joints. Our method has three steps, learning dictionaries of sparse codes of 3D movements of joints, sparse coding, and classification. In the first step, space-time volumes of 3D movements of body joints are obtained via dense sampling and independent component analysis is then performed to construct a dictionary of sparse codes for each activity. In the second step, the space-time volumes are projected to the dictionaries and a set of sparse histograms of the projection coefficients are constructed as feature representations of the activities. Finally, the sparse histograms are used as inputs to a support vector machine to recognize human activities. We tested this model on three databases of human activities and found that it outperforms the state-of-the-art algorithms. Thus, this model can be used for real-time human activity recognition in many applications.

Real-time human activity recognition plays an essential role in real-world human-centered robotics applications, such as assisted living and human-robot collaboration. Although previous methods based on skeletal data to encode human poses showed promising results on real-time activity recognition, they lacked the capability to consider the context provided by objects within the scene and in use by the humans, which can provide a further discriminant between human activity categories. In this paper, we propose a novel approach to real-time human activity recognition, through simultaneously learning from observations of both human poses and objects involved in the human activity. We formulate human activity recognition as a joint optimization problem under a unified mathematical framework, which uses a regression-like loss function to integrate human pose and object cues and defines structured sparsity-inducing norms to identify discriminative body joints and object attributes. To evaluate our method, we perform extensive experiments on two benchmark datasets and a physical robot in a home assistance setting. Experimental results have shown that our method outperforms previous methods and obtains real-time performance for human activity recognition with a processing speed of 104 Hz.

Analyzing physical activities through wearable devices is a promising research area for improving health assessment. This research focuses on the development of an affordable and real-time Human Activity Recognition (HAR) sys... | Find, read and cite all the research you need on Tech Science Press

This paper presents an efficient technique for real-time recognition of human activities by using accelerometer and photoplethysmography (PPG) data. It is based on singular value decomposition (SVD) and truncated Karhunen-Loeve transform (KLT) for feature extraction and reduction, and Bayesian classification for class recognition. Due to the nature of signals, and being the algorithm independent from the orientation of the inertial sensor, this technique is particularly suitable for implementation in smartwatches in order to both recognize the exercise being performed and improve the motion artifact (MA) removal from PPG signal for accurate heart rate (HR) estimation. In order to demonstrate the validity of this methodology, it has been successfully applied to a database of accelerometer and PPG data derived from four dynamic activities.

This paper presents an effective approach for recognising human actions from depth video sequences by employing depth motion maps (DMMs) and convolutional neural networks (CNNs). Depth maps are projected onto three orthogonal planes, and frame differences under each view (front/side/top) are then accumulated through an entire depth video sequence generating a DMM. We build a model architecture of multi-view convolutional neural network (MV-CNN) containing multiple networks to deal with three DMMs (DMMf, DMMs, DMMt). The output of full-connected layer under each view is integrated as feature representation, which is then learned in the last softmax regression layer to predict human actions. Experimental results on MSR-Action3D dataset and UTD-MHAD dataset indicate that the proposed approach achieves state-of-the-art recognition performance and is appropriate for real-time recognition.

The paper presents a novel lightweight framework for real-time human detection, tracking, and action recognition in low-light environments, addressing critical challenges of poor visibility and noise in surveillance and autonomous systems. Key problems tackled are insufficient image contrast, real-time detection misses, high computational load, and class imbalance. We propose: Zero-Reference Deep Curve Estimation (Zero-DCE) for dark region contrast enhancement; an optimized YOLOv5 detector with cascaded 3×3 max-pooling stacks; a core Epsilon sampling strategy ensuring temporal diversity and computational efficiency by strategically selecting frames to avoid omitting semantically critical content; a ResNet-34-based R(2+1)D network combined with a Transformer-style BERT module for robust action recognition under occlusion/low contrast; and a “deep compression” pipeline. Focused loss and data augmentation mitigate class imbalance. Evaluated on low-light datasets (ARID, HMDB51, synthetic HMDB51-dark), our framework achieves state-of-the-art performance. It reduces model size by 50% and inference latency by 28% while maintaining near-original accuracy.

A Lightweight Deep Learning Model for Human Activity Recognition on Edge Devices

Heart disease is a heart condition that sometimes causes a person to die suddenly. One indication is a rhythm disorder known as arrhythmia. Multi-class Arrhythmia Detection has followed: QRS complex detection procedure and arrhythmia classification based on the QRS complex morphology. We proposed an edge device that detects QRS complexes based on variance analysis (QVAT) and the arrhythmia classification based on the QRS complex spectrogram. The classifier uses two-dimensional convolutional neural network (2D CNN) deep learning. We use a single board computer and neural network compute stick to implement the edge device. The outcomes are a prototype device cardiologists use as a supporting tool for analysing ECG signals, and patients can also use it for self-tests to figure out their heart health. To evaluate the performance of our edge device, we tested using the MIT-BIH database because other methods also use the data. The QVAT sensitivity and predictive positive are 99.81% and 99.90%, respectively. Our classifier's accuracy, sensitivity, predictive positive, specificity, and F1-score are 99.82%, 99.55%, 99.55%, 99.89%, and 99.55%, respectively. The experiment result of arrhythmia classification shows that our method outperforms the others. Still, for r-peak detection, the QVAT implemented in an edge device is comparable to the other methods. In future work, we can improve the performance of r-peak detection using the double-check algorithm in QVAT and cross-check the QRS complex detection by adding 1 class to the classifier, namely the non-QRS class.

Deep neural networks (DNNs) have seen resurgent attraction to be implemented in edge applications. However, such implementations are not easy to achieve because execution of DNNs often require more resources than those provided by individual edge devices. On the other hand, relying on model-level distribution methods to implement a DNN on connected edge devices leads to costly communication overheads. To utilize available in-the-edge resources with less communication overhead, we propose using edge-tailored models comprised of nearly-independent narrow DNNs, the inference of which are accelerated using small cost-efficient RISC-based engines. We implement these engines on PYNQ boards as a platform that mimics the limited resources of edge devices. We create the narrow DNNs based on the available resources of PYNQ boards, and allocate each narrow DNN to one engine, implemented in an FPGA. We compare the communication overhead of our implantation against the state-of-the-art model-level distribution methods.

Advances in deep learning, especially Convolutional Neural Networks (CNNs) have revolutionized intelligent frame-works such as Human Activity Recognition (HAR) systems by effectively and efficiently inferring human activity from various modalities of data. However, the training and inference of CNNs are often resource-intensive. Recent research developments are focused on bringing the effectiveness of CNNs in resource con-strained edge devices through Tiny Machine Learning (TinyML). However, this is extremely hard to achieve due to the limitations in memory, compute power, and energy of resource constrained edge devices. This paper provides a benchmark to understand these trade-offs among variations of CNN network architectures, different training methodologies, and different modalities of data in the context of HAR, TinyML, and edge devices. We tested and reported the performance of CNN and Depthwise Separable CNN (DSCNN) models as well as two training methodologies: Quantization Aware Training (QAT) and Post-training Quantization (PTQ) on five commonly used benchmark datasets containing image and time-series data: UP-Fall, Fall Detection Dataset (FDD), PAMAP2, UCI-HAR, and WISDM. We also deployed and tested the performance of the model-based standalone applications on multiple commonly available resource constrained edge devices in terms of inference time and power consumption. The experimental results demonstrate the effectiveness and feasibility of Tiny ML for HAR in edge devices.

Human activity recognition (HAR) plays a critical role in a wide range of real-world applications, and it is traditionally achieved via wearable sensing. Recently, to avoid the burden and discomfort caused by wearable devices, device-free approaches exploiting radio-frequency (RF) signals arise as a promising alternative for HAR. Most of the latest device-free approaches require training a large deep neural network model in either time or frequency domain, entailing extensive storage to contain the model and intensive computations to infer human activities. Consequently, even with some major advances on device-free HAR, current device-free approaches are still far from practical in real-world scenarios where the computation and storage resources possessed by, for example, edge devices, are limited. To overcome these weaknesses, we introduce HAR-SAnet which is a novel RF-based HAR framework. It adopts an original signal adapted convolutional neural network architecture: instead of feeding the handcraft features of RF signals into a classifier, HAR-SAnet fuses them adaptively from both time and frequency domains to design an end-to-end neural network model. We apply point-wise grouped convolution and depth-wise separable convolutions to confine the model scale and to speed up the inference execution time. The experiment results show that the recognition accuracy of HAR-SAnet substantially outperforms the state-of-the-art algorithms and systems.

DeepEdgeSoC: End-to-end deep learning framework for edge IoT devices

Machine learning has surged in popularity in recent years, with deep neural networks (DNNs) becoming a cornerstone of applications such as autonomous driving, augmented reality, and natural language processing, owing to their remarkable accuracy and scalability. As the demand for greater portability and flexibility grows, there is a strong push to optimize these applications for resource-constrained edge devices like smartphones and IoT devices.However, deploying DNN models on such devices presents significant challenges, primarily due to the large size of these models, which is a key contributor to their high performance. For instance, the DeepSpeech 2 model outperforms its predecessor, DeepSpeech 1, but its size is eight times larger, encompassing approximately 68 million parameters. Similarly, other networks designed for various tasks, including convolutional neural networks like VGG and transformer-based networks like BERT and GPT, also exhibit substantial sizes. For example, the GPT-3 model, a predecessor to ChatGPT, consists of an astonishing 175 billion parameters. Consequently, fitting DNN models onto resource-limited edge devices is a formidable challenge. My thesis addresses these challenges with a three-part approach: * High-Performance DNN Accelerators: I will develop DNN accelerators that excel in high performance, low latency, and energy efficiency through an algorithm and hardware co-design across diverse FPGA architectures (Chapter 2). * Real-Time DNN Execution: I aim to achieve exceptional accuracy and real-time execution for various large-scale DNNs on off-the-shelf mobile devices, IoT devices, and other platforms (Chapter 3). * CompVQC Framework: I will explore a systematic framework, CompVQC, to reduce the circuit length of quantum neural networks (QNNs) through model compression, which can achieve up to a 20% improvement in accuracy. This reveals that compilation-aware compression can enhance the robustness of QNNs on near-term noisy quantum devices (Chapter 4). The field of AI has recently undergone a significant transformation, ushering in the era of generative AI, where models like ChatGPT and Stable Diffusion create realistic and creative content for a wide range of applications. As generative AI becomes more prevalent, it is crucial to adapt algorithm-hardware co-design strategies to meet the specific needs of these models, particularly on edge devices. In the final chapter (Chapter 5), I will introduce my ongoing and future work focused on activating generative AI on the edge.--Author's abstract

Efficient deployment of Deep Neural Networks (DNNs) on edge devices (i.e., FPGAs and mobile platforms) is very challenging, especially under a recent witness of the increasing DNN model size and complexity. Model compression strategies, including weight quantization and pruning, are widely recognized as effective approaches to significantly reduce computation and memory intensities, and have been implemented in many DNNs on edge devices. However, most state-of-the-art works focus on ad hoc optimizations, and there lacks a thorough study to comprehensively reveal the potentials and constraints of different edge devices when considering different compression strategies. In this article, we qualitatively and quantitatively compare the energy efficiency of FPGA-based and mobile-based DNN executions using mobile GPU and provide a detailed analysis. Based on the observations obtained from the analysis, we propose a unified optimization framework using block-based pruning to reduce the weight storage and accelerate the inference speed on mobile devices and FPGAs, achieving high hardware performance and energy-efficiency gain while maintaining accuracy.