Human action recognition using genetic algorithms and convolutional neural networks

The conventional Human Activity Recognition (HAR) systems use smartwatches, smartphones, and other wearable devices to autonomously derive consumer physical activities. However, implementing real-time HAR in resource-constrained applications is challenging. Computational offloading approaches must address network reliance, latency, and data privacy, which affect system performance. The present study proposes a low-cost, wearable HAR device that collects and transmits the Inertial Measurement Unit (IMU) telemetry signals to smartphones, enabling real-time on-device activity detection. Human actions are subjective and affected by physiological, environmental, and behavioural factors, complicating HAR systems' performance in personalized contexts. Generalized methods for existing systems reduce performance for new consumers or due to alterations in consumer locomotive signatures. Thus, our second goal is to enable the on-device personalization of wearable HAR models with minimum customer calibration. The HAR device, which is in the form of a smart-band, transfers data to and fro smartphones through Bluetooth Low Energy (BLE). A lightweight 1D convolutional neural network (CNN) is built and, using transfer learning, the network model is fine-tuned from real-time sensed data. On-device HAR inferences ensure real-time processing on smartphones without computational offloading. Smartphone-based on-device training for fine-tuning the model allows the HAR system to get customized without compromising privacy or computational costs. The experimental results show that the personalized classification achieves on average 98% accuracy when assessed using four benchmark datasets.

Recently, the continuous evolution of deep learning has opened up promising avenues to groundbreaking advancements in wireless sensing systems, which significantly enhance the practical applications of WiFi-based Human Activity Recognition (HAR) systems. However, despite these strides, such systems remain susceptible to adversarial attacks. This article unveils the vulnerability of existing WiFi-based HAR systems to common adversaries, revealing their insufficient robustness. While the intuitive approach is to employ adversarial training to fortify the models, our investigation exposes inherent deficiencies in the current approach. Specifically, we confirm that the strength of perturbations directly influences training outcomes. Moreover, even when confined within a specified perturbation radius, the perturbation strength exhibits variability within a prescribed range, potentially giving rise to “extreme” samples that could compromise training results. To address this challenge, we propose a two-stage Selective Adversarial Training (SAT) approach that integrates model confidence calibration and sample selection. Specifically, we start with calibrating the model and then selectively choose samples from all adversarial examples based on the calibrated confidence outputs that align with the desired criteria for adversarial training. This sample-wise perturbation intensity control effectively prevents the inclusion of inappropriate samples in training, a capability lacking in previous domain-wise perturbation control. Our experiments demonstrate that the proposed fine-grained training method, SAT, is both straightforward and effective in augmenting adversarial training results.

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

Modern monostatic radar-based human activity recognition (HAR) systems perform very well as long as the direction of human activities is either towards or away from the radar. The monostatic single-input single-output (SISO) and monostatic multiple-input multiple-output (MIMO) radar systems cannot detect motion of an object that moves perpendicularly to the radar’s boresight axis. Due to this physical layer limitation, today’s radar-based HAR systems fail to classify multi-directional human activities. In this paper, we resolve this typical but critical physical layer problem of contemporary HAR systems. We propose a HAR system underlying a distributed MIMO radar configuration, where multiple antennas of a millimeter wave MIMO radar system (Ancortek SDR-KIT 2400T2R4) are distributed in an indoor environment. In our proposed HAR system, we have two independent and identical monostatic radar subsystems that irradiate and capture the multi-directional human movement from two perspectives, which allows to compute two distinct time-variant radial velocity distributions. A feature extraction network extracts numerous features from the measured time-variant radial velocity distributions, which are then fused by a multiclass classifier to detect five types of human activities. The proposed multi-perspective MIMO-radar-based HAR system achieves a classification accuracy of 98.52%, which surpasses the accuracy of SISO radar-based HAR system by more than 9%. Our approach resolves the physical layer limitations of modern HAR systems that are based on either monostatic SISO or monostatic MIMO radar systems.

A multi-resolution fusion approach for human activity recognition from video data in tiny edge devices

Accelerometer-based human activity recognition (HAR) wearable systems are location-centric and noisy, needing multiple sensors with complex signal processing and filtering mechanisms. A recently reported alternative approach using a wearable suspender integrated with strain sensors and machine learning presented a viable option for nonlocalized measurement with less noise and better recognition capabilities. The washability and wearability of the strain sensor instrumented suspenders due to the physical wires are limited, and the power consumption is higher, which needs to be minimized to extend the battery life of the wearable device. This article proposes an improved body-worn suspender-based HAR system built using a conductive knit jersey fabric material that overcomes the existing strain sensor-based wearable device’s limitations and at the same time provides improved sensitivity. The proposed suspender system recognizes 14 human activities using machine learning and deep learning algorithms with the best accuracy of 98.11%. A performance comparison of machine learning models based on two dimensionality reduction techniques using kernel and linear discriminatory analysis was conducted. The kernel-based method outperformed the linear one in recognizing human activities across all classifiers. The durability of the wearable is tested by washing the sensor, and the recognition capabilities were consistent before and after the wash.

With the rapid increase in digital technology, most research areas are involved in human activity recognition, which can help to analyze the activities of patients. A novel approach for human activity recognition in egocentric video has been invoked in this research article. Generally, only the objects are identified, but the actions are not recognized. With this motivation and new trends, this paper presents an efficient technique to recognize the activities. In our approach, first the various activity dataset is trained, and the feature vector values are stored for various activities, which are applied to the testing inputs. Here, we use a filtering technique, i.e., a median filter followed by a segmentation method using watershed and feature extraction, such as a Histogram of Oriented Gradient (HOG), Color and GiST and a combination of all Features. Features are reduced using a genetic algorithm, and classification is done using Support Vector Machine (SVM) and a Random Forest classifier. The experimental results demonstrate that the Random Forest classifier outperformed the SVM classifier.

Advances in deep learning, especially Convolutional Neural Networks (CNNs) have revolutionized intelligent frame-works such as Human Activity Recognition (HAR) systems by effectively and efficiently inferring human activity from various modalities of data. However, the training and inference of CNNs are often resource-intensive. Recent research developments are focused on bringing the effectiveness of CNNs in resource con-strained edge devices through Tiny Machine Learning (TinyML). However, this is extremely hard to achieve due to the limitations in memory, compute power, and energy of resource constrained edge devices. This paper provides a benchmark to understand these trade-offs among variations of CNN network architectures, different training methodologies, and different modalities of data in the context of HAR, TinyML, and edge devices. We tested and reported the performance of CNN and Depthwise Separable CNN (DSCNN) models as well as two training methodologies: Quantization Aware Training (QAT) and Post-training Quantization (PTQ) on five commonly used benchmark datasets containing image and time-series data: UP-Fall, Fall Detection Dataset (FDD), PAMAP2, UCI-HAR, and WISDM. We also deployed and tested the performance of the model-based standalone applications on multiple commonly available resource constrained edge devices in terms of inference time and power consumption. The experimental results demonstrate the effectiveness and feasibility of Tiny ML for HAR in edge devices.

Human Activity Recognition (HAR) has become one of the most important research fields to achieve real-time monitoring of human activities for timely decision making in various applications like fall detection, elderly care etc. Now-a-days, most people use smartphones which come with various embedded inertial sensors like accelerometer and gyroscope to monitor acceleration and angular velocity. These smartphone-based sensors have proven to be cost-effective solution in identification of activities belonging to ADL (Activities of Daily Living). Various Machine Learning, Deep learning and hybrid models have been proposed and implemented for HAR. This paper also proposes various hybrid deep learning approaches which combine Deep Neural Networks with other models like LSTM (Long Short Term Memory) Model and GRU (Gated Recurrent Unit) for effective classification of engineered features from CNN (Convolutional Neural Network) Model. A novel architecture that integrates CNN with Random Forest Classifier (DeepCNN-RF) is proposed to add randomness to the model. The proposed models have been tested on publicly available HAR Datasets like UCI HAR and WISDM Activity Recognition Datasets. Experimental results show that the hybrid models outperform the state-of-the-art data mining, machine learning techniques in UCI HAR and WISDM with an overall maximum accuracy of 97.77% and 98.2% respectively.

In this paper, a stereo camera-based novel approach for Human Activity Recognition (HAR) is presented using robust 3-D human body joint features and joint-specific Hidden Markov Models (HMMs). At f...

Human action recognition has been an active research area in recent years. However, building a robust human action recognition system still remains a challenging task due to the large variations in action classes, varying human appearances, illumination changes, camera motion, occlusions and background clutter. Most previous work focus on the goal of improving recognition rates. This paper describes a computationally fast votingbased approach for human action recognition, in which the action in the video sequence is recognized based on the support of the local spatio-temporal features. The proposed technique requires no parameter tuning and can produce recognition rates that are comparable to those in recent published literature. Moreover, the technique can localize the single human action in the video sequence without much additional computation. Recognition results on the KTH and Weizmann action dataset are presented.

Human action recognition is a research hotspot in the field of ambient intelligence, serving as a foundation for the Internet of Healthcare Things (IoHT) and smart home wellness, with extensive application value. Currently, most research primarily focuses on the performance of single-modal approach for human action recognition. However, single-modal data cannot adequately capture the action characteristics of the human body. For instance, camera data lacks detailed micro-motion information, while radar data lacks visual appearance information. This limitation makes human action recognition systems susceptible to complex covariate influences. To address this issue, this paper proposes a Multi-Modal Fusion Model, M2FM, to accurately recognize various complex human action from millimeter-wave radar signals and video data. In the millimeter-wave radar branch, a LFNet network is constructed to capture richly hierarchical human action representation by extracting micro-Doppler feature and cadence velocity feature simultaneously. Specially, a Linear Feedforward Neural Network (LFNN) module is designed for modeling global-local feature of human action. In the video branch, a lightweight Transformer-based video analysis and action recognition network, STL-Former, is developed. Specially, an Agent downSampling (AS) attention module and a Linear Maxpooling (LM) attention module are designed to achieve context-aware downsampling with global receptive field and spatial vector sequence downsampling, respectively. Additionally, a multi-scale Linear attention (Litner) module is designed for modeling the spatio-temporal information of human action. Finally, the outputs from two branches are fused by a filter-based Dempster-Shafer theory. The experimental results on a self-collected Multi-modal Human Action Dataset, JH-MHAD, show that the M2FM model outperforms other advanced models, with an average recognition accuracy of 99.3% for eight different human actions, and it demonstrates strong adaptability under different conditions.

Wireless body area network (WBAN) is a novel technology with the incorporation of numerous types of devices, which is also employed in health monitoring applications. Human activity recognition (HAR) receives more interest in recent times along with wearable sensors. HAR system provides information about a person's identity, personality, and psychological state. In the present scenario, it becomes important to model the active learning paradigms with the help of wearable sensors for analysing human activities. Although various deep learning models and existing algorithms secure better outcomes through the sensor data analysis regarding HAR, the decision‐making evaluation seems to be a complex one. The main intent of this paper is to implement the WBAN‐based HAR system using the improved deep learning model. By connecting the wearable sensors, the signals are gathered regarding human activity from diverse benchmark sources. After collecting the required signals, the pre‐processing of the input signals is done using artefact removal and median filtering. Further, the feature extraction is performed, which intends to extract a set of features by utilizing short‐time Fourier transform (STFT) and statistical features. For reducing the feature‐length, a multi‐objective‐based optimal feature selection is adopted. Human activities such as ‘walking, walking upstairs, walking downstairs, sitting, standing, lying, and jogging’ are recognized with help of selected optimal features. The optimized probabilistic neural network (PNN) and convolutional neural network (CNN) are combined and named as adaptive probabilistic‐based CNN (AP‐CNN). The effective performance of optimal feature selection and recognition is accomplished by incorporating the developed backward updating position‐based sea lion optimization algorithm (BU‐SLnO). Finally, the performance of the BU‐SLnO‐AP‐CNN‐based suggested model is analysed, which shows 1.001%, 1.237%, 0.811%, and 0.859% advanced than SLnO‐AP‐CNN, feed‐forward‐AP‐CNN (FF‐AP‐CNN), grey wolf optimization‐AP‐CNN (GWO‐AP‐CNN), particle swarm optimization‐AP‐CNN (PSO‐AP‐CNN) when observing the dataset 1. The experimental outcomes from comparison with various classification techniques demonstrate the efficiency of the developed technique.

Human Activity Recognition (HAR) systems are designed to read sensor data and analyse it to classify any detected movement and respond accordingly. However, there is a need for more responsive and near real-time systems to distinguish between false and true alarms. To accurately determine alarm triggers, the motion pattern of legitimate users need to be stored over a certain period and used to train the system to recognise features associated with their movements. This training process is followed by a testing cycle that uses actual data of different patterns of activity that are either similar or different to the training data set. This paper evaluates the use of a combined Convolutional Neural Network (CNN) and Naive Bayes for accuracy and robustness to correctly identify true alarm triggers in the form of a buzzer sound for example. It shows that pattern recognition can be achieved using either of the two approaches, even when a partial motion pattern is derived as a subset out of a full-motion path.

Human Action Recognition (HAR)is a challenging time series classification problem that has received significant attention from computer vision researchers. In this paper, different techniques used for human activities are investigated, and a human action recognition approach using a Convolutional Neural Network (CNN) with Long Short Term Memory (LSTM) and generative adversarial network is proposed. The proposed research evaluates the performance of cross-entropy and adversarial loss function for HAR analysis. Two different datasets UFC101 and the classic KTH dataset, are used for experimental purposes. The UFC101 dataset contains 13k videos in which 101 human actions are included i.e., playing instrument, makeup, etc. In contrast, KTH dataset contains 600 videos containing six human activities, including walking, running, jogging, hand clapping and hand waving, performed by 25 different persons. Also, demonstrates the process of HAR by mixing both datasets and evaluate the performance. The GAN enhances the model robustness by applying adversarial training which fully discovers the underlying connections in both intra-view and cross-view aspects.