Human Activity Recognition: From Controlled Lab Experiments to Competitive Live Evaluation

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

There are several large datasets available that are captured from motion tracking systems which could be useful to train wearable human activity recognition (HAR) systems, if only their spatial data could be mapped into the equivalent inertial measurement unit (IMU) data that would be sensed on the body. In this paper, we describe a mapping from 3D Vicon motion tracking data to data collected from a BlueSense on-body IMU. We characterise the error incurred in order to discern the extent to which it is possible to generate useful training data for a wearable activity recognition system from data collected with a motion capture system. We analyse this by mapping Vicon motion tracking data to rotational velocity and linear acceleration at the head, and compare this to actual gyroscope and accelerometer data collected by an IMU mounted on the head. In a 15 min dataset comprising three static activities—sitting, standing and lying down—we find that 95% of the reconstructed gyroscope data is within an error of [−7.25;+7.46] \(deg \cdot s^{-1}\), while 95% of the reconstructed accelerometer data was contained within [−96.1;+72.9] \(m \cdot G\). However, when we introduce more movement by including data collected while walking this increases to [−19.0;+18.2] \(deg \cdot s^{-1}\) for the gyroscope and [−208;+186] \(m \cdot G\) for the accelerometer. We conclude that generating accurate IMU data from motion capture datasets is possible and could be useful in providing larger volumes of data for activity recognition tasks and in helping enable advanced, data-hungry techniques such as deep learning to be employed on a larger scale within the domain of human activity recognition.

Human activity recognition is becoming the vital underpinning for a myriad of emerging applications in the field of human-computer interaction, mobile computing, and smart grid. Besides the utilization of up-to-date sensing techniques, modern activity recognition systems also require a machine learning (ML) algorithm that leverages the sensory data for identification purposes. In view of the unique characteristics of the measurement data and the ML challenges thereof, we propose a non-intrusive human activity recognition system that only uses existing commodity WiFi routers. The core of our system is a novel multiple kernel representation learning (MKRL) framework that automatically extracts and combines informative patterns from the Channel State Information (CSI) measurements. The MKRL firstly learns a kernel string representation from time, frequency, wavelet, and shape domains with an efficient greedy algorithm. Then it performs information fusion from diverse perspectives based on multi-view kernel learning. Moreover, different stages of MKRL can be seamlessly integrated into a multiple kernel learning framework to build up a robust and comprehensive activity classifier. Extensive experiments are conducted in typical indoor environments and the experimental results demonstrate that the proposed system outperforms existing methods and achieves a 98\% activity recognition accuracy.

Human Activity Recognition (HAR) systems are designed to read sensor data and analyse it to classify any detected movement and respond accordingly. However, there is a need for more responsive and near real-time systems to distinguish between false and true alarms. To accurately determine alarm triggers, the motion pattern of legitimate users need to be stored over a certain period and used to train the system to recognise features associated with their movements. This training process is followed by a testing cycle that uses actual data of different patterns of activity that are either similar or different to the training data set. This paper evaluates the use of a combined Convolutional Neural Network (CNN) and Naive Bayes for accuracy and robustness to correctly identify true alarm triggers in the form of a buzzer sound for example. It shows that pattern recognition can be achieved using either of the two approaches, even when a partial motion pattern is derived as a subset out of a full-motion path.

Human activity recognition is a key technology for ICT-based (infomation and communication technologies) assistive applications. The most successful activity recognition systems for intelligent environments in terms of performance rely on supervised learning techniques. However, those techniques demand large labelled data sets for specific sensor deployments and monitored person. Such requirements make supervised learning techniques not to scale well to real world deployments, where different sensor infrastructures may be used to monitor different users. In this paper, we present a novel activity recognition system, based on a combination of unsupervised learning techniques and knowledge-based activity models. First, we use a domain-specific data mining algorithm previously developed by Cook et al. to extract the most frequent action sequences executed by a person. Second, we insert knowledge-based activity models in a novel matching algorithm with the aim of inferring what activities are being performed in a given action sequence. The approach results on a scalable activity recognition system, which has been tested on three real data sets. The obtained performance is comparable to supervised learning techniques.

Healthcare using body sensor data has been getting huge research attentions by a wide range of researchers because of its good practical applications such as smart health care systems. For instance, smart wearable sensor-based behavior recognition system can observe elderly people in a smart eldercare environment to improve their lifestyle and can also help them by warning about forthcoming unprecedented events such as falls or other health risk, to prolong their independent life. Although there are many ways of using distinguished sensors to observe behavior of people, wearable sensors mostly provide reliable data in this regard to monitor the individual’s functionality and lifestyle. In this paper, we propose a body sensor-based activity modeling and recognition system using time-sequential information-based deep Neural Structured Learning (NSL), a promising deep learning algorithm. First, we obtain data from multiple wearable sensors while the subjects conduct several daily activities. Once the data is collected, the time-sequential information then go through some statistical feature processing. Furthermore, kernel-based discriminant analysis (KDA) is applied to see the better clustering of the features from different activity classes by minimizing inner-class scatterings while maximizing inter-class scatterings of the samples. The robust time-sequential features are then applied with Neural Structured Learning (NSL) based on Long Short-Term Memory (LSTM), for activity modeling. The proposed approach achieved around 99% recall rate on a public dataset. It is also compared to existing different conventional machine learning methods such as typical Deep Belief Network (DBN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) where they yielded the maximum recall rate of 94%. Furthermore, a fast and efficient explainable Artificial Intelligence (XAI) algorithm, Local Interpretable Model-Agnostic Explanations (LIME) is used to explain and check the machine learning decisions. The robust activity recognition system can be adopted for understanding peoples' behavior in their daily life in different environments such as homes, clinics, and offices.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Application of DNN for radar micro-doppler signature-based human suspicious activity recognition

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

The research area of ambient assisted living has led to the development of activity recognition systems (ARS) based on human activity recognition (HAR). These systems improve the quality of life and the health care of the elderly and dependent people. However, before making them available to end users, it is necessary to evaluate their performance in recognizing activities of daily living, using data set benchmarks in experimental scenarios. For that reason, the scientific community has developed and provided a huge amount of data sets for HAR. Therefore, identifying which ones to use in the evaluation process and which techniques are the most appropriate for prediction of HAR in a specific context is not a trivial task and is key to further progress in this area of research. This work presents a systematic review of the literature of the sensor-based data sets used to evaluate ARS. On the one hand, an analysis of different variables taken from indexed publications related to this field was performed. The sources of information are journals, proceedings, and books located in specialized databases. The analyzed variables characterize publications by year, database, type, quartile, country of origin, and destination, using scientometrics, which allowed identification of the data set most used by researchers. On the other hand, the descriptive and functional variables were analyzed for each of the identified data sets: occupation, annotation, approach, segmentation, representation, feature selection, balancing and addition of instances, and classifier used for recognition. This paper provides an analysis of the sensor-based data sets used in HAR to date, identifying the most appropriate dataset to evaluate ARS and the classification techniques that generate better results.

PurposeTo find a successful human action recognition system (HAR) for the unmanned environments.Design/methodology/approachThis paper describes the key technology of an efficient HAR system. In this paper, the advancements for three key steps of the HAR system are presented to improve the accuracy of the existing HAR systems. The key steps are feature extraction, feature descriptor and action classification, which are implemented and analyzed. The usage of the implemented HAR system in the self-driving car is summarized. Finally, the results of the HAR system and other existing action recognition systems are compared.FindingsThis paper exhibits the proposed modification and improvements in the HAR system, namely the skeleton-based spatiotemporal interest points (STIP) feature and the improved discriminative sparse descriptor for the identified feature and the linear action classification.Research limitations/implicationsThe experiments are carried out on captured benchmark data sets and need to be analyzed in a real-time environment.Practical implicationsThe middleware support between the proposed HAR system and the self-driven car system provides several other challenging opportunities in research.Social implicationsThe authors’ work provides the way to go a step ahead in machine vision especially in self-driving cars.Originality/valueThe method for extracting the new feature and constructing an improved discriminative sparse feature descriptor has been introduced.

This paper brings to light one of the most prominent applications of human activity recognition which is the anomaly detection. Providing security to an individual is a major concern of any society today due to the constantly increasing actions causing threats, starting from deliberate violence to an injury caused by an accident. Mere installation of a traditional closed-circuit television is not enough as it requires a human being to constantly remain alert and monitor the cameras, which is quite inefficient. This calls for the need to develop an automated security system that identifies abnormal activities in real time and brings immediate help to the victims. But it is time-consuming to recognize activities from lengthy surveillance videos and hence a new implementation idea which adaptively compresses the videos before passing it through the activity recognition system has been proposed. Adaptive video compression is a unique compression technology that compresses only the least significant parts of the video, retaining the objects of interest. By combining adaptive video compression and the convolutional 3D network with contextual multiple scales based on the temporal features (Xu et al. in Learning deep representations of appearance and motion for anomalous event detection. In: Computer Vision and Pattern Recognition (CVPR), 2015), the paper aims to provide an accurate anomalous human activity recognition system that works in real time. The method is implemented on the UCF101 crime dataset with around 13 different anomalies (Learning Spatiotemporal Features with 3D Convolutional Networks. In: International Conference on Computer Vision (ICCV), IEEE, 2015). It resulted in improved recognition performance compared to state-of-the-art techniques.

The conventional Human Activity Recognition (HAR) systems use smartwatches, smartphones, and other wearable devices to autonomously derive consumer physical activities. However, implementing real-time HAR in resource-constrained applications is challenging. Computational offloading approaches must address network reliance, latency, and data privacy, which affect system performance. The present study proposes a low-cost, wearable HAR device that collects and transmits the Inertial Measurement Unit (IMU) telemetry signals to smartphones, enabling real-time on-device activity detection. Human actions are subjective and affected by physiological, environmental, and behavioural factors, complicating HAR systems' performance in personalized contexts. Generalized methods for existing systems reduce performance for new consumers or due to alterations in consumer locomotive signatures. Thus, our second goal is to enable the on-device personalization of wearable HAR models with minimum customer calibration. The HAR device, which is in the form of a smart-band, transfers data to and fro smartphones through Bluetooth Low Energy (BLE). A lightweight 1D convolutional neural network (CNN) is built and, using transfer learning, the network model is fine-tuned from real-time sensed data. On-device HAR inferences ensure real-time processing on smartphones without computational offloading. Smartphone-based on-device training for fine-tuning the model allows the HAR system to get customized without compromising privacy or computational costs. The experimental results show that the personalized classification achieves on average 98% accuracy when assessed using four benchmark datasets.

Automatic analysis of the video is one of most complex problems in the fields of computer vision and machine learning. A significant part of this research deals with (human) activity recognition (HAR) since humans, and the activities that they perform, generate most of the video semantics. Video-based HAR has applications in various domains, but one of the most important and challenging is HAR in sports videos. Some of the major issues include high inter- and intra-class variations, large class imbalance, the presence of both group actions and single player actions, and recognizing simultaneous actions, i.e., the multi-label learning problem. Keeping in mind these challenges and the recent success of CNNs in solving various computer vision problems, in this work, we implement a 3D CNN based multi-label deep HAR system for multi-label class-imbalanced action recognition in hockey videos. We test our system for two different scenarios: an ensemble of $k$ binary networks vs. a single k-output network, on a publicly available dataset. We also compare our results with the system that was originally designed for the chosen dataset. Experimental results show that the proposed approach performs better than the existing solution.