Human activity recognition with smartphone sensors using deep learning neural networks

In recent years, many researchers have focused on using convolutional neural networks to perform human activity recognition as evidenced by the emergence of a number of convolutional neural network architectures such as LeNet-5, AlexNet and VGG16 and modern architectures such as ResNet, Inception V3, Inception-ResNet, MobileNet V2, NASNet and PNASNet. The main characteristic of a convolutional neural network (CNN) is its ability to extract features automatically from input images, which facilitates the processes of activity recognition and classification. Convolutional networks indeed derive more relevant and complex features with every additional layer. In addition, CNNs have achieved perfect classification on highly similar activities that were previously extremely difficult to classify. In this paper, we evaluate modern convolutional neural networks in terms of their human activity recognition accuracy, and we compare the results with the state of the art methods. In our research, we used two public data sets, HMDB (Shooting gun, kicking, falling to the floor, punching) and the Weizman dataset (walking, running, jumping, bending, one hand waving, two-hand waving, jumping in place, jumping jack, skipping). Our experimental results indicated that the CNN with NASNet architecture achieves the best performance of the six CNN architectures on both human activity data sets (HMDB and Weizman).

Encoding human activities using multimodal wearable sensory data

In recent years, channel state information (CSI) in WiFi 802.11n has been increasingly used to collect data pertaining to human activity. Such raw data are then used to enhance human activity recognition. Activities such as lying down, falling, walking, running, sitting down, and standing up can now be detected with the use of information collected through CSI. Human activity recognition has a multitude of applications, such as home monitoring of patients. Four deep learning models are presented in this paper, namely: a convolution neural network (CNN) with a Gated Recurrent Unit (GRU); a CNN with a GRU and attention; a CNN with a GRU and a second CNN, and a CNN with Long Short-Term Memory (LSTM) and a second CNN. Those models were trained to perform Human Activity Recognition (HAR) using CSI amplitude data collected by a CSI tool. Experiments conducted to test the efficacy of these models showed superior results compared with other recent approaches. This enhanced performance of our models may be attributable the ability of our models to make full use of available data and to extract all data features, including high dimensionality and time sequence. The highest average recognition accuracy reached by the proposed models was achieved by the CNN-GRU, and the CNN-GRU with attention models, standing at 99.31% and 99.16%, respectively. In addition, the performance of the models was evaluated for unseen CSI data by training our models using a random split-of-dataset method (70% training and 30% testing). Our models achieved impressive results with accuracies reaching 100% for nearly all activities. For the lying down activity, accuracy obtained from the CNN-GRU model stood at 99.46%; slightly higher than the 99.05% achieved by the CNN-GRU with attention model. This confirmed the robustness of our models against environmental changes.

Smartphone-Based Human Activity Recognition Using Bagging and Boosting

Wearable sensors are essential for recognizing human activity in sports, healthcare, and smart surroundings applications. Robust classification models and effective feature selection directly affect recognition accuracy. This paper proposes a novel approach called Chaotic Logistic Map-based Grey Wolf Optimization with Decision Tree (CLM-GWO-DT) to predict human activity recognition. The proposed technique improves the GWO algorithm by using chaotic logistic maps to enhance its exploration and exploitation abilities. CLM-GWO is used to find the most informative features in raw sensor data, thereby reducing dimensionality and enhancing relevant patterns. A Decision Tree (DT) classifier is then applied to the retrieved data to ensure accurate and interpretable identification of human activity. The experiments employed two popular datasets: UCI Human Activity Recognition (HAR) and Wireless Sensor Data Mining (WISDM). The results indicate that the proposed model exceeds the performance of existing methods in the literature concerning accuracy, precision, recall, F-Score, and Matthews Correlation Coefficient (MCC).

Enhancing lifestyle and health monitoring of elderly populations using CSA-TkELM classifier

Smartphones are increasing ubiquitously due to the need and demand in the modern era. The world is transmuting into a global village with the cumulation of smart devices. Nowadays, smartphones are enriched with inertial sensors which can be used to recognize physical human activities in the wild. Human activity recognition (HAR) lies at the core of many applications like health monitoring, fall detection, road safety, personal assistance, and behavior-based context awareness. Context-based HAR is a new dimension that provides fine-grained information about the action being performed and leads us towards the automated and intelligent system design, which is useful to furnish the smart solution to real-life problems. In this paper, smartphone sensors are utilized to propose a framework for human activity and context recognition. This framework performs activity-dependent context recognition on the ExtraSensory dataset using random forest, decision tree, and k-nearest neighbor classifiers, and achieves an accuracy of 93.10%, demonstrating the effectiveness of the proposed framework.

Wearable inertial measurement unit (IMU) sensors are powerful enablers for acquisition of motion data. Specifically, in human activity recognition (HAR), IMU sensor data collected from human motion are categorically combined to formulate datasets that can be used for learning human activities. However, successful learning of human activities from motion data involves the design and use of proper feature representations of IMU sensor data and suitable classifiers. Furthermore, the scarcity of labelled data is an impeding factor in the process of understanding the performance capabilities of data-driven learning models. To tackle these challenges, two primary contributions are in this article: first; by using raw IMU sensor data, a spectrogram-based feature extraction approach is proposed. Second, an ensemble of data augmentations in feature space is proposed to take care of the data scarcity problem. Performance tests were conducted on a deep long term short term memory (LSTM) neural network architecture to explore the influence of feature representations and the augmentations on activity recognition accuracy. The proposed feature extraction approach combined with the data augmentation ensemble produces state-of-the-art accuracy results in HAR. A performance evaluation of each augmentation approach is performed to show the influence on classification accuracy. Finally, in addition to using our own dataset, the proposed data augmentation technique is evaluated against the University of California, Irvine (UCI) public online HAR dataset and yields state-of-the-art accuracy results at various learning rates.

Human activity recognition is concerned with identifying the specific movement of a person based on sensor data. In recent years, many different techniques have been proposed for modelling and recognising human activities, with a specific focus on the development of approaches to classifying human activities using deep learning techniques. The research presented in this paper proposes a fuzzy feature representation approach to represent occupancy sensor data, along with a Convolutional Neural Network Classifier (CNNC) for human activity modelling and recognition. Sensory data gathered from a home environment are converted into occupancy data representing human activities and then fuzzified before being fed as inputs into the CNNC. The learning capability of CNNC allows the model to learn the relationship between the fuzzified inputs and their corresponding output activities during training mode. The relations learned in the trained CNNC model are then used to identify human activity patterns and classify these when the testing dataset is applied. The proposed method is evaluated using a dataset representing activities of daily living for a single user gathered from a real-home environment.

Activity recognition commonly made use of hidden Markov models (HMMs) to exploit temporal dependencies between activities. The emission distribution of HMMs could be represented by generative models, such as Gaussian mixture models (GMMs), or discriminative models, such as random forest (RF). These models, especially discriminative ones, needed to manually extract features from the sensor data, which relied on the experience of the researchers, and usually was a time-consuming task when complicated features are extracted. Furthermore, with these methods, the process of quantization of the sensor data, i.e., manual feature extraction, might lose much useful information and thus led to a performance debasement. In this paper, we recommend deep neural networks (DNNs) for modeling the emission distribution of HMMs, which automatically learn features suitable for classification from the raw sensor data and then estimate the posterior probabilities of the HMM states. We collected a dataset of daily activities and based on which experiments were performed to compare our HMM-DNN model with both HMM-GMM and HMM-RF. The results illustrated that HMM-DNN outperformed both HMM-GMM and HMM-RF.

Traditional indoor human activity recognition (HAR) has been defined as a time-series data classification problem and requires feature extraction. The current indoor HAR systems still lack transparent, interpretable, and explainable approaches that can generate human-understandable information. This paper proposes a new approach, called Human Activity Recognition on Signal Images (HARSI), which defines the HAR problem as an image classification problem to improve both explainability and recognition accuracy. The proposed HARSI method collects sensor data from the Internet of Things (IoT) environment and transforms the raw signal data into some visual understandable images to take advantage of the strengths of convolutional neural networks (CNNs) in handling image data. This study focuses on the recognition of symmetric human activities, including walking, jogging, moving downstairs, moving upstairs, standing, and sitting. The experimental results carried out on a real-world dataset showed that a significant improvement (13.72%) was achieved by the proposed HARSI model compared to the traditional machine learning models. The results also showed that our method (98%) outperformed the state-of-the-art methods (90.94%) in terms of classification accuracy.

Human Activity Recognition (HAR) systems are designed to read sensor data and analyse it to classify any detected movement and respond accordingly. However, there is a need for more responsive and near real-time systems to distinguish between false and true alarms. To accurately determine alarm triggers, the motion pattern of legitimate users need to be stored over a certain period and used to train the system to recognise features associated with their movements. This training process is followed by a testing cycle that uses actual data of different patterns of activity that are either similar or different to the training data set. This paper evaluates the use of a combined Convolutional Neural Network (CNN) and Naive Bayes for accuracy and robustness to correctly identify true alarm triggers in the form of a buzzer sound for example. It shows that pattern recognition can be achieved using either of the two approaches, even when a partial motion pattern is derived as a subset out of a full-motion path.

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

Lifelog, Human Activity Recognition, Smartphones, Embedded Sensors, Smart-City, Heterogeneous Sensory Data Analytics. This paper introduces a new idea for sensor data analytics, named PHASOR, that can recognize and stream individual human activities online. The proposed sensor concept can be utilized to solve some emerging problems in smartcity domain such as health care, urban mobility, or security by creating a lifelog of human activities. PHASOR is created from three ‘components’: ID, model, and Sensor. The first component is to identify which sensor is used to monitor which object (e.g., group of users, individual users, type of smart- phone). The second component decides suitable classifiers for human activities recognition. The last one includes two types: (1) physical sensors that utilize embedded sensors in smartphones to recognize human activities, (2) human factors that uses human interaction to personally increase the accuracy of the detection. The advantage of PHASOR is the error signal is inversely proportional to its lifetime, which is well-suited for lifelogging applications. The proposed concept is evaluated and compared to de-facto datasets as well as state-of-the-art of Human Activity Recognition (HAR) using smartphones, confirming that applying PHASOR can improves the accuracy of HAR.

Neuromorphic computing with hybrid CNN–Stochastic Reservoir for time series WiFi based human activity recognition