Towards real-time sensor-based human activity recognition: a re-parameterized multidimensional feature communication fusion framework

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

Due to the breadth of its application domains, Hu-man Activity Recognition (HAR) is a problematic area of human-computer interaction. HAR can be used in remote monitoring of senior healthcare and concern situations in intelligent man-ufacturing, among other applications. HAR based on wearable inertial sensors has been researched identification efficiency in various kinds of human actions considerably more than vision-based HAR. The sensor-based HAR is generally applicable to indoor and outdoor locations without privacy considerations of implementation. In this research, we explore the recognition performance of multiple deep learning (DL) models to recognize everyday living human activities. We developed a deep residual neural network that employed aggregated multi-branch transformation to boost identification performance. The proposed model is called the ResNeXt model. To evaluate its performance, three standard DL models (CNN, LSTM, and CNN-LSTM) are investigated and compared to our proposed model using a standard HAR dataset called Daily Living Activity dataset. These datasets gathered mobility signal data from multimodal sensors (accelerometer, gyroscope, and magnetometer) in three distinct body areas (wrist, hip, and ankle). The experimental findings reveal that the proposed model surpasses other benchmark DL models with maximum accuracy and F1-scores. Furthermore, the findings show that the ResNeXt model is more resistant than other models with fewer training parameters.

Recently, deep learning, which are able to extract automatically features from data, has achieved state-of-the-art performance across a variety of sensor based human activity recognition (HAR) tasks. However, the existing deep neural networks are usually trained with a global loss, and all hidden layer weights have to be always kept in memory before the forward and backward pass has completed. The backward locking phenomenon prevents the reuse of memory, which is a crucial limitation for wearable activity recognition. In the paper, we proposed a layer-wise convolutional neural networks (CNN) with local loss for the use of HAR task. To our knowledge, this paper is the first that uses local loss based CNN for HAR in ubiquitous and wearable computing arena. We performed experiments on five public HAR datasets including UCI HAR dataset, OPPOTUNITY dataset, UniMib-SHAR dataset, PAMAP dataset, and WISDM dataset. The results show that local loss works better than global loss for tested baseline architectures. At no extra cost, the local loss can approach the state-of-the-arts on a variety of HAR datasets, even though the number of parameters was smaller. We believe that the layer-wise CNN with local loss can be used to update the existing deep HAR methods.

In this work, we propose a Human Activity Prediction (HAP) system using activity sequence spanning trees constructed from a life-log created by a video sensor-based daily Human Activity Recognition (HAR) system using time-sequential Independent Component (IC)-based depth silhouette features with Hidden Markov Models (HMMs). In the daily HAR system, the IC features are extracted from the collection of the depth silhouettes containing various daily human activities such as walking, sitting, lying, cooking, eating etc. Using these features, HMMs are used to model the time sequential features and recognize various human activities. The depth silhouette-based human activity recognition system is used to recognize daily human activities automatically in real time, which creates a life-log of daily activity events. In this work, we propose a method for human activity prediction using fixed-length activity sequence spanning trees based on the life-log. Utilizing the consecutive activities recorded in an activity sequence database (i.e. life-log) for a specific period of time of each day over a period such as a month, the fixed-length spanning trees can be constructed for the sequences starting with each activity where the leaf nodes contain the frequency of the fixed-length consecutive activity sequences. Once the trees are constructed, to predict an activity after a sequence of activities, we traverse the spanning trees until a path up to the previous node of the leaf nodes is matched with the testing pattern. Finally, we can predict the next activity based on the highest frequency of the leaf nodes along the matched path. The prediction experiments over the computer simulated data which is based on the daily logs show satisfactory results. Our video sensor-based human activity recognition and prediction systems can be utilized for practical applications such as smart and proactive healthcare.

Sensor-based human activity recognition (HAR) aims to recognize a human's physical actions by using sensors attached to different body parts. As a user-specific application, HAR often suffers poor generalization from training on an individual to testing on another individual, or from one body part to another body part. To tackle this cross-domain HAR problem, this article proposes a domain adaptation (DA) method called local domain adaptation (LDA), whose core is to align cluster-to-cluster distributions between the source domain and the target domain. On the one hand, LDA differs from existing set-to-set alignment by reducing the distribution discrepancy at a finer granularity. On the other hand, LDA is superior to the class-to-class alignment because it can provide more accurate soft labels for the target domain. Specifically, LDA contains three main steps: 1) groups the activity class into several high-level abstract clusters; 2) maps the original data of each cluster in both domains into the same low-dimension subspace to align the intracluster data distribution; 3) predicts the class labels for target domain in the low-dimension subspace. Experimental results on two public HAR benchmark datasets show that LDA outperforms state-of-the-art DA methods for the cross-domain HAR.

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Sensor-based Human Activity Recognition (HAR) is a research field that has been widely studied in Pervasive Computing. Due to its many applications from healthcare to well-being, sensor-based HAR has been proposed to recognize high-level activities considering smart-homes environmental sensors as well as low-level activities using wearable sensors. The most effective solutions presented so far rely on supervised learning approaches, that have the potential of reaching high recognition rates. However, such approaches assume that large amounts of labeled data are available. Annotating human activities is costly, time-consuming, intrusive, and often prohibitive. Considering the high variability of activity execution among different subjects, methods that are capable of personalizing to each subject with limited labeled data are needed. In the last few years, several research groups are indeed working hard trying to mitigate this problem. However, data scarcity still remains an open challenge in sensor-based HAR. This talk presents latest research efforts on these topics. In particular, the talk will cover: a) purely knowledge-based approaches enhanced with active learning for smart-home sensor data, b) hybrid data-driven and knowledge-based approaches relying on context data to mitigate data scarcity in mobile settings, and c) semi-supervised federated learning approaches. We will discuss the above-mentioned challenges, report our experience and identify critical aspects which still need to be investigated.

A survey on wearable sensor modality centred human activity recognition in health care

Sensor-based Human Activity Recognition (HAR) has been applied across various domains, including healthcare monitoring, fitness tracking, and smart home systems. These applications require the ability to accurately detect and respond to a wide range of human activities, each with varying distributions, which imposes a significant challenge. The task-incremental learning paradigm can address this problem by enabling HAR systems to adapt to changes in distribution and learn new activities over time without forgetting the previously learned ones. Continual adaptation is essential for maintaining high performance, as it allows the system to effectively respond to these changes. Although several strategies in the continual learning literature have been evaluated for task-incremental scenarios in HAR, there is still room for improvement, as the results are not as good as those achieved with conventional approaches. This work proposes two new neuroplasticity-inspired techniques that can be integrated with any learning strategy. Inspired by the brain’s ability to reorganize and strengthen connections over time, these methods focus on enhancing the model’s flexibility and long-term knowledge retention. The proposed techniques were evaluated alongside the WA-ADB and WA-MDF strategies on well-known HAR datasets. Experimental results demonstrated that the new techniques significantly enhanced the models’ ability to retain knowledge, which holds significant potential for improving the robustness and longevity of HAR systems in real-world applications.

Recently, domain generalization (DG) has emerged as a promising solution to mitigate distribution-shift issue in sensor-based human activity recognition (HAR) scenario. However, most existing DG-based works have merely focused on modeling statistical dependence between sensor data and activity labels, neglecting the importance of intrinsic casual mechanism. Intuitively, every sensor input can be viewed as a mixture of causal (category-aware) and non-causal factors (domain-specific), where only the former affects activity classification judgment. In this paper, by casting such DG-based HAR as a casual inference problem, we propose a causality-inspired representation learning algorithm for cross-domain activity recognition. To this end, an early-forking two-branch framework is designed, where two separate branches are respectively responsible for learning casual and non-causal features, while an independence-based Hilbert-Schmidt Information Criterion is employed to implicitly disentangling them. Additionally, an inhomogeneous domain sampling strategy is designed to enhance disentanglement, while a category-aware domain perturbation layer is performed to prevent representation collapse. Extensive experiments on several public HAR benchmarks demonstrate that our causality-inspired approach significantly outperforms eleven related state-of-the-art baselines under cross-person, cross-dataset, and cross-position settings. Detailed ablation and visualizations analyses reveal underlying casual mechanism, indicating its effectiveness, efficiency, and universality in cross-domain activity recognition scenario.

One of the major open problems in sensor-based Human Activity Recognition (HAR) is the scarcity of labeled data. Among the many solutions to address this challenge, semi-supervised learning approaches represent a promising direction. However, their centralized architecture incurs in the scalability and privacy problems that arise when the process involves a large number of users. Federated learning (FL) is a promising paradigm to address these problems. However, the FL methods that have been proposed for HAR assume that the participating users can always obtain labels to train their local models (i.e., they assume a fully supervised setting). In this work, we propose FedAR: a novel hybrid method for HAR that combines semi-supervised and federated learning to take advantage of the strengths of both approaches. FedAR combines active learning and label propagation to semi-automatically annotate the local streams of unlabeled sensor data, and it relies on FL to build a global activity model in a scalable and privacy-aware fashion. FedAR also includes a transfer learning strategy to fine-tune the global model on each user. We evaluated our method on two public datasets, showing that FedAR reaches recognition rates and personalization capabilities similar to state-of-the-art FL supervised approaches. As a major advantage, FedAR only requires a very limited number of annotated data to populate a pre-trained model and a small number of active learning questions that quickly decrease while using the system, leading to an effective and scalable solution for the data scarcity problem of HAR.

With the rise of artificial intelligence, sensor-based human activity recognition (S-HAR) is increasingly being employed in healthcare monitoring for the elderly, fitness tracking, and patient rehabilitation using smart devices. Inertial sensors have been commonly used for S-HAR, but wearable devices have been demanding more comfort and flexibility in recent years. Consequently, there has been an effort to incorporate stretch sensors into S-HAR with the advancement of flexible electronics technology. This paper presents a deep learning network model, utilizing aggregation residual transformation, that can efficiently extract spatial–temporal features and perform activity classification. The efficacy of the suggested model was assessed using the w-HAR dataset, which included both inertial and stretch sensor data. This dataset was used to train and test five fundamental deep learning models (CNN, LSTM, BiLSTM, GRU, and BiGRU), along with the proposed model. The primary objective of the w-HAR investigations was to determine the feasibility of utilizing stretch sensors for recognizing human actions. Additionally, this study aimed to explore the effectiveness of combining data from both inertial and stretch sensors in S-HAR. The results clearly demonstrate the effectiveness of the proposed approach in enhancing HAR using inertial and stretch sensors. The deep learning model we presented achieved an impressive accuracy of 97.68%. Notably, our method outperformed existing approaches and demonstrated excellent generalization capabilities.

The recent development of machines exhibiting intelligent characteristics involves numerous techniques including computer hardware and software architecture development. Many different hardware devices, wearable sensors, machine learning, and deep learning model implementations are being applied in human activity recognition (HAR) applications in recent times. However, to develop high accuracy classification systems for activity recognition using ilow-cost hardware technology is of significant importance. To achieve this goal this study uses sensor data from two low-cost sensors, gyroscope and accelerometer along with the implementation of an Artificial Neural Network (ANN) based deep learning model for HAR. In particular, Deep Stacked Multilayered Perceptron (DS-MLP) has been proposed. In the implementation of DS-MLP, an ANN model has been used as a meta-learner while five MLP models have been used as base-learners. In this study, these base-learners and meta-learner have been combined using a stack ensemble technique. The performance evaluations have been done first on the applicability of individual base-models followed by the application of DS-MLP, the results prove the high accuracy of 97.3% and 99.4% for heterogeneous datasets used for testing. The performance of the proposed DS-MLP models has been compared to some existing machine learning classifiers and several state-of-the-art activity recognition systems. The comparative result analysis also proves that the proposed system performed better than these classification approaches in terms of important performance metrics such as accuracy, precision, recall, F-score, Cohen's Kappa, and Mathew correlation coefficient.

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.