Hierarchical‐Split Multi‐Scale Convolution Network With Multi‐Task Learning for Human Activity Recognition in Wearable Devices

Human activity recognition (HAR) is an important research topic in the field of human-computer interaction (HCI), and wearable HAR based on convolutional neural networks (CNNs) has achieved the most advanced performance. In this study, wearable plastic optical fiber (POF) sensors based on GAF-ConvNeXt algorithm are used for HAR for the first time. The GAF-ConvNeXt algorithm, combined with the Gramian Angular Field (GAF) and the latest CNN model ConvNeXt, can convert motion signal of a one-dimensional time series into a two-dimensional image, which can not only retain the complete information of the signal, but also maintain its dependence on time, thereby presenting more depth features and facilitating accurate recognition of human activities. Furthermore, transfer learning (TL) was introduced to further optimize the GAF-ConvNeXt model, which effectively reduces the number of training samples, improves the recognition accuracy and accelerates the convergence speed. The GAF-ConvNeXt algorithm based on TL can recognize seven motion modes with an accuracy of 99.75%, and has great application potential in the fields of human motion recognition and HCI.

Federated Learning (FL) has recently attracted great interest in sensor-based human activity recognition (HAR) tasks. However, in real-world environment, sensor data on devices is non-independently and identically distributed (Non-IID), e.g., activity data recorded by most devices is sparse, and sensor data distribution for each client may be inconsistent. As a result, the traditional FL methods in the heterogeneous environment may incur a drifted global model that causes slow convergence and a heavy communication burden. Although some FL methods are gradually being applied to HAR, they are designed for overly ideal scenarios and do not address such Non-IID problem in the real-world setting. It is still a question whether they can be applied to cross-device FL. To tackle this challenge, we propose ProtoHAR, a prototype-guided FL framework for HAR, which aims to decouple the representation and classifier in the heterogeneous FL setting efficiently. It leverages the global prototype to correct the activity feature representation to make the prototype knowledge flow among clients without leaking privacy while solving a better classifier to avoid excessive drift of the local model in personalized training. Extensive experiments are conducted on four publicly available datasets: USC-HAD, UNIMIB-SHAR, PAMAP2, and HARBOX, which are collected in both controlled environments and real-world scenarios. The results show that compared with the state-of-the-art FL algorithms, ProtoHAR achieves the best performance and faster convergence speed in HAR datasets.

Learning underlying patterns from sensory data is crucial in the Human Activity Recognition (HAR) task to avoid poor generalization when coping with unseen data. A key solution to such an issue is representation learning, which becomes essential when input signals contain activities with similar patterns or when patterns generated by different subjects for the same activity vary. To address these issues, we seek a solution to increase generalization by learning the underlying factors of each sensor signal. We develop a novel multi-channel asymmetric auto-encoder to recreate input signals precisely and extract indicative unsupervised futures. Further, we investigate the role of various activation functions in signal reconstruction to ensure the model preserves the patterns of each activity in the output. Our main contribution is that we propose a multi-task learning model to enhance representation learning through shared layers between signal reconstruction and the HAR task to improve the robustness of the model in coping with users not included in the training phase. The proposed model learns shared features between different tasks that are indeed the underlying factors of each input signal. We validate our multi-task learning model using several publicly available HAR datasets, UCI-HAR, MHealth, PAMAP2, and USC-HAD, and an in-house alpine skiing dataset collected in the wild, where our model achieved 99%, 99%, 95%, 88%, and 92% accuracy. Our proposed method shows consistent performance and good generalization on all the datasets compared to the state of the art.

INTRODUCTION: A solution to subject-independent HAR prediction through machine learning classification algorithms using statistical equivalency for comparative analysis between independent groups with non-subject training dependencies.OBJECTIVES: To indicate that the multinomial predictive classification model that was trained and optimized on the one-subject control group is at least partially extensible to multiple independent experiment groups for at least one activity class.METHODS: Gradient boosted machine multinomial classification algorithm is trained on a single individual with the classifier trained on all activity classes as a multinomial classification problem.RESULTS: Levene-Wellek-Welch (LWW) Statistic calculated as 0.021, with a Critical Value for LWW of 0.026, using an alpha of 0.05.CONCLUSION: Confirmed falsifiability that incorporates reproducible methods into the quasi-experiment design applied to the field of machine learning for human activity recognition.

Human Activity Recognition(HAR) based on Channel State Information(Csi)plays an increasingly impor-tant role in human-computer interaction. Traditional research requires a large amount of activity sample to train network model. However, collecting a great many of data causes waste of time and manpower. Some models can well identify the categories of activities in this scene, but when another scene is tested, the identification accuracy of the model will be reduced. Therefore it needs to re-collect data to retrain the model. We proposed a method which can transfer the knowledge learned from a scenario to a new scenario. It can also facilitate the model's knowledge learning from the source domain and quickly generalize to new tasks that contain only a small number of samples. Through this method, the model that maintains high accuracy and scalability to identify new category, we added attention mechanism can automatically extract features that are useful to the model and ignore some noise that negatively affects the model, meanwhile improve system stability and the effectiveness of activity recognition. We also performed the scaling and shifting(SS) transformation on the network, which could reduce the parameters of the model, improve the training speed, and avoid overfitting.

Unsupervised learning for human activity recognition using smartphone sensors

With the advancement and ubiquitousness of wearable devices, wearable sensor-based human activity recognition (HAR) has become a prominent research area in the healthcare domain and human-computer interaction. Inertial measurement unit (IMU) which can provide a wide range of information such as acceleration, angular velocity has become one of the most commonly used sensors in HAR. Recently, with the growing demand for soft and flexible wearable devices, mountable stretch sensors have become a new promising modality in wearable sensor-based HAR. In this paper, we propose a deep learning-based multi-modality HAR framework which consists of three IMUs and two fabric stretch sensors in order to evaluate the potential of stretch sensors independently and in combination with IMU sensors for the activity recognition task. Three different deep learning algorithms: long short-term memory (LSTM), convolutional neural network (CNN) and hybrid CNN-LSTM are deployed to the sensor data for automatically extracting deep features and performing activity classification. The impact of sensor type on recognition accuracy of different activities is also examined in this study. A dataset collected from the proposed framework, namely iSPL IMU-Stretch and a public dataset called w-HAR are used for experiments and performance evaluation.

The conventional Human Activity Recognition (HAR) systems use smartwatches, smartphones, and other wearable devices to autonomously derive consumer physical activities. However, implementing real-time HAR in resource-constrained applications is challenging. Computational offloading approaches must address network reliance, latency, and data privacy, which affect system performance. The present study proposes a low-cost, wearable HAR device that collects and transmits the Inertial Measurement Unit (IMU) telemetry signals to smartphones, enabling real-time on-device activity detection. Human actions are subjective and affected by physiological, environmental, and behavioural factors, complicating HAR systems' performance in personalized contexts. Generalized methods for existing systems reduce performance for new consumers or due to alterations in consumer locomotive signatures. Thus, our second goal is to enable the on-device personalization of wearable HAR models with minimum customer calibration. The HAR device, which is in the form of a smart-band, transfers data to and fro smartphones through Bluetooth Low Energy (BLE). A lightweight 1D convolutional neural network (CNN) is built and, using transfer learning, the network model is fine-tuned from real-time sensed data. On-device HAR inferences ensure real-time processing on smartphones without computational offloading. Smartphone-based on-device training for fine-tuning the model allows the HAR system to get customized without compromising privacy or computational costs. The experimental results show that the personalized classification achieves on average 98% accuracy when assessed using four benchmark datasets.

Human activity recognition (HAR) has attracted considerable research attention in the past decade with the development of wearable sensor technology and deep learning algorithms. However, most of the existing HAR methods ignored the spatial relationship of features, which may lead to recognition errors. In this paper, a novel model based on a modified capsule network (MCN) is proposed to accurately recognize various human activities. This novel model is composed of a convolution block and a capsule block, which can achieve end-to-end intelligent recognition. In the meantime, the spatial information among features is preserved through a dynamic routing process. To validate the effectiveness of the model, a human activity dataset is constructed by placing an inertial measurement unit (IMU) on the calf of the volunteers to collect their activity data in daily life, including walking, jogging, upstairs, downstairs, up-ramps, and down-ramps. The recognition accuracy of this novel approach can reach 96.08%, which performs better than the convolutional neural network (CNN) with an accuracy of 91.62%. In addition, it is evaluated on two public datasets named WISDM and UCI-HAR, and the accuracies achieve 98.21% and 95.28%, respectively, which presents higher accuracy than the reported results obtained from benchmark algorithms like CNN. The experimental results show that the proposed model has better activity detection capability and achieves outstanding performance for HAR.

Human Activity Recognition (HAR) has rapidly gained interest over the years due to its wide range of applications in AI-based systems, particularly healthcare monitoring. HAR methods typically involve extracting relevant features from data provided by wearable sensors, smartphone sensors, cameras, or their combinations to classify different activities. Nevertheless, a major challenge lies in achieving high classification accuracy with limited data samples, particularly when distinguishing between activities with similar signal attributes. To address this challenge, we propose a novel HAR method called BinAry sequence-augmented spectrograM with Self-attention deep learning (BAMS). Our proposed method leverages only basic wearable sensor data. It utilizes short-time Fourier transform spectrograms to extract spatio-temporal sensor information. The spectrogram is integrated with a binary sequence that captures movement direction. We integrate a scaled dot-product self-attention mechanism into the model to prioritize data from wearable sensors, thereby enhancing the model's performance. The proposed method is evaluated on a public dataset using leave-one-subject-out cross-validation for efficacy and robustness. The method is found to achieve significant improvement over other state-of-the-art methods with the classification accuracy percentage and weighted F-1 scores of 88.06±5.11 and 87.36±5.96, respectively, for a twelve-activity classification.

Quantum contrastive learning for human activity recognition

In recent years, channel state information (CSI) in WiFi 802.11n has been increasingly used to collect data pertaining to human activity. Such raw data are then used to enhance human activity recognition. Activities such as lying down, falling, walking, running, sitting down, and standing up can now be detected with the use of information collected through CSI. Human activity recognition has a multitude of applications, such as home monitoring of patients. Four deep learning models are presented in this paper, namely: a convolution neural network (CNN) with a Gated Recurrent Unit (GRU); a CNN with a GRU and attention; a CNN with a GRU and a second CNN, and a CNN with Long Short-Term Memory (LSTM) and a second CNN. Those models were trained to perform Human Activity Recognition (HAR) using CSI amplitude data collected by a CSI tool. Experiments conducted to test the efficacy of these models showed superior results compared with other recent approaches. This enhanced performance of our models may be attributable the ability of our models to make full use of available data and to extract all data features, including high dimensionality and time sequence. The highest average recognition accuracy reached by the proposed models was achieved by the CNN-GRU, and the CNN-GRU with attention models, standing at 99.31% and 99.16%, respectively. In addition, the performance of the models was evaluated for unseen CSI data by training our models using a random split-of-dataset method (70% training and 30% testing). Our models achieved impressive results with accuracies reaching 100% for nearly all activities. For the lying down activity, accuracy obtained from the CNN-GRU model stood at 99.46%; slightly higher than the 99.05% achieved by the CNN-GRU with attention model. This confirmed the robustness of our models against environmental changes.

In recent years, many researchers have focused on using convolutional neural networks to perform human activity recognition as evidenced by the emergence of a number of convolutional neural network architectures such as LeNet-5, AlexNet and VGG16 and modern architectures such as ResNet, Inception V3, Inception-ResNet, MobileNet V2, NASNet and PNASNet. The main characteristic of a convolutional neural network (CNN) is its ability to extract features automatically from input images, which facilitates the processes of activity recognition and classification. Convolutional networks indeed derive more relevant and complex features with every additional layer. In addition, CNNs have achieved perfect classification on highly similar activities that were previously extremely difficult to classify. In this paper, we evaluate modern convolutional neural networks in terms of their human activity recognition accuracy, and we compare the results with the state of the art methods. In our research, we used two public data sets, HMDB (Shooting gun, kicking, falling to the floor, punching) and the Weizman dataset (walking, running, jumping, bending, one hand waving, two-hand waving, jumping in place, jumping jack, skipping). Our experimental results indicated that the CNN with NASNet architecture achieves the best performance of the six CNN architectures on both human activity data sets (HMDB and Weizman).

<p>Human activity recognition (HAR) based on wearable devices is an emerging field of great interest. HAR can provide additional information on a human subject’s physical status. Utilising new technologies for HAR will become very meaningful with the development of deep learning. This study aims to mine deep learning models for HAR prediction with the highest accuracy on the basis of time-series data collected by mobile wearable devices. To this end, convolutional neural networks (CNN) and long short-term memory neural networks (LSTM) are combined in a deep network model to extract behavioural facts. The proposed CNN model contains two convolutional layers and a maximum pooling layer, and batch normalisation is added after each convolutional layer to improve convergence speed and avoid overfitting. This structure yields significant results in terms of performance. The model is evaluated on the MHEALTH dataset with a test set accuracy of 99.61% and can be used for the intelligent recognition of human activity. The results of this study show that the proposed model has better robustness and motion pattern detection capability compared to other models.</p> <p> </p>

Due to rapid development of sensor technology, human activity recognition (HAR) using wearable inertial sensors has recently become a new research hotspot. Deep learning, especially convolutional neural network (CNN) that can automatically learn intricate activity features have gained a lot of attention in ubiquitous HAR task. Most existing CNNs process sensor input by extracting channel-wise features, and the information from each channel can be separately propagated in a hierarchical way from lower layers to higher layers. As a result, they typically overlook information exchange among channels within the same layer. In this article, we first propose a shallow CNN that considers cross-channel communication in HAR scenario, where all channels in the same layer have a comprehensive interaction to capture more discriminative features of sensor input. One channel can communicate with all other channels by graph neural network to remove redundant information accumulated among channels, which is more beneficial for deploying lightweight deep models. Extensive experiments are conducted on multiple benchmark HAR datasets, namely UCI-HAR, OPPORTUNITY, PAMAP2 and UniMib-SHAR, which indicates that the proposed method enables shallower CNNs to aggregate more useful information, and surpasses baseline deep networks and other competitive methods. The inference speed is evaluated via deploying the HAR systems on an embedded system.