MhaGNN: A Novel Framework for Wearable Sensor-Based Human Activity Recognition Combining Multi-Head Attention and Graph Neural Networks

Improving worker safety and productivity is of paramount importance in the manufacturing industry, driving the adoption of advanced sensing and control systems. This concern is particularly relevant within the framework of Industry 5.0. In this context, wearable sensors offer a promising solution by enabling continuous and unobtrusive monitoring of workers’ activities in the manufacturing line. This book chapter focuses on wearable sensor-based human activity recognition and its role in promoting worker safety in manufacturing environments. Specifically, we present a case study on wearable sensor-based worker activity recognition in a manufacturing line with a mobile robot. As wearable sensors comprise various sensor types, we investigate and compare sensor data fusion approaches using neural network models to effectively handle the multimodal sensor data. In addition, we introduce several deep learning-based techniques to improve the performance of human activity recognition. By harnessing wearable sensors for human activity recognition, this book chapter provides valuable insights into improving worker safety on the manufacturing line, aligning with the principles of the Industry 5.0 paradigm. The chapter sheds light on the potential of wearable sensor technologies and offers avenues for future research in this field.

Human Activity Recognition (HAR) has become one of the most important research fields to achieve real-time monitoring of human activities for timely decision making in various applications like fall detection, elderly care etc. Now-a-days, most people use smartphones which come with various embedded inertial sensors like accelerometer and gyroscope to monitor acceleration and angular velocity. These smartphone-based sensors have proven to be cost-effective solution in identification of activities belonging to ADL (Activities of Daily Living). Various Machine Learning, Deep learning and hybrid models have been proposed and implemented for HAR. This paper also proposes various hybrid deep learning approaches which combine Deep Neural Networks with other models like LSTM (Long Short Term Memory) Model and GRU (Gated Recurrent Unit) for effective classification of engineered features from CNN (Convolutional Neural Network) Model. A novel architecture that integrates CNN with Random Forest Classifier (DeepCNN-RF) is proposed to add randomness to the model. The proposed models have been tested on publicly available HAR Datasets like UCI HAR and WISDM Activity Recognition Datasets. Experimental results show that the hybrid models outperform the state-of-the-art data mining, machine learning techniques in UCI HAR and WISDM with an overall maximum accuracy of 97.77% and 98.2% respectively.

Novel Human Activity Recognition by graph engineered ensemble deep learning model

With the advancement and ubiquitousness of wearable devices, wearable sensor-based human activity recognition (HAR) has become a prominent research area in the healthcare domain and human-computer interaction. Inertial measurement unit (IMU) which can provide a wide range of information such as acceleration, angular velocity has become one of the most commonly used sensors in HAR. Recently, with the growing demand for soft and flexible wearable devices, mountable stretch sensors have become a new promising modality in wearable sensor-based HAR. In this paper, we propose a deep learning-based multi-modality HAR framework which consists of three IMUs and two fabric stretch sensors in order to evaluate the potential of stretch sensors independently and in combination with IMU sensors for the activity recognition task. Three different deep learning algorithms: long short-term memory (LSTM), convolutional neural network (CNN) and hybrid CNN-LSTM are deployed to the sensor data for automatically extracting deep features and performing activity classification. The impact of sensor type on recognition accuracy of different activities is also examined in this study. A dataset collected from the proposed framework, namely iSPL IMU-Stretch and a public dataset called w-HAR are used for experiments and performance evaluation.

Data preprocessing, feature selection and classification algorithms usually occupy the bulk of surveys on human activity recognition (HAR). This paper instead gives a brief review on the data acquisition which is a critical stage of the wearable data-driven-based HAR. The review focuses on the determination of sensor types, modality of sensor devices, sensor deployment and data collection. The work aims to provide a comprehensive and detailed guidance for the fundamental part in HAR, also to highlight the challenges related to the topics reviewed.

Recently, deep learning, which are able to extract automatically features from data, has achieved state-of-the-art performance across a variety of sensor based human activity recognition (HAR) tasks. However, the existing deep neural networks are usually trained with a global loss, and all hidden layer weights have to be always kept in memory before the forward and backward pass has completed. The backward locking phenomenon prevents the reuse of memory, which is a crucial limitation for wearable activity recognition. In the paper, we proposed a layer-wise convolutional neural networks (CNN) with local loss for the use of HAR task. To our knowledge, this paper is the first that uses local loss based CNN for HAR in ubiquitous and wearable computing arena. We performed experiments on five public HAR datasets including UCI HAR dataset, OPPOTUNITY dataset, UniMib-SHAR dataset, PAMAP dataset, and WISDM dataset. The results show that local loss works better than global loss for tested baseline architectures. At no extra cost, the local loss can approach the state-of-the-arts on a variety of HAR datasets, even though the number of parameters was smaller. We believe that the layer-wise CNN with local loss can be used to update the existing deep HAR methods.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

To better understand and evaluate the GNSS positioning performance, it is convenient to adopt corresponding measures to reduce the impact of errors on positioning. A GNSS positioning error estimation scheme based on an improved multi-layer perceptron model is proposed. The multi-head attention mechanism and gating operation are integrated into the multi-layer perceptron model to adaptively select and filter features, enhancing the model’s ability to understand input features. First, the original positioning error of the satellite is obtained through the Kalman filter positioning method. The data are then preprocessed to extract available features. Finally, the features are input into the constructed model for training and testing to obtain the estimated positioning error value. Two types of comparative experiments were completed. The performance of the presented model is evaluated by the root mean square error. Experimental results show that the proposed method performs well in terms of performance indicators, and has obvious advantages over other state-of-the-art methods. In particular, the root mean square error of the presented method in the first dataset is 0.239 m, which is 39.2% and 17% lower than the current state-of-the-art long short-term memory network and convolutional neural network, respectively. The presented method can provide higher-precision estimated values for studying the GNSS positioning error estimation problem.

To overcome the limitations of existing deep learning-based image recognition approaches for diagnosing the operating conditions of pumping wells–namely, slow processing speed and high parameter complexity–this paper proposes a lightweight diagnostic method based on a one-dimensional convolutional neural network (1D CNN). Unlike traditional image-based techniques, the proposed method eliminates the need to convert raw data into dynamometer card images, thereby reducing computational overhead and significantly decreasing the number of model parameters. The architecture integrates 1D CNN with a bidirectional long short-term memory (BiLSTM) network to effectively capture both local patterns and global temporal dependencies inherent in different operating states of pumping wells. Additionally, a multi-head attention (MHA) mechanism is incorporated to enhance the representation of critical time points, thereby improving the model's sensitivity to key features and its overall classification performance. Experiments conducted on an augmented dataset–generated via a simulation-based algorithm–demonstrate that the proposed 1DCNN-BiLSTM-MHA model achieves a classification accuracy of 99.22%. Experimental results demonstrate that the proposed model possesses strong feature learning capabilities. Comparative analysis with several baseline models highlights its superior diagnostic performance and reliability, along with reduced parameter count and improved computational speed.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.

This study proposes a user influence propagation model based on dynamic heterogeneous network representation learning. The model combines multi-type nodes and semantic edge types, and adopts a learnable temporal embedding strategy to construct a dynamic heterogeneous graph, so as to realize the time-aware representation of nodes. In the representation learning stage, a multi-head attention mechanism is introduced to enhance context modeling and propagation path awareness. Meanwhile, structural contrastive learning is used as an auxiliary task to improve the discriminability of node representation, and a joint training strategy of node classification and edge prediction is adopted to enhance the generalization ability of the model. In the experimental evaluation, the proposed model is compared with several representative Graph Neural Network (GNN) models, including Graph Convolutional Network (GCN), Temporal Graph Attention Network (TGAT), and Heterogeneous Graph Transformer (HGT). In the overall prediction task, the proposed model achieves a Mean Squared Error (MSE) of 0.1214, a Mean Absolute Error (MAE) of 0.2398, and an R² of 0.701. All these metrics are significantly better than those of TGAT (MSE: 0.1597, MAE: 0.2785, R²: 0.603) and HGT (MSE: 0.1429, MAE: 0.2652, R²: 0.642). Within a 14-day prediction window, the model maintains an error rate of 0.1390, demonstrating superior temporal generalization capability. For identifying high-influence user groups, the model achieves an MSE of 0.1268, significantly better than HGT’s 0.1456, indicating higher sensitivity in modeling strong propagation nodes. This shows that the combination of these methods enables the proposed model to demonstrate stronger temporal modeling capability and heterogeneous relationship understanding capability compared with TGAT and HGT, in terms of capturing users’ historical behaviors and predicting future influence.

Recent advancement in smart phones and computing technologies has played a vital role in people’s life. Develop a model to detect the human basic dynamic activities such as Amble, Climb stairs, coming down the stairs into the floor and human basic static activities like Sitting, Standing or Laying using the person’s smart phone and computers are the major work of this paper. Conventional Machine learning models like Logistic Regression, SVC, Decision tree, etc. results are compared with a recurrent deep neural network model named as Long Short Term Memory (LSTM). LSTM is proposed to detect the human behavior based on Human Activity Recognition (HAR) dataset. The data is monitored and recorded with the aid of sensors like accelerometer and Gyroscope in the user smart phone. HAR dataset is collected from 30 persons, performing different activities with a smart phone to their waists. The testing of the model is evaluated with respect to accuracy and efficiency. The designed activity recognition system can be manipulated in other activities like predicting abnormal human actions, disease by human actions, etc. The overall accuracy has improved to 95.40%.

Cross-subject transfer learning in human activity recognition systems using generative adversarial networks

With the widespread application of the Go language, the demand for vulnerability detection in Go programs is increasing. Existing detection models and methods have deficiencies in extracting source code features of Go programs and mainly focus on detecting concurrency vulnerabilities. In response to these issues, we propose a Go program vulnerability detection method based on a graph neural network (GNN). The core of this approach is to utilize GraphSAGE to extract the global structure and deep semantic information of each concurrent function, maximizing the learning of concurrency vulnerability features. To capture contextual information of fine-grained code fragments in source code, we employ taint analysis to extract taint propagation chains and use a Transformer model with a multi-head attention mechanism, based on lexical analysis, to extract fine-grained vulnerability features. We integrate graph-level and token-level features to maximize the detection of various complex types of vulnerabilities in Go source code. Experimental results on a real-world vulnerability dataset demonstrate that our model outperforms existing detection methods and tools, achieving an F1-score of 91.35%. Furthermore, ablation experiments confirm that the proposed feature fusion method effectively extracts deep vulnerability features.

Human activity recognition (HAR) based on wearable sensors has emerged as a low-cost key-enabling technology for applications such as human–computer interaction and healthcare. In wearable sensor-based HAR, deep learning is desired for extracting human active features. Due to the spatiotemporal dynamic of human activity, a special deep learning network for recognizing the temporal continuous activities of humans is required to improve the recognition accuracy for supporting advanced HAR applications. To this end, a residual multifeature fusion shrinkage network (RMFSN) is proposed. The RMFSN is an improved residual network which consists of a multi-branch framework, a channel attention shrinkage block (CASB), and a classifier network. The special multi-branch framework utilizes a 1D-CNN, a lightweight temporal attention mechanism, and a multi-scale feature extraction method to capture diverse activity features via multiple branches. The CASB is proposed to automatically select key features from the diverse features for each activity, and the classifier network outputs the final recognition results. Experimental results have shown that the accuracy of the proposed RMFSN for the public datasets UCI-HAR, WISDM, and OPPORTUNITY are 98.13%, 98.35%, and 93.89%, respectively. In comparison with existing advanced methods, the proposed RMFSN could achieve higher accuracy while requiring fewer model parameters.