Activity Recognition Using a Multi-head Convolution Neural Network Integrated with Attention Mechanism

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Currently, identification of complex human activities is experiencing exponential growth through the use of deep learning algorithms. Conventional strategies for recognizing human activity generally rely on handcrafted characteristics from heuristic processes in time and frequency domains. The advancement of deep learning algorithms has addressed most of these issues by automatically extracting features from multimodal sensors to correctly classify human physical activity. This study proposed an attention-based bidirectional gated recurrent unit as Att-BiGRU to enhance recurrent neural networks. This deep learning model allowed flexible forwarding and reverse sequences to extract temporal-dependent characteristics for efficient complex activity recognition. The retrieved temporal characteristics were then used to exemplify essential information through an attention mechanism. A human activity recognition (HAR) methodology combined with our proposed model was evaluated using the publicly available datasets containing physical activity data collected by accelerometers and gyroscopes incorporated in a wristwatch. Simulation experiments showed that attention mechanisms significantly enhanced performance in recognizing complex human activity.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.

Human activity recognition is concerned with identifying the specific movement of a person based on sensor data. In recent years, many different techniques have been proposed for modelling and recognising human activities, with a specific focus on the development of approaches to classifying human activities using deep learning techniques. The research presented in this paper proposes a fuzzy feature representation approach to represent occupancy sensor data, along with a Convolutional Neural Network Classifier (CNNC) for human activity modelling and recognition. Sensory data gathered from a home environment are converted into occupancy data representing human activities and then fuzzified before being fed as inputs into the CNNC. The learning capability of CNNC allows the model to learn the relationship between the fuzzified inputs and their corresponding output activities during training mode. The relations learned in the trained CNNC model are then used to identify human activity patterns and classify these when the testing dataset is applied. The proposed method is evaluated using a dataset representing activities of daily living for a single user gathered from a real-home environment.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

Human motion detection and activity recognition are becoming vital for the applications in smart homes. Traditional Human Activity Recognition (HAR) mechanisms use special devices to track human motions, such as cameras (vision-based) and various types of sensors (sensor-based). These mechanisms are applied in different applications, such as home security, Human–Computer Interaction (HCI), gaming, and healthcare. However, traditional HAR methods require heavy installation, and can only work under strict conditions. Recently, wireless signals have been utilized to track human motion and HAR in indoor environments. The motion of an object in the test environment causes fluctuations and changes in the Wi-Fi signal reflections at the receiver, which result in variations in received signals. These fluctuations can be used to track object (i.e., a human) motion in indoor environments. This phenomenon can be improved and leveraged in the future to improve the internet of things (IoT) and smart home devices. The main Wi-Fi sensing methods can be broadly categorized as Received Signal Strength Indicator (RSSI), Wi-Fi radar (by using Software Defined Radio (SDR)) and Channel State Information (CSI). CSI and RSSI can be considered as device-free mechanisms because they do not require cumbersome installation, whereas the Wi-Fi radar mechanism requires special devices (i.e., Universal Software Radio Peripheral (USRP)). Recent studies demonstrate that CSI outperforms RSSI in sensing accuracy due to its stability and rich information. This paper presents a comprehensive survey of recent advances in the CSI-based sensing mechanism and illustrates the drawbacks, discusses challenges, and presents some suggestions for the future of device-free sensing technology.

Real-time human activity recognition plays an essential role in real-world human-centered robotics applications, such as assisted living and human-robot collaboration. Although previous methods based on skeletal data to encode human poses showed promising results on real-time activity recognition, they lacked the capability to consider the context provided by objects within the scene and in use by the humans, which can provide a further discriminant between human activity categories. In this paper, we propose a novel approach to real-time human activity recognition, through simultaneously learning from observations of both human poses and objects involved in the human activity. We formulate human activity recognition as a joint optimization problem under a unified mathematical framework, which uses a regression-like loss function to integrate human pose and object cues and defines structured sparsity-inducing norms to identify discriminative body joints and object attributes. To evaluate our method, we perform extensive experiments on two benchmark datasets and a physical robot in a home assistance setting. Experimental results have shown that our method outperforms previous methods and obtains real-time performance for human activity recognition with a processing speed of 104 Hz.

Smartphones are increasing ubiquitously due to the need and demand in the modern era. The world is transmuting into a global village with the cumulation of smart devices. Nowadays, smartphones are enriched with inertial sensors which can be used to recognize physical human activities in the wild. Human activity recognition (HAR) lies at the core of many applications like health monitoring, fall detection, road safety, personal assistance, and behavior-based context awareness. Context-based HAR is a new dimension that provides fine-grained information about the action being performed and leads us towards the automated and intelligent system design, which is useful to furnish the smart solution to real-life problems. In this paper, smartphone sensors are utilized to propose a framework for human activity and context recognition. This framework performs activity-dependent context recognition on the ExtraSensory dataset using random forest, decision tree, and k-nearest neighbor classifiers, and achieves an accuracy of 93.10%, demonstrating the effectiveness of the proposed framework.

With the rapid development of the computer and sensor field, inertial sensor data have been widely used in human activity recognition. At present, most relevant studies divide human activities into basic actions and transitional actions, in which basic actions are classified by unified features, while transitional actions usually use context information to determine the category. For the existing single method that cannot well realize human activity recognition, this paper proposes a human activity classification and recognition model based on smartphone inertial sensor data. The model fully considers the feature differences of different properties of actions, uses a fixed sliding window to segment the human activity data of inertial sensors with different attributes and, finally, extracts the features and recognizes them on different classifiers. The experimental results show that dynamic and transitional actions could obtain the best recognition performance on support vector machines, while static actions could obtain better classification effects on ensemble classifiers; as for feature selection, the frequency-domain feature used in dynamic action had a high recognition rate, up to 99.35%. When time-domain features were used for static and transitional actions, higher recognition rates were obtained, 98.40% and 91.98%, respectively.

Sensor-based Human Activity Recognition (HAR) has been applied across various domains, including healthcare monitoring, fitness tracking, and smart home systems. These applications require the ability to accurately detect and respond to a wide range of human activities, each with varying distributions, which imposes a significant challenge. The task-incremental learning paradigm can address this problem by enabling HAR systems to adapt to changes in distribution and learn new activities over time without forgetting the previously learned ones. Continual adaptation is essential for maintaining high performance, as it allows the system to effectively respond to these changes. Although several strategies in the continual learning literature have been evaluated for task-incremental scenarios in HAR, there is still room for improvement, as the results are not as good as those achieved with conventional approaches. This work proposes two new neuroplasticity-inspired techniques that can be integrated with any learning strategy. Inspired by the brain’s ability to reorganize and strengthen connections over time, these methods focus on enhancing the model’s flexibility and long-term knowledge retention. The proposed techniques were evaluated alongside the WA-ADB and WA-MDF strategies on well-known HAR datasets. Experimental results demonstrated that the new techniques significantly enhanced the models’ ability to retain knowledge, which holds significant potential for improving the robustness and longevity of HAR systems in real-world applications.

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

Human Activity Recognition (HAR) is pivotal in ubiquitous computing, offering benefits to human-centric services such as health monitoring, smart homes, and eldercare systems. HAR leverages smartphones, smartwatches, and other wearable devices to collect sensory data annotated with activity labels, which are then used to train machine learning or deep learning models for automatic activity recognition. Effective HAR systems must integrate information from multiple modalities to accurately assist users. This paper introduces the XAttentionHAR model, an innovative cross-modality attention-based ensemble model for HAR. Our approach utilizes a self-attention module to extract features within each modality and an inter-domain cross-attention module to capture and integrate long-term dependencies across domains. The cross-modality attention mechanism enhances the fusion of diverse modalities, enriching the semantic information. We conducted extensive experiments on the WISDM public dataset, which includes accelerometer and gyroscope data from smartwatches and smartphones. Our results demonstrate that XAttentionHAR outperforms other state-of-the-art methods in activity recognition, achieving 98.48% accuracy for smartphone-based HAR and 98.73% accuracy for smartwatch-based HAR, paving the way for improved human-centric services.

Emotions and activities are an integral part of human lifestyles and daily living. They occur all the time in everyone as humans are always in an emotional state and also performing an activity. This paper proposes an architecture that combines Human Emotion Recognition (HER) and Human Activity Recognition (HAR) to monitor people. It utilizes emotion physiological signals and accelerometer data harvested from individuals to recognize human emotions and activities respectively in addition to location information in order to remotely monitor such individuals and trigger alerts if necessary especially in a distress state situation that requires emergency services. The basic concepts of accelerometer data based HAR and physiological signals based HER and how they can be combined to monitor individuals in an emergency response system are succinctly presented.

Human Activity Recognition (HAR) and gender recognition have become pivotal in advancing intelligent systems, as they enable tailored user experiences and enhance automation in healthcare, security, and personalized technology. By accurately identifying human activities and gender, systems can proactively adapt to user needs, improving human-computer interaction, reducing response times in emergency detection, and enhancing the quality of life in smart home and assistive living applications. In this study, a Kolmogorov-Arnold Network model was created to predict daily living activities and the gender of the person performing an activity. The proposed Kolmogorov-Arnold Network (KAN) classifier outperformed the previous studies with 94.5% and 95.6% in terms of overall accuracy for multi-class HAR and gender recognition tasks, respectively. Additionally, the Chi-square test results demonstrated that there is a statistically significant difference between the performances. It can be concluded that KAN method is a robust classifier especially for detecting activities that have a minor number of samples on the utilized dataset.

Human activity recognition (HAR) can play a vital role in biomedical and health informatics by enabling the monitoring of human daily activities and health behaviors. Accurate HAR can provide valuable insights into patients' physical activity levels, and this helps to manage chronic conditions and promote healthy lifestyles. In this paper, we propose a deep learning model, DKInception, designed for HAR tasks. DKInception integrates deep convolutional residual networks with an attention mechanism and leverages multi-scale convolution kernels to efficiently extract temporal features for activity identification. This model is built on the Inception ResNet architecture and extends its capabilities with effective, fast convergence and robust scaling properties. To evaluate the performance of DKInception, we conduct extensive experiments on four benchmark HAR datasets: UCI-HAR, Opportunity, Daphnet, and PAMAP2. Our comparative analysis with several existing models shows that DKInception outperforms these models using various evaluation metrics. These results demonstrate the model recorded high accuracy of 95.70%, 87.48%, 94.00% and 89.72%, for UCI-HAR, Opportunity, Daphnet, and PAMAP2, respectively.