Multiple Kernel Representation Learning for WiFi-Based Human Activity Recognition

Human motion detection and activity recognition are becoming vital for the applications in smart homes. Traditional Human Activity Recognition (HAR) mechanisms use special devices to track human motions, such as cameras (vision-based) and various types of sensors (sensor-based). These mechanisms are applied in different applications, such as home security, Human–Computer Interaction (HCI), gaming, and healthcare. However, traditional HAR methods require heavy installation, and can only work under strict conditions. Recently, wireless signals have been utilized to track human motion and HAR in indoor environments. The motion of an object in the test environment causes fluctuations and changes in the Wi-Fi signal reflections at the receiver, which result in variations in received signals. These fluctuations can be used to track object (i.e., a human) motion in indoor environments. This phenomenon can be improved and leveraged in the future to improve the internet of things (IoT) and smart home devices. The main Wi-Fi sensing methods can be broadly categorized as Received Signal Strength Indicator (RSSI), Wi-Fi radar (by using Software Defined Radio (SDR)) and Channel State Information (CSI). CSI and RSSI can be considered as device-free mechanisms because they do not require cumbersome installation, whereas the Wi-Fi radar mechanism requires special devices (i.e., Universal Software Radio Peripheral (USRP)). Recent studies demonstrate that CSI outperforms RSSI in sensing accuracy due to its stability and rich information. This paper presents a comprehensive survey of recent advances in the CSI-based sensing mechanism and illustrates the drawbacks, discusses challenges, and presents some suggestions for the future of device-free sensing technology.

Human activity recognition (HAR) is an important research topic in the field of human-computer interaction (HCI), and wearable HAR based on convolutional neural networks (CNNs) has achieved the most advanced performance. In this study, wearable plastic optical fiber (POF) sensors based on GAF-ConvNeXt algorithm are used for HAR for the first time. The GAF-ConvNeXt algorithm, combined with the Gramian Angular Field (GAF) and the latest CNN model ConvNeXt, can convert motion signal of a one-dimensional time series into a two-dimensional image, which can not only retain the complete information of the signal, but also maintain its dependence on time, thereby presenting more depth features and facilitating accurate recognition of human activities. Furthermore, transfer learning (TL) was introduced to further optimize the GAF-ConvNeXt model, which effectively reduces the number of training samples, improves the recognition accuracy and accelerates the convergence speed. The GAF-ConvNeXt algorithm based on TL can recognize seven motion modes with an accuracy of 99.75%, and has great application potential in the fields of human motion recognition and HCI.

Healthcare using body sensor data has been getting huge research attentions by a wide range of researchers because of its good practical applications such as smart health care systems. For instance, smart wearable sensor-based behavior recognition system can observe elderly people in a smart eldercare environment to improve their lifestyle and can also help them by warning about forthcoming unprecedented events such as falls or other health risk, to prolong their independent life. Although there are many ways of using distinguished sensors to observe behavior of people, wearable sensors mostly provide reliable data in this regard to monitor the individual’s functionality and lifestyle. In this paper, we propose a body sensor-based activity modeling and recognition system using time-sequential information-based deep Neural Structured Learning (NSL), a promising deep learning algorithm. First, we obtain data from multiple wearable sensors while the subjects conduct several daily activities. Once the data is collected, the time-sequential information then go through some statistical feature processing. Furthermore, kernel-based discriminant analysis (KDA) is applied to see the better clustering of the features from different activity classes by minimizing inner-class scatterings while maximizing inter-class scatterings of the samples. The robust time-sequential features are then applied with Neural Structured Learning (NSL) based on Long Short-Term Memory (LSTM), for activity modeling. The proposed approach achieved around 99% recall rate on a public dataset. It is also compared to existing different conventional machine learning methods such as typical Deep Belief Network (DBN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) where they yielded the maximum recall rate of 94%. Furthermore, a fast and efficient explainable Artificial Intelligence (XAI) algorithm, Local Interpretable Model-Agnostic Explanations (LIME) is used to explain and check the machine learning decisions. The robust activity recognition system can be adopted for understanding peoples' behavior in their daily life in different environments such as homes, clinics, and offices.

In this work, we address the issue of Wi-Fi-based human activity recognition (HAR) system in densely deployed Wi-Fi environments. With the benefit of sufficient information provided by Wi-Fi channel state information (CSI), HAR based on Wi-Fi has become an active research area in recent years. Traditional Wi-Fi CSI-based HAR applications usually focus on utilizing one Wi-Fi transmitter and one or several Wi-Fi receivers to extract the activity-related features, ignoring the communication among multiple Wi-Fi devices in the real world. In this paper, we present a novel Wi-Fi link selection model on the basis of continuous state decision-making process in which CSI is modeled as a part of the state. The model, referred to as WiAgent, takes an action of selecting one Wi-Fi link according to current state, and then updates the state for the choice of the next action. From extensive experiment results, our method performs better than other solutions in a given environment where multiple Wi-Fi transmitters exist.

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

Automatic analysis of the video is one of most complex problems in the fields of computer vision and machine learning. A significant part of this research deals with (human) activity recognition (HAR) since humans, and the activities that they perform, generate most of the video semantics. Video-based HAR has applications in various domains, but one of the most important and challenging is HAR in sports videos. Some of the major issues include high inter- and intra-class variations, large class imbalance, the presence of both group actions and single player actions, and recognizing simultaneous actions, i.e., the multi-label learning problem. Keeping in mind these challenges and the recent success of CNNs in solving various computer vision problems, in this work, we implement a 3D CNN based multi-label deep HAR system for multi-label class-imbalanced action recognition in hockey videos. We test our system for two different scenarios: an ensemble of $k$ binary networks vs. a single k-output network, on a publicly available dataset. We also compare our results with the system that was originally designed for the chosen dataset. Experimental results show that the proposed approach performs better than the existing solution.

Human activity recognition is a basic building block in numerous healthcare systems, mainly because the ability to understand the user's situation and context. This paper presents a solution to the general problem with evaluation of human activity recognition systems, i.e., an activity recognition system may perform perfectly in controlled lab experiments, but significantly worse once applied to more realistic conditions. The solution is presented through the practical experience gained with the creation of our RAReFall activity recognition system. Although the system was awarded the first place at the EvAAL-AR live competition, the recognition accuracy at the competition was significantly lower compared to the controlled lab experiments performed just before the competition. To overcome the encountered problem we developed an automatic calibration method, which solves the encountered problem by adapting and re-calibrating the accelerometer data in real-time while the user is performing everyday activities. The method increased the overall accuracy for 8 percentage points and for 51 percentage points for the sitting activity.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

The recent development of machines exhibiting intelligent characteristics involves numerous techniques including computer hardware and software architecture development. Many different hardware devices, wearable sensors, machine learning, and deep learning model implementations are being applied in human activity recognition (HAR) applications in recent times. However, to develop high accuracy classification systems for activity recognition using ilow-cost hardware technology is of significant importance. To achieve this goal this study uses sensor data from two low-cost sensors, gyroscope and accelerometer along with the implementation of an Artificial Neural Network (ANN) based deep learning model for HAR. In particular, Deep Stacked Multilayered Perceptron (DS-MLP) has been proposed. In the implementation of DS-MLP, an ANN model has been used as a meta-learner while five MLP models have been used as base-learners. In this study, these base-learners and meta-learner have been combined using a stack ensemble technique. The performance evaluations have been done first on the applicability of individual base-models followed by the application of DS-MLP, the results prove the high accuracy of 97.3% and 99.4% for heterogeneous datasets used for testing. The performance of the proposed DS-MLP models has been compared to some existing machine learning classifiers and several state-of-the-art activity recognition systems. The comparative result analysis also proves that the proposed system performed better than these classification approaches in terms of important performance metrics such as accuracy, precision, recall, F-score, Cohen's Kappa, and Mathew correlation coefficient.

SUMMARYRemote surveillance of large‐scale equipment such as power plants and building complexes is important to prevent serious attacks and troubles. Automatic human action recognition can reduce the burdens of the surveillance. Multiview video is useful for human action recognition, because it provides robustness to the changes of people's appearance by orientation and occlusion. One problem of conventional multiview action recognition is that it requires both detection and tracking before action recognition. Human pose and motion vary depending on the person's action, and such variances may cause detection and tracking error. To solve this problem, previous work has proposed simultaneous action recognition and location estimation for single‐view videos using Hough voting. In this paper, we extend the Hough voting approach to simultaneous multiview action recognition and location estimation. Our proposed method independently casts votes for the action labels and spatiotemporal reference positions of actions in each view and integrates them using homographical transformations in the multiview extension. We evaluated our method and confirmed that it achieved high accuracy in action recognition and location estimation. The contribution of this paper is that it enables multiview action recognition without prior human detection and tracking.

This paper brings to light one of the most prominent applications of human activity recognition which is the anomaly detection. Providing security to an individual is a major concern of any society today due to the constantly increasing actions causing threats, starting from deliberate violence to an injury caused by an accident. Mere installation of a traditional closed-circuit television is not enough as it requires a human being to constantly remain alert and monitor the cameras, which is quite inefficient. This calls for the need to develop an automated security system that identifies abnormal activities in real time and brings immediate help to the victims. But it is time-consuming to recognize activities from lengthy surveillance videos and hence a new implementation idea which adaptively compresses the videos before passing it through the activity recognition system has been proposed. Adaptive video compression is a unique compression technology that compresses only the least significant parts of the video, retaining the objects of interest. By combining adaptive video compression and the convolutional 3D network with contextual multiple scales based on the temporal features (Xu et al. in Learning deep representations of appearance and motion for anomalous event detection. In: Computer Vision and Pattern Recognition (CVPR), 2015), the paper aims to provide an accurate anomalous human activity recognition system that works in real time. The method is implemented on the UCF101 crime dataset with around 13 different anomalies (Learning Spatiotemporal Features with 3D Convolutional Networks. In: International Conference on Computer Vision (ICCV), IEEE, 2015). It resulted in improved recognition performance compared to state-of-the-art techniques.

An increase in life expectancy is leading to an aging society, where gradual degradation in physical/cognitive health could result in fall injuries, cardiovascular, neurological/psychological problems, or a sedentary lifestyle. Sensor-based human activity recognition (HAR) is found to be a potential assistive technology in this context to improve the quality of life of an aging society. This chapter presents the state of the art and future trends of Internet of Things (IoT)-enabled sensor data-based Human Activity Recognition (HAR) systems, where machine learning (ML)/deep learning (DL) is used to classify human actions in the context of healthcare applications. In particular, the impact of IoT on the healthcare industry is so immense that a dedicated specialization, termed as Internet of Healthcare Things (IoHT), has emerged. Smart healthcare systems are equipped with different categories of sensors (wearable, wireless, and ambient). The huge amount of sensor data acquired by IoHT sensors allows for performing accurate HAR. The HAR involves detection/classification problems, e.g. fall detection and seizer detection, where ML is used extensively, at its heart. A lot of ML/DL techniques are being used in HAR including support vector machine (SVM), artificial neural network (ANN), recurrent neural network (RNN), convolutional neural network (CNN), deep neural network (DNN), long short-term memory (LSTM), etc. However, there are a few special aspects of HAR sensor data that make it difficult to handle. Involved research is still being pursued across the globe to solve these problems.KeywordsIoTIoHTHealth careHARSensorMachine learning (ML)Deep learning (DL)

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Deep learning techniques for human activity recognition are gaining popularity due to their effectiveness in identifying intricate tasks and their cost-effectiveness compared to conventional machine learning methods. Human Activity Recognition (HAR) is a research domain concerned with detecting the everyday activities performed by individuals using time-series data captured by sensors. HAR encompasses a wide range of applications, including surveillance, baby monitoring, elderly healthcare, and smart driving. This article provides a brief introduction to the application of deep learning in HAR. It covers the fundamental concepts of CNNs and LSTMs, their strengths in capturing spatial and temporal features, and their integration for enhanced activity recognition. Different approaches are employed in HAR to address problems with efficiency and precision. Conventional human activity recognition (HAR) systems rely on wearable devices like IMUs and stretch sensors to identify different activities. These systems have proven to be effective in recognizing simple user actions like sitting, standing, and walking. However, when it comes to more intricate activities like running, jumping, wrestling, and swinging, sensor-based HAR systems encounter greater misclassification rates due to inaccuracies in sensor readings. These errors significantly impact the accuracy of the HAR system, resulting in suboptimal classification outcomes. In contrast, employing vision-based HAR systems enables improved accuracy in identifying complex activities, leading to enhanced overall performance.

Application of DNN for radar micro-doppler signature-based human suspicious activity recognition