Online Personalisation of Deep Mobile Activity Recognisers

Regularized multi-task learning (MTL) algorithms have been exploited in the field of pattern recognition and computer vision gradually, which can fully excavate the relationships of different related tasks. Therefore, many dramatically favorable approaches based on regularized MTL have been proposed. In the past decades, although the promising results about human action recognition have been achieved, most of existing action recognition algorithms focus on action descriptors, single/multi-view and multi-modality action recognition, and few works are related with MTL, especial of lacking the systematic evaluation of existing MTL algorithms for human action recognition. Thus, in the paper, seven popular regularized MTL algorithms in which different actions are considered as different tasks, are systematically exploited on two public multi-view action datasets. In detail, dense trajectory features are firstly extracted for each view, and then the shared codebook are constructed for all views by k-means, and then each video is coded by the shared codebook. Moreover, according to different regularized MTL algorithms, all actions or part of actions are considered as related, and then these actions are set to different tasks in MTL. Further, the effectiveness of different number of training samples from different action views is also evaluated for MTL. Large scale experimental results show that: 1) Regularized MTL is very useful for action recognition which can dig the latent relationship among different actions; 2) Not of all human actions are related, if irrelative actions are put together in MTL, its performance will fall; 3) With the increase of the training samples from different views, the relationships about different actions can be fully exploited, and it promotes the accuracy improvement of action recognition.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

Smartphones are increasing ubiquitously due to the need and demand in the modern era. The world is transmuting into a global village with the cumulation of smart devices. Nowadays, smartphones are enriched with inertial sensors which can be used to recognize physical human activities in the wild. Human activity recognition (HAR) lies at the core of many applications like health monitoring, fall detection, road safety, personal assistance, and behavior-based context awareness. Context-based HAR is a new dimension that provides fine-grained information about the action being performed and leads us towards the automated and intelligent system design, which is useful to furnish the smart solution to real-life problems. In this paper, smartphone sensors are utilized to propose a framework for human activity and context recognition. This framework performs activity-dependent context recognition on the ExtraSensory dataset using random forest, decision tree, and k-nearest neighbor classifiers, and achieves an accuracy of 93.10%, demonstrating the effectiveness of the proposed framework.

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

Video-based human action recognition is one of the most important and challenging areas of research in the field of computer vision. Human action recognition has found many pragmatic applications in video surveillance, human-computer interaction, entertainment, autonomous driving, etc. Owing to the recent development of deep learning methods for human action recognition, the performance of action recognition has significantly enhanced for challenging datasets. Deep learning techniques are mainly used for recognizing actions in images and videos comprising of Euclidean data. A recent development in deep learning methods is the extension of these techniques to non-Euclidean data or graph data with many nodes and edges. Human body skeleton resembles a graph, therefore, the graph convolutional network (GCN) is applicable to the non-Euclidean body skeleton. In the past few years, GCN has emerged as an important tool for skeleton-based action recognition. Therefore, we conduct a survey using GCN methods for action recognition. Herein, we present a comprehensive overview of recent GCN techniques for action recognition, propose a taxonomy for the categorization of GCN techniques for action recognition, carry out a detailed study of the benchmark datasets, enlist relevant resources and open-source codes, and finally provide an outline for future research directions and trends. To the best of authors' knowledge, this is the first survey for action recognition using GCN techniques.

Human Activity Recognition (HAR) with body-worn sensors has been studied intensively in the past decade. Existing approaches typically rely on data from inertial sensors. This paper explores the potential of using point cloud data gathered from wearable depth cameras for on-body activity recognition. We discuss effects of different granularity in the depth information and compare their performance to inertial sensor based HAR. We evaluated our approach with a total of sixteen participants performing nine distinct activity classes in three home environments. 10-fold cross-validation results of KNN and Random Forests classification exhibit a significant increase in F-score from inertial data to depth information (by > 12 percentage points) and show a further improvement when combining low-resolution depth matrices and sensor data. We discuss the performance of the different sensor types for different contexts and show that overall, depth sensors prove to be suitable for HAR.

Human Activity Recognition (HAR) is pivotal in ubiquitous computing, offering benefits to human-centric services such as health monitoring, smart homes, and eldercare systems. HAR leverages smartphones, smartwatches, and other wearable devices to collect sensory data annotated with activity labels, which are then used to train machine learning or deep learning models for automatic activity recognition. Effective HAR systems must integrate information from multiple modalities to accurately assist users. This paper introduces the XAttentionHAR model, an innovative cross-modality attention-based ensemble model for HAR. Our approach utilizes a self-attention module to extract features within each modality and an inter-domain cross-attention module to capture and integrate long-term dependencies across domains. The cross-modality attention mechanism enhances the fusion of diverse modalities, enriching the semantic information. We conducted extensive experiments on the WISDM public dataset, which includes accelerometer and gyroscope data from smartwatches and smartphones. Our results demonstrate that XAttentionHAR outperforms other state-of-the-art methods in activity recognition, achieving 98.48% accuracy for smartphone-based HAR and 98.73% accuracy for smartwatch-based HAR, paving the way for improved human-centric services.

This paper reports the scaling behavior of the fuzzy c-means based classifier (FCMC) with many parameters. FCMC is a classifier based on clustering approaches. The classification accuracy on test sets (i.e., the generalization capability) is not necessarily improved by increasing the number of clusters. Especially when the number of training samples is relatively small, not only the classification boundary over-fits the data, but also covariance matrices and cluster centers are computed incorrectly, since the number of samples in each cluster becomes smaller. Hence, the test set accuracy deteriorates. The performance of FCMC with two clusters in each class and the number of training samples less than 1000, was reported in the literature. This paper reports the scaling behavior of FCMC by testing with variously-sized training samples. The number of clusters of FCMC is increased up to eight. The number of clusters used in this paper is not very large but the number of parameters is relatively large. So, the parameters are optimized to training sets. LibSVM is one of the widely known state of the art tools for support vector machines (SVM). The test set accuracy, training time and testing time (i.e., the detection time) of FCMC are compared with LibSVM by varying the size of training sets. FCMC shows a good generalization capability, though the parameters are optimized to training sets. When the number of training samples is increased by 10 times, the training time of FCMC increases by 10 times, but that of LibSVM increases by a factor of 100. The testing time is also much shorter than LibSVM when the size of the training set is large.

Human motion detection and activity recognition are becoming vital for the applications in smart homes. Traditional Human Activity Recognition (HAR) mechanisms use special devices to track human motions, such as cameras (vision-based) and various types of sensors (sensor-based). These mechanisms are applied in different applications, such as home security, Human–Computer Interaction (HCI), gaming, and healthcare. However, traditional HAR methods require heavy installation, and can only work under strict conditions. Recently, wireless signals have been utilized to track human motion and HAR in indoor environments. The motion of an object in the test environment causes fluctuations and changes in the Wi-Fi signal reflections at the receiver, which result in variations in received signals. These fluctuations can be used to track object (i.e., a human) motion in indoor environments. This phenomenon can be improved and leveraged in the future to improve the internet of things (IoT) and smart home devices. The main Wi-Fi sensing methods can be broadly categorized as Received Signal Strength Indicator (RSSI), Wi-Fi radar (by using Software Defined Radio (SDR)) and Channel State Information (CSI). CSI and RSSI can be considered as device-free mechanisms because they do not require cumbersome installation, whereas the Wi-Fi radar mechanism requires special devices (i.e., Universal Software Radio Peripheral (USRP)). Recent studies demonstrate that CSI outperforms RSSI in sensing accuracy due to its stability and rich information. This paper presents a comprehensive survey of recent advances in the CSI-based sensing mechanism and illustrates the drawbacks, discusses challenges, and presents some suggestions for the future of device-free sensing technology.

In robotics, recognition of human activity has been used extensively for robot task learning through imitation and demonstration. However, there has not been much work performed on modeling and recognition of activities that involve object manipulation and grasping. In this work, we deal with single arm/hand actions which are very similar to each other in terms of arm/hand motions. The approach is based on the hypothesis that actions can be represented as sequences of motion primitives. Given this, a set of five different manipulation actions of different levels of complexity are investigated. To model the process, we use a combination of discriminative support vector machines and generative hidden Markov models. The experimental evaluation, performed with 10 people, investigates both the definition and structure of primitive motions, as well as the validity of the modeling approach taken.

Wearable sensors are monumental for human activity recognition. Researchers are continuously inventing new technology to detect human activity properly. Earable opens up interesting possibilities of monitoring personal scale behavioral activities. In this paper, we explore earables device 'eSense' multisensory stereo device for personal scale behavior analysis. We propose an activity recognition framework by exploiting eSense based multi-sensory device. It has a microphone, 6-axis inertial measurement unit, and a dual-mode Bluetooth. We use eSense accelerometer sensor data for detecting head and mouth related behavioral activities. We develop a data collection framework from the eSense through our smartphone application via Bluetooth. Then from the collected data, a few statistical features are computed to classify six personal scale activities related to head and neck movement such as speaking, eating, headshaking and head nodding, as well as, stay and walk. We aggregate the time series data intzo different action labels that summarize the user activity over a time interval. After, we train the data to induce a predictive model for activity recognition. We explore both machine learning and deep learning approach for data classification. For classification, we use the Support Vector Machine, Random Forest, and K-Nearest Neighbor and Convolutional Neural Network and achieve satisfactory recognition accuracy. The findings provide promising prospect for eSense for personal scale activity recognition in healthcare monitoring service. Based on our study, this kid of work is done for the first time with satisfactory findings.