Multimodal Attribute and Feature Embedding for Activity Recognition

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Encoding human activities using multimodal wearable sensory data

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.

Human activity recognition is one of the most challenging research areas in human-machine interaction. Activity recognition by wearable sensors produces significant information about a person's functional ability and lifestyle. In this paper, we present an inertial sensors-based approach for activity and postural transition recognition. Efficient features are extracted from the sensor data. Then, the features are normalized using Z-score followed by Generalized Discriminant Analysis (GDA) to make them more robust. Finally, the features are trained with a Recurrent Neural Network (RNN) for activity and postural transition recognition. The proposed method recognizes 12 activities with an overall accuracy of 97%. In the activity recognition system, the multimodal input data is collected from only one accelerometer and one gyroscope sensor embedded in the user's smartphone. The proposed system can contribute in research fields such as human-robot interaction for practical applications such as eldercare in smart homes.

Human activity recognition is concerned with identifying the specific movement of a person based on sensor data. In recent years, many different techniques have been proposed for modelling and recognising human activities, with a specific focus on the development of approaches to classifying human activities using deep learning techniques. The research presented in this paper proposes a fuzzy feature representation approach to represent occupancy sensor data, along with a Convolutional Neural Network Classifier (CNNC) for human activity modelling and recognition. Sensory data gathered from a home environment are converted into occupancy data representing human activities and then fuzzified before being fed as inputs into the CNNC. The learning capability of CNNC allows the model to learn the relationship between the fuzzified inputs and their corresponding output activities during training mode. The relations learned in the trained CNNC model are then used to identify human activity patterns and classify these when the testing dataset is applied. The proposed method is evaluated using a dataset representing activities of daily living for a single user gathered from a real-home environment.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

Recently, a significant amount of literature concerning machine learning techniques has focused on automatic recognition of activities performed by people. The main reason for this considerable interest is the increasing availability of devices able to acquire signals which, if properly processed, can provide information about human activities of daily living (ADL). The recognition of human activities is generally performed by machine learning techniques that process signals from wearable sensors and/or cameras appropriately arranged in the environment. Whatever the type of sensor, activities performed by human beings have a strong subjective characteristic that is related to different factors, such as age, gender, weight, height, physical abilities, and lifestyle. Personalization models have been studied to take into account these subjective factors and it has been demonstrated that using these models, the accuracy of machine learning algorithms can be improved. In this work we focus on the recognition of human activities using signals acquired by the accelerometer embedded in a smartphone. The contributions of this research are mainly three. A first contribution is the definition of a clear validation model that takes into account the problem of personalization and which thus makes it possible to objectively evaluate the performances of machine learning algorithms. A second contribution is the evaluation, on three different public datasets, of a personalization model which considers two aspects: the similarity between people related to physical aspects (age, weight, and height) and similarity related to intrinsic characteristics of the signals produced by these people when performing activities. A third and last contribution is the development of a personalization model that considers both the physical and signal similarities. The experiments show that the employment of personalization models improves, on average, the accuracy, thus confirming the soundness of the approach and paving the way for future investigations on this topic.

Smartphones are increasing ubiquitously due to the need and demand in the modern era. The world is transmuting into a global village with the cumulation of smart devices. Nowadays, smartphones are enriched with inertial sensors which can be used to recognize physical human activities in the wild. Human activity recognition (HAR) lies at the core of many applications like health monitoring, fall detection, road safety, personal assistance, and behavior-based context awareness. Context-based HAR is a new dimension that provides fine-grained information about the action being performed and leads us towards the automated and intelligent system design, which is useful to furnish the smart solution to real-life problems. In this paper, smartphone sensors are utilized to propose a framework for human activity and context recognition. This framework performs activity-dependent context recognition on the ExtraSensory dataset using random forest, decision tree, and k-nearest neighbor classifiers, and achieves an accuracy of 93.10%, demonstrating the effectiveness of the proposed framework.

With the rapid development of the computer and sensor field, inertial sensor data have been widely used in human activity recognition. At present, most relevant studies divide human activities into basic actions and transitional actions, in which basic actions are classified by unified features, while transitional actions usually use context information to determine the category. For the existing single method that cannot well realize human activity recognition, this paper proposes a human activity classification and recognition model based on smartphone inertial sensor data. The model fully considers the feature differences of different properties of actions, uses a fixed sliding window to segment the human activity data of inertial sensors with different attributes and, finally, extracts the features and recognizes them on different classifiers. The experimental results show that dynamic and transitional actions could obtain the best recognition performance on support vector machines, while static actions could obtain better classification effects on ensemble classifiers; as for feature selection, the frequency-domain feature used in dynamic action had a high recognition rate, up to 99.35%. When time-domain features were used for static and transitional actions, higher recognition rates were obtained, 98.40% and 91.98%, respectively.

Currently, identification of complex human activities is experiencing exponential growth through the use of deep learning algorithms. Conventional strategies for recognizing human activity generally rely on handcrafted characteristics from heuristic processes in time and frequency domains. The advancement of deep learning algorithms has addressed most of these issues by automatically extracting features from multimodal sensors to correctly classify human physical activity. This study proposed an attention-based bidirectional gated recurrent unit as Att-BiGRU to enhance recurrent neural networks. This deep learning model allowed flexible forwarding and reverse sequences to extract temporal-dependent characteristics for efficient complex activity recognition. The retrieved temporal characteristics were then used to exemplify essential information through an attention mechanism. A human activity recognition (HAR) methodology combined with our proposed model was evaluated using the publicly available datasets containing physical activity data collected by accelerometers and gyroscopes incorporated in a wristwatch. Simulation experiments showed that attention mechanisms significantly enhanced performance in recognizing complex human activity.

Human activity recognition is a very active research on pervasive computing and mobile health application. Many human activity systems based on inertial measurement unit (IMU) sensor data were proposed in the past few years. These systems mainly use IMU sensor placed on he torso and limbs to collect data and utilize supervised machine learning algorithms on sensor data. One main issue of these systems is that wearing multiple on-body IMU sensors may bring inconvenience to users' daily life. The other issue of these exiting methods is that an activity recognition model that is trained on a specific subject does not work well when being applied to predict another subject's activities since IMU activity data always carry information that is specific to the human subject who conducts the activities. In our work, inspired by the principle of domain adaption, we proposed a new deep-learning activity recognition model based on an adversarial network which can remove the subject-specific information within the IMU activity data and extract subject-independent features shared by the data collected on different subjects. We also for the first time use data collected from insole based IMU sensors on 8 participants for 5 common activities to build a new real world human activity dataset which can minimize the inconvenience for users to wear. We conducted experiments with our new real-world dataset. Results show that our subject independent activity recognition model outperforms state-of-art supervised learning techniques and eliminates the effects of individual differences between subjects successfully. The average recognition accuracy under the leave-one-out (L1O) condition achieves 99.0% which is higher than the performance of traditional human activity recognition system based on CNNs.

Human motion detection and activity recognition are becoming vital for the applications in smart homes. Traditional Human Activity Recognition (HAR) mechanisms use special devices to track human motions, such as cameras (vision-based) and various types of sensors (sensor-based). These mechanisms are applied in different applications, such as home security, Human–Computer Interaction (HCI), gaming, and healthcare. However, traditional HAR methods require heavy installation, and can only work under strict conditions. Recently, wireless signals have been utilized to track human motion and HAR in indoor environments. The motion of an object in the test environment causes fluctuations and changes in the Wi-Fi signal reflections at the receiver, which result in variations in received signals. These fluctuations can be used to track object (i.e., a human) motion in indoor environments. This phenomenon can be improved and leveraged in the future to improve the internet of things (IoT) and smart home devices. The main Wi-Fi sensing methods can be broadly categorized as Received Signal Strength Indicator (RSSI), Wi-Fi radar (by using Software Defined Radio (SDR)) and Channel State Information (CSI). CSI and RSSI can be considered as device-free mechanisms because they do not require cumbersome installation, whereas the Wi-Fi radar mechanism requires special devices (i.e., Universal Software Radio Peripheral (USRP)). Recent studies demonstrate that CSI outperforms RSSI in sensing accuracy due to its stability and rich information. This paper presents a comprehensive survey of recent advances in the CSI-based sensing mechanism and illustrates the drawbacks, discusses challenges, and presents some suggestions for the future of device-free sensing technology.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Emotions and activities are an integral part of human lifestyles and daily living. They occur all the time in everyone as humans are always in an emotional state and also performing an activity. This paper proposes an architecture that combines Human Emotion Recognition (HER) and Human Activity Recognition (HAR) to monitor people. It utilizes emotion physiological signals and accelerometer data harvested from individuals to recognize human emotions and activities respectively in addition to location information in order to remotely monitor such individuals and trigger alerts if necessary especially in a distress state situation that requires emergency services. The basic concepts of accelerometer data based HAR and physiological signals based HER and how they can be combined to monitor individuals in an emergency response system are succinctly presented.