Convolutional Neural Network Classifier with Fuzzy Feature Representation for Human Activity Modelling

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Human activity recognition and behavior monitoring in a home setting using wireless sensor networks (WSNs) provide a great potential for ambient assisted living (AAL) applications, ranging from health and wellbeing monitoring to resource consumption monitoring. However, due to the limitations of the sensor devices, challenges in wireless communication and the challenges in processing large amounts of sensor data in order to recognize complex human activities, WSN-based AAL systems are not effectively integrated in the home environment. Additionally, given the variety of sensor types and activities, selecting the most suitable set of sensors in the deployment is an important task. In order to investigate and propose solutions to such challenges, we introduce a WSN-based multimodal AAL system compatible for homes with multiple residents. Particularly, we focus on the details of the system architecture, including the challenges of sensor selection, deployment, networking and data collection and provide guidelines for the design and deployment of an effective AAL system. We also present the details of the field study we conducted, using the systems deployed in two different real home environments with multiple residents. With these systems, we are able to collect ambient sensor data from multiple homes. This data can be used to assess the wellbeing of the residents and identify deviations from everyday routines, which may be indicators of health problems. Finally, in order to elaborate on the possible applications of the proposed AAL system and to exemplify directions for processing the collected data, we provide the results of several human activity inference experiments, along with examples on how such results could be interpreted. We believe that the experiences shared in this work will contribute towards accelerating the acceptance of WSN-based AAL systems in the home setting.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.

With the rapid development of the computer and sensor field, inertial sensor data have been widely used in human activity recognition. At present, most relevant studies divide human activities into basic actions and transitional actions, in which basic actions are classified by unified features, while transitional actions usually use context information to determine the category. For the existing single method that cannot well realize human activity recognition, this paper proposes a human activity classification and recognition model based on smartphone inertial sensor data. The model fully considers the feature differences of different properties of actions, uses a fixed sliding window to segment the human activity data of inertial sensors with different attributes and, finally, extracts the features and recognizes them on different classifiers. The experimental results show that dynamic and transitional actions could obtain the best recognition performance on support vector machines, while static actions could obtain better classification effects on ensemble classifiers; as for feature selection, the frequency-domain feature used in dynamic action had a high recognition rate, up to 99.35%. When time-domain features were used for static and transitional actions, higher recognition rates were obtained, 98.40% and 91.98%, respectively.

Human Activity Recognition is the process of identifying the activity of a person by analyzing continuous frames of a video. In many application areas, human activity identification is either a direct goal or it is a key segment of a bigger objective. Some of the examples are surveillance system, elder healthcare monitoring system, abnormal activity detection systems such as fight detection, theft detection etc. Robust and accurate activity recognition is a challenging task due to diverse reasons, such as changing ambient illumination, noise, background turbulence, camera placements etc. Existing literatures discuss some techniques for identifying human activity but these approaches are restricted to the case of videos recorded from static camera. The aim of the proposed approach is to fill this gap. In this proposed method, a new skeleton based feature for human activity recognition- “Orientation Invariant Skeleton Feature (OISF)”- is introduced and used to train Random Forest (RF) classifier for Human Activity Recognition. Efficiency of newly introduced feature OISF is analyzed for the videos recorded with multiple cameras positioned at two different slant angles. Experimental results reveal that the newly introduced feature OISF has minimal dependency on variations of camera orientation. Accuracy achieved is ≈ 99.30% with ViHASi dataset, ≈ 96.85% with KTH dataset and ≈ 98.34% with in-house dataset which is higher than those achieved by other researches with existing features. The improved result of human activity recognition in terms of accuracy proves the appropriateness of the proposed research in being used commercially.

Encoding human activities using multimodal wearable sensory data

The enormous applicability of advanced human-computer interaction technology in daily life has facilitated the interest of the researchers toward the development of more intelligent autonomous systems. These human-computer interaction systems can be successful in real-life by addressing the gap among the existing techniques. This research work focuses on one of the prominent human-computer interaction applications of human activity recognition. Human activity recognition (HAR) is the process to detect human gestures, actions, and different types of interactions. The HAR process requires competent knowledge about the day to day human activities and advanced technology to recognize their activities. The conventional pattern recognition techniques are significant to recognize the human activities using the machine learning techniques but only within the controlled environment for the recognition of limited actions. In recent years, deep learning techniques are developed that can learn the deep attributes of the problem application and determine the outcomes with promising performance. The present paper has presented a systematic review of the deep learning models for video-based human activity recognition. The work describes the recent developments in the field for the analysis of different models. The paper also discusses the process of human activity recognition and the eminent datasets available for experimentation. The summarization of the work is illustrated with the future directions in the field.KeywordsDeep learningHuman computer interactionHuman activity recognitionVision based human activity recognitionVideo surveillanceVideo dataset

Human activity recognition is a very active research on pervasive computing and mobile health application. Many human activity systems based on inertial measurement unit (IMU) sensor data were proposed in the past few years. These systems mainly use IMU sensor placed on he torso and limbs to collect data and utilize supervised machine learning algorithms on sensor data. One main issue of these systems is that wearing multiple on-body IMU sensors may bring inconvenience to users' daily life. The other issue of these exiting methods is that an activity recognition model that is trained on a specific subject does not work well when being applied to predict another subject's activities since IMU activity data always carry information that is specific to the human subject who conducts the activities. In our work, inspired by the principle of domain adaption, we proposed a new deep-learning activity recognition model based on an adversarial network which can remove the subject-specific information within the IMU activity data and extract subject-independent features shared by the data collected on different subjects. We also for the first time use data collected from insole based IMU sensors on 8 participants for 5 common activities to build a new real world human activity dataset which can minimize the inconvenience for users to wear. We conducted experiments with our new real-world dataset. Results show that our subject independent activity recognition model outperforms state-of-art supervised learning techniques and eliminates the effects of individual differences between subjects successfully. The average recognition accuracy under the leave-one-out (L1O) condition achieves 99.0% which is higher than the performance of traditional human activity recognition system based on CNNs.

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

Smartphones are increasing ubiquitously due to the need and demand in the modern era. The world is transmuting into a global village with the cumulation of smart devices. Nowadays, smartphones are enriched with inertial sensors which can be used to recognize physical human activities in the wild. Human activity recognition (HAR) lies at the core of many applications like health monitoring, fall detection, road safety, personal assistance, and behavior-based context awareness. Context-based HAR is a new dimension that provides fine-grained information about the action being performed and leads us towards the automated and intelligent system design, which is useful to furnish the smart solution to real-life problems. In this paper, smartphone sensors are utilized to propose a framework for human activity and context recognition. This framework performs activity-dependent context recognition on the ExtraSensory dataset using random forest, decision tree, and k-nearest neighbor classifiers, and achieves an accuracy of 93.10%, demonstrating the effectiveness of the proposed framework.

Human motion detection and activity recognition are becoming vital for the applications in smart homes. Traditional Human Activity Recognition (HAR) mechanisms use special devices to track human motions, such as cameras (vision-based) and various types of sensors (sensor-based). These mechanisms are applied in different applications, such as home security, Human–Computer Interaction (HCI), gaming, and healthcare. However, traditional HAR methods require heavy installation, and can only work under strict conditions. Recently, wireless signals have been utilized to track human motion and HAR in indoor environments. The motion of an object in the test environment causes fluctuations and changes in the Wi-Fi signal reflections at the receiver, which result in variations in received signals. These fluctuations can be used to track object (i.e., a human) motion in indoor environments. This phenomenon can be improved and leveraged in the future to improve the internet of things (IoT) and smart home devices. The main Wi-Fi sensing methods can be broadly categorized as Received Signal Strength Indicator (RSSI), Wi-Fi radar (by using Software Defined Radio (SDR)) and Channel State Information (CSI). CSI and RSSI can be considered as device-free mechanisms because they do not require cumbersome installation, whereas the Wi-Fi radar mechanism requires special devices (i.e., Universal Software Radio Peripheral (USRP)). Recent studies demonstrate that CSI outperforms RSSI in sensing accuracy due to its stability and rich information. This paper presents a comprehensive survey of recent advances in the CSI-based sensing mechanism and illustrates the drawbacks, discusses challenges, and presents some suggestions for the future of device-free sensing technology.

This paper presents a method based on feature selection to obtain sets of Human Activity Recognizers of different complexity . Classifiers for human activity recognition are built exploring a space of candidate feature subsets, trying to maximize the accuracy of a classifier trained with them. At the same time, the size of the selected feature subset is minimized. The accuracy of a classifier tends to grow with the number of features, but in a real time task, like Human Activity Recognition, the number of features used has to be minimized, because its growing involves a slower processing rate. A set of solutions with different trade-offs between accuracy and number of features may be achieved modeling the problem of feature selection using Multiobjective Optimization (MO), where both measures are optimized at the same time. To solve the MO problem, Multiobjective Optimization Evolutionary Algorithms (MOEA) are going to be used. MOEA methods based on Pareto dominance not only find an optimal solution for the problem, they find a set of different optimal solutions so called Pareto-optimal set. A set of activity recognizers of different complexities is found using this approach. Having a set of different solutions allows the designer to choose the one that best fits its requirements. The method will be applied using a Hidden Markov Model as classifier. Results of the use of the method for recognizing different instantaneous human activities are discussed.

In recent years, it has been observed that many researchers have been working on different areas of detection, recognition and monitoring of human activities. The automatic determination of human physical activities is often referred to as human activity recognition (HAR). One of the most important technology that detects and tracks the activity of the human body is sensor-based HAR technology. In recent days, sensor-based HAR attracts attention in the field of computers due to its wide use in daily life and is a rapidly growing field of research. Activity recognition (AR) application is carried out by evaluating the signals obtained from various sensors placed in the human body. In this study, a new approach is proposed to extract features from sensor signals using HAR. The proposed approach is inspired by the Gray Level Co-Occurrence Matrix (GLCM) method, which is widely used in image processing, but it is applied to one-dimensional signals, unlike GLCM. Two datasets were used to test the proposed approach. The datasets were created from the signals obtained from the accelerometer, gyro and magnetometer sensors. Heralick features were obtained from co-occurrence matrix created after 1D-GLCM (One (1) Dimensional-Gray Level Co-Occurrence Matrix) was applied to the signals. HAR operation has been carried out for different scenarios using these features. Success rates of 96.66 and 93.88% were obtained for two datasets, respectively. It has been observed that the new approach proposed within the scope of the study provides high success rates for HAR applications. It is thought that the proposed approach can be used in the classification of different signals.

Human activity recognition (HAR) is used to support older adults to live independently in their own homes. Once activities of daily living (ADL) are recognised, gathered information will be used to identify abnormalities in comparison with the routine activities. Ambient sensors, including occupancy sensors and door entry sensors, are often used to monitor and identify different activities. Most of the current research in HAR focuses on a single-occupant environment when only one person is monitored, and their activities are categorised. The assumption that home environments are occupied by one person all the time is often not true. It is common for a resident to receive visits from family members or health care workers, representing a multi-occupancy environment. Entropy analysis is an established method for irregularity detection in many applications; however, it has been rarely applied in the context of ADL and HAR. In this paper, a novel method based on different entropy measures, including Shannon Entropy, Permutation Entropy, and Multiscale-Permutation Entropy, is employed to investigate the effectiveness of these entropy measures in identifying visitors in a home environment. This research aims to investigate whether entropy measures can be utilised to identify a visitor in a home environment, solely based on the information collected from motion detectors [e.g., passive infra-red] and door entry sensors. The entropy measures are tested and evaluated based on a dataset gathered from a real home environment. Experimental results are presented to show the effectiveness of entropy measures to identify visitors and the time of their visits without the need for employing extra wearable sensors to tag the visitors. The results obtained from the experiments show that the proposed entropy measures could be used to detect and identify a visitor in a home environment with a high degree of accuracy.

Emotions and activities are an integral part of human lifestyles and daily living. They occur all the time in everyone as humans are always in an emotional state and also performing an activity. This paper proposes an architecture that combines Human Emotion Recognition (HER) and Human Activity Recognition (HAR) to monitor people. It utilizes emotion physiological signals and accelerometer data harvested from individuals to recognize human emotions and activities respectively in addition to location information in order to remotely monitor such individuals and trigger alerts if necessary especially in a distress state situation that requires emergency services. The basic concepts of accelerometer data based HAR and physiological signals based HER and how they can be combined to monitor individuals in an emergency response system are succinctly presented.