Learning Spatio-Temporal Representations for Action Recognition: A Genetic Programming Approach.

Activity recognition from smartphone sensor inputs is of great importance to enhance user experience. Our study aims to investigate the applicability of Genetic Programming (GP) approach on this complex real world problem. Traditional methods often require substantial human efforts to define good features. Moreover the optimal features for one type of activity may not be suitable for another. In comparison, our GP approach does not require such feature extraction process, hence, more suitable for complex activities where good features are difficult to be pre-defined. To facilitate this study we therefore propose a benchmark of activity data collected from various smartphone sensors, as currently there is no existing publicly available database for activity recognition. In this study, a GP-based approach is applied to nine types of activity recognition tasks by directly taking raw data instead of features. The effectiveness of this approach can be seen by the promising results. In addition our benchmark data provides a platform for other machine learning algorithms to evaluate their performance on activity recognition.

Nonlinear model identification of an experimental ball-and-tube system using a genetic programming approach

<p>Symbolic regression (SR) is a function identification process, the task of which is to identify and express the relationship between the input and output variables in mathematical models. SR is named to emphasise its ability to find the structure and coefficients of the model simultaneously. Genetic Programming (GP) is an attractive and powerful technique for SR, since it does not require any predefined model and has a flexible representation. However, GP based SR generally has a poor generalisation ability which degrades its reliability and hampers its applications to science and real-world modeling. Therefore, this thesis aims to develop new GP approaches to SR that evolve/learn models exhibiting good generalisation ability. This thesis develops a novel feature selection method in GP for high-dimensional SR. Feature selection can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalisation ability. However, feature selection is seldom considered in GP for high-dimensional SR. The proposed new feature selection method utilises GP’s built-in feature selection ability and relies on permutation to detect the truly relevant features and discard irrelevant/noisy features. The results confirm the superiority of the proposed method over the other examined feature selection methods including random forests and decision trees on identifying the truly relevant features. Further analysis indicates that the models evolved by GP with the proposed feature selection method are more likely to contain only the truly relevant features and have better interpretability. To address the overfitting issue of GP when learning from a relatively small number of instances, this thesis proposes a new GP approach by incorporating structural risk minimisation (SRM), which is a framework to estimate the generalisation performance of models, into GP. The effectiveness of SRM highly depends on the accuracy of the Vapnik-Chervonenkis (VC) dimension measuring model complexity. This thesis significantly extends an experimental method (instead of theoretical estimation) to measure the VC-dimension of a mixture of linear and nonlinear regression models in GP for the first time. The experimental method has been conducted using uniform and non-uniform settings and provides reliable VC-dimension values. The results show that our methods have an impressively better generalisation gain and evolve more compact model, which have a much smaller behavioural difference from the target models than standard GP and GP with bootstrap, The proposed method using the optimised non-uniform setting further improves the one using the uniform setting. This thesis employs geometric semantic GP (GSGP) to tackle the unsatisfied generalisation performance of GP for SR when no overfitting occurs. It proposes three new angle-awareness driven geometric semantic operators (GSO) including selection, crossover and mutation to further explore the geometry of the semantic space to gain a greater generalisation improvement in GP for SR. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, to be resistant to overfitting. The results show that compared with two kinds of state-of-the-art GSOs, the proposed new GSOs not only drive the evolutionary process fitting the target semantics more efficiently but also significantly improve the generalisation performance. A further comparison on the evolved models shows that the new method generally produces simpler models with a much smaller size and containing important building blocks of the target models.</p>

An automatic human action representation and recognition technique is proposed in this paper. Appearance-change problem due to human wearing dresses and body shapes is also investigated in this study for automatic human action recognition. A tuned eigenspace technique is proposed for automatic human posture and/or motion recognition that successfully overcome the preceding problems. We employ image pre-processing by Gaussian and Sobel edge filter, called the first stage tuning, for reducing a dress effect, and a mean eigenspace produced by taking a mean of the similar postures, called the second stage tuning, for avoiding the preceding problems. An eigenspace called a tuned eigenspace is obtained from the mentioned processes and it is used for further recognition of unfamiliar postures and actions. The proposed method is compared with a related technique and the robustness of this approach is presented.

Skeleton-based human action recognition has been a hot topic in recent years. Most existing studies are based on the skeleton data obtained from Kinect, which is noisy and unstable, in particular, in the case of occlusions. To cope with the noisy skeleton data and variation of viewpoints, this paper presents a view-invariant method for human action recognition by recovering the corrupted skeletons based on a 3D bio-constrained skeleton model and visualizing those body-level motion features obtained during the recovery process with images. The bio-constrained skeleton model is defined with two types of constraints: 1) constant bone lengths and 2) motion limits of joints. Based on the bio-constrained model, an effective method is proposed for skeleton recovery. Two types of new motion features, the Euclidean distance matrix between joints (JEDM), which contains the global structure information of the body, and the local dynamic variation of the joint Euler angles (JEAs) are used in describing human action. These two types of features are encoded into different motion images, which are fed into a two-stream convolutional neural network for learning different action patterns. The experiments on three benchmark datasets achieve better accuracy than the state-of-the-art approaches, which demonstrates the effectiveness of the proposed method.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

Scene image and human skeleton-based dual-stream human action recognition

Applications like smart homes, security surveillance, healthcare, and workplace safety now depend heavily on Human Activity Recognition (HAR) and monitoring. Automated systems can respond quickly, stop dangerous situations, and provide individualized services when they can precisely identify and categorize human activity. The intricate temporal dependencies present in human motion are frequently difficult to capture by conventional HAR techniques that rely on hand-crafted features and shallow learning models [1]. This paper investigates the effectiveness of a pure Recurrent Neural Network (RNN) architecture for activity recognition using smartphone sensor data, despite the fact that deep learning models like CNNs and LSTMs have dominated recent HAR research. Without the need for manually created features, we suggest a simplified RNN-based framework that can identify temporal dependencies in unprocessed accelerometer and gyroscope data. Pure RNNs can attain competitive accuracy, as shown by experiments on the UCI HAR and WISDM datasets, which yielded results of 92.4% and 90.7%, respectively. The results imply that, when properly optimized, RNNs despite their simplicity remain feasible for HAR tasks.

Two different classifier representations based on dynamic Artificial Neural Networks (ANNs) and Genetic Programming (GP) are being compared on a human action recognition task by an ubiquitous mobile robot. The classification methodologies used, process time series generated by an indoor ubiquitous 3D tracker which generates spatial points based on 23 reflectable markers attached on a human body. This investigation focuses mainly on class discrimination of normal and aggressive action recognition performed by an architecture which implements an interconnection between an ubiquitous 3D sensory tracker system and a mobile robot to perceive, process, and classify physical human actions. The 3D tracker and the robot are used as a perception-to-action architecture to process physical activities generated by human subjects. Both classifiers process the activity time series to eventually generate surveillance assessment reports by generating evaluation statistics indicating the classification accuracy of the actions recognized.

A framework of human action recognition using length control features fusion and weighted entropy-variances based feature selection

This paper presents a novel nonparametric efficiency analysis technique based on the Genetic Programming (GP) in order to measure efficiency of Iran electric power plants. GP model was used to predict the output of power plants with respect to input data. The method, we presented here, is capable of finding a best performance among power plant based on the set of input data, GP predicted results and real outputs. The advantage of using GP over traditional statistical methods is that in prediction with GP, the researcher doesn’t need to assume the data characteristic of the dependent variable or output and the independent variable or input. In this proposed methodology to calculate the efficiency scores, a novel algorithm was introduced which worked on the basis of predicted and real output values. To validate our model, the results of proposed algorithm for calculating efficiency rank of power plants were compared to traditional method. Real data was presented for illustrative our proposed methodology. Results showed that by utilizing the capability of input-output pattern recognition of GP, this method provides more realistic results and outperform in identification of efficient units than the conventional methods.

Human action recognition from 2D videos is a demanding area due to its broad applications. Many methods have been proposed by the researchers for recognizing human actions. The improved accuracy in identifying human actions is desirable. This paper presents an improved method of human action recognition using support vector machine (SVM) classifier. This paper proposes a novel feature descriptor constructed by fusing the various investigated features. The handcrafted features such as scale invariant feature transform (SIFT) features, speed up robust features (SURF), histogram of oriented gradient (HOG) features and local binary pattern (LBP) features are obtained on online 2D action videos. The proposed method is tested on different action datasets having both static and dynamically varying backgrounds. The proposed method achieves shows best recognition rates on both static and dynamically varying backgrounds. The datasets considered for the experimentation are KTH, Weizmann, UCF101, UCF sports actions, MSR action and HMDB51.The performance of the proposed feature fusion model with SVM classifier is compared with the individual features with SVM. The fusion method showed best results. The efficiency of the classifier is also tested by comparing with the other state of the art classifiers such as k-nearest neighbors (KNN), artificial neural network (ANN) and Adaboost classifier. The method achieved an average of 94.41% recognition rate.

Transfer learning is a common solution to address cross-domain identification problems in Human Activity Recognition (HAR). Most existing approaches typically perform cross-subject transferring while ignoring transfers between different sensors or body parts, which limits the application scope of these models. Only a few approaches have been made to design a versatile HAR approach (cross-subject, cross-sensor and cross-body-part). Unfortunately, these existing approaches depend on complex handcrafted features and ignore the inequality of samples for positive transfer, which will hinder the transfer performance. In this paper, we propose a framework for versa-tile cross-domain activity recognition. Specifically, the proposed framework allows end-to-end implementation by exploiting adaptive features from activity image instead of extracting handcrafted features. And the framework uses a two-stage adaptation strategy consisting of pretraining stage and re-weighting stage to perform knowledge transfer. The pretraining stage ensures transferability of the source domain as well as separability of the target domain, and the re-weighting stage rebalances the contribution of the two domain samples. These two stages enhance the ability of knowledge transfer. We evaluate the performance of the proposed framework by conducting comprehensive experiments on three public HAR datasets (DSADS, OPPORTUNITY, and PAMAP2), and the experimental results demonstrate the effectiveness of our framework in versatile cross-domain HAR.

In this paper, we present a home-monitoring oriented human activity recognition benchmark database, based on the combination of a color video camera and a depth sensor. Our contributions are two-fold: 1) We have created a publicly releasable human activity video database (i.e., named as RGBD-HuDaAct), which contains synchronized color-depth video streams, for the task of human daily activity recognition. This database aims at encouraging more research efforts on human activity recognition based on multi-modality sensor combination (e.g., color plus depth). 2) Two multi-modality fusion schemes, which naturally combine color and depth information, have been developed from two state-of-the-art feature representation methods for action recognition, i.e., spatio-temporal interest points (STIPs) and motion history images (MHIs). These depth-extended feature representation methods are evaluated comprehensively and superior recognition performances over their uni-modality (e.g., color only) counterparts are demonstrated.

Human activity modeling and recognition using wearable sensors is important in pervasive healthcare, with applications including quantitative assessment of motor function, rehabilitation, and elder care. Previous human activity recognition techniques use a whole-motion model in which continuous sensor streams are divided into windows with a fixed time duration whose length is chosen such that all the relevant information in each activity signal can be extracted from each window. In this paper, we present a statistical motion primitive-based framework for human activity representation and recognition. Our framework is based on Bag-of-Features (BoF), which builds activity models using histograms of primitive symbols. We experimentally validate the effectiveness the BoF-based framework for recognizing nine activity classes and evaluate six factors which impact the performance of the framework. The factors include window size, choices of features, methods to construct motion primitives, motion vocabulary size, weighting schemes of motion primitive assignments, and learning machine kernel functions. Finally, we demonstrate that our statistical BoF-based framework can achieve much better performance compared to a non-statistical string-matching-based approach.