3D human action recognition using Gaussian processes dynamical models

The discriminative power of modern deep learning models for 3D human action recognition is growing ever so potent. In conjunction with the recent resurgence of 3D human action representation with 3D skeletons, the quality and the pace of recent progress have been significant. However, the inner workings of state-of-the-art learning based methods in 3D human action recognition still remain mostly black-box. In this work, we propose to use a new class of models known as Temporal Convolutional Neural Networks (TCN) for 3D human action recognition. TCN provides us a way to explicitly learn readily interpretable spatio-temporal representations for 3D human action recognition. Through this work, we wish to take a step towards a spatio-temporal model that is easier to understand, explain and interpret. The resulting model, Res-TCN, achieves state-of-the-art results on the largest 3D human action recognition dataset, NTU-RGBD.

This doctoral thesis presents a framework for machine based interpretation of Malaysian Sign Language (MSL). The goal is to explore and develop necessary core algorithms that are capable of recognizing hand signs, based on the monocular/2D visual input of a low-cost hardware (Digital Webcam), without the assistance of visual-markers on hands. Successful hand sign interpretation with the aforementioned constrained visual system is challenging. Economical hardware devices often provide noisy/cluttered raw data with limited useful information. Non-involvement of visual markers imposes the requirement of additional strategies to disambiguate occlusions and background complexities. Natural variations in hand posture/gesture performances further convolute the recognition task. Addressing these challenges require computationally heavy algorithms in image processing, machine learning and pattern recognition. Thus vision-based sign language translator frameworks have to be multifaceted and versatile, in order to address innumerable practical concerns. In this research, hand postures interpretation is achieved by statistical projection methods. Analysis of high dimensional hand shape data is a complex task. Thus reducing dimensions via known projection methods aids in alleviating some of these complexities. Locality Preserving Projection (LPP) has been employed to accomplish this goal, while Principal Component Analysis (PCA) has been used as the benchmark method. Both methods were extensively tested against a hand posture database of 24 alphabet signs from MSL. As for the hand gesture interpretation, parameter-less Gaussian process based machine learning methods are proposed, as an alternative to current status quo. A variation of the Gaussian latent variable model – Gaussian Process Dynamic Model (GPDM) – has been employed for hand gesture interpretation. This has been comparatively studied against the established Hidden Markov Model (HMM). These machine learning methods were thoroughly tested against a hand gesture database of 64 signs from MSL. Grammatical sentence construction offers both intelligible sentences, and a mean of identifying erroneous sign classifications. In this thesis, a proof-of-concept on: use of Context Free Grammar (CFG) in successful parsing of meaningful sentences based on recognised gesture words, and how it can be extended in identifying (and correcting) erroneous sign classifications, is demonstrated with 92 sentences and the output of GPDM gesture classification. The novelty and contribution of this research: this is the first undertaking of markerless vision-based translation for MSL. Both LPP and GPDM are introduced to the sign language research field as an outcome of this investigation. The preliminary study and demonstration of grammatical sentence construction for translated gesture signs, is the first of its kind to the best of author's knowledge.

Human action recognition based on RGB-D sequences is an important research direction in the field of computer vision. In this work, we incorporate the skeleton on the Grassmann manifold in order to model the human action as a trajectory. Given the couple of matched points on the Grassmann manifold, we introduce the special orthogonal group SO(3) to exploit the rotation ignored by the Grassmann manifold. In fact, our objective is to define the best weighted linear combination between distances in Grassmann and SO(3) manifolds according to the nature of action, while modeling human actions by temporal trajectories and finding the best weighted combination. The effectiveness of combining the two non-Euclidean spaces was validated on three standard challenging 3D human action recognition datasets (G3D-Gaming, UTD-MHAD multimodal action and Florence3D-Action), and the preliminary results confirm the accuracy of the proposed method comparatively to relevant methods from the state of the art.

A new method of Gaussian process dynamic model (GPDM), named class-aware shared GPDM (CSGPDM), is presented in this paper. One of the most difference between our CSGPDM and existing GPDM is considering class information which helps to build the class label-based latent space being effective for the following class-related tasks. In terms of representation learning, CSGPDM is optimized by considering not only a non-linear relationship but also time-series relation and discriminative information of each class label. Then CSGPDM can reflect the following three points to the estimated latent space: i) the relationship between heterogeneous input sets, ii) time-series relations lurked in each input data, and iii) class information. Therefore, when input heterogeneous sets of features have time-series relations and class information, the above CSGPDM-based latent space can be beneficial for the obtaining the new CSGPDM-based feature sets for the post classification and estimating one side of the lacking samples by bridging the input heterogeneous feature sets via the latent space. Experimental results show that the estimated CSGPDM-based latent space outperformed those of GPDM and shared GPDM (SGPDM).

We present an alternative approach to generate gait motion at arbitrary speed for gait rehabilitation robots. The methodology utilizes Gaussian process dynamical model (GPDM), which is a nonlinear dimensionality reduction technique. GPDM consists of a dynamics in low-dimensional latent space and a mapping from the space to configuration space, and GPDM learning results in the low-dimensional representation of training data and parameters for the dynamics and mapping. We use second-order Markov process dynamics model, and hence given a pair of initial points, the dynamics generates a latent trajectory at arbitrary speed. We use linear regression to obtain the initial points. Mapping from the latent to configuration spaces constructs trajectories of walking motion. We verify the algorithm with motion capture data from 50 healthy subjects, who walked on a treadmill at 1, 2, and 3 km/h. We show examples and compare the original and interpolated trajectories to prove the efficacy of the algorithm.

Nonlinear dynamic modeling and model reduction strategy are studied in this paper for a rotating thin cylindrical shell. The nonlinear dynamic model is first established in terms of ordinary differential equations, in which the effects of Coriolis and centrifugal forces are considered, as well as the initial hoop tension due to rotation. This model describes both the in‐plane vibrations and the flexural vibration and reflects the coupling effects of those deformations. Based on this original model, a novel model reduction strategy is proposed to reduce the degrees of freedom by neglecting vibration modes predominated by in‐plane vibrations. Meanwhile, for the reduced‐order model, the in‐plane vibrations’ contributions to the rotating shell’s response are still preserved. To validate the dynamic model and the model reduction strategy, comparisons and simulations are carried out. Subsequently, nonlinear dynamic behaviors are investigated preliminarily by analyzing the rotating cylindrical shell’s amplitude‐frequency responses under different excitation levels.

Human action recognition is challenging, due to large temporal and spatial variations in actions performed by humans. These variations include significant nonlinear temporal stretching. In this paper, we propose an intuitively simple method to extract action templates from 3D human joint data that is insensitive to nonlinear stretching. The extracted action templates are used as the training instances of the actions to train multiple classifiers including a multi-class SVM classifier. Given an unknown action, we first extract and classify all its constituent atomic actions and then assign the action label via an equal voting scheme. We have tested the method on two public datasets that contain 3D human skeleton data. The experimental results show the proposed method can obtain a comparable or better performance than published state-of-the-art methods. Additional experiments also demonstrate the method works robustly on randomly stretched actions.

: This report summarizes the major findings from our research on a semantic information representation framework (SIRF) for visual sensing scenarios. First, the concept and architecture of a cognitive linguistic (CL) based SIRF is introduced. Two levels of information abstraction are proposed within this framework. At the syntactic level, a probabilistic contest free grammar (PCFG) method is employed for information compression and summarization. At the semantic level, a Bayesian network approach is used to achieve semantic concept inference and reasoning. To facilitate the functions of this SIRF, several conceptual primitive modeling methods are proposed, which include a dynamic structure preserving map (DSPM) for individual human action recognition, a Gaussian Process Dynamic Model with Social Network Analysis (GPDM-SNA) for a small human group action recognition, an extended GPDM-SNA method for human object interaction (HOI) recognition, and a pyramid histogram of gradient (pHOG) method for human object recognition based on gait images. In addition to these conceptual primitive models, two quantities sensing modality utility assessment methods are introduced. They are essentially feature selection methods, one is based sparse imputation and one is based on 11 graph. Extensive experiments on publicly available datasets have been conducted to assess the effectiveness of the proposed methods, and highly competitive and promising results have been observed.

This paper addresses a learning-based human action recognition system from multiple images based on integrating features of segmented 3D human body parts such as face, torso, and limbs. The innovation of our proposed 3D human action recognition system consists of three parts: (1) 3D reconstruction of the target object by tracking the position of a target object in a scene to voxelize the accurate 3D human model, (2) Human body model segmentation into several human body parts using ellipsoidal models in the space of second-order three dimensional diffusion tensor fields, and (3) Classification and recognition of human actions from features of the segmented human model using Multiple-Kernel based Support Vector Machine. Experimental results on a set of test volume data show that our proposed method is very efficient to visualize and recognize the human action using few parameters which are independent to partial occlusion, dimension, and viewpoint.

Combustion instability analysis in an ethylene-fueled scramjet combustor under various fuel penetration height conditions using an image-based nonlinear dimensionality reduction method

Most 3D skeleton feature-based human action recognition methods are sensitive to changes in viewpoints, motion scales, and human scales. In addition, acquiring depth information from a real scene in outdoor environments results in poor precision or high computational costs. To address these drawbacks, in this study, we propose a new RGB video and 2D skeleton-based action recognition method including local joint trajectory volume representation and feature coding. First, a video is transferred to a set of volumes, which are called the local joint trajectory volumes. Then, hand-crafted and convolutional networks are used to calculate the features of each volume-based RGB image sequence. Different from most works that use convolutional networks to learn global video features, in this paper, the problem of using a convolutional network to represent local video regions is discussed. Finally, the feature set of each joint is encoded into the Fisher vector as an action feature. The classifier is trained by a linear SVM. The experimental results show that skeleton joint-based features result in a more compact and effective action representation approach than other approaches.

Accurate objective, automated limb lameness detection and pose estimation play an important role for animal well-being and precision livestock farming. We present a wearable sensor-based limb lameness detection and pose estimation for horse walk and trot locomotion. The gait event and lameness detection are first built on a recurrent neural network (RNN) with long short-term memory (LSTM) cells. Its outcomes are used in the limb pose estimation. A learned low-dimensional motion manifold is parameterized by a phase variable with a Gaussian process dynamic model. We compare the RNN-LSTM-based lameness detection method with a feature-based multi-layer classifier (MLC) and a multi-class classifier (MCC) that are built on support vector machine/K-nearest-neighbors and deep convolutional neural network methods, respectively. Experimental results show that using only accelerometer measurements, the RNN-LSTM-based approach achieves 95% lameness detection accuracy and also outperforms the feature-based MLC or MCC in terms of several assessment criteria. The pose estimation scheme can predict the 24 limb joint angles in the sagittal plane with average errors less than 5 and 10 degs under normal and induced lameness conditions, respectively. The presented work demonstrate the successful use of machine learning techniques for high performance lameness detection and pose estimation in equine science. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Automation technologies are increasingly used for precision agriculture but few have focused on monitoring individual animals in open field for precision livestock farming. Limb lameness detection and pose estimation in open field is labor-intensive, unsafe for farmers, and inefficient. The presented machine learning-enabled, wearable inertial sensor-based design provides an effective and efficient approach for horse limb lameness detection and pose estimation applications. We present an RNN-LSTM for lameness detection and an integrated manifold learning model is used to predict the horse limb joint angles in walk and trot gaits under normal and induced lameness conditions. We also present a systematic analysis and experiments to demonstrate the impacts of the wearable sensor locations and signal information on lameness detection and pose estimation performance. Several other machine learning-based lameness detection methods are also presented and compared. The extensive multi-horse testing results are presented to demonstrate the superior accuracy and higher performance than other types of machine learning methods. One attractive feature of the proposed design lies in its high performance and fast computational capability for potential real-time applications in open field.

Real-time gait detection and pose estimation are critical for safety monitoring and prevention of work-related musculoskeletal disorders for construction workers. We present a single wearable inertial measurement unit (IMU)-based gait detection and pose estimation for human walking on flat and sloped surfaces. The gait detection algorithm is built on a recurrent neural network-based method and its outcome is then used in the full-body pose estimation. The detection scheme also predicts the terrain slope information in real-time. The pose estimation is obtained through learned motion manifold in latent space with the Gaussian process dynamic model. Extensive experiments of different walking patterns and speeds on the level and sloped surfaces are conducted to validate and demonstrate the design. The proposed algorithm can detect gait activities with 96% accuracy, the estimated human pose errors are within 8.30 degs, and the detection latency is within 18.6 ms using only a single IMU attached to a human shank.

In this paper, a dynamic subgrid scale (SGS) stress model is proposed by modifying the existing quadratic nonlinear model. Unlike the conventional eddy viscosity models, the nonlinear model suggests an anisotropic tensorial polynomial relation between the SGS stress and the resolved strain rate tensor. Examined in the rotating turbulent channel flow, the previous nonlinear model is found unable to successfully predict the turbulent kinetic energy and the Reynolds shear stress. In the previous nonlinear model, an excessive backward energy transfer from the SGS to the resolved scale is found, so it is removed in the new model to provide sufficient SGS dissipation. The dynamic method is reconsidered, based on the analysis of the transport equation of the resolved Reynolds shear stress. The new dynamic nonlinear model is examined in the rotating turbulent channel flow at Re = 7000 and various rotation numbers (Ro), ranging from 0.0 to 0.6. The new dynamic procedure determines a more proper model coefficient for the nonlinear term, so the modified nonlinear model predicts superior results to the previous nonlinear models. The Vreman's approach of the eddy viscosity is implemented into the nonlinear model, to compare with the conventional dynamic Smagorinsky type nonlinear model in the rotating turbulent channel flow.

With the popularization of the Kinect sensor, human actions can be recognized based on the 3D skeletal information. In this paper, the Spatial-Temporal Moving Skeleton Descriptor (STMSD) is proposed by the fusion of three complementary features which are the Relative Geometric Velocity (RGV) between body parts, Relative Joint Positions (RJP), and Joint Angles (JA). The STMSD descriptor gives a complete view of the body skeleton in space and time. Among the three features, the Relative Geometric Velocity (RGV) is first proposed in our work. Inspired by the relative geometry using the Lie group and the Lie algebra, RGV describes the variation rates of body transformations which include 3D rotations and translations. Then interpolation and normalization are applied in frame descriptors. After the temporal modeling, Principal Component Analysis (PCA) is utilized. Experimental results on three datasets show that our approach performs better than existing action recognition approaches, including skeleton-based and other types.