Recognizing Gaits on Spatio-Temporal Feature Domain

Understanding highly accurate and real-time violent actions from surveillance videos is a demanding challenge. Our primary contribution of this work is divided into two parts. Firstly, we propose a computationally efficient Bag-of-Words (BoW) pipeline along with improved accuracy of violent videos classification. The novel pipeline's feature extraction stage is implemented with densely sampled Histogram of Oriented Gradients (HOG) and Histogram of Optical Flow (HOF) descriptors rather than Space-Time Interest Point (STIP) based extraction. Secondly, in encoding stage, we propose Outlier-Resistant VLAD (OR-VLAD), a novel higher order statistics-based feature encoding, to improve the original VLAD performance. In classification, efficient Linear Support Vector Machine (LSVM) is employed. The performance of the proposed pipeline is evaluated with three popular violent action datasets. On comparison, our pipeline achieved near perfect classification accuracies over three standard video datasets, outperforming most state-of-the-art approaches and having very low number of vocabulary size compared to previous BoW Models.

It is well recognized that gait is an important biometric feature to identify a person at a distance, e.g., in video surveillance application. However, in reality, change of viewing angle causes significant challenge for gait recognition. A novel approach using regression-based view transformation model (VTM) is proposed to address this challenge. Gait features from across views can be normalized into a common view using learned VTM(s). In principle, a VTM is used to transform gait feature from one viewing angle (source) into another viewing angle (target). It consists of multiple regression processes to explore correlated walking motions, which are encoded in gait features, between source and target views. In the learning processes, sparse regression based on the elastic net is adopted as the regression function, which is free from the problem of overfitting and results in more stable regression models for VTM construction. Based on widely adopted gait database, experimental results show that the proposed method significantly improves upon existing VTM-based methods and outperforms most other baseline methods reported in the literature. Several practical scenarios of applying the proposed method for gait recognition under various views are also discussed in this paper.

It is well recognized that gait is an important biometric feature to identify a person at a distance, such as in video surveillance application. However, in reality, a change of viewing angle causes a significant challenge for gait recognition. In this paper, a novel approach is proposed for multiview gait recognition with the view angle of a probe gait sequence unknown. We formulate a new patch distribution feature based classification framework to estimate the view angle of each probe gait sequence. In this method, each gait energy image is represented as a set of dual-tree complex wavelet transform (DTCWT) features derived from different scales and orientations together with the x-y coordinates. Then, a two-stage Gaussian mixture model is presented that can represent each DTCWT based gait feature with a set of patch distribution parameters. A simple nearest-neighbor classifier is employed for view classification. To measure the similarity of gait sequences, we also propose a sparse local discriminant canonical correlation analysis algorithm to model the correlation of gait features from different views and use the correlation strength as similarity measure. An uncorrelated multilinear SLDCCA (UMSLDCCA) framework is further presented that aims to extract uncorrelated discriminative features directly from multidimensional gait features through solving a tensor-to-vector projection. The solution consists of sequential iterative processes based on the alternating projection method. Different from existing approaches, UMSLDCCA considers the spatial structure information within each gait sample and local geometry information among multiple gait samples. Moreover, our approach does not need explicit reconstruction and is robust against feature noise. Extensive experiments have been performed on two benchmark gait databases. The results demonstrate that our method outperforms the state-of-the-art methods in terms of accuracy and efficiency.

We propose a new gait feature extraction method for gait recognition that uses the dyadic wavelet transform (DYWT) and structural similarity (SSIM). Gait recognition is a person authentication technique that uses the gait features obtained from a monitoring camera video. Most existing gait recognition methods use gait features based on the silhouette image sequence. However, the gait recognition accuracy decreases if the resolution of the silhouette image is low. We developed a new method to extract gait features from silhouette images having a low-resolution gait period sequence that uses the DYWT and SSIM. For our experiment, we prepared two types of silhouette image sequences of 100 subjects for use as the probe and gallery images, respectively. The sizes of the silhouette images were 64 × 44, 32 × 22, and 16 × 11. We describe our proposed method in detail and present our experimental results demonstrating that the extracted gait features are effective.

Location-based services and identification hold promise for future smart home applications. Through them, we can provide customized services for specific users in current locations. Recent studies have demonstrated that Wi-Fi signals can be leveraged to achieve device-free tracking and gait recognition. Despite their good performance, these two technologies are not effectively integrated for the following reasons: First, the device-free tracking method might yield tracking results that conflict with human gait. Second, extracting gait features relies on knowing or accurately estimating the user's trajectory. Consequently, gait recognition and tracking are inherently linked, but there has been no effective approach to integrate these two techniques. In this paper, we present STAGR, a system capable of <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</u>imultaneous <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</u>racking <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</u>nd <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</u>ait <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">R</u>ecognition. The main contribution of our technique is that we establish a theoretical model that reveals how to transform path-dependent spectra into path-independent spectra directly. Specifically, we conduct a preliminary study to demonstrate the need for simultaneous tracking and gait recognition. Second, we propose a novel method to extract path-independent gait features, which can significantly save execution time compared with the learning-based method. Third, we design a polar-coordinate filtering method to retain the gait features while correcting the trajectory. We implement a prototype STAGR system and conduct extensive experiments to verify the proposed mechanism. The experimental results show that we can realize simultaneous tracking and gait recognition. The median tracking error is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ 0.45m$</tex-math></inline-formula>, while the recognition accuracy is 95.3% for 6 users.

Recognizing people by their gait has become more and more popular nowadays due to the following reasons. First, gait recognition can work well remotely. Second, gait recognition can be done from low-resolution videos and with simple instrumentation. Third, gait recognition can be done without the cooperation of individuals. Fourth, gait recognition can work well while other features such as faces and fingerprints are hidden. Finally, gait features are typically difficult to be impersonated.Recent ubiquity of smartphones that capture gait patterns through accelerometers and gyroscope and advances in machine learning have opened new research directions and applications in gait recognition. A timely survey that addresses current advances is missing.In this article, we survey research works in gait recognition. In addition to recognition based on video, we address new modalities, such as recognition based on floor sensors, radars, and accelerometers; new approaches that include machine learning methods; and examine challenges and vulnerabilities in this field. In addition, we propose a set of future research directions. Our review reveals the current state-of-art and can be helpful to both experts and newcomers of gait recognition. Moreover, it lists future works and publicly available databases in gait recognition for researchers.

View difference is a factor that degrades the accuracy of gait recognition. A solution to reducing accuracy degradation is applying a view transformation model (VTM) that encodes a joint subspace of multi-view gait features trained from multiple training subjects. In the VTM framework, once an intrinsic vector of a test subject in the joint subspace is estimated from a gait feature with a source view (e.g. probe view), a gait feature with a destination view (e.g. gallery view) is generated for the same-view matching. Although this family of methods can improve the total accuracy, the quality of generated gait features depends on a test gait feature, and may be relevant to the accuracy of gait recognition. We therefore propose a method of incorporating the quality measure of the generated gait feature into the VTM framework. We employ the projection error into the joint subspace as the quality measure. A posterior probability is then computed by incorporating the quality measure. The accuracy evaluation against a subset of a public database collected from 1,912 subjects shows that the proposed method further improves the accuracy.

Gait recognition has become a mainstream technology for identification, as it can recognize the identity of subjects from a distance without any cooperation. However, when subjects wear coats (CL) or backpacks (BG), their gait silhouette will be occluded, which will lose some gait information and bring great difficulties to the identification. Another important challenge in gait recognition is that the gait silhouette of the same subject captured by different camera angles varies greatly, which will cause the same subject to be misidentified as different individuals under different camera angles. In this article, we try to overcome these problems from three aspects: data augmentation, feature extraction, and feature refinement. Correspondingly, we propose gait sequence mixing (GSM), multigranularity feature extraction (MFE), and feature distance alignment (FDA). GSM is a method that belongs to data enhancement, which uses the gait sequences in NM to assist in learning the gait sequences in BG or CL, thus reducing the influence of lost gait information in abnormal gait sequences (BG or CL). MFE explores and fuses different granularity features of gait sequences from different scales, and it can learn as much useful information as possible from incomplete gait silhouettes. FDA refines the extracted gait features with the help of the distribution of gait features in real world and makes them more discriminative, thus reducing the influence of various camera angles. Extensive experiments demonstrate that our method has better results than some state-of-the-art methods on CASIA-B and mini-OUMVLP. We also embed the GSM module and FDA module into some state-of-the-art methods, and the recognition accuracy of these methods is greatly improved.

A new gait recognition method based on Zernike moments and BP neural network is proposed. Zernike moments are calculated to extract gait features based on the introduced concept of normalized gait cycle. All gait Zernike moments compose the gait feature space. PCA algorithm is used to compress Zernike moments and a new lower dimension feature space containing gait spatio-temporal features is generated. Each normalized gait cycle's Zernike moments are mapped to this new feature space and compose an eigen-matrix, whose row square error vectors are used as the gait recognition eigenvectors. BP neural network is used to classify the gait features. To increase recognition accuracy, multiple training samples and multiple inputs are used for each to be recognized gait class. Experimental results show that the method can obtain accurate gait recognition in relatively simple scenes.

Gait recognition is an emerging biometric technology that can be used to protect the privacy of wearable device owners. To improve the performance of the existing gait recognition method based on wearable devices and to reduce the memory size of the model and increase its robustness, a new identification method based on multimodal fusion of gait cycle data is proposed. In addition, to preserve the time-dependence and correlation of the data, we convert the time-series data into two-dimensional images using the Gramian angular field (GAF) algorithm. To address the problem of high model complexity in existing methods, we propose a lightweight double-channel depthwise separable convolutional neural network (DC-DSCNN) model for gait recognition for wearable devices. Specifically, the time series data of gait cycles and GAF images are first transferred to the upper and lower layers of the DC-DSCNN model. The gait features are then extracted with a three-layer depthwise separable convolutional neural network (DSCNN) module. Next, the extracted features are transferred to a softmax classifier to implement gait recognition. To evaluate the performance of the proposed method, the gait dataset of 24 subjects were collected. Experimental results show that the recognition accuracy of the DC-DSCNN algorithm is 99.58%, and the memory usage of the model is only 972 KB, which verifies that the proposed method can enable gait recognition for wearable devices with lower power consumption and higher real-time performance.

Background subtraction method is widely used in most of the video motion detection algorithms especially for video surveillance application. Background subtraction is used to extract moving or static foreground objects from the background scene. The efficiency of foreground-background segmentation heavily relies on background model which must be able to cope with changes in the scene and granularity of the foreground objects. The robust background model can produce good foreground segmentation results and it is still a great challenge to get accurate and high performance result today. In this paper, a video foreground-background segmentation approach is proposed. This approach is based on Codebook (CB) model with Kalman Filter. This approach can be used to extract foreground objects from the video stream. The Lab color space is used in this approach to calculate color difference between two pixels using CIEDE2000 color difference formula. Extracted foreground object from video sequence using this approach is useful for object detection in video surveillance applications. This approach has been tested with PETS and CDnet2014 datasets and segmentation results accuracy are evaluated compare with ground truth.

Rapidly development in the realm of biometrics in recent years results in the unprecedented growth with the number of researchers and interesters. As one of the biometrics, gait recognition has many advantages such as from a distance, lower quality video, hard to disguised comparing with others. However, nearly all studies on gait recognition are 2D methods based on analysis of image sequences captured by a monocular camera. This paper presents a novel 3D method for automatic gait recognition by analyzing the changes of a 1D silhouette signal. Binarized silhouette of a moving individual is firstly achieved. Secondly contour is extracted for next stereo matching. Afterward 3D contour is obtained through a step of contour based stereo matchin. Then, stereo silhouette vector (SSV) is extracted from 3D contour and transformed to a 1D silhouette signal as the stereo gait feature (SGF). K-L transform is applied for reducing the dimensionality of feature and the complexity of computing. Finally, kNN and NN are applied to gait classification and recognition. A stereo gait database named PRLAB is established for gait recognition based on stereo vision. Experimental results on PRLAB showed the efficiency and robustness of our algorithm.

Gait recognition is a distinctive biometric technique that can identify pedestrians by their walking patterns from considerable distances. A critical challenge in gait recognition lies in effectively acquiring discriminative spatial-temporal representations from silhouettes that exhibit invariance to disturbances. In this paper, we present a novel gait recognition network by aggregating features in the spatial-temporal and view domains, which consists of two-path spatial-temporal feature fusion module and view embedding module. Specifically, two-path spatial-temporal feature fusion module firstly utilizes multi-scale feature extraction (MSFE) to enrich the input features with multiple convolution kernels of various sizes. Then, frame-level spatial feature extraction (FLSFE) and multi-scale temporal feature extraction (MSTFE) are parallelly constructed to capture spatial and temporal gait features of different granularities and these features are fused together to obtain muti-scale spatial-temporal features. FLSFE is designed to extract both global and local gait features by employing a specially designed residual operation. Simultaneously, MSTFE is applied to adaptively interact multi-scale temporal features and produce suitable motion representations in temporal domain. Taking into account the view information, we introduce a view embedding module to reduce the impact of differing viewpoints. Through the extensive experimentation over CASIA-B and OU-MVLP datasets, the proposed method has achieved superior performance to the other state-of-the-art gait recognition approaches.

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a "dynamic" image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

A new approach for abnormal behavior detection was proposed using causality analysis and sparse reconstruction. To effectively represent multiple-object behavior, low level visual features and causality features were adopted. The low level visual features, which included trajectory shape descriptor, speeded up robust features and histograms of optical flow, were used to describe properties of individual behavior, and causality features obtained by causality analysis were introduced to depict the interaction information among a set of objects. In order to cope with feature noisy and uncertainty, a method for multiple-object anomaly detection was presented via a sparse reconstruction. The abnormality of the testing sample was decided by the sparse reconstruction cost from an atomically learned dictionary. Experiment results show the effectiveness of the proposed method in comparison with other state-of-the-art methods on the public databases for abnormal behavior detection.