Human action recognition framework by fusing multiple features

Multiple depth-levels features fusion enhanced network for action recognition

Feature fusion methods have been demonstrated to be effective for many computer vision based applications. These methods generally use multiple hand-crafted features. However, in recent days, features extracted through transfer leaning procedures have been proved to be robust than the hand-crafted features in myriad applications, such as object classification and recognition. The transfer learning is a highly appreciated strategy in deep convolutional neural networks (DCNNs) due to its multifaceted benefits. It heartens us to explore the effect of fusing multiple transfer learning features of different DCNN architectures. Thus, in this work, we extract features of image statistics by exploiting three different pre-trained DCNNs through transfer learning. Then, we transform the features into a generalized subspace through a recently introduced Autoencoder network and fuse them to form intra-class invariant feature vector that is used to train a multi-class Support Vector Machine (SVM). The experimental results on various datasets, including object and action image statistics show that the fusion of multiple transfer learning features improves classification accuracy as compared to fusion of multiple hand-crafted features and usage of single component transfer learning features.

A palmprint image contains abundant features, such as the ridges, lines and textures, etc. Different features reflect the different characteristic of the palmprint. Fusion of multiple palmprint features may enhance the performance of a palmprint authentication system. In this paper, we investigate the fusion of two novel palmprint representations: orientationCode and diffCode. OrientationCode are defined using four directional templates and diffCode is computed using differential operation. Then these two codes are fused to measure the similarity of palmprints. The experimental results show that the proposed approach can get a very high accuracy.

Multiple color representation and fusion for diabetes mellitus diagnosis based on back tongue images

This paper presents an image segmentation method which utilizes image multiple features fusion, superpixel segmentation technique and hypergraph theory. Based on the image superpixel structure, our method maps each superpixel to a vertex and the similarity between two superpixels to an edge, to form a graph structure which can be represented as an affinity matrix. Hyperedges can be generated by doing spectral clustering on this affinity graph. Similarity of superpixels can be defined by different feature, then multiple features can be used to generate the hyperedges, and finally construct a hypergraph. Our approach uses the brightness, color and edge features. Then we use normalized hypergraph cut algorithm and the K-means clustering method to obtain the final segmentation result. The method was evaluated in Berkeley Segmentation Database. Experimental results show that our method can give better segmentations for the natural images. The computational efficiency and robustness of the method was verified through a comparative analysis of some state-of-the-art segmentation algorithms.KeywordsImage segmentationHypergraphSuperpixelMultiple features fusion

Multiple features data fusion method in color texture analysis

Recognizing human actions is crucial for an effective and safe collaboration between humans and robots. For example, in a collaborative assembly task, human workers can use gestures to communicate with the robot, and the robot can use the recognized actions to anticipate the next steps in the assembly process, leading to improved safety and productivity. In this work, we propose a general framework for human action recognition based on 3D pose estimation and ensemble techniques, which allows to recognize both body actions and hand gestures. The framework relies on OpenPose and 2D to 3D lifting methods to estimate 3D joints for the human body and the hands, feeding then these joints into a set of graph convolutional networks based on the Shift-GCN architecture. The output scores of all networks are combined using an ensemble approach to predict the final human action. The proposed framework was evaluated on a custom dataset designed for human–robot collaboration tasks, named IAS-Lab Collaborative HAR dataset. The results showed that using an ensemble of action recognition models improves the accuracy and robustness of the overall system; moreover, the proposed framework can be easily specialized on different scenarios and achieve state-of-the-art results on the HRI30 dataset when coupled with an object detector or classifier.

Depth motion maps (DMM), containing abundant information on appearance and motion, are captured from the absolute difference between two consecutive depth video sequences. In this paper, each depth frame is first projected onto three orthogonal planes (front, side, top). Then the DMMf, DMMs and DMMt are generated under the three projection view respectively. In order to describe DMM in local and global, histogram of oriented gradient (HOG), local binary patterns (LBP), a local Gist feature description based on a dense grid are computed respectively. Considering the advantages of features fusion and information entropy quantitative evaluation of the Principal Component Analysis (PCA), three descriptors are weighted and fused based on information entropy improved PCA to represent the depth video. A reconstruction error adaptively weighted combination collaborative classifier based on $l$ 1 -norm and $l$ 2 -norm is employed for action recognition, the adaptively weights are determined by Entropy Method. Experimental results on MSR Action3D dataset show that the present approach has strong robustness, discriminability and stability.

Feature fusion for basic behavior unit segmentation from video sequences

Real-time identification of human activities in urban environments is increasingly becoming important in the context of public safety and national security. Distributed camera networks that provide multiple views of a scene are ideally suited for real-time action recognition. However, deployments of multi-camera based real-time action recognition systems have thus far been inhibited because of several practical issues and restrictive assumptions that are typically made such as the knowledge of a subjects orientation with respect to the cameras, the duration of each action and the conformation of a network deployment during the testing phase to that of a training deployment. In reality, action recognition involves classification of continuously streaming data from multiple views which consists of an interleaved sequence of various human actions. While there has been extensive research on machine learning techniques for action recognition from a single view, the issues arising in the fusion of data from multiple views for reliable action recognition have not received as much attention. In this thesis, I have developed a fusion framework for human action recognition using a multi-camera network that addresses these practical issues of unknown subject orientation, unknown view configurations, action interleaving and variable duration actions.;The proposed framework consists of two components: (1) a score-fusion technique that utilizes underlying view-specific supervised learning classifiers to classify an action within a given set of frames and (2) a sliding window technique that is used to parse a sequence of frames into multiple actions. The use of a score-fusion technique as opposed to a feature-level fusion of data from multiple views allows us to robustly classify actions even when camera configurations are arbitrary and different from training phase and at the same time reduces the required network bandwidth for data transmission permitting wireless deployments. Moreover, the proposed framework is independent of the underlying classifier that is used to generate scores for each action snippet and thus offers more flexibility compared to sequential approaches like Hidden Markov Models. The amount of training and parameterization is also significantly lower compared to HMM-based approaches. This Real-Time recognition system has been tested on 4 classifiers which are Linear Discriminant Analysis, Multinomial Naive Bayes, Logistic Regression and Support Vector Machines. Over 90% accuracy has been achieved by this system in Real-Time recognizing variable duration actions performed by the subject. The performance of the system is also shown to be robust to camera failures.

Fusion of multiple features can boost the performance of large-scale visual classification and detection tasks like TRECVID Multimedia Event Detection (MED) competition [1]. In this paper, we propose a novel feature fusion approach, namely Feature Weighting via Optimal Thresholding (FWOT) to effectively fuse various features. FWOT learns the weights, thresholding and smoothing parameters in a joint framework to combine the decision values obtained from all the individual features and the early fusion. To the best of our knowledge, this is the first work to consider the weight and threshold factors of fusion problem simultaneously. Compared to state-of-the-art fusion algorithms, our approach achieves promising improvements on HMDB [8] action recognition dataset and CCV [5] video classification dataset. In addition, experiments on two TRECVID MED 2011 collections show that our approach outperforms the state-of-the-art fusion methods for complex event detection.

Understanding semantic meaning of human actions captured in unconstrained environments has broad applications in fields ranging from patient monitoring, human-computer interaction, to surveillance systems. However, while great progresses have been achieved on automatic human action detection and recognition in videos that are captured in controlled/constrained environments, most existing approaches perform unsatisfactorily on videos with uncontrolled/unconstrained conditions (e.g., significant camera motion, background clutter, scaling, and light conditions). To address this issue, the authors propose a robust human action detection and recognition framework that works effectively on videos taken in controlled or uncontrolled environments. Specifically, the authors integrate the optical flow field and Harris3D corner detector to generate a new spatial-temporal information representation for each video sequence, from which the general Gaussian mixture model (GMM) is learned. All the mean vectors of the Gaussian components in the generated GMM model are concatenated to create the GMM supervector for video action recognition. They build a boosting classifier based on a set of sparse representation classifiers and hamming distance classifiers to improve the accuracy of action recognition. The experimental results on two broadly used public data sets, KTH and UCF YouTube Action, show that the proposed framework outperforms the other state-of-the-art approaches on both action detection and recognition.

This paper presents a novel framework for human action recognition based on a newly proposed mid-level feature representation method named Lie Algebrized Guassians (LAG). As an action sequence can be treated as a 3D object in space-time space, we address the action recognition problem by recognizing 3D objects and characterize 3D objects by the probability distributions of local spatio-temporal features. First, for each video, we densely sample local spatio-temporal features (e.g. HOG3D) at multiple scales confined in bounding boxes of human body. Moreover, normalized spatial coordinates are appended to local descriptor in order to capture spatial position information. Then the distribution of local features in each video is modeled by a Gaussian Mixture Model (GMM). To estimate the parameters of video-specific GMMs, a global GMM is trained using all training data and video-specific GMMs are adapted from the global GMM. Then the LAG is adopted to vectorize those video-specific GMMs. Finally, linear SVM is employed for classification. Experimental results on the KTH and UCF Sports dataset show that our method achieves state-of-the-art performance.

Human detection and action recognition form the basis for understanding human behaviors. Human detection is used to detect the positions of humans, and action recognition is able to recognize the action of specific humans. However, numerous approaches have been used to handle action recognition and human detection separately. Therefore, three main issues still exist when independent methods of human detection and action recognition are combined, 1) intrinsic errors in object detection impact the performance of action recognition, 2) features common to action recognition and object detection are missed, 3) the combination also has an impact on processing speed. We propose a single framework for human detection and action recognition to solve these issues. It is based on a hierarchical structure called Boosted Randomized Trees. The nodes are trained such that the upper nodes detect humans from the background, while the lower nodes recognize action. We were able to improve both human detection and action recognition rates over earlier hierarchical structure approaches with the proposed method.

Malaysia has been recognized as one of the twelve nations endowed with rich biodiversity. Such huge number of species in the rain forest and sea are an important asset that need to be properly documented. Responding to these important needs, we have designed and evaluated a content based image retrieval system catered for marine life images. This paper investigates the effectiveness of various low level image descriptors, which includes the colour, shape and texture features in representing the semantic categories of the marine life images. This is a challenging task since the images are taken from different viewpoints and marine species do not possess consistent colour, shape and textural appearance. For the purpose of evaluating the overall image retrieval system effectiveness, we design and implement an image retrieval system which support image query by example. Experiment is conducted to evaluate the effectiveness of various low level image descriptors and the fusion of multiple features. The experiment result on the image retrieval performance is presented.