Integrating appearance features and soft biometrics for person re-identification

Person re-identification has been widely studied due to its importance in surveillance and forensics applications. In practice, gallery images are high resolution (HR), while probe images are usually low resolution (LR) in the identification scenarios with large variation of illumination, weather, or quality of cameras. Person re-identification in this kind of scenarios, which we call super-resolution (SR) person re-identification, has not been well studied. In this paper, we propose a semi-coupled low-rank discriminant dictionary learning (SLD2L) approach for SR person re-identification task. With the HR and LR dictionary pair and mapping matrices learned from the features of HR and LR training images, SLD2L can convert the features of the LR probe images into HR features. To ensure that the converted features have favorable discriminative capability and the learned dictionaries can well characterize intrinsic feature spaces of the HR and LR images, we design a discriminant term and a low-rank regularization term for SLD2L. Moreover, considering that low resolution results in different degrees of loss for different types of visual appearance features, we propose a multi-view SLD2L (MVSLD2L) approach, which can learn the type-specific dictionary pair and mappings for each type of feature. Experimental results on multiple publicly available data sets demonstrate the effectiveness of our proposed approaches for the SR person re-identification task.

Person re-identification plays important roles in many practical applications. Due to various human poses, complex backgrounds and similarity of person clothes, person re-identification is still a challenging task. In this paper, we mainly focus on the robust and discriminative appearance feature representation and proposed a novel multi-appearance method for person re-identification. First, we proposed a deep feature fusion method and get the multi-appearance feature by combining two Convolutional Neural Networks. Then, in order to further enhance the representation of the appearance feature, the multi-part model was constructed by combining the whole body and the six body parts. Additionally, we optimized the feature extraction process by adding a pooling layer. Comprehensive and comparative experiments with the state-of-the-art methods over publicly available datasets demonstrated that the proposed method can get promising results.

Person re-identification is to match pedestrian targets observed from different camera views of multi-camera surveillance systems. Aiming at the issues that the color similarity pedestrian affect the recognition result, a person re-identification method based on multi-feature fusion in perceptual uniform color space is proposed, which is according to the characteristics of human vision system. Firstly, the color space which is suitable for similar pedestrian recognition is selected from five color spaces CIELAB, S-CIELAB, IPT, LAB2000HL and CAM02-SCD. Secondly, three kinds of pedestrian appearance features are extracted including spatial weighted histogram, local color and shape feature and global texture feature. Different distance measure methods are used to calculate the similarity of three different features. Finally, the linear fusion is performed by the adaptive weights. The experimental results based on VIPeR database and ETHZ database show that the proposed method is effective and can be used for the recognition of pedestrians with visual similarity.

Towards fast and kernelized orthogonal discriminant analysis on person re-identification

People re-identification is a difficult problem in non-overlapping video surveillance, because pedestrian images contain variations in view angle, lighting, background clutter and occlusion. This paper presents an approach for person re-identification in surveillance system by taking pedestrian sequence (set) as processing element. The distance metric is learnt from relative and irrelative set-to-set pairs by extending relative distance comparison. Combining the appearance features, the person re-identification is transformed as a set-to-set matching based on the learnt distance. The approach proposed in this paper is compared with several state-of-the-art algorithms on NLPR_MCT dataset. Experimental results show that this algorithm has a more superior performance than the state-of-the-art algorithms.

Shape Context for soft biometrics in person re-identification and database retrieval

Person re-identification aims to associate images of the same person over multiple non-overlapping camera views at different times. Depending on the human operator, manual re-identification in large camera networks is highly time consuming and erroneous. Automated person re-identification is required due to the extensive quantity of visual data produced by rapid inflation of large scale distributed multi-camera systems. The state-of-the-art works focus on learning and factorize person appearance features into latent discriminative factors at multiple semantic levels. We propose Deep Parallel Feature Consensus Network (DeepPFCN), a novel network architecture that learns multi-scale person appearance features using convolutional neural networks. This model factorizes the visual appearance of a person into latent discriminative factors at multiple semantic levels. Finally consensus is built. The feature representations learned by DeepPFCN are more robust for the person re-identification task, as we learn discriminative scale-specific features and maximize multi-scale feature fusion selections in multi-scale image inputs. We further exploit average and max pooling in separate scale for person-specific task to discriminate features globally and locally. We demonstrate the re-identification advantages of the proposed DeepPFCN model over the state-of-the-art re-identification methods on three benchmark datasets - Market1501, DukeMTMCreID, and CUHK03. We have achieved mAP results of 75.8%, 64.3%, and 52.6% respectively on these benchmark datasets. KeywordsPerson re-identificationDeep learningArchitecture

Person reidentification has attracted extensive research efforts in recent years. It is challenging due to the varied visual appearance from illumination, view angle, background, and possible occlusions, leading to the difficulties when measuring the relevance, i.e., similarities, between probe and gallery images. Existing methods mainly focus on pairwise distance metric learning for person reidentification. In practice, pairwise image matching may limit the data for comparison (just the probe and one gallery subject) and yet lead to suboptimal results. The correlation among gallery data can be also helpful for the person reidentification task. In this paper, we propose to investigate the high-order correlation among the probe and gallery data, not the pairwise matching, to jointly learn the relevance of gallery data to the probe. Recalling recent progresses on feature representation in person reidentification, it is difficult to select the best feature and each type of feature can benefit person description from different aspects. Under such circumstances, we propose a multihypergraph joint learning algorithm to learn the relevance in corporation with multiple features of the imaging data. More specifically, one hypergraph is constructed using one type of feature and multiple hypergraphs can be generated accordingly. Then, the learning process is conducted on the multihypergraph structure, and the identity of a probe is determined by its relevance to each gallery data. The merit of the proposed scheme is twofold. First, different from pairwise image matching, the proposed method jointly explores the relationships among different images. Second, multimodal data, i.e., different features, can be formulated in the multihypergraph structure, which can convey more information in the learning process and can be easily extended. We note that the proposed method is a general framework to incorporate with any combination of features, and thus is flexible in practice. Experimental results and comparisons with the state-of-the-art methods on three public benchmarking data sets demonstrate the superiority of the proposed method.

Person Re-ID (Re-ID) has been an emerging topic and there is a need for non-contextual, long term person Re-ID algorithm since most of the crimes occur in public places such as airports, railway stations where the video is recorded for a long duration across arbitrary camera views. The proposed work comprises of two fusion frameworks. First, the people detection is accomplished by fusing Histogram of Oriented Gradients (HOG) and extended Center Symmetric Local Binary Pattern (XCS-LBP) features which overcomes several disadvantages like missing detection and false detection due to sudden illumination, and near/far field of view changes. Next, each body part of the tracked person is learned using Deformable Part Model (DPM) which is robust to different viewpoints. Secondly, the feature level fusion of Gabor (appearance feature) and Skeleton Recurrent Motion Image (SRMI Gait biometric feature) is proposed to overcome the homogeneity issue of distinguishing people when both the texture and color of the people attire are similar. Finally, person Re-ID is achieved by relevance metric learning method with list wise constraints (RMLLCs). The performance measure, Cumulative Matching Curve (CMC) Rate shows the improved matching accuracy compared to other state-of-the-art algorithms with benchmark datasets.KeywordsPerson Re-identificationLong-termSkeleton recurrent motion image sudden illuminationGait and appearance features

We propose an open-ended multimodal video question answering (VideoQA) method that predicts textual answers by referring to multimodal information derived from videos. Most current open-ended VideoQA methods focus on motion and appearance features from videos and ignore the audio features that are useful for understanding video content in more detail. A few prior works that use motion, appearance, and audio features showed poor results on public benchmarks since they failed to (e.g., region or grid-level) multimodal features effectively fuse the features with details for video reasoning. We overcame these limitations with multi-stream 3-dimensional convolutional networks (3D ConvNets) and a transformer-based modulator for VideoQA. Our network represents detailed motion and appearance features as well as an audio feature on multiple 3D ConvNets and modulates each intermediate representation with question information to extract their relevant spatiotemporal features over the frames. Based on the question content, our network fuses the multimodal information of 3D ConvNets and predicts the final answers. Our VideoQA method, which effectively combined multimodal data yields, outperformed both a previous multimodal VideoQA method and a state-of-the-art method on standard benchmarks. Visualization suggests that our method can predict the correct answers by listening to the audio information, even when the motion and appearance features are inadequate for understanding the video constant.

In this paper, we propose a novel coupled dictionary learning method, namely coupled analysis-synthesis dictionary learning, to improve the performance of person re-identification in the non-overlapping fields of different camera views. Most of the existing coupled dictionary learning methods train a coupled synthesis dictionary directly on the original feature spaces, which limits the representation ability of the dictionary. To handle the diversities of different original spaces, We first employ local Fisher discriminant analysis (LFDA) to learn a common feature space for close relationship of the same people in different views. In order to enhance the representation power of the coupled synthesis dictionary, we then learn a coupled analysis dictionary by transforming the common feature space into the coupled feature space. Experimental results on two publicly available VIPeR and CUHK01 datasets have validated the effectiveness of the proposed method.

Person re-identification is a topic which has potential to be used for applications within forensics, flow analysis and queue monitoring. It is the process of matching persons across two or more camera views, most often by extracting colour and texture based hand-crafted features, to identify similar persons. Because of challenges regarding changes in lighting between views, occlusion or even privacy issues, more focus has turned to overhead and depth based camera solutions. Therefore, we have developed a system, based on a Convolutional Neural Network (CNN) which is trained using both depth and RGB modalities to provide a fused feature. By training on a locally collected dataset, we achieve a rank-1 accuracy of 74.69%, increased by 16.00% compared to using a single modality. Furthermore, tests on two similar publicly available benchmark datasets of TVPR and DPI-T show accuracies of 77.66% and 90.36%, respectively, outperforming state-of-the-art results by 3.60% and 5.20%, respectively.

Person re-identification has become a hot research topic due to its importance in surveillance and forensics applications. The purpose of person re-identification is to find the same person from disjoint camera views at different time. Most of the existing methods try to identify the person by measuring the similarity of two still images from different camera views, which only uses intraimage features such as color, shape and texture. In this paper, we propose a person re-identification architecture which analyzes a sequence pair while not an image pair, so not only intra-image features but also the gait feature is also considered. In contrast to existing works that use handcrafted features, our method automatically learns spatio-temporal features optimal for the person re-identification task with a deep convolutional network. To learn a discriminant metric, we use a subspace and metric learning method called Cross-view Quadratic Discriminant Analysis (XQDA).The process of XQDA is also called transfer learning. Experiments show that our method significantly outperforms the state of the art on both a large dataset (CUHK03) and a medium-sized dataset (CUHK01). We also get a better performance on a small dataset (VIPeR) with a pre-trained network without fine-tuning.

Soft biometrics enable the identification of subjects based on semantic descriptions collected from eyewitnesses allowing people to search in surveillance databases. Although research has recently shown an increased interest in soft biometrics, not much of the work have used crowdsourcing, and it did not investigate the impact of feature selection on identification. In this paper, we introduce a new set of facial soft biometrics and labels with a novel description for the eyebrow region. Also, we examine the use of crowdsourcing for labelling the comparative facial soft biometrics and assess its impact on the identification. Moreover, we explore the impact of feature selection with our biometric measures and evaluate the effect of label scale compression. Experiments based on the Southampton biometric tunnel database demonstrate a 100% rank-1 identification rate using 20 features only.

Researches on person re-identification have become more prevalent as a core technique in visual surveillance systems. While a large volume of person re-identification (Re-ID) methods adopt experimental settings with a large patch size, relatively small ones are not much considered. Thus, this work focuses on investigating comparison of person Re-ID performance when color and shape features-based are adopted for small patch size images with low resolution. First, the deep decompositional network is used to divide the person into upper and lower body parts. Then, color and shape features are extracted. Finally, using single or combination of features, similarity-based ranked matching scores are computed. The person Re-ID performance is evaluated based on VIPeR dataset. From the experiment results, we found that color-based feature is better than shape-based features, and the combination of color and shape features-based can be meaningful.