- Research Article
73
- 10.1016/j.patcog.2017.07.005
Multi-type attributes driven multi-camera person re-identification
- Sep 12, 2017
- Pattern Recognition
- Chi Su + 4 more +4
Multi-type attributes driven multi-camera person re-identification
Driven by growing public security demands and the advancement of intelligent surveillance systems, person re-identification (ReID) has emerged as a prominent research focus in the field of computer vision. However, this task presents challenges due to its high sensitivity to variations in visual appearance caused by factors such as body pose and camera parameters. Although deep learning-based methods have achieved marked progress in ReID, the high cost of annotation remains a challenge that cannot be overlooked. To address this, we propose an unsupervised attribute learning framework that eliminates the need for costly manual annotations while maintaining high accuracy. The framework learns the mid-level human attributes (such as clothing type and gender) that are robust to substantial visual appearance variations and can hence boost the accuracy of attributes with a small amount of labeled data. To carry out our framework, we present a part-based convolutional neural network (CNN) architecture, which consists of two components for image and body attribute learning on a global level and upper- and lower-body image and attribute learning at a local level. The proposed architecture is trained to learn attribute-semantic and identity-discriminative feature representations simultaneously. For model learning, we first train our part-based network using a supervised approach on a labeled attribute dataset. Then, we apply an unsupervised clustering method to assign pseudo-labels to unlabeled images in a target dataset using our trained network. To improve feature compatibility, we introduce an attribute consistency scheme for unsupervised domain adaptation on this unlabeled target data. During training on the target dataset, we alternately perform three steps: extracting features with the updated model, assigning pseudo-labels to unlabeled images, and fine-tuning the model. Through a unified framework that fuses complementary attribute-label and identity label information, our approach achieves considerable improvements of 10.6% and 3.91% mAP on Market-1501→DukeMTMC-ReID and DukeMTMC-ReID→Market-1501 unsupervised domain adaptation tasks, respectively.
Multi-type attributes driven multi-camera person re-identification
Multi-type attributes driven multi-camera person re-identification
Deep Attributes Driven Multi-camera Person Re-identification
The visual appearance of a person is easily affected by many factors like pose variations, viewpoint changes and camera parameter differences. This makes person Re-Identification (ReID) among multiple cameras a very challenging task. This work is motivated to learn mid-level human attributes which are robust to such visual appearance variations. And we propose a semi-supervised attribute learning framework which progressively boosts the accuracy of attributes only using a limited number of labeled data. Specifically, this framework involves a three-stage training. A deep Convolutional Neural Network (dCNN) is first trained on an independent dataset labeled with attributes. Then it is fine-tuned on another dataset only labeled with person IDs using our defined triplet loss. Finally, the updated dCNN predicts attribute labels for the target dataset, which is combined with the independent dataset for the final round of fine-tuning. The predicted attributes, namely \emph{deep attributes} exhibit superior generalization ability across different datasets. By directly using the deep attributes with simple Cosine distance, we have obtained surprisingly good accuracy on four person ReID datasets. Experiments also show that a simple metric learning modular further boosts our method, making it significantly outperform many recent works.
Read morePerson Re-Identification With Joint Verification and Identification of Identity-Attribute Labels
One of the major challenges in person Re-Identification (ReID) is the inconsistent visual appearance of a person. Current works on visual feature and distance metric learning have achieved significant achievements, but still suffer from the limited robustness to pose variations, viewpoint changes, etc. This makes person ReID among multiple cameras still challenging. This work is motivated to learn mid-level human attributes which are robust to visual appearance variations and could be used as efficient features for person matching. We propose a supervised multi-task learning framework which considers attribute label information with joint identification-verification network to simultaneously learn an attribute-semantic and identity-discriminative feature representation. Specifically, this framework adopts the part-based deep neural network and learn three different tasks simultaneously: person identification, person verifications and attribute identification, so as to discover and capture concurrently complementary discriminative information about a person image from global and local image features and mid-level attribute features in one deep neural network. With the multi-task learning architecture, we obtain a discriminative model that reaches a synergy in distinguishing different person images, as manifested with the competitive accuracy on three person ReID datasets: Market1501, DukeMTMC-reID and VIPeR.
Read moreLabel-efficient visual recognition with transfer learning
Visual recognition, encompassing tasks such as image classification, object detection, and semantic segmentation, has been a fundamental challenge in computer vision research. It also serves as the foundation for numerous applications, including autonomous driving, remote sensing, and robotics. The emergence of deep learning has significantly advanced this field, enabling remarkable achievements through the use of end-to-end trainable deep neural networks (DNNs). However, this remarkable progress comes at the cost of requiring vast amounts of annotated training data, the collection of which is often extremely labor-intensive and time-consuming. This constraint presents a significant barrier to the scalability and adaptability of various deep learning approaches. Transfer learning, which aims to transfer the off-the-shelf learned knowledge from one or multiple related source domains to target domains, is a promising way to mitigate this constraint. In this thesis, we explore label-efficient visual recognition with various transfer learning techniques, where the objective is to learn effective visual representations for target domains without any annotations by transferring the off-the-shelf knowledge learned in different but related source domains. Specifically, we investigate three representative types of transfer learning problems, including unsupervised domain adaptation, black-box unsupervised domain adaptation and test-time prompt tuning. As one of the most typical transfer learning problems, unsupervised domain adaptation aims to adapt a visual recognition model that is trained on labeled source-domain data to perform well on a related but unlabeled target-domain data. The primary challenge is to mitigate the distribution differences between the source and target domains, which are also referred to as domain gap or domain discrepancy, such that the model can generalize well on the target domain. In this thesis, we propose three innovative unsupervised domain adaptation techniques designed to efficiently transfer knowledge from labeled source domains to unlabeled target domains across a range of visual recognition tasks. Specifically, we explore diverse approaches to align domains from multiple perspectives (e.g., data or feature space alignment), thereby reducing the domain gap and enabling to perform well on unlabeled target domain. Black-box unsupervised domain adaptation aims to learn visual recognition model for target domain with the initial predictions of target data obtained by black-box source-domain model. In another word, during the adaptation process, only the model predictions of target data are available while either source data or source models are inaccessible. Compared to conventional unsupervised domain adaptation, black-box unsupervised domain adaptation offers notable advantages in terms of data privacy and flexibility, allowing adaptation to different target models regardless of the architecture or details of the source-trained black-box model. In this thesis, we analyze the challenges in black-box domain adaptation and design a novel approach for effectively exploiting the initial predictions of target data, leading to better adapted model on unlabeled target data. We further explore label-efficient visual recognition in the recent era of vision-language models. Despite the impressive zero-shot generalization capabilities of vision-language models, they always face unneglectable gaps while applied to target datasets, i.e., a target dataset typically features unique image styles and text formats tailored to its specific tasks. To circumvent this limitation, we explore an effective transfer learning approach for vision-language models, i.e., test-time prompt tuning, where the prompts are optimized as learnable vectors during the test stage without requiring any annotations for target data. Specifically, we examine the limitation of existing test-time prompt tuning methods, and propose a novel approach that could effectively exploit the learned knowledge in vision-language models and learn prompts for target data on-the-fly. Extensive experiments conducted over various visual recognition benchmarks indicate that our proposed methods achieve superior performance over target datasets, enabling to largely mitigate the dependence of labeled data across various visual recognition tasks.
Read moreProgressive Unsupervised Domain Adaptation for Image-based Person Re-Identification
Unsupervised domain adaptation (UDA) has emerged as an effective paradigm for reducing the huge manual annotation cost for Person Re-Identification (Re-ID). Many of the recent UDA methods for Re-ID are clustering-based and select all the pseudo-label samples in each iteration for the model training. However, there are many wrong labeled samples that will mislead the model optimization under this circumstance. To solve this problem, we propose a Progressive Unsupervised Domain Adaptation (PUDA) framework for image-based Person Re-ID to reduce the negative effect of wrong pseudo-label samples on the model training process. Specifically, we first pretrain a CNN model on a labeled source dataset, then finetune the model on unlabeled target dataset with the following three steps iteratively: 1) estimating pseudo-labels for all the images in the target dataset with the model trained in the last iteration; 2) extending the training set by adding pseudo-label samples with higher label confidence; 3) updating the CNN model with the expanded training set in a supervised manner. During the iteration process, the number of pseudo-label samples added increased progressively. In particular, a Moderate Initial Selections (MIS) strategy for pseudo-label sampling is also proposed to reduce the negative impacts of random noise features in the early iterations and mislabeled samples in the late iterations on the model. The proposed framework with MIS strategy is validated on the Duke-to-Market, Market-to-Duke unsupervised domain adaptation tasks and achieves improvements of 4.2 points (absolute, i.e., 80.0% vs. 75.8%) and 1.7 points (absolute, i.e., 70.7% vs. 69.0%) in mAP correspondingly.
Read moreProgressive Unsupervised Domain Adaptation for Image-based Person Re-Identification
Unsupervised domain adaptation (UDA) has emerged as an effective paradigm for reducing the huge manual annotation cost for Person Re-Identification (Re-ID). Many of the recent UDA methods for Re-ID are clustering-based and select all the pseudo-label samples in each iteration for the model training. However, there are many wrong labeled samples that will mislead the model optimization under this circumstance. To solve this problem, we propose a Progressive Unsupervised Domain Adaptation (PUDA) framework for image-based Person Re-ID to reduce the negative effect of wrong pseudo-label samples on the model training process. Specifically, we first pretrain a CNN model on a labeled source dataset, then finetune the model on unlabeled target dataset with the following three steps iteratively: 1) estimating pseudo-labels for all the images in the target dataset with the model trained in the last iteration; 2) extending the training set by adding pseudo-label samples with higher label confidence; 3) updating the CNN model with the expanded training set in a supervised manner. During the iteration process, the number of pseudo-label samples added increased progressively. In particular, a Moderate Initial Selections (MIS) strategy for pseudo-label sampling is also proposed to reduce the negative impacts of random noise features in the early iterations and mislabeled samples in the late iterations on the model. The proposed framework with MIS strategy is validated on the Duke-to-Market, Market-to-Duke unsupervised domain adaptation tasks and achieves improvements of 4.2 points (absolute, i.e., 80.0% vs. 75.8%) and 1.7 points (absolute, i.e., 70.7% vs. 69.0%) in mAP correspondingly.
Read moreTriplet CNN and pedestrian attribute recognition for improved person re-identification
In this paper, we propose a pedestrian attribute recognition approach and a CNN-based person re-identification framework enhanced by pedestrian attributes. The knowledge of person attributes can help video surveillance tasks like person re-identification as well as person search, semantic video indexing and retrieval to overcome viewpoint changes with their robustness to the inherent visual appearance variations. Compared to previous approaches, our attribute recognition method using Local Maximal Occurrence (LOMO) features and a Multi-Label Multi-Layer Perceptron (MLMLP) classifier proves to be more robust to different view points and is computationally more efficient. The experiments on three public benchmarks show that the proposed method improves the state-of-the art on attribute recognition. Furthermore, we integrate our attribute recognition algorithm into a triplet CNN similarity learning framework for person re-identification fusing both learned CNN features and attributes. This fusion leads to an overall improvement, and we achieve state-of-the-art results on person re-identification.
Read moreKnowledge Distillation for Multi-Target Domain Adaptation in Real-Time Person Re-Identification
Despite the recent success of deep learning architectures, person re-identification (ReID) remains a challenging problem in real-word applications. Several unsupervised single-target domain adaptation (STDA) methods have recently been proposed to limit the decline in ReID accuracy caused by the domain shift that typically occurs between source and target video data. Given the multimodal nature of person ReID data (due to variations across camera viewpoints and capture conditions), training a common CNN backbone to address domain shifts across multiple target domains, can provide an efficient solution for real-time ReID applications. Although multi-target domain adaptation (MTDA) has not been widely addressed in the ReID literature, a straightforward approach consists in blending different target datasets, and performing STDA on the mixture to train a common CNN. However, this approach may lead to poor generalization, especially when blending a growing number of distinct target domains to train a smaller CNN. To alleviate this problem, we introduce a new MTDA method based on knowledge distillation (KD-ReID) that is suitable for real-time person ReID applications. Our method adapts a common lightweight student backbone CNN over the target domains by alternatively distilling from multiple specialized teacher CNNs, each one adapted on data from a specific target domain. Extensive experiments <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> conducted on several challenging person ReID datasets indicate that our approach outperforms state-of-art methods for MTDA, including blending methods, particularly when training a compact CNN backbone like OSNet. Results suggest that our flexible MTDA approach can be employed to design cost-effective ReID systems for real-time video surveillance applications.
Read moreUnsupervised Domain Adaption based on metric learning for Person Re-Identification
Person re-identification(ReID) with deep convolutional neural networks(CNNs) has attracted increasing interest in computer vision due to its wide potential applications in visual surveillance and has achieved high performance in recent years using a lot of techniques to overcome the challenges such as variations in view angle, lighting, image occlusion. Another main challenge in person re-identification(ReID) is the cross domain adaptation. Due to different domains, a person re-identification model trained on one dataset with good performance often fails to achieve same or better performance on other datasets. We propose a method which is about both the source and target datasets. We fine-tune the deep CNN model on the labeled source dataset in a supervised manner by using distance metric learning and the unlabeled target dataset in an unsupervised manner simultaneously.
Read moreExploiting low-rank structures from cross-camera images for robust person re-identification
Matching individuals across non-overlapping camera views is known as the problem of person re-identification. In addition to significant visual appearance variations due to lighting, view angle, etc. changes, one might encounter corrupted data due to background clutter and occlusion, or even missing data at some camera views in practical scenarios. To address the above challenges, we present a novel approach to robust person re-identification, particularly aiming at handling missing and corrupted image data across camera views. Based on the technique of low-rank matrix decomposition, our proposed algorithm observes the low-rank structure of cross-view data, which is able to disregard extreme/sparse errors while the missing instances can be recovered automatically. Our experiments will confirm the effectiveness and robustness of our method, which is shown to outperform several baseline and state-of-the-art person re-identification approaches.
Read moreUnsupervised Domain Adaptation on Person Reidentification Via Dual-Level Asymmetric Mutual Learning.
Unsupervised domain adaptation (UDA) person reidentification (Re-ID) aims to identify pedestrian images within an unlabeled target domain with an auxiliary labeled source-domain dataset. Many existing works attempt to recover reliable identity information by considering multiple homogeneous networks. And take these generated labels to train the model in the target domain. However, these homogeneous networks identify people in approximate subspaces and equally exchange their knowledge with others or their mean net to improve their ability, inevitably limiting the scope of available knowledge and putting them into the same mistake. This article proposes a dual-level asymmetric mutual learning (DAML) method to learn discriminative representations from a broader knowledge scope with diverse embedding spaces. Specifically, two heterogeneous networks mutually learn knowledge from asymmetric subspaces through the pseudo label generation in a hard distillation manner. The knowledge transfer between two networks is based on an asymmetric mutual learning (AML) manner. The teacher network learns to identify both the target and source domain while adapting to the target domain distribution based on the knowledge of the student. Meanwhile, the student network is trained on the target dataset and employs the ground-truth label through the knowledge of the teacher. Extensive experiments in Market-1501, CUHK-SYSU, and MSMT17 public datasets verified the superiority of DAML over state-of-the-arts (SOTA).
Read moreRepeated Pattern Detection Using CNN Activations
We propose a new approach for detecting repeated patterns on a grid in a single image. To do so, we detect repetitions in the space of pre-trained deep CNN filter responses at all layer levels. These encode features at several conceptual levels (from low-level patches to high-level semantics) as well as scales (from local to global). As a result, our repeated pattern detector is robust to challenging cases where repeated tiles show strong variation in visual appearance due to occlusions, lighting or background clutter. Our method contrasts with previous approaches that rely on keypoint extraction, description and clustering or on patch correlation. These generally only detect low-level feature clusters that do not handle variations in visual appearance of the patterns very well. Our method is simpler, yet incorporates high level features implicitly. As such, we can demonstrate detections of repetitions with strong appearance variations, organized on a nearly-regular axis-aligned grid Results show robustness and consistency throughout a varied database of more than 150 images.
Read morePerson Re-identification and Tracking in Video Surveillance
Video surveillance system is one of the most essential topics in the computer vision field. As the rapid and continuous increasement of using video surveillance cameras to obtain portrait information in scenes, it becomes a very important system for security and criminal investigations. Video surveillance system includes many key technologies, including the object recognition, the object localization, the object re-identification, object tracking, and by which the system can be used to identify or suspect the movements of the objects and persons. In recent years, person re-identification and visual object tracking have become hot research directions in the computer vision field. The re-identification system aims to recognize and identify the target of the required attributes, and the tracking system aims at following and predicting the movement of the target after the identification process. Researchers have used deep learning and computer vision technologies to significantly improve the performance of person re-identification. However, the study of person re-identification is still challenging due to complex application environments such as lightning variations, complex background transformations, low-resolution images, occlusions, and a similar dressing of different pedestrians. The challenge of this task also comes from unavailable bounding boxes for pedestrians, and the need to search for the person over the whole gallery images. To address these critical issues in modern person identification applications, we propose an algorithm that can accurately localize persons by learning to minimize intra-person feature variations. We build our model upon the state-of-the-art object detection framework, i.e., faster R-CNN, so that high-quality region proposals for pedestrians can be produced in an online manner. In addition, to relieve the negative effects caused by varying visual appearances of the same individual, we introduce a novel center loss that can increase the intra-class compactness of feature representations. The engaged center loss encourages persons with the same identity to have similar feature characteristics. Besides the localization of a single person, we explore a more general visual object tracking problem. The main task of the visual object tracking is to predict the location and size of the tracking target accurately and reliably in subsequent image sequences when the target is given at the beginning of the sequence. A visual object tracking algorithm with high accuracy, good stability, and fast inference speed is necessary. In this thesis, we study the updating problem for two kinds of tracking algorithms among the mainstream tracking approaches, and improve the robustness and accuracy. Firstly, we extend the siamese tracker with a model updating mechanism to improve their tracking robustness. A siamese tracker uses a deep convolutional neural network to obtain features and compares the new frame features with the target features in the first frame. The candidate region with the highest similarity score is considered as the tracking result. However, these kinds of trackers are not robust against large target variation due to the no-update matching strategy during the whole tracking process. To combat this defect, we propose an ensemble siamese tracker, where the final similarity score is also affected by the similarity with tracking results in recent frames instead of solely considering the first frame. Tracking results in recent frames are used to adjust the model for a continuous target change. Meanwhile, we combine adaptive candidate sampling strategy and large displacement optical flow method to improve its performance further. Secondly, we investigate the classic correlation filter based tracking algorithm and propose to provide a better model selection strategy by reinforcement learning. Correlation filter has been proven to be a useful tool for a number of approaches in visual tracking, particularly for seeking a good balance between tracking accuracy and speed. However, correlation filter based models are susceptible to wrong updates stemming from inaccurate tracking results. To date, little effort has been devoted to handling the correlation filter update problem. In our approach, we update and maintain multiple correlation filter models in parallel, and we use deep reinforcement learning for the selection of an optimal correlation filter model among them. To facilitate the decision process efficiently, we propose a decision-net to deal with target appearance modeling, which is trained through hundreds of challenging videos using proximal policy optimization and a lightweight learning network. An exhaustive evaluation of the proposed approach on the OTB100 and OTB2013 benchmarks show the effectiveness of our approach.
Read moreSelf-Supervised Person Re-Identification with Channel-Wise Transformer
Unsupervised domain adaptive (UDA) person Re-Identification aims to improve the model’s generalization capability from labeled source domain to unlabeled target domain. To this end, a strong and robust method is required to extract discriminative features of pedestrians. Recently, transformer-based method achieves great performance on person Re-Identification (ReID). However, due to the domain gap between ImageNet and ReID datasets, it requires a large pre-training dataset to boost performance on Vision Transformer (ViT). To this end, we first investigate self-supervised learning methods with ViTs pretrained on LUPerson datasets, and find it significantly outperforms ImageNet supervised pre-training models on ReID tasks. A Catastrophic Forgetting Score (CFS) is also used to select a subset of LUPerson, which reduces the training time and improves performance. We then proposed a channel-wise self-attention module to reduce the computing cost on the class token. A dual prototype contrastive learning is proposed to fully exploit the hard feature on memory bank under unsupervised domain adaptation. Finally, we achieve state-of-the-art performance on Market-1501 and MSMT17. Our model achieves 91.5%/69.6% mAP accuracy on Market-1501/MSMT17 for supervised ReID, and 90.7%/57.4% mAP for MS2MA/MA2MS UDA ReID.
Read morePerson re-identification via region-of-interest based features
Person re-identification is still a challenging task due to large visual appearance variations caused by illumination, background, viewpoints and poses in multi-camera surveillance. To address these challenges, many methods have been proposed. In this paper, we present an efficient method, called Region-of-Interest based Features (ROIF), via combining textural and chromatic features. It consists of two main phases - region-of-interest exploration from image and features extraction from ROI. Experimental results on the database VIPeR show that our method can yield promising accuracy with a quite cheap time cost.
Read more