Thermal Infrared Tracking using Multi-stages Deep Features Fusion

To acquire the better object tracking performance in this paper, the object of interest is modeled by its RGB-color histogram feature together with deep convolutional neural network feature (deep CNN feature) and sparse representation. Because the information of the objects of interest cannot be obtained ahead in practical use, the deep CNN features should be abstracted through the pre-trained VGG network. Then the deep feature and RGB-color histogram are combined to model the object in adaptive mode. And then sparse representation is used to express the model. Above all, an adaptive particle filter algorithm based on deep CNN feature together with RGB-color feature and sparse representation is proposed to track the object of interest. Extensive experiment results demonstrate the effectiveness of our proposed method under the serious object occlusions and appearance changes.

Recognizing human actions in videos is a challenging problem in computer vision. Recently, convolutional neural network based deep features have shown promising results for action recognition. In this paper, we investigate the problem of fusing deep appearance and motion cues for action recognition. We propose a video representation which combines deep appearance and motion based local convolutional features within the bag-of-deep-features framework. Firstly, dense deep appearance and motion based local convolutional features are extracted from spatial (RGB) and temporal (flow) networks, respectively. Both visual cues are processed in parallel by constructing separate visual vocabularies for appearance and motion. A category-specific appearance map is then learned to modulate the weights of the deep motion features. The proposed representation is discriminative and binds the deep local convolutional features to their spatial locations. Experiments are performed on two challenging datasets: JHMDB dataset with 21 action classes and ACT dataset with 43 categories. The results clearly demonstrate that our approach outperforms both standard approaches of early and late feature fusion. Further, our approach is only employing action labels and without exploiting body part information, but achieves competitive performance compared to the state-of-the-art deep features based approaches.

To address the problem that the performance of hyperspectral target tracking will be degraded when facing background clutter, this paper proposes a novel hyperspectral target tracking algorithm based on the deep edge convolution feature (DECF) and an improved context filter (ICF). DECF is a fusion feature via deep features convolving 3D edge features, which makes targets easier to distinguish under complex backgrounds. In order to reduce background clutter interference, an ICF is proposed. The ICF selects eight neighborhoods around the target as the context areas. Then the first four areas that have a greater interference in the context areas are regarded as negative samples to train the ICF. To reduce the tracking drift caused by target deformation, an adaptive scale estimation module, named the region proposal module, is proposed for the adaptive estimation of the target box. Experimental results show that the proposed algorithm has satisfactory tracking performance against background clutter challenges.

Segmentation ability map: Interpret deep features for medical image segmentation.

The goal of infrared and visible image fusion is to generate a fused image that possesses the prominent attributes of infrared image and the rich textures of visible image. Current deep learning-based methods typically involve three main steps: feature extraction, feature fusion, and reconstruction. Extracting multi-scale features during the feature extraction stage is beneficial for fully utilizing deep features. Shallow image features exhibit higher resolution and contain more textures, while deep image features have lower resolution but stronger semantic information after more pooling and convolution. Existing methods use the same network to process features of different scales, while ignoring their differences. Moreover, they lack carefully designed texture preservation modules, leading to insufficient preservation of texture in the fused images. To overcome these issues, we propose a novel end-to-end fusion network for infrared and visible images where we develop a scaling-down network and texture attention. Based on the characteristics of features at different scales, we design a scaling-down fusion network that use deeper and more complex network to process shallow features, while using more streamlined network to process deep features. In order to better preserve image textures, we design texture attention in a relatively gentle way that focus on feature channels have rich textures to achieve the goal. We conducted experiments on publicly available datasets, and the results demonstrate that our method surpasses eleven state-of-the-art methods in terms of fusion performance. This conclusion has been verified through both subjective evaluation and objective evaluation.

This paper addresses two key issues which concern the performance of nonparametric scene parsing: (1) the semantic quality of image retrieval; and (2) the accuracy in label transfer. First, because nonparametric methods annotate a query image through transferring labels from retrieved images, the task of image retrieval should find a set of "semantically similar" images to the query. Second, with the retrieval set, a good strategy should be developed to transfer semantic labels in pixel-level accuracy. In this paper, we focus on improving scene parsing accuracy in these two issues. We propose using the state-of-the-art deep convolutional features as image descriptors to improve the semantic quality of retrieved images. In addition, we include dense alignment into the Markov Random Field inference framework to transfer labels at pixel-level accuracy. Our experiments on the SIFT Flow dataset shows the improvement of the proposed approach over other nonparametric methods.

Visual object tracking is a significant technology for camera-based sensor networks applications. Multilayer convolutional features comprehensively used in correlation filter (CF)-based tracking algorithms have achieved excellent performance. However, there are tracking failures in some challenging situations because ordinary features are not able to well represent the object appearance variations and the correlation filters are updated irrationally. In this paper, we propose a local–global multiple correlation filters (LGCF) tracking algorithm for edge computing systems capturing moving targets, such as vehicles and pedestrians. First, we construct a global correlation filter model with deep convolutional features, and choose horizontal or vertical division according to the aspect ratio to build two local filters with hand-crafted features. Then, we propose a local–global collaborative strategy to exchange information between local and global correlation filters. This strategy can avoid the wrong learning of the object appearance model. Finally, we propose a time-space peak to sidelobe ratio (TSPSR) to evaluate the stability of the current CF. When the estimated results of the current CF are not reliable, the Kalman filter redetection (KFR) model would be enabled to recapture the object. The experimental results show that our presented algorithm achieves better performances on OTB-2013 and OTB-2015 compared with the other latest 12 tracking algorithms. Moreover, our algorithm handles various challenges in object tracking well.

Visual tracking is an open issue in computer vision. Designing an robust tracking algorithm is a challenging problem due to complicated appearance variations, such as partial occlusion, illumination variations, fast motion and out-of-plane rotation. Some tracking algorithms use a set of templates to represent a target candidate. Such representations are not robust to drastic appearance variations. In order to alleviate the influence of appearance variations and outliers, we propose a novel tracking algorithm based on a robust Bayesian matrix factorization, where sample data is decomposed into a basis matrix and a coefficient matrix. The sample data is consist of previous tracking results. A column vector is stacked by a tracking result in a frame. A target candidate is represented by a linear combinations of the columns in the basis matrix. The matrix factorization can capture the correlation between different features in sample data. Also the appearance variations in tracking processing can be learnt in matrix factorization. Experimental results demonstrate the appearance models is effective and the proposed tracker in a particle filter tracking framework is robust to outliers and complicated appearance variations.

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multi-background model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multi-background model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

The hashing technique has been extensively used in large-scale image retrieval applications due to its low storage and fast computing speed. Most existing deep hashing approaches cannot fully consider the global semantic similarity and category-level semantic information, which result in the insufficient utilization of the global semantic similarity for hash codes learning and the semantic information loss of hash codes. To tackle these issues, we propose a novel deep hashing approach with triplet labels, namely, deep category-level and regularized hashing (DCRH), to leverage the global semantic similarity of deep feature and category-level semantic information to enhance the semantic similarity of hash codes. There are four contributions in this article. First, we design a novel global semantic similarity constraint about the deep feature to make the anchor deep feature more similar to the positive deep feature than to the negative deep feature. Second, we leverage label information to enhance category-level semantics of hash codes for hash codes learning. Third, we develop a new triplet construction module to select good image triplets for effective hash functions learning. Finally, we propose a new triplet regularized loss (Reg-L) term, which can force binary-like codes to approximate binary codes and eventually minimize the information loss between binary-like codes and binary codes. Extensive experimental results in three image retrieval benchmark datasets show that the proposed DCRH approach achieves superior performance over other state-of-the-art hashing approaches.

Visual object tracking is a challenging problem due to appearance variation of target. Correlation filter (CF) -based trackers have shown competing results for visual object tracking. However, they perform poorly in the case of abrupt motion and heavy background clutter due to less use of contextual information. In this paper, we solve this problem by explicitly incorporating contextual information into a context-aware (CA) framework. Under this framework, deep features from higher convolutional layers encode more semantic information of target which are robust to appearance variations, and features from lower layers locate the target more precise. Compared with handcrafted features, DL-based representation learning require less human interventions and provide much better performance. Extensive experimental results on largescale benchmark datasets show that the proposed algorithm performs favorably against state-of-the-art methods.

Contour detection serves as the basis of a variety of computer vision tasks such as image segmentation and object recognition. The mainstream works to address this problem focus on designing engineered gradient features. In this work, we show that contour detection accuracy can be improved by instead making the use of the deep features learned from convolutional neural networks (CNNs). While rather than using the networks as a blackbox feature extractor, we customize the training strategy by partitioning contour (positive) data into subclasses and fitting each subclass by different model parameters. A new loss function, named positive-sharing loss, in which each subclass shares the loss for the whole positive class, is proposed to learn the parameters. Compared to the sofmax loss function, the proposed one, introduces an extra regularizer to emphasizes the losses for the positive and negative classes, which facilitates to explore more discriminative features. Our experimental results demonstrate that learned deep features can achieve top performance on Berkeley Segmentation Dataset and Benchmark (BSDS500) and obtain competitive cross dataset generalization result on the NYUD dataset.

In this paper, we propose an effective visual tracking method based on candidates selected by deep features and CNSGM. Higher level deep features contain semantic information, which enables the tracker focus more on the target than on the background. A certain convolutional layer contains many neurons. Different neurons response different things in the image. We pick the neuron has larger response on the target to select the object-related candidates for CNSGM, which can reduce the calculation load on local sparse representation in CNSGM to improve tracking speed by around 6 times. Turn out the deep features can also improve the tracking accuracy by choosing the object-related candidates. Experiment results on TB-76 video database proved that the proposed method has better performance in general.

A multi-resolution deep feature framework for dynamic displacement measurement of bridges using vision-based tracking system

Aggregating deep convolutional features into a global image vector has attracted sustained attention in image retrieval. In this paper, we propose an efficient unsupervised aggregation method that uses an adaptive Gaussian filter and an element-value sensitive vector to co-weight deep features. Specifically, the Gaussian filter assigns large weights to features of region-of-interests (RoI) by adaptively determining the RoI's center, while the element-value sensitive channel vector suppresses burstiness phenomenon by assigning small weights to feature maps with large sum values of all locations. Experimental results on benchmark datasets validate the proposed two weighting schemes both effectively improve the discrimination power of image vectors. Furthermore, with the same experimental setting, our method outperforms other very recent aggregation approaches by a considerable margin.