Visual object tracking via collaborative correlation filters

In recent years, discriminative correlation filter (DCF) based algorithms have significantly advanced the state of the art in visual object tracking. The key to the success of DCF is an efficient discriminative regression model trained with powerful multi-cue features, including both hand-crafted and deep neural network features. However, the tracking performance is hindered by their inability to respond adequately to abrupt target appearance variations. This issue is posed by the limited representation capability of fixed image features. In this work, we set out to rectify this shortcoming by proposing a complementary representation of a visual content. Specifically, we propose the use of a collaborative representation between successive frames to extract the dynamic appearance information from a target with rapid appearance changes, which results in suppressing the undesirable impact of the background. The resulting collaborative representation coefficients are combined with the original feature maps using a spatially regularised DCF framework for performance boosting. The experimental results on several benchmarking datasets demonstrate the effectiveness and robustness of the proposed method, as compared with a number of state-of-the-art tracking algorithms.

Indoor environment is a common scene in our everyday life, and detecting and tracking multiple targets in this environment is a key component for many applications. However, this task still remains challenging due to limited space, intrinsic target appearance variation, e.g. full or partial occlusion, large pose deformation, and scale change. In the proposed approach, we give a novel framework for detection and tracking in indoor environments, and extend it to robot navigation. One of the key components of our approach is a virtual top view created from an RGB-D camera, which is named ground plane projection (GPP). The key advantage of using GPP is the fact that the intrinsic target appearance variation and extrinsic noise is far less likely to appear in GPP than in a regular side-view image. Moreover, it is a very simple task to determine free space in GPP without any appearance learning even from a moving camera. Hence GPP is very different from the top-view image obtained from a ceiling mounted camera. We perform both object detection and tracking in GPP. Two kinds of GPP images are utilized: gray GPP, which represents the maximal height of 3D points projecting to each pixel, and binary GPP, which is obtained by thresholding the gray GPP. For detection, a simple connected component labeling is used to detect footprints of targets in binary GPP. For tracking, a novel Pixel Level Association (PLA) strategy is proposed to link the same target in consecutive frames in gray GPP. It utilizes optical flow in gray GPP, which to our best knowledge has never been done before. Then we "back project" the detected and tracked objects in GPP to original, side-view (RGB) images. Hence we are able to detect and track objects in the side-view (RGB) images. Our system is able to robustly detect and track multiple moving targets in real time. The detection process does not rely on any target model, which means we do not need any training process. Moreover, tracking does not require any manual initialization, since all entering objects are robustly detected. We also extend the novel framework to robot navigation by tracking. As our experimental results demonstrate, our approach can achieve near prefect detection and tracking results. The performance gain in comparison to state-of-the-art trackers is most significant in the presence of occlusion and background clutter.

Template-matching methods for visual tracking have gained popularity recently due to their comparable performance and fast speed. However, they lack effective ways to adapt to changes in the target object's appearance, making their tracking accuracy still far from state-of-the-art. In this paper, we propose a dynamic memory network to adapt the template to the target's appearance variations during tracking. An LSTM is used as a memory controller, where the input is the search feature map and the outputs are the control signals for the reading and writing process of the memory block. As the location of the target is at first unknown in the search feature map, an attention mechanism is applied to concentrate the LSTM input on the potential target. To prevent aggressive model adaptivity, we apply gated residual template learning to control the amount of retrieved memory that is used to combine with the initial template. Unlike tracking-by-detection methods where the object's information is maintained by the weight parameters of neural networks, which requires expensive online fine-tuning to be adaptable, our tracker runs completely feed-forward and adapts to the target's appearance changes by updating the external memory. Moreover, unlike other tracking methods where the model capacity is fixed after offline training --- the capacity of our tracker can be easily enlarged as the memory requirements of a task increase, which is favorable for memorizing long-term object information. Extensive experiments on OTB and VOT demonstrates that our tracker MemTrack performs favorably against state-of-the-art tracking methods while retaining real-time speed of 50 fps.

In order to cope with the complex variation of target appearance during visual tracking, a robust tracking algorithm based on multi-scale kernelized least squares (KLS) is proposed. First, by showing that the dense sampling set of translated patches is circulant, using the well-established theory of circulant matrices, kernelized least squares is efficient computed with fast Fourier transform (FFT). Second, we propose a method to judge whether target and background changed significantly by the response variance of all candidate targets, and use it to improve the model updating. Meanwhile, robust scale estimation based on a scale pyramid representation is proposed to handle large scale variations. Extensive experimental results show that our approach outperforms state-of-the-art methods, while operating at real-time.

Sparse representation (SR)-based tracking systems have become popular in the past recent years for its effectiveness. However, the underlying assumption of these tracking systems is that the target appearance can be linearly represented by a sparse approximation over a set of templates and a residual term, which usually needs to solve $\ell_1$ norm minimization for many times and brings a heavy computational cost. This paper introduces an efficient tracking system by discovering orthogonalized templates. By orthogonalizing templates from previous frames and removing their correlation, we show that the sparsity of template weights is not necessary in target appearance modeling and thus a least squares regularization can be employed. We also decompose the residual term into two components in observation model to take occlusion cases into consideration. We demonstrate that, in comparison with the SR-based tracking systems that use $\ell_1$ learning, our tracking system is much more computationally efficient while getting an even better performance. Experiments on a variety of challenging video sequences demonstrate both the effectiveness and efficiency of the system.

In this paper, we propose a novel occlusion detection algorithm based on depth-layer information for robust visual tracking. The scene can be classified into the near, the target and the far layer. We find that when occlusion happens, some background patches in the near layer will move into the target region and hence occlude the target. Based on this feature of occlusion, we propose an algorithm which exploits both temporal and spatial context information to discriminate occlusion from target appearance variation. Using the framework of particle filter, our algorithm divides the background region around the target into multiple patches and tracks each of them. The background patch that occludes the target is identified collaboratively by the tracking results of both background and target trackers. Then the occlusion is evaluated with the target visibility function. If occlusion is detected, the target template stops updating. Comprehensive experiments in OTB-2013 and VOT-2015 show that our tracker achieves comparable performance with other state-of-art trackers.

Bi-RRNet: Bi-level recurrent refinement network for camouflaged object detection

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multi-background model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multi-background model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Siamese trackers demonstrated high performance in object tracking due to their balance between accuracy and speed. Unlike classification-based CNNs, deep similarity networks are specifically designed to address the image similarity problem and thus are inherently more appropriate for the tracking task. However, Siamese trackers mainly use the last convolutional layers for similarity analysis and target search, which restricts their performance. In this paper, we argue that using a single convolutional layer as feature representation is not an optimal choice in a deep similarity framework. We present a Multiple Features-Siamese Tracker (MFST), a novel tracking algorithm exploiting several hierarchical feature maps for robust tracking. Since convolutional layers provide several abstraction levels in characterizing an object, fusing hierarchical features allows to obtain a richer and more efficient representation of the target. Moreover, we handle the target appearance variations by calibrating the deep features extracted from two different CNN models. Based on this advanced feature representation, our method achieves high tracking accuracy, while outperforming the standard siamese tracker on object tracking benchmarks.The source code and trained models are available at https://github.com/zhenxili96/MFST.

SummaryThe compressive tracking algorithm introduces a compressive sensing theory into the target tracking field and produces good real‐time performance. However, the original compressive tracking algorithm ignores the fact that individual samples make different contributions to the target and that the learning factor is an empirical value that remains constant when the template is updated. Therefore, adverse factors (such as noise) and errors can infiltrate into the parametric model during the updating of the model when the object is obscured or receives interference from external factors, which will lead to tracking drift. In view of these problems, the weights of samples are given according to the distance between the sample and the target when training the Naive Bayesian classifier; hence, the stability of the tracking is improved. While the introduction of the Bhattacharyya coefficient is utilized to adjust the learning factor, this can help parameters to self‐adapt effectively. Experimental results show that the improved tracking algorithm has a better adaption to the target appearance variations, illumination changes, occlusion, and so on, and has better robustness than the original algorithm.

Recently, the Siamese network based visual tracking methods have shown great potentials in balancing the tracking accuracy and computational efficiency. These methods use two-branch convolutional neural networks (CNNs) to generate a response map between the target exemplar and each of candidate patches in the search region. However, since these methods have not fully exploit the target-specific information contained in the CNN features during the computation of the response map, they are less effective to cope with target appearance variations and background clutters. In this paper, we propose a Target-Specific Response Attention (TSRA) module to enhance the discriminability of these methods. In TSRA, a channel-wise cross-correlation operation is used to produce a multi-channel response map, where different channels correspond to different semantic information. Then, TSRA uses an attention network to dynamically re-weight the multi-channel response map at every frame. Moreover, we introduce a shortcut connection strategy to generate a residual multi-channel response map for more discriminative tracking. Finally, we integrate the proposed TSRA into the classical Siamese based tracker (i.e., SiamFC) to propose a new tracker (called TSRA-Siam). Experimental results on three popular benchmark datasets show that the proposed TSRA-Siam outperforms the baseline tracker (i.e., SiamFC) by a large margin and obtains competitive performance compared with several state-of-the-art trackers.

Recently, the compressive tracking (CT) method has attracted much attention due to its high efficiency, but it cannot well deal with the large scale target appearance variations due to its data-independent random projection matrix that results in less discriminative features. To address this issue, in this paper, we propose an adaptive CT approach, which selects the most discriminative features to design an effective appearance model. Our method significantly improves CT in three aspects. First, the most discriminative features are selected via an online vector boosting method. Second, the object representation is updated in an effective online manner, which preserves the stable features while filtering out the noisy ones. Furthermore, a simple and effective trajectory rectification approach is adopted that can make the estimated location more accurate. Finally, a multiple scale adaptation mechanism is explored to estimate object size, which helps to relieve interference from background information. Extensive experiments on the CVPR2013 tracking benchmark and the VOT2014 challenges demonstrate the superior performance of our method.

In visual tracking, developing an efficient appearance model is a challenging task due to the influence of various factors, such as illumination variation, occlusion, background clutter, and so on. Existing tracking algorithms use appearance samples from previous frames to form a template set upon which target appearance models are built. However, these appearance models are data-dependent, so they may be corrupted by significant appearance variation. It is difficult to update the templates in challenging environments. In this paper, we propose a robust visual tracking algorithm with an adaptive appearance model using a point-to-set metric learning technique. To do this, we first model a target representation using a set of target templates and a regularized affine hull (RAH) spanned by the target templates. Then, we learn a point-to-set distance metric, which is incorporated into the optimization process to obtain an adaptive target representation. The RAH model covers unseen target appearances by affine combinations of the target templates. Based on the proposed target appearance model, we design an effective template update scheme by adjusting the weights of the target templates. Experimental results on challenging video sequences with comparisons to several state-of-the-art tracking algorithms demonstrate the effectiveness and robustness of the proposed tracking algorithm.

Visual object tracking is a challenging problem due to appearance variation of target. Correlation filter (CF) -based trackers have shown competing results for visual object tracking. However, they perform poorly in the case of abrupt motion and heavy background clutter due to less use of contextual information. In this paper, we solve this problem by explicitly incorporating contextual information into a context-aware (CA) framework. Under this framework, deep features from higher convolutional layers encode more semantic information of target which are robust to appearance variations, and features from lower layers locate the target more precise. Compared with handcrafted features, DL-based representation learning require less human interventions and provide much better performance. Extensive experimental results on largescale benchmark datasets show that the proposed algorithm performs favorably against state-of-the-art methods.

Accurate location estimation of a target is a classical and very popular problem in visual object tracking, for which correlation filters have been proven highly effective in real-time scenarios. However, the great variation of the target's appearance and the surrounding background throughout a video sequence would lead to failure tracking for the sake of the model drift, using trackers based on correlation filters. In our approach, we present a simple and fast method to improve the robustness of the model based on sum of template and pixel-wise learners (Staple). On the one hand, a confidence regression model is established to adjust adaptively the model online learning rate to alleviate the model drift. On the other hand, instead of likelihood, the scale with maximal posterior probability is selected as the target scale to obtain the more accurate estimation. Extensive experimental results demonstrate that the proposed approach performs favorably against several state-of-the-art algorithms on large-scale challenging benchmark data sets at speed in excess of 42 frames/s.