Robust Long-Term Visual Object Tracking via Low-Rank Sparse Learning for Re-Detection

As a fundamental task in computer vision, visual object tracking has received much attention in recent years. Most studies focus on short-term visual tracking which addresses shorter videos and always-visible targets. However, long-term visual tracking is much closer to practical applications with more complicated challenges. There exists a longer duration such as minute-level or even hour-level in the long-term tracking task, and the task also needs to handle more frequent target disappearance and reappearance. In this paper, we provide a thorough review of long-term tracking, summarizing long-term tracking algorithms from two perspectives: framework architectures and utilization of intermediate tracking results. Then we provide a detailed description of existing benchmarks and corresponding evaluation protocols. Furthermore, we conduct extensive experiments and analyse the performance of trackers on six benchmarks: VOTLT2018, VOTLT2019 (2020/2021), OxUvA, LaSOT, TLP and the long-term subset of VTUAV-V. Finally, we discuss the future prospects from multiple perspectives, including algorithm design and benchmark construction. To our knowledge, this is the first comprehensive survey for long-term visual object tracking. The relevant content is available at https://github.com/wangdong-dut/Long-term-Visual-Tracking.

Combining complementary trackers for enhanced long-term visual object tracking

In this paper, a long-term visual tracking scheme of an unmanned surface vehicle (USV) based on adaptive multi-feature fusion is proposed for visual tracking in complex environment. The salient features of the proposed approach are: (1) A scale filter is used to solve the problem of performance degradation caused by scale change of the USV in visual tracking and the dimension reduction strategy is applied to the scale filter to accelerate scale detection in the tracking process. (2) To alleviate problem such as illumination variation, deformation and background color similarity in complex environment, the Histogram of Gradient (HOG) and Color Names (CN) features are successfully combined to improve reliability and accuracy of the visual tracking scheme. (3) The anti-interference ability of visual tracking is enhanced by training Support Vector Machine (SVM) re-detection module online in order to eliminate external factors such as occlusion and moving out of sight during long-term visual tracking thereby strengthening robustness of the visual tracking scheme. Experimental results and comparative studies demonstrate that the proposed scheme is superior in tracking.

Chapter 10 - Long-term deep object tracking

In order to accomplish the long term visual tracking task in complex scenes, solve problems of scale variation, appearance variation and tracking failure, a long term tracking algorithm is given based on the framework of collaborative correlation tracking. Firstly, we integrate several powerful features to boost the represent ability based on the kernel correlation filter, and extend the filter by embedding a scale factor into the kernelized matrix to handle the scale variation. Then, we use the Peak-Sidelobe Ratio to decide whether the object is tracked successfully, and a CUR filter for re-detection the object in case of tracking failure is learnt with random sampling. Corresponding experiment is performed on 17 challenging benchmark video sequences. Compared with the 8 existing state-of-the-art algorithms based on discriminative learning method, the results show that the proposed algorithm improves the tracking performance on several indexes, and is robust to complex scenes for long term visual tracking.

Long-term object tracking is key for a higher level of semantic interpretation of driving environment. One of the state-of-the-art approaches for long-term tracking is Tracking-Learning-Detection (TLD), which, however, suffers from variability of on-road objects and moving cluttered background. This paper presents a long-term object tracking method for intelligent driving based on Stixel World to address the drifting problem. First, this method adopts TLD framework, and integrates intensity and depth cues into the detector and learning component. Next, this method introduces Stixel World for compact medium-level representation of the 3D world, and Attention Guiding Filter (AGF) is proposed to focus on relevant areas in the image. Experiments in real traffic scene show the outstanding long-term tracking performance for intelligent driving.

High tracking frame rates have been achieved based on traditional tracking methods which however would fail due to drifts of the object template or model, especially when the object disappears from the camera’s field of view. To deal with it, tracking-and-detection-combination has become more and more popular for long-term unknown object tracking, whose detector almost does not drift and can regain the disappeared object when it comes back. However, for online machine learning and multiscale object detection, expensive computing resources and time are required. So it is not a good idea to combine tracking and detection sequentially like Tracking-Learning-Detection algorithm. Inspired from parallel tracking and mapping, this article proposes a framework of parallel tracking and detection for unknown object tracking. The object tracking algorithm is split into two separate tasks—tracking and detection which can be processed in two different threads, respectively. One thread is used to deal with the tracking between consecutive frames with a high processing speed. The other thread runs online learning algorithms to construct a discriminative model for object detection. Using our proposed framework, high tracking frame rates and the ability of correcting and recovering the failed tracker can be combined effectively. Furthermore, our framework provides open interfaces to integrate state-of-the-art object tracking and detection algorithms. We carry out an evaluation of several popular tracking and detection algorithms using the proposed framework. The experimental results show that different tracking and detection algorithms can be integrated and compared effectively by our proposed framework, and robust and fast long-term object tracking can be realized.

In recent years, discriminative correlation filter (DCF) based algorithms have significantly advanced the state of the art in visual object tracking. The key to the success of DCF is an efficient discriminative regression model trained with powerful multi-cue features, including both hand-crafted and deep neural network features. However, the tracking performance is hindered by their inability to respond adequately to abrupt target appearance variations. This issue is posed by the limited representation capability of fixed image features. In this work, we set out to rectify this shortcoming by proposing a complementary representation of a visual content. Specifically, we propose the use of a collaborative representation between successive frames to extract the dynamic appearance information from a target with rapid appearance changes, which results in suppressing the undesirable impact of the background. The resulting collaborative representation coefficients are combined with the original feature maps using a spatially regularised DCF framework for performance boosting. The experimental results on several benchmarking datasets demonstrate the effectiveness and robustness of the proposed method, as compared with a number of state-of-the-art tracking algorithms.

Despite remarkable progress, visual object tracking is still a challenging task as objects usually suffer from significant appearance changes, fast motion, and serious occlusion. In this paper, we propose an anti-occlusion correlation filter-based tracking method (AO-CF) for robust visual tracking. We first propose an occlusion criterion based on continuous response values. Based on the criterion, objects are divided into four categories to adaptively identify the occlusion of objects. Then we propose a new detection condition for detecting proposals. When the occlusion criterion is triggered, the re-detection mechanism is executed and the tracker is commanded to stop, and then the re-detector selects the most reliable proposal to reinitialize the tracker. Experimental results show that our method outperforms other state-of-the-art trackers in terms of both precision rate and success rate on the widely used object tracking benchmark dataset. In addition, AO-CF is able to achieve real-time tracking speed.

One of the most important and challenging research topics in the area of computer vision is visual object tracking, which is relevant to many real-world applications. Recently, discriminative correlation filters (DCF) have been demonstrated to overcome the problems in visual object tracking efficiently. So far, only single-resolution feature maps have been utilized in DCF. Owing to this limitation, the potential of DCF has not been exploited. Moreover, convolutional features have demonstrated a better performance for visual tracking than histogram of oriented gradients (HOG) features and color features. Based on these facts, in this paper, we propose collaborative learning based on multi-resolution feature maps for DCF, employing convolutional features. Further, the confidence score, which represents the location of the target object, is selected from various candidates based on certain rules. In addition, the continuous filters are trained to handle the variations of appearance of the target. The extensive experimental results obtained using VOT2015 and OTB-100 benchmark datasets show that the proposed algorithm performs favorably against state-of-the-art tracking algorithms.

Recently, Discriminative Correlation Filters (DCF) have shown excellent performance in visual object tracking. The correlation for a computing response map can be conducted efficiently in Fourier domain by Discrete Fourier Transform (DFT) of inputs, where the DFT of an image has symmetry on the Fourier domain. To enhance the robustness and discriminative ability of the filters, many efforts have been devoted to optimizing the learning process. Regularization methods, such as spatial regularization or temporal regularization, used in existing DCF trackers aim to enhance the capacity of the filters. Most existing methods still fail to deal with severe appearance variations—in particular, the large scale and aspect ratio changes. In this paper, we propose a novel framework that employs adaptive spatial regularization and temporal regularization to learn reliable filters in both spatial and temporal domains for tracking. To alleviate the influence of the background and distractors to the non-rigid target objects, two sub-models are combined, and multiple features are utilized for learning of robust correlation filters. In addition, most DCF trackers that applied 1-dimensional scale space search method suffered from appearance changes, such as non-rigid deformation. We proposed a 2-dimensional scale space search method to find appropriate scales to adapt to large scale and aspect ratio changes. We perform comprehensive experiments on four benchmarks: OTB-100, VOT-2016, VOT-2018, and LaSOT. The experimental results illustrate the effectiveness of our tracker, which achieved a competitive tracking performance. On OTB-100, our tracker achieved a gain of 0.8% in success, compared to the best existing DCF trackers. On VOT2018, our tracker outperformed the top DCF trackers with a gain of 1.1% in Expected Average Overlap (EAO). On LaSOT, we obtained a gain of 5.2% in success, compared to the best DCF trackers.

Appearance change of moving targets is a challenging problem in visual tracking. In this paper, we present a novel visual object tracking algorithm based on the observation dependent hidden Markov model (OD-HMM) framework. The observation dependency is computed by structure complexity coefficients (SCC) which is defined to predict the target appearance change. Unlike conventional methods addressing the appearance change problem by investigating different online appearance models, we handle this problem by addressing the fundamental reason of motion -related appearance change during visual tracking. Based on the analysis of motion-related appearance change, we investigate the relationship between the structure of the object surface and the appearance stability. The appearance of complex structural regions is easier to change compared with that of smooth structural regions with object moving. Based on this, we define SCC to predict the appearance stability of moving objects. Different from the standard HMM-based tracking algorithms where observations between different frames are assumed to be independent, we consider the observation dependency between consecutive frames with the information provided by SCC. Moreover , we present a novel outlier removing method in appearance model updating which helps to avoid error accumulation. Experimental results on challenging video sequences demonstrate that the proposed visual tracking algorithm with OD-HMM and SCC achieves better performance than existing related tracking algorithms.

Correlation filter (CF) has received increasing attention in online visual object tracking. By accelerating the correlation in the frequency domain, CF trackers have achieved superior performance. However, existing CF trackers have a common shortcoming in that tracking models are prone to drift due to error accumulation. To address this problem, we propose a two-stage cascaded framework for accurate object modeling. In the first stage, we propose a passive–aggressive correlation filter (PACF) tracker to reduce error accumulation. In the subsequent stage, an online refinement algorithm is used to calibrate the tracking model by exploiting both long-term and short-term cues. In order to achieve high efficiency, our scheme reuses the PACF tracking response in the following stage. Extensive experiments were conducted on both long-term and short-term visual tracking benchmarks. The experimental results demonstrate that our tracker outperforms the state-of-the-art online tracking schemes in both long-term and short-term settings. Finally, we present a comprehensive analysis to validate the efficacy of our proposed method.

The performance of player object tracking in soccer match video is seriously affected by challenges like occlusion, out-of-view, similarity interference, low resolution, etc. To solve this problem, we propose a long-term tracking algorithm based on kernelized correlation filter. Firstly, the algorithm fuses shape, color and grayscale features to enhance the object representation ability and introduces scale filter to realize real-time scale estimation to improve the tracking accuracy and robustness. Secondly, the algorithm monitors tracking status in real time by the tracking result evaluation function. If the status judged good, the multi-peak re-detection of response map is used to review the tracking result. Otherwise, the object is re-detected through sliding windows to realize long-term tracking. The experimental results tested on Benchmark for Soccer Player Tracking (BSPT) demonstrate that the proposed tracker can achieve an accurate, real-time and long-term visual tracking for soccer player while runs at speed near 80 FPS.

In long-term video object tracking (VOT) tasks, most long-term trackers are modified from short-term trackers, which contain more and more machine learning modules to improve their performance. However, we empirically find that more modules do not necessarily lead to better results. In this paper, we make the long-term tracking framework simple by carefully selecting the cutting-edge trackers. Specifically, we propose a new long-term VOT framework that combines the benefits of two mainstream short-term tracking pipelines, i.e., the discriminative online tracker and the one-shot Siamese tracker, with a global re-detector awakened when the target is lost. Such a framework fully exploits existing advanced works from three complementary perspectives. Experimental results show that by exploiting the capabilities of existing methods instead of designing new neural networks, we can still achieve remarkable results on seven long-term VOT datasets. By introducing a continuous adjustable speed control parameter, our tracker reaches 20+FPS with only a small performance loss. The refine module not only improves the bounding box estimations but also outputs segmentation masks, so that our framework can handle the video object segmentation (VOS) tasks by using only VOT trackers. We obtain a trade-off between time and accuracy on two representative VOS datasets by only using bounding boxes as the initial input.