Switch and Refine: A Long-Term Tracking and Segmentation Framework

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multi-background model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multi-background model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Segmenting and tracking of objects in video is of great importance for video-based encoding, surveillance, and retrieval. However, the inherent difficulty of object segmentation and tracking is to distinguish changes in the displacement of objects from disturbing effects such as noise and illumination changes. Therefore, in this paper, we formulate a color-based deformable model which is robust against noisy data and changing illumination. Computational methods are presented to measure color constant gradients. Further, a model is given to estimate the amount of sensor noise through these color constant gradients. The obtained uncertainty is subsequently used as a weighting term in the deformation process. Experiments are conducted on image sequences recorded from three-dimensional scenes. From the experimental results, it is shown that the proposed color constant deformable method successfully finds object contours robust against illumination, and noisy, but homogeneous regions.

As a fundamental task in computer vision, visual object tracking has received much attention in recent years. Most studies focus on short-term visual tracking which addresses shorter videos and always-visible targets. However, long-term visual tracking is much closer to practical applications with more complicated challenges. There exists a longer duration such as minute-level or even hour-level in the long-term tracking task, and the task also needs to handle more frequent target disappearance and reappearance. In this paper, we provide a thorough review of long-term tracking, summarizing long-term tracking algorithms from two perspectives: framework architectures and utilization of intermediate tracking results. Then we provide a detailed description of existing benchmarks and corresponding evaluation protocols. Furthermore, we conduct extensive experiments and analyse the performance of trackers on six benchmarks: VOTLT2018, VOTLT2019 (2020/2021), OxUvA, LaSOT, TLP and the long-term subset of VTUAV-V. Finally, we discuss the future prospects from multiple perspectives, including algorithm design and benchmark construction. To our knowledge, this is the first comprehensive survey for long-term visual object tracking. The relevant content is available at https://github.com/wangdong-dut/Long-term-Visual-Tracking.

High tracking frame rates have been achieved based on traditional tracking methods which however would fail due to drifts of the object template or model, especially when the object disappears from the camera’s field of view. To deal with it, tracking-and-detection-combination has become more and more popular for long-term unknown object tracking, whose detector almost does not drift and can regain the disappeared object when it comes back. However, for online machine learning and multiscale object detection, expensive computing resources and time are required. So it is not a good idea to combine tracking and detection sequentially like Tracking-Learning-Detection algorithm. Inspired from parallel tracking and mapping, this article proposes a framework of parallel tracking and detection for unknown object tracking. The object tracking algorithm is split into two separate tasks—tracking and detection which can be processed in two different threads, respectively. One thread is used to deal with the tracking between consecutive frames with a high processing speed. The other thread runs online learning algorithms to construct a discriminative model for object detection. Using our proposed framework, high tracking frame rates and the ability of correcting and recovering the failed tracker can be combined effectively. Furthermore, our framework provides open interfaces to integrate state-of-the-art object tracking and detection algorithms. We carry out an evaluation of several popular tracking and detection algorithms using the proposed framework. The experimental results show that different tracking and detection algorithms can be integrated and compared effectively by our proposed framework, and robust and fast long-term object tracking can be realized.

The Visual Object Tracking challenge VOT2020 is the eighth annual tracker benchmarking activity organized by the VOT initiative. Results of 58 trackers are presented; many are state-of-the-art trackers published at major computer vision conferences or in journals in the recent years. The VOT2020 challenge was composed of five sub-challenges focusing on different tracking domains: (i) VOT-ST2020 challenge focused on short-term tracking in RGB, (ii) VOT-RT2020 challenge focused on “real-time” short-term tracking in RGB, (iii) VOT-LT2020 focused on long-term tracking namely coping with target disappearance and reappearance, (iv) VOT-RGBT2020 challenge focused on short-term tracking in RGB and thermal imagery and (v) VOT-RGBD2020 challenge focused on long-term tracking in RGB and depth imagery. Only the VOT-ST2020 datasets were refreshed. A significant novelty is introduction of a new VOT short-term tracking evaluation methodology, and introduction of segmentation ground truth in the VOT-ST2020 challenge – bounding boxes will no longer be used in the VOT-ST challenges. A new VOT Python toolkit that implements all these novelites was introduced. Performance of the tested trackers typically by far exceeds standard baselines. The source code for most of the trackers is publicly available from the VOT page. The dataset, the evaluation kit and the results are publicly available at the challenge website (http://votchallenge.net).

The Discriminative Correlation Filter (DCF) has been universally recognized in visual object tracking, thanks to its excellent accuracy and high speed. Nevertheless, these DCF-based trackers perform poorly in long-term tracking. The reasons include the following aspects—first, they have low adaptability to significant appearance changes in long-term tracking and are prone to tracking failure; second, these trackers lack a practical re-detection module to find the target again after tracking failure. In our work, we propose a new long-term tracking strategy to solve these issues. First, we make the best of the static and dynamic information of the target by introducing the motion features to our long-term tracker and obtain a more robust tracker. Second, we introduce a low-rank sparse dictionary learning method for re-detection. This re-detection module can exploit a correlation among these training samples and alleviate the impact of occlusion and noise. Third, we propose a new reliability evaluation method to model an adaptive update, which can switch expediently between the tracking module and the re-detection module. Massive experiments demonstrate that our proposed approach has an obvious improvement in precision and success rate over these state-of-the-art trackers.

Long-term object tracking is key for a higher level of semantic interpretation of driving environment. One of the state-of-the-art approaches for long-term tracking is Tracking-Learning-Detection (TLD), which, however, suffers from variability of on-road objects and moving cluttered background. This paper presents a long-term object tracking method for intelligent driving based on Stixel World to address the drifting problem. First, this method adopts TLD framework, and integrates intensity and depth cues into the detector and learning component. Next, this method introduces Stixel World for compact medium-level representation of the 3D world, and Attention Guiding Filter (AGF) is proposed to focus on relevant areas in the image. Experiments in real traffic scene show the outstanding long-term tracking performance for intelligent driving.

The performance of player object tracking in soccer match video is seriously affected by challenges like occlusion, out-of-view, similarity interference, low resolution, etc. To solve this problem, we propose a long-term tracking algorithm based on kernelized correlation filter. Firstly, the algorithm fuses shape, color and grayscale features to enhance the object representation ability and introduces scale filter to realize real-time scale estimation to improve the tracking accuracy and robustness. Secondly, the algorithm monitors tracking status in real time by the tracking result evaluation function. If the status judged good, the multi-peak re-detection of response map is used to review the tracking result. Otherwise, the object is re-detected through sliding windows to realize long-term tracking. The experimental results tested on Benchmark for Soccer Player Tracking (BSPT) demonstrate that the proposed tracker can achieve an accurate, real-time and long-term visual tracking for soccer player while runs at speed near 80 FPS.

Chapter 10 - Long-term deep object tracking

The Visual Object Tracking challenge VOT2022 is the tenth annual tracker benchmarking activity organized by the VOT initiative. Results of 93 entries are presented; many are state-of-the-art trackers published at major computer vision conferences or in journals in recent years. The VOT2022 challenge was composed of seven sub-challenges focusing on different tracking domains: (i) VOT-STs2022 challenge focused on short-term tracking in RGB by segmentation, (ii) VOT-STb2022 challenge focused on short-term tracking in RGB by bounding boxes, (iii) VOT-RTs2022 challenge focused on “real-time” short-term tracking in RGB by segmentation, (iv) VOT-RTb2022 challenge focused on “real-time” short-term tracking in RGB by bounding boxes, (v) VOT-LT2022 focused on long-term tracking, namely coping with target disappearance and reappearance, (vi) VOT-RGBD2022 challenge focused on short-term tracking in RGB and depth imagery, and (vii) VOT-D2022 challenge focused on short-term tracking in depth-only imagery. New datasets were introduced in VOT-LT2022 and VOT-RGBD2022, VOT-ST2022 dataset was refreshed, and a training dataset was introduced for VOT-LT2022. The source code for most of the trackers, the datasets, the evaluation kit and the results are publicly available at the challenge website (http://votchallenge.net).KeywordsVisual Object Tracking challengeVOTShort-term trackingLong-term trackingPerformance evaluation

In order to accomplish the long term visual tracking task in complex scenes, solve problems of scale variation, appearance variation and tracking failure, a long term tracking algorithm is given based on the framework of collaborative correlation tracking. Firstly, we integrate several powerful features to boost the represent ability based on the kernel correlation filter, and extend the filter by embedding a scale factor into the kernelized matrix to handle the scale variation. Then, we use the Peak-Sidelobe Ratio to decide whether the object is tracked successfully, and a CUR filter for re-detection the object in case of tracking failure is learnt with random sampling. Corresponding experiment is performed on 17 challenging benchmark video sequences. Compared with the 8 existing state-of-the-art algorithms based on discriminative learning method, the results show that the proposed algorithm improves the tracking performance on several indexes, and is robust to complex scenes for long term visual tracking.

Combining complementary trackers for enhanced long-term visual object tracking

This paper presents a method to address the problem of long-term robust object tracking in unconstrained environments. An enhanced random fern is proposed and integrated into our tracking framework...

We propose an algorithm of semiautomatic video segmentation based background constructing. Firstly, we obtain the video object of the first frame by employing manual method, then, construct the background of the first frame, finally, the video object tracking is completed based on background updating and motion information of the video object. Our algorithm obtains a good tracking effect when the video object rapidly motions or is still. Experimental results are compared with the results of other algorithm, and demonstrate the performance of our algorithm.

One of the major objectives in multimedia technology is to be able to segment objects automatically from a video sequence, for a diverse range of applications from video surveillance and object tracking through to content-based video retrieval, coding and medical imaging. Probabilistic spatio-temporal (PST) video object segmentation has been shown to be of pivotal importance in achieving better segmentation, because it considers space, colour and time features conjointly in a spatio-temporal framework. Existing PST techniques however, incur high computational expense as they normally have to process large dimensional feature vectors. This paper addresses this problem by presenting a computationally efficient PST video object segmentation algorithm that has reduced dimensionality, with experimental results confirming that for various standard video test sequences, a significant reduction in computational complexity is achieved compared with the existing PST technique, without compromising perceptual picture quality.