Context-Aware Correlation Filter for Visual Tracking with Deep Convolution Features

In recent years, discriminative correlation filter (DCF) based algorithms have significantly advanced the state of the art in visual object tracking. The key to the success of DCF is an efficient discriminative regression model trained with powerful multi-cue features, including both hand-crafted and deep neural network features. However, the tracking performance is hindered by their inability to respond adequately to abrupt target appearance variations. This issue is posed by the limited representation capability of fixed image features. In this work, we set out to rectify this shortcoming by proposing a complementary representation of a visual content. Specifically, we propose the use of a collaborative representation between successive frames to extract the dynamic appearance information from a target with rapid appearance changes, which results in suppressing the undesirable impact of the background. The resulting collaborative representation coefficients are combined with the original feature maps using a spatially regularised DCF framework for performance boosting. The experimental results on several benchmarking datasets demonstrate the effectiveness and robustness of the proposed method, as compared with a number of state-of-the-art tracking algorithms.

Visual object tracking is a significant technology for camera-based sensor networks applications. Multilayer convolutional features comprehensively used in correlation filter (CF)-based tracking algorithms have achieved excellent performance. However, there are tracking failures in some challenging situations because ordinary features are not able to well represent the object appearance variations and the correlation filters are updated irrationally. In this paper, we propose a local–global multiple correlation filters (LGCF) tracking algorithm for edge computing systems capturing moving targets, such as vehicles and pedestrians. First, we construct a global correlation filter model with deep convolutional features, and choose horizontal or vertical division according to the aspect ratio to build two local filters with hand-crafted features. Then, we propose a local–global collaborative strategy to exchange information between local and global correlation filters. This strategy can avoid the wrong learning of the object appearance model. Finally, we propose a time-space peak to sidelobe ratio (TSPSR) to evaluate the stability of the current CF. When the estimated results of the current CF are not reliable, the Kalman filter redetection (KFR) model would be enabled to recapture the object. The experimental results show that our presented algorithm achieves better performances on OTB-2013 and OTB-2015 compared with the other latest 12 tracking algorithms. Moreover, our algorithm handles various challenges in object tracking well.

Accurate location estimation of a target is a classical and very popular problem in visual object tracking, for which correlation filters have been proven highly effective in real-time scenarios. However, the great variation of the target's appearance and the surrounding background throughout a video sequence would lead to failure tracking for the sake of the model drift, using trackers based on correlation filters. In our approach, we present a simple and fast method to improve the robustness of the model based on sum of template and pixel-wise learners (Staple). On the one hand, a confidence regression model is established to adjust adaptively the model online learning rate to alleviate the model drift. On the other hand, instead of likelihood, the scale with maximal posterior probability is selected as the target scale to obtain the more accurate estimation. Extensive experimental results demonstrate that the proposed approach performs favorably against several state-of-the-art algorithms on large-scale challenging benchmark data sets at speed in excess of 42 frames/s.

One of the most important and challenging research topics in the area of computer vision is visual object tracking, which is relevant to many real-world applications. Recently, discriminative correlation filters (DCF) have been demonstrated to overcome the problems in visual object tracking efficiently. So far, only single-resolution feature maps have been utilized in DCF. Owing to this limitation, the potential of DCF has not been exploited. Moreover, convolutional features have demonstrated a better performance for visual tracking than histogram of oriented gradients (HOG) features and color features. Based on these facts, in this paper, we propose collaborative learning based on multi-resolution feature maps for DCF, employing convolutional features. Further, the confidence score, which represents the location of the target object, is selected from various candidates based on certain rules. In addition, the continuous filters are trained to handle the variations of appearance of the target. The extensive experimental results obtained using VOT2015 and OTB-100 benchmark datasets show that the proposed algorithm performs favorably against state-of-the-art tracking algorithms.

Recently, Discriminative Correlation Filters (DCF) have shown excellent performance in visual object tracking. The correlation for a computing response map can be conducted efficiently in Fourier domain by Discrete Fourier Transform (DFT) of inputs, where the DFT of an image has symmetry on the Fourier domain. To enhance the robustness and discriminative ability of the filters, many efforts have been devoted to optimizing the learning process. Regularization methods, such as spatial regularization or temporal regularization, used in existing DCF trackers aim to enhance the capacity of the filters. Most existing methods still fail to deal with severe appearance variations—in particular, the large scale and aspect ratio changes. In this paper, we propose a novel framework that employs adaptive spatial regularization and temporal regularization to learn reliable filters in both spatial and temporal domains for tracking. To alleviate the influence of the background and distractors to the non-rigid target objects, two sub-models are combined, and multiple features are utilized for learning of robust correlation filters. In addition, most DCF trackers that applied 1-dimensional scale space search method suffered from appearance changes, such as non-rigid deformation. We proposed a 2-dimensional scale space search method to find appropriate scales to adapt to large scale and aspect ratio changes. We perform comprehensive experiments on four benchmarks: OTB-100, VOT-2016, VOT-2018, and LaSOT. The experimental results illustrate the effectiveness of our tracker, which achieved a competitive tracking performance. On OTB-100, our tracker achieved a gain of 0.8% in success, compared to the best existing DCF trackers. On VOT2018, our tracker outperformed the top DCF trackers with a gain of 1.1% in Expected Average Overlap (EAO). On LaSOT, we obtained a gain of 5.2% in success, compared to the best DCF trackers.

Indoor environment is a common scene in our everyday life, and detecting and tracking multiple targets in this environment is a key component for many applications. However, this task still remains challenging due to limited space, intrinsic target appearance variation, e.g. full or partial occlusion, large pose deformation, and scale change. In the proposed approach, we give a novel framework for detection and tracking in indoor environments, and extend it to robot navigation. One of the key components of our approach is a virtual top view created from an RGB-D camera, which is named ground plane projection (GPP). The key advantage of using GPP is the fact that the intrinsic target appearance variation and extrinsic noise is far less likely to appear in GPP than in a regular side-view image. Moreover, it is a very simple task to determine free space in GPP without any appearance learning even from a moving camera. Hence GPP is very different from the top-view image obtained from a ceiling mounted camera. We perform both object detection and tracking in GPP. Two kinds of GPP images are utilized: gray GPP, which represents the maximal height of 3D points projecting to each pixel, and binary GPP, which is obtained by thresholding the gray GPP. For detection, a simple connected component labeling is used to detect footprints of targets in binary GPP. For tracking, a novel Pixel Level Association (PLA) strategy is proposed to link the same target in consecutive frames in gray GPP. It utilizes optical flow in gray GPP, which to our best knowledge has never been done before. Then we "back project" the detected and tracked objects in GPP to original, side-view (RGB) images. Hence we are able to detect and track objects in the side-view (RGB) images. Our system is able to robustly detect and track multiple moving targets in real time. The detection process does not rely on any target model, which means we do not need any training process. Moreover, tracking does not require any manual initialization, since all entering objects are robustly detected. We also extend the novel framework to robot navigation by tracking. As our experimental results demonstrate, our approach can achieve near prefect detection and tracking results. The performance gain in comparison to state-of-the-art trackers is most significant in the presence of occlusion and background clutter.

In recent years, visual object tracking has become a very active research field which is mainly divided into the correlation filter-based tracking and deep learning (e.g., deep convolutional neural network and Siamese neural network) based tracking. For target tracking algorithms based on deep learning, a large amount of computation is required, usually deployed on expensive graphics cards. However, for the rich monitoring devices in the Internet of Things, it is difficult to capture all the moving targets in each device in real time, so it is necessary to perform hierarchical processing and use tracking based on correlation filtering in insensitive areas to alleviate the local computing pressure. In sensitive areas, upload the video stream to a cloud computing platform with a faster computing speed to perform an algorithm based on deep features. In this paper, we mainly focus on the correlation filter-based tracking. In the correlation filter-based tracking, the discriminative scale space tracker (DSST) is one of the most popular and typical ones which is successfully applied to many application fields. However, there are still some improvements that need to be further studied for DSST. One is that the algorithms do not consider the target rotation on purpose. The other is that it is a very heavy computational load to extract the histogram of oriented gradient (HOG) features from too many patches centered at the target position in order to ensure the scale estimation accuracy. To address these two problems, we introduce the alterable patch number for target scale tracking and the space searching for target rotation tracking into the standard DSST tracking method and propose a visual object multimodality tracker based on correlation filters (MTCF) to simultaneously cope with translation, scale, and rotation in plane for the tracked target and to obtain the target information of position, scale, and attitude angle at the same time. Finally, in Visual Tracker Benchmark data set, the experiments are performed on the proposed algorithms to show their effectiveness in multimodality tracking.

As the development of Artificial Intelligent, visual object tracking plays a key role in computer vision area with numerous real-world applications. This paper proposes a novel approach with the combination of deep spatial transformer features and discriminative correlation filters-based tracking frameworks for the visual tracking problem. The deep spatial transformer features have several advantages compared to the standard deep features (fully connected layers). Firstly, they are more robust when object is suffering affine transformation because of the specific characteristic. Secondly, they have low dimensionality with deeper layer. Lastly, some structural information is contained during tracking procedure. We perform comprehensive experiments on two benchmark datasets: OTB and VOT2015. Surprisingly, compared with some traditional approach based on hand-crafted features, our results further show the affine robustness. We also maintain the accuracy and tracking speed compared with several discriminative correlation filter-based tracker at the same time. Finally, results comparable to state-of-the-art trackers are obtained on all two benchmark datasets.

Correlation filter (CF) theory has gained a sustainable attractiveness on the tracking field by virtue of its efficiency in training and decision stage. The seminal idea of cyclic shift operations on image patch makes the CF-based trackers accomplish the dense sampling scheme in an efficient manner. Due to all shifted samples merely provide the single appearance of the target in the previous frame, the filters cannot acquire sufficient information of the target appearance, which leads to the tracking failed. To tackle this problem, we designed a novel tracker using collaborative correlation filters, which collaboratively exploit the historical appearances of the target and the surrounding contexts of the previous frame for the filters learning. Therefore, the collaborative correlation filter simultaneously acquires the variations in target appearance and promotes discriminability when the target suffers from occlusion or motion blur. In addition, a unified optimization procedure is proposed to enable abundant information to be embedded in filter learning, which obtains a closed-form solution. Since the historical appearances of the target are collected from previous tracking results and the target appearance may change drastically or suffer from occlusion, using all of these templates for training is inappropriate. To eliminate the useless samples from the template set, we provide a template updating strategy based on collaborative representation. The experimental results on the standard $$\hbox {OTB}100$$ benchmark data sets illustrate the robustness of the proposed tracker.

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multi-background model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multi-background model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

Chapter 10 - Long-term deep object tracking

This work deals with visual object tracking. The well known discriminant correlation filter (DCF) based approach is improved by multi-layer CNN features, spatial reliability (through a foreground mask) and conditionally model updating strategy. In the training stage, by calculating a foreground mask using the color histograms, for each chosen CNN layer, a correlation filter is trained under the foreground constraint to construct a weak tracker. In next frame, the tracking position is from the weighting of weak trackers, for which the weights are computed by Hedge method. The response peak and oscillation are both considered to estimate the confidence criteria. The model and weight of each weak tracker are updated only when the tracking is high-confident. We analyze and evaluate our system on OTB-13 dataset, and show that our approach performs superiorly against several state-of-the-art methods.

In recent times, the topic of human facial age estimation attracted much attention. This is due to its ability to improve biometrics systems. Recently, several applications that are based on the demographic attributes estimation have been developed. These include law enforcement, re-identification in videos, planed marketing, intelligent advertising, social media, and human-computer interaction. The main contributions of the paper are as follows. Firstly, it extends some handcrafted models that are based on the Pyramid Multi Level (PML) face representation. Secondly, it evaluates the performance of two different kinds of features that are handcrafted and deep features. It compares handcrafted and deep features in terms of accuracy and computational complexity. The proposed scheme of study includes the following three main steps: 1) face preprocessing; 2) feature extraction (two different kinds of features are studied: handcrafted and deep features); 3) feeding the obtained features to a linear regressor. In addition, we investigate the strengths and weaknesses of handcrafted and deep features when used in facial age estimation. Experiments are run on three public databases (FG-NET, PAL and FACES). These experiments show that both handcrafted and deep features are effective for facial age estimation.

Context information brings new opportunities for efficient and effective system resource management of mobile devices. In this work, we focus on the use of context information to achieve energy-efficient, ubiquitous wireless connectivity. Our field-collected data show that the energy cost of network interfaces poses a great challenge to ubiquitous connectivity, despite decent availability of cellular networks. We propose to leverage the complementary strengths of Wi-Fi and cellular interfaces by automatically selecting the most efficient one based on context information. We formulate the selection of wireless interfaces as a statistical decision problem. The challenge is to accurately estimate Wi-Fi network conditions without powering up the network interface. We explore the use of different context information, including time, history, cellular network conditions, and device motion, to statistically estimate Wi-Fi network conditions with negligible overhead. We evaluate several context-based algorithms for the estimation and prediction of current and future network conditions. Simulations using field-collected traces show that our network estimation algorithms can improve the average battery lifetime of a commercial mobile phone for an ECG reporting application by 40 percent, very close to the estimated theoretical upper bound of 42 percent. Furthermore, our most effective algorithm can predict Wi-Fi availability for one and ten hours into the future with 95 and 90 percent accuracy, respectively.

Siamese networks have been successfully introduced into visual tracking, which match the best candidate and a target template via a couple of networks with shared parameters. However, most Siamese network-based trackers (SNTs) are tailored to best match the canonical posture of the template and the search-region images, resulting in inferior performance when the target objects have large-scale pose variations. Besides, SNTs fail to discriminate distractors well because they only leverage high-level semantic features as target representations that cannot well tell from different targets of the same category. To address these issues, this paper presents an efficient and effective SNT that is based on feature alignment and aggregation networks. Specifically, we first design an effective feature alignment network module to calibrate the search-region image. This module results in a more reliable matching response that is robust to severe target pose variations. Then, we develop an effective shallow-level and high-level feature aggregation network module to complement the feature characteristics, making the learned feature representation not only well differentiate the target from distractors, but also robust to target appearance variations. Afterwards, we employ a channel-attention mechanism to further strengthen the discriminative capability of the aggregated feature representation. Finally, both the alignment and the aggregation modules are seamlessly integrated into the Siamese networks for robust tracking. Meanwhile, we offline learn the network parameters end-to-end without time-consuming fine-tuning. Extensive evaluations on a variety of benchmarks including VOT-2017, OTB-100, UAV123 and GOT-10k demonstrate favorable performance of our tracker against state-of-the-art ones with a speed of 60 fps.