Coupling detection and data association for multiple object tracking

Global optimization for coupled detection and data association in multiple object tracking

In agriculture, automating the accurate tracking of fruits, vegetables, and fiber is a very tough problem. The issue becomes extremely challenging in dynamic field environments. Yet, this information is critical for making day-to-day agricultural decisions, assisting breeding programs, and much more. To tackle this dilemma, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NTrack</i> , a novel multiple object tracking framework based on the linear relationship between the locations of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">neighboring</i> tracks. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NTrack</i> computes dense optical flow and utilizes particle filtering to guide each tracker. Correspondences between detections and tracks are found through data association via direct observations and indirect cues, which are then combined to obtain an updated observation. Our modular multiple object tracking system is independent of the underlying detection method, thus allowing for the interchangeable use of any off-the-shelf object detector. We show the efficacy of our approach on the task of tracking and counting infield cotton bolls. Experimental results show that our system exceeds contemporary tracking and cotton boll-based counting methods by a large margin. Furthermore, we publicly release the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">first</i> annotated cotton boll video dataset to the research community. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the need to provide highly-accurate estimates of the total number of cotton bolls across an entire farm. We provide a multiple object tracking framework for automating the counting process using a dynamic motion model that can reidentify severely occluded objects. This information is immensely beneficial to agronomists and breeders. For example, it can allow them to accelerate the selection of genotypes and identify cotton cultivars that exhibit tolerance to adverse environmental conditions (e.g., drought, poor soil quality, etc.) via yield prediction. Since the performance of our tracker is tied to the accuracy of the object detector, the ability to swap detectors is important for enhancing the usability of the system. Our dataset structure was modeled after similar multiple object tracking datasets for the purpose of increasing adoption among practitioners. To implement our system, it is assumed that the image/video capture device is of high resolution and data is acquired under ideal lighting conditions. The framework can be run on any commercial-off-the-shelf platform (e.g., ground-based robots, unmanned aerial vehicles, etc.), with sufficient computing and memory resources, using a vision-based sensor.

Photoactivated Localization Microscopy (PALM) is a technique which allows the localization of particles smaller than the resolution of the microscope and can be used to analyze intracellular particle motion. Images acquired with this technique, however, are noisy, which complicates particle detection, and tracking the particles is complicated due to the presence of multiple objects at any given time. Additionally, the microscope head may drift by small amounts, which reduces the precision of the localization method. This paper proposes solutions to these problems based on the PHD Filter. To begin, a method for extracting protein positions from microscopy images is proposed. Tracking is provided on the extracted data using the PHD Filter framework for multiple object tracking, and a specially adapted particle filter for bias estimation is developed which exploits the PHD filter to estimate the likeliest position of the microscope. Results are shown using simulated data, and data acquired from a fluorescence microscopy experiment.

This work presents a computational framework for the analysis and design of large-scale algorithms utilized in the estimation of acoustic, doubly-dispersive, randomly time-variant, underwater communication channels. Channel estimation results are used, in turn, in the proposed framework for the development of efficient high performance algorithms, based on fast Fourier transformations, for the search, detection, estimation and tracking (SDET) of underwater moving objects through acoustic wavefront signal analysis techniques associated with real-time electronic surveillance and acoustic monitoring (eSAM) operations. Particular importance is given in this work to the estimation of the range and speed of deep underwater moving objects modeled as point targets. The work demonstrates how to use Kronecker products signal algebra (KSA), a branch of finite-dimensional tensor signal algebra, as a mathematical language for the formulation of novel variants of parallel orthogonal matching pursuit (POMP) algorithms, as well as a programming aid for mapping these algorithms to large-scale computational structures, using a modified Kuck’s paradigm for parallel computation.

In the tracking‐by‐detection approach of online multiple object tracking (MOT), a major challenge is how to associate object detections on the new video frame with previously tracked objects. Two important aspects that directly influence the performance of MOT are quality of detection and accuracy in data association. The authors propose an efficient and unified MOT framework for improved object detection, followed by enhanced object tracking. The object detection and tracking are considered as two independent functions in the tracking‐by‐detection paradigm. In this study, object detection accuracy has been increased by employing a faster region‐based convolutional neural network (Faster R‐CNN) modified with the feedback region proposals from the tracker. Target association is performed by the correlation filter‐based Siamese CNN model, which finds the similarity score between the input image patches. The Siamese CNN is trained using a supervised hard sample mining strategy. An optical flow‐based motion model is employed to predict the next probable location of the targets from the tracker and these region proposals are fed back to the classifier module of Faster R‐CNN. The authors’ extensive analysis of publicly available MOT benchmark datasets and comparison with the state‐of‐the‐art tracking methods demonstrate competitive tracking performance of the proposed MOT framework.

Traditional Multiple Object Tracking (MOT) methods in satellite videos mostly follow the Detection-Based Tracking (DBT) framework. However, the DBT framework assumes that all objects are correctly recognized and localized by the detector. In practice, the low resolution of satellite videos, small objects, and complex backgrounds inevitably leads to a decline in detector performance. To alleviate the impact of detector degradation on track, we propose Coarse-Fine Tracker, a framework that integrates the MOT framework with the Tracking Any Point (TAP) method CoTracker for the first time, leveraging TAP’s persistent point correspondence modeling to compensate for detector failures. In our Coarse-Fine Tracker, we divide the satellite video into sub-videos. For one sub-video, we first use ByteTrack to track the outputs of the detector, referred to as coarse tracking, which involves the Kalman filter and box-level motion features. Given the small size of objects in satellite videos, we treat each object as a point to be tracked. We then use CoTracker to track the center point of each object, referred to as fine tracking, by calculating the appearance feature similarity between each point and its neighboring points. Finally, the Consensus Fusion Strategy eliminates mismatched detections in coarse tracking results by checking their geometric consistency against fine tracking results and recovers missed objects via linear interpolation or linear fitting. This method is validated on the VISO and SAT-MTB datasets. Experimental results in VISO show that the tracker achieves a multi-object tracking accuracy (MOTA) of 66.9, a multi-object tracking precision (MOTP) of 64.1, and an IDF1 score of 77.8, surpassing the detector-only baseline by 11.1% in MOTA while reducing ID switches by 139. Comparative experiments with ByteTrack demonstrate the robustness of our tracking method when the performance of the detector deteriorates.

Multiple object tracking (MOT) in videos captured by unmanned aerial vehicle (UAV) is a fundamental aspect of computer vision. Recently, the one-shot tracking paradigm integrates the detection and re-identification (ReID) tasks, striking a balance between tracking accuracy and inference speed. This paradigm alleviates task conflicts and achieves remarkable results through various feature decoupling methods. However, in challenging scenarios like drone movements, lighting changes and object occlusion, it still encounters issues with detection failures and identity switches. In addition, traditional feature decoupling methods directly employ channel-based attention to decompose the detection and ReID branches, without a meticulous consideration of the specific requirements of each branch. To address the above problems, we introduce an asymmetric feature enhancement network with a global coordinate-aware enhancement (GCAE) module and an embedding feature aggregation (EFA) module, aiming to optimize the two branches independently. On the one hand, we develop the GCAE module for the detection branch, which effectively merges rich semantic information within the feature space to improve detection accuracy. On the other hand, we introduce the EFA module for the ReID branch, which highlights the significance of pixel-level features and acquires discriminative identity embedding through a local feature aggregation strategy. By efficiently incorporating the GCAE and EFA modules into the one-shot tracking pipeline, we present a novel MOT framework, named AsyUAV. Extensive experiments have demonstrated the effectiveness of our proposed AsyUAV. In particular, it achieves a MOTA of 38.3% and IDF1 of 51.7% on VisDrone2019, and a MOTA of 48.0% and IDF1 of 67.5% on UAVDT, outperforming existing state-of-the-art trackers.

The one-shot multiple object tracking (MOT) framework has drawn more and more attention in the MOT research community due to its advantage in inference speed. However, the tracking accuracy of current one-shot approaches could lead to an inferior performance compared with their two-stage counterparts. The reasons are two-fold: one is that motion information is often neglected due to the single-image input. The other is that detection and re-identification (ReID) are two different tasks with different focuses. Joining detection and re-identification at the training stage could lead to a suboptimal performance. To alleviate the above limitations, we propose a one-shot network named Motion and Correlation-Multiple Object Tracking (MAC-MOT). MAC-MOT introduces a motion enhance attention module (MEA) and a dual correlation attention module (DCA). MEA performs differences on adjacent feature maps which enhances the motion-related features while suppressing irrelevant information. The DCA module focuses on decoupling the detection task and re-identification task to strike a balance and reduce the competition between these two tasks. Moreover, symmetry is a core design idea in our proposed framework which is reflected in Siamese-based deep learning backbone networks, the input of dual stream images, as well as a dual correlation attention module. Our proposed approach is evaluated on the popular multiple object tracking benchmarks MOT16 and MOT17. We demonstrate that the proposed MAC-MOT can achieve a better performance than the baseline state of the arts (SOTAs).

End-to-end Multiple Object Tracking (MOT) frameworks integrate detection and tracking into a unified model, avoiding intermediate information loss and complicated post-processing. However, existing end-to-end MOT trackers rely on track queries of the previous frame to provide prior information. Their limited short-term temporal modeling struggle to cope with high dynamic tracking scenarios, where inter-frame target variations exhibit significant heterogeneity. To address these shortcomings, we propose a scene-perception MOT framework (SP-MOT) that encodes scene context understanding into long-term embedding and adaptively complements it with short-term cues, enabling discriminative and flexible instance representations. Specifically, SP-MOT introduces: (1) a learnable scene query that globally profiles foreground and background to capture short-term scene-level features; (2) a context understanding module to uncover long-term stable relationships across dynamic scenes based on multiple historical scene features; (3) scene-adaptive augmented decoding that leverages scene information as guidance, adaptively aggregating long-term and short-term information into object embeddings, improving the model's association ability and fault-tolerance. Extensive experiments on MOT benchmarks demonstrate that SP-MOT outperforms state-of-the-art end-to-end trackers across multiple metrics, particularly in challenging scenarios with high dynamics.

In this paper, we propose detection based simplified multiple object tracking model with handling stationary object detection and occlusion problem for real-time intelligent surveillance system. In order to solve detection of slow and stationary object problem in Gaussian Mixture Model(GMM) based adaptive background model, we presents controlling learning rate mechanism using tracked region information. And, the simple primitive multi-features are applied for real-time multiple object tracking. As well, we proposed modified moving average filter for predicting next position of moving object to handle occlusion problems. Computational and real-target experiment results show that the proposed model can successfully track moving object within 45ms per frame for 640×480 image size on Intel® Core(TM) i7 CPU 1.6GHz in a real indoor scene including occlusion situation.

ReMOT: A model-agnostic refinement for multiple object tracking

Chapter 11 - Object detection and tracking

Video object segmentation and tracking are two essential building blocks of smart surveillance systems. However, there are several issues that need to be resolved. Threshold decision is a difficult problem for video object segmentation with a multi-background model. In addition, some conditions make robust video object tracking difficult. These conditions include nonrigid object motion, target appearance variations due to changes in illumination, and background clutter. In this paper, a video object segmentation and tracking framework is proposed for smart cameras in visual surveillance networks with two major contributions. First, we propose a robust threshold decision algorithm for video object segmentation with a multi-background model. Second, we propose a video object tracking framework based on a particle filter with the likelihood function composed of diffusion distance for measuring color histogram similarity and motion clue from video object segmentation. The proposed framework can track nonrigid moving objects under drastic changes in illumination and background clutter. Experimental results show that the presented algorithms perform well for several challenging sequences, and our proposed methods are effective for the aforementioned issues.

In this paper, a novel multiple objects detection and tracking approach based on support vector machine and particle filter is proposed to track players in broadcast sports video. Compared with previous work, the contributions of this paper are focused on three aspects. First, an improved particle filter called SVR particle filter is proposed as the player tracker by integrating support vector regression (SVR) into sequential Monte Carlo framework. SVR particle filter enhances the performance of classical particle filter with small sample set and improves the efficiency of tracking system. Second, support vector classification combined with playfield segmentation is employed to automatically detect the players in sports video as the initialization of tracker. Third, a unified framework for automatic object detection and tracking is proposed based on support vector machine and particle filter. The experimental results are encouraging and demonstrate that our approach is effective

In this paper, we propose an approach which uses compressive sensing features to improve Markov Decision Process (MDP) tracking framework. First, we design a single object tracker which integrates compressive tracking into Tracking-Learning-Detection (TLD) framework to complement each other. Then we apply this tracker into the MDP tracking framework to improve the multi-object tracking performance. A discriminative model is built for each object and updated online. With the built discriminative model, the features used for data association are also enhanced. In order to validate our method, we first test the designed single object tracker with a common dataset. Then we use the validation set from the multiple object tracking (MOT) training dataset to analyze each part of our method. Finally, we test our approach in the MOT benchmark. The results show our approach improves the original method and performs superiorly against several state-of-the-art online multi-object trackers.