Multi-object tracking through learning relational appearance features and motion patterns

Occlusion, a pervasive challenge in Multi-Object Tracking (MOT) within complex scenes, severely degrades tracking performance. Current methods still face numerous problems when handling occlusions. Motion prediction struggles to accommodate diverse motion patterns, resulting in failure during short-term occlusions. Concurrently, appearance features possess insufficient discriminative power under occluded conditions, which frequently leads to identity switches following long-term occlusion. To enhance the performance of MOT under such challenging conditions, we propose an innovative Occlusion-Aware Multi-Object Tracking via Joint Diffusion Motion Prediction and Appearance Purification (OAMOT). For short-term occlusions, a Diffusion Motion Recovery Model (DMRM) is developed, which integrates residual and noise diffusion branches to recover trajectories precisely under varied motion patterns. For long-term occlusions, the SAM Appearance Purification for ReID (SAPR) module is proposed; the module leverages the mask generation mechanism of SAM as foreground attention to enhance feature discriminability. Furthermore, a Lightweight Attention Predictor (LAP) is integrated into the ReID network to achieve SAM-quality attention during inference without significant computational overhead. Experimental results on the MOT17, MOT20, and DanceTrack datasets demonstrate that the proposed OAMOT method outperforms current state-of-the-art multi-object tracking techniques across multiple evaluation metrics. The code can be available on https://github.com/wangtuo111/OAMOT.

It is essential for them to accurately and efficiently estimate both their own pose and the poses of surrounding dynamic objects in real time during autonomous exploration. Although most existing approaches achieve accurate self-localization or object detection and tracking individually, there is limited research on effectively integrating these two tasks, despite their inherently coupled and interactive nature. To address this challenge, this paper proposes a multi-object tracking and trajectory optimization framework based on enhanced data association and a two-stage optimization strategy, enabling robust localization of both the ego-agent and surrounding dynamic objects in highly dynamic urban environments. In the front-end, we incorporate an improved data association algorithm into a multi-object tracking pipeline and combine it with a GICP-based fast registration method to achieve efficient dynamic tracking. In the back-end, based on the front-end tracking results, we model the states of moving objects as dynamic tracking factors and integrate them into a global factor graph. This allows for joint optimization of the ego-pose and the poses of dynamic objects, resulting in high-precision multi-object tracking. Extensive comparative experiments on public high-dynamic datasets demonstrate the state-of-the-art performance and advantages of the proposed method in multi-object tracking and self-localization, where the accuracy of self-localization and tracking has increased by 19.7% and 3%, respectively, compared with the most advanced method.

Autonomous driving is conducted in complex scenarios, which requires to detect 3D objects in real time scenarios as well as accurately track these 3D objects in order to get such information as location, size, trajectory, velocity. MOT (Multi-Object Tracking) performance is heavily dependent on object detection. Once object detection gives false alarms or missing alarms, the multi-object tracking would be automatically influenced. In this paper, we propose a coupling system which combines 3D object detection and multi-object tracking into one framework. We use the tracked objects as a reference in 3D object detection, in order to locate objects, reduce false or missing alarms in a single frame, and weaken the impact of false and missing alarms on the tracking quality. Our method is evaluated on kitti dataset and is proved effective.

A robust method for multi object tracking in autonomous ship navigation systems

In recent years, deep learning has made dramatic advances in computer vision field, especially in improving the performance of object detection as well as instance semantic segmentation. Still, multi-object tracking (MOT) remains a very challenging issue. Even in state-of-the-art deep learning-based object detectors, a preferred paradigm for MOT: tracking-by-detection, can only slightly improve the tracking performance. Pixel-level information is considered more precise and useful for tracking performance improvement than using conventional information, such as foreground or background content in a bounding box. However, the performance of current state-of-the-art models for automatically annotating pixel-level information is still far from the expectation of human beings. Therefore, we shall explore how multi-object tracking and segmentation (MOTS) is affected when the information obtained after applying instance semantic segmentation is incomplete. We propose a mask-guided two-streamed augmentation learning (MGTSAL) algorithm, which can be applied to TrackR-CNN to alleviate significant drop of MOTS performance when encountering incompletely segmented information. We evaluate the proposed approach on MOTS KITTI dataset, and our approach outperforms the baseline model TrackR-CNN in all our experimental settings. The promising experimental results and ablation study validate the effectiveness of the proposed approach.

The multi-object tracking (MOT) algorithms of the joint detection and embedding (JDE) approach estimate bounding boxes and re-identification (re-ID) features of objects with the single network, which balance the tracking accuracy and inference speed. However, when the appearance information between different objects is highly similar, these algorithms are usually easy to cause identity switches, and the comprehensive tracking performance is poor in crowded scenes. Aiming at the above problems, we propose a stronger multi-object tracking algorithm termed as ReIMOT, based on FairMOT. A joint loss function of combining normalized Softmax Loss and the center distance penalty term is designed to supervise the re-ID branch, which increases the intra-class similarity and makes the extracted appearance features more discriminative. To further improve the tracking performance, we introduce coordinate attention to make the encoder-decoder network focus more on features of interest. The experimental results show that the proposed ReIMOT is more effective than the other advanced multi-object tracking algorithms, and decreases the number of ID switches by 13.8% compared to FairMOT on the MOT17 dataset.

Multi-object tracking (MOT) is a crucial task in computer vision, enabling the tracking of multiple objects across frames in video sequences while preserving their identities. This problem is fundamental in applications like autonomous driving, video surveillance, and robotic systems. This paper provides a concise overview of the current state of MOT, exploring key advancements and methods developed over recent years. We focus on the widely adopted tracking-by-detection paradigm, where object detection is followed by the association of detections across frames. Modern MOT systems, such as ByteTrack, BoT-SORT, and the integration of advanced deep learning models like YOLO, have significantly improved tracking performance by addressing challenges like occlusion, motion blur, and identity switching. Additionally, the integration of real-time systems, such as UAV-based MOT, utilizing big data technologies, highlights the practical scalability of these methods. We present a survey of these systems, examine their limitations, and suggest potential directions for future research.

Object detection and data association are critical components in multi-object tracking (MOT) systems. Despite the fact that the two components are dependent on each other, prior works often design detection and data association modules separately which are trained with separate objectives. As a result, one cannot back-propagate the gradients and optimize the entire MOT system, which leads to sub-optimal performance. To address this issue, recent works simultaneously optimize detection and data association modules under a joint MOT framework, which has shown improved performance in both modules. In this work, we propose a new instance of joint MOT approach based on Graph Neural Networks (GNNs). The key idea is that GNNs can model relations between variablesized objects in both the spatial and temporal domains, which is essential for learning discriminative features for detection and data association. Through extensive experiments on the MOT15/16/17/20 datasets, we demonstrate the effectiveness of our GNN-based joint MOT approach and show state-of-the-art performance for both detection and MOT tasks.

The human brain can effortlessly perform vision processes using the visual system, which helps solve multi-object tracking (MOT) problems. However, few algorithms simulate human strategies for solving MOT. Therefore, devising a method that simulates human activity in vision has become a good choice for improving MOT results, especially occlusion. Eight brain strategies have been studied from a cognitive perspective and imitated to build a novel algorithm. Two of these strategies gave our algorithm novel and outstanding results, rescuing saccades and stimulus attributes. First, rescue saccades were imitated by detecting the occlusion state in each frame, representing the critical situation that the human brain saccades toward. Then, stimulus attributes were mimicked by using semantic attributes to reidentify the person in these occlusion states. Our algorithm favourably performs on the MOT17 dataset compared to state-of-the-art trackers. In addition, we created a new dataset of 40,000 images, 190,000 annotations and 4 classes to train the detection model to detect occlusion and semantic attributes. The experimental results demonstrate that our new dataset achieves an outstanding performance on the scaled YOLOv4 detection model by achieving a 0.89 mAP 0.5.

Multi-object model-free tracking is challenging because the tracker is not aware of the objects' type (not allowed to use object detectors), and needs to distinguish one object from background as well as other similar objects. Most existing methods keep updating their appearance model individually for each target, and their performance is hampered by sudden appearance change and/or occlusion. We propose to use both appearance model and motion model to overcome this issue. We introduce an indicator variable to predict sudden appearance change and occlusion. When they happen, our model stops updating the appearance model to avoid parameter update based on background or incorrect object, and rely more on motion model to track. Moreover, we consider the correlation among all targets, and seek the joint optimal locations for all target simultaneously. We formulate the problem of finding the most likely locations jointly as a graphical model inference problem, and learn the joint parameters for both appearance model and motion model in an online fashion in the framework of LaRank. Experiment results show that our method outperforms the state-of-the-art.

Multi-object tracking (MOT) is a crucial technology for security surveillance, which is computationally intensive due to the requirement of processing a large number of video streams within low latency in practice. The input video streams of MOT are processed on a cloud computing center with abundant computational capability, posing heavy pressures on delivering video streams to the cloud. Recent advances in the Internet of Things (IoT) technology provide edge-computing-based solutions for video analytics at scale. However, the gap between MOT’s high computational capability demand and IoT devices’ resource-constrained nature remains significant. In this paper, a resource-efficient multi-object tracking method (REMOT) is proposed for real-time surveillance on IoT embedded devices, including an affinity measurement based on an appearance model with angular triplet loss and a motion association that substitutes the time-consuming graph-based data association stage. Considering the trade-off between latency and accuracy, we design an optimization strategy on the parallel processing of deep learning models’ layers to accelerate the inference speed with less accuracy loss. Besides, we employ a model compression strategy for model size reduction. Experiments on MOT16 and MOT17 benchmarks demonstrate that REMOT reduces 2.4x latency compared with the original implementation and achieves a running speed of 81 frames per second (fps) on an embedded device with only a marginal accuracy loss (6%), which meets the requirements of real-time processing and low-latency response for surveillance.

3D Multi-object tracking (MOT) is crucial to autonomous systems. Recent work uses a standard tracking-by-detection pipeline, where feature extraction is first performed independently for each object in order to compute an affinity matrix. Then the affinity matrix is passed to the Hungarian algorithm for data association. A key process of this standard pipeline is to learn discriminative features for different objects in order to reduce confusion during data association. In this work, we propose two techniques to improve the discriminative feature learning for MOT: (1) instead of obtaining features for each object independently, we propose a novel feature interaction mechanism by introducing the Graph Neural Network. As a result, the feature of one object is informed of the features of other objects so that the object feature can lean towards the object with similar feature (i.e., object probably with a same ID) and deviate from objects with dissimilar features (i.e., object probably with different IDs), leading to a more discriminative feature for each object; (2) instead of obtaining the feature from either 2D or 3D space in prior work, we propose a novel joint feature extractor to learn appearance and motion features from 2D and 3D space simultaneously. As features from different modalities often have complementary information, the joint feature can be more discriminate than feature from each individual modality. To ensure that the joint feature extractor does not heavily rely on one modality, we also propose an ensemble training paradigm. Through extensive evaluation, our proposed method achieves state-of-the-art performance on KITTI and nuScenes 3D MOT benchmarks. Our code will be made available at https://github.com/xinshuoweng/GNN3DMOT

Multi-object tracking (MOT) in satellite videos is a challenging task due to the small size and blurry features of objects, which often lead to intermittent detection and tracking instability. Many existing object detection and tracking models often struggle with these issues, as they are not designed to effectively handle the unique characteristics of satellite videos. To address these challenges, we propose LocaLock, a joint detection and tracking framework for MOT that incorporates feature matching concepts from single object tracking (SOT) to enhance tracking stability and reduce intermittent tracking results. Specifically, LocaLock utilizes an anchor-free detection backbone for efficiency and employs a local cost volume (LCV) module to perform precise feature matching in the local area. This provides valuable object priors to the detection head, enabling the model to “lock” onto objects with greater accuracy and mitigate the instability associated with small object detection. Additionally, the local computation within the LCV module ensures low computational complexity and memory usage. Furthermore, LocaLock incorporates a novel motion flow (MoF) module to accumulate and exploit temporal information, further enhancing feature robustness and consistency across frames. Rigorous evaluations on the VISO dataset demonstrate the superior performance of LocaLock, surpassing existing methods in tracking accuracy and precision within the demanding satellite video analysis domain. Notably, LocaLock achieved state-of-the-art performance on the VISO benchmark, achieving a multi-object tracking accuracy (MOTA) of 62.6 while ensuring fast running speed.

Football video content analysis is a rapidly evolving field aiming to enrich the viewing experience of football matches. Current research often focuses on specific tasks like player and/or ball detection, tracking, and localisation in top-down views. Our study strives to integrate these efforts into a comprehensive Multi-Object Tracking (MOT) model capable of handling perspective transformations. Our framework, FootyVision, employs a YOLOv7 backbone trained on an extended player and ball dataset. The MOT module builds a gallery and assigns identities via the Hungarian algorithm based on feature embeddings, bounding box intersection over union, distance, and velocity. A novel component of our model is the perspective transformation module that leverages activation maps from the YOLOv7 backbone to compute homographies using lines, intersection points, and ellipses. This method effectively adapts to dynamic and uncalibrated video data, even in viewpoints with limited visual information. In terms of performance, FootyVision sets new benchmarks. The model achieves a mean average precision (mAP) of 95.7% and an F1-score of 95.5% in object detection. For MOT, it demonstrates robust capabilities, with an IDF1 score of approximately 93% on both ISSIA and SoccerNet datasets. For SoccerNet, it reaches a MOTA of 94.04% and shows competitive results for ISSIA. Additionally, FootyVision scores a HOTA(0) of 93.1% and an overall HOTA of 72.16% for the SoccerNet dataset. Our ablation study confirms the effectiveness of the selected tracking features and identifies key attributes for further improvement. While the model excels in maintaining track accuracy throughout the testing dataset, we recognise the potential to enhance spatial-location accuracy.

Drones, or general UAVs, equipped with cameras have been fast deployed with a wide range of applications, including agriculture, aerial photography, and surveillance. Consequently, automatic understanding of visual data collected from drones becomes highly demanding, bringing computer vision and drones more and more closely. To promote and track the developments of object detection and tracking algorithms, we have organized three challenge workshops in conjunction with ECCV 2018, ICCV 2019 and ECCV 2020, attracting more than 100 teams around the world. We provide a large-scale drone captured dataset, VisDrone, which includes four tracks, i.e., (1) image object detection, (2) video object detection, (3) single object tracking, and (4) multi-object tracking. In this paper, we first present a thorough review of object detection and tracking datasets and benchmarks, and discuss the challenges of collecting large-scale drone-based object detection and tracking datasets with fully manual annotations. After that, we describe our VisDrone dataset, which is captured over various urban/suburban areas of 14 different cities across China from North to South. Being the largest such dataset ever published, VisDrone enables extensive evaluation and investigation of visual analysis algorithms for the drone platform. We provide a detailed analysis of the current state of the field of large-scale object detection and tracking on drones, and conclude the challenge as well as propose future directions. We expect the benchmark largely boost the research and development in video analysis on drone platforms. All the datasets and experimental results can be downloaded from https://github.com/VisDrone/VisDrone-Dataset.