Multi-Object tracking using multi-channel part appearance representation

Multi-object video segmentation and multi-object tracking are very similar in the aspect that both determine the locations and maintain the identities of the objects of interest (targets) in each frame of the video. Our approach takes advantage of this fact and uses the strengths of one task to improve the accuracy of the other. In our framework, the multi-object tracking and segmentation modules initially produce results on our dataset independently. The tracking module enforces higher-order smoothness constraints on the object trajectories and uses Lagrangian relaxation to get an iterative solution method. The segmentation module forms superpixels through clustering, trains a linear SVM using Lab color to obtain the foreground and background segmentation and assigns ID labels based on color and optical flow. The results of these two modules are then jointly processed and updated. The locations of the tracking bounding boxes are refined with the help of the segmentation results, so that they are more precisely centered on the targets. The tracking module is more accurate in terms of ID assignment and hence, its results are used to correct errors in ID labeling in the segmentation module. Both modules identify and add any target detections they initially missed to their results using the results of the other component. Hence, this joint processing increases the accuracy of both the tracking and the segmentation results as can be seen from our experimental results. Our approach is comparable to state-of-the-art tracking and segmentation techniques.

A robust method for multi object tracking in autonomous ship navigation systems

This paper presents a high-accuracy and robust multi-object tracking method for maritime vessel detection and tracking in complex marine environments, characterized by dense targets, large-scale variations, and frequent occlusions. The proposed approach adopts an enhanced YOLOv8-based detector with lightweight feature enhancement and attention mechanisms to improve its capability in detecting small-scale vessels and complex scenes. Furthermore, a tracking framework combining BOTSORT with an OSNet-based re-identification (ReID) model is employed to achieve stable and reliable vessel association. Experimental results on the Near-Infrared On-Shore (NIR) dataset demonstrate that the proposed method improves Precision, Recall, mAP@0.5, and mAP@0.5:0.95 by approximately 4.0%, 5.0%, 5.1%, and 5.4%, respectively, compared with the baseline YOLOv8, while reducing parameter count and model size by 11.6% and 6.5%. On the Visible On-Shore (VIS) dataset, the proposed method outperforms state-of-the-art approaches in detection accuracy and robustness, further validating its effectiveness and cross-modal generalization capability. In multi-object tracking tasks, the proposed CPM-YOLO and BOTSORT framework demonstrates clear advantages in trajectory continuity, occlusion handling, and identity preservation compared with mainstream tracking algorithms. On the NIR dataset, the proposed method achieves a competitive inference speed of 188 FPS, while running at 187 FPS on the VIS dataset, demonstrating that the accuracy improvements are achieved without sacrificing real-time performance. Overall, the proposed method achieves a favorable balance between detection accuracy, tracking robustness, and model efficiency, making it well-suited for practical maritime applications.

Most multi-object tracking algorithms are developed within the tracking-by-detection framework that consider the pairwise appearance similarities between detection responses or tracklets within a limited temporal window, and thus less effective in handling long-term occlusions or distinguishing spatially close targets with similar appearance in crowded scenes. In this work, we propose an algorithm that formulates the multi-object tracking task as one to exploit hierarchical dense structures on an undirected hypergraph constructed based on tracklet affinity. The dense structures indicate a group of vertices that are inter-connected with a set of hyperedges with high affinity values. The appearance and motion similarities among multiple tracklets across the spatio-temporal domain are considered globally by exploiting high-order similarities rather than pairwise ones, thereby facilitating distinguish spatially close targets with similar appearance. In addition, the hierarchical design of the optimization process helps the proposed tracking algorithm handle long-term occlusions robustly. Extensive experiments on various challenging datasets of both multi-pedestrian and multi-face tracking tasks, demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods.

It is essential for them to accurately and efficiently estimate both their own pose and the poses of surrounding dynamic objects in real time during autonomous exploration. Although most existing approaches achieve accurate self-localization or object detection and tracking individually, there is limited research on effectively integrating these two tasks, despite their inherently coupled and interactive nature. To address this challenge, this paper proposes a multi-object tracking and trajectory optimization framework based on enhanced data association and a two-stage optimization strategy, enabling robust localization of both the ego-agent and surrounding dynamic objects in highly dynamic urban environments. In the front-end, we incorporate an improved data association algorithm into a multi-object tracking pipeline and combine it with a GICP-based fast registration method to achieve efficient dynamic tracking. In the back-end, based on the front-end tracking results, we model the states of moving objects as dynamic tracking factors and integrate them into a global factor graph. This allows for joint optimization of the ego-pose and the poses of dynamic objects, resulting in high-precision multi-object tracking. Extensive comparative experiments on public high-dynamic datasets demonstrate the state-of-the-art performance and advantages of the proposed method in multi-object tracking and self-localization, where the accuracy of self-localization and tracking has increased by 19.7% and 3%, respectively, compared with the most advanced method.

In recent years, deep learning has made dramatic advances in computer vision field, especially in improving the performance of object detection as well as instance semantic segmentation. Still, multi-object tracking (MOT) remains a very challenging issue. Even in state-of-the-art deep learning-based object detectors, a preferred paradigm for MOT: tracking-by-detection, can only slightly improve the tracking performance. Pixel-level information is considered more precise and useful for tracking performance improvement than using conventional information, such as foreground or background content in a bounding box. However, the performance of current state-of-the-art models for automatically annotating pixel-level information is still far from the expectation of human beings. Therefore, we shall explore how multi-object tracking and segmentation (MOTS) is affected when the information obtained after applying instance semantic segmentation is incomplete. We propose a mask-guided two-streamed augmentation learning (MGTSAL) algorithm, which can be applied to TrackR-CNN to alleviate significant drop of MOTS performance when encountering incompletely segmented information. We evaluate the proposed approach on MOTS KITTI dataset, and our approach outperforms the baseline model TrackR-CNN in all our experimental settings. The promising experimental results and ablation study validate the effectiveness of the proposed approach.

DCTracker: Rethinking MOT in soccer events under dual views via cascade association

This paper presents a comparative analysis of state-of-the-art multi-object tracking algorithms applied in UAV-based video surveillance systems. The performance results of three advanced tracking methods – DeepSORT, ByteTrack, and StrongSORT – integrated with the YOLOv8 object detector are presented. A mathematical description and experimental simulations were conducted to evaluate the accuracy, stability, and computational performance of the algorithms in dynamic and complex scenes. The obtained results indicate that the StrongSORT + YOLOv8 combination provides the best balance between accuracy and robustness, whereas the ByteTrack method demonstrates high track continuity in high-density environments. The proposed approach can be utilized to enhance the efficiency of UAV-based autonomous monitoring systems.

To adjust management practices and improve wine marketing strategies, accurate vineyard yield estimation early in the growing season is essential. Conventional methods for yield forecasting rely on phenotypic features' manual assessment, which is time- and labor-intensive and often destructive. We combined a deep object segmentation method, mask region-based convolutional neural network (Mask R-CNN), with two potential multi-object tracking algorithms, simple online and real-time tracking (SORT) and intersection-over-union (IOU) trackers to develop a complete visual system that can automatically detect and track individual inflorescences, enabling the assessment of the number of inflorescences per vineyard row from vineyard video footage. The performance of the two tracking algorithms was evaluated using our vineyard dataset, which is more challenging than conventional tracking benchmark datasets owing to environmental factors. Our evaluation dataset consists of videos of four vineyard rows, including 221 vines that were automatically acquired under unprepared field conditions. We tracked individual inflorescences across video image frames with a 92.1% multi-object tracking accuracy (MOTA) and an 89.6% identity F1 score (IDF1). This allowed us to estimate inflorescence count per vineyard row with a 0.91 coefficient of determination (R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) between the estimated count and manual-annotated ground truth count. The impact of leaf occlusions on inflorescence visibility was lessened by processing multiple successive image frames with minimal displacements to construct multiple camera views. This study demonstrates the use of deep learning and multi-object tracking in creating a low-cost (requiring only an RGB camera), high-throughput phenotyping system for precision viticulture.

Multi-object tracking has been a difficult problem in recent years, especially in complex scenes such as player tracking in sports videos. Player movements are often complex and abrupt. In this paper, we focus on the problem of tracking multiple players in beach volleyball videos. To handle the difficulties of player tracking, we follow the popular tracking-by-detection framework in multi-object tracking and adopt the multiple hypotheses tracking (MHT) algorithm to solve the data association problem. To improve the efficiency of the MHT, we use motion information from Kalman filter and train an online appearance model of each track hypothesis. An auxiliary particle filter method is adopted to handle the missing detection problem. Furthermore, we obtain the significant performance on our beach volleyball datasets, which demonstrate the effectiveness and efficiency of the proposed method.

Pig husbandry constitutes a significant segment within the broader framework of livestock farming, with porcine well-being emerging as a paramount concern due to its direct implications on pig breeding and production. An easily observable proxy for assessing the health of pigs lies in their daily patterns of movement. The daily movement patterns of pigs can be used as an indicator of their health, in which more active pigs are usually healthier than those who are not active, providing farmers with knowledge of identifying pigs’ health state before they become sick or their condition becomes life-threatening. However, the conventional means of estimating pig mobility largely rely on manual observations by farmers, which is impractical in the context of contemporary centralized and extensive pig farming operations. In response to these challenges, multi-object tracking and pig behavior methods are adopted to monitor pig health and welfare closely. Regrettably, these existing methods frequently fall short of providing precise and quantified measurements of movement distance, thereby yielding a rudimentary metric for assessing pig health. This paper proposes a novel approach that integrates optical flow and a multi-object tracking algorithm to more accurately gauge pig movement based on both qualitative and quantitative analyses of the shortcomings of solely relying on tracking algorithms. The optical flow records accurate movement between two consecutive frames and the multi-object tracking algorithm offers individual tracks for each pig. By combining optical flow and the tracking algorithm, our approach can accurately estimate each pig’s movement. Moreover, the incorporation of optical flow affords the capacity to discern partial movements, such as instances where only the pig’s head is in motion while the remainder of its body remains stationary. The experimental results show that the proposed method has superiority over the method of solely using tracking results, i.e., bounding boxes. The reason is that the movement calculated based on bounding boxes is easily affected by the size fluctuation while the optical flow data can avoid these drawbacks and even provide more fine-grained motion information. The virtues inherent in the proposed method culminate in the provision of more accurate and comprehensive information, thus enhancing the efficacy of decision-making and management processes within the realm of pig farming.

Multi-object tracking aims to estimate the complete trajectories of objects in a scene. Distinguishing among objects efficiently and correctly in complex environments is a challenging problem. In this paper, a Siamese network with an auto-encoding constraint is proposed to extract discriminative features from detection responses in a tracking-by-detection framework. Different from recent deep learning methods, the simple two layers stacked auto-encoder structure enables the Siamese network to operate efficiently only with small-scale online sample data. The auto-encoding constraint reduces the possibility of overfitting during small-scale sample training. Then, the proposed Siamese network is improved to extract the previous-appearance-next vector from tracklet for better association. The new feature integrates the appearance, previous, and next stage motions of an element in a tracklet. With the new features, an online incremental learned tracking framework is established. It contains reliable tracklet generation, data association to generate complete object trajectories, and tracklet growth to deal with missing detections and to enhance the new feature for tracklet. Benefiting from discriminative features, the final trajectories of objects can be achieved by an efficient iterative greedy algorithm. Feature experiments show that the proposed Siamese network has advantages in terms of both discrimination and correctness. The system experiments show the improved tracking performance of the proposed method.

Events cameras are ideal sensors for enabling robots to detect and track objects in highly dynamic environments due to their low latency output, high temporal resolution, and high dynamic range. In this paper, we present the Asynchronous Event Multi-Object Tracking (AEMOT) algorithm for detecting and tracking multiple objects by processing individual raw events asynchronously. AEMOT detects salient event blob features by identifying regions of consistent optical flow using a novel Field of Active Flow Directions built from the Surface of Active Events. Detected features are tracked as candidate objects using the recently proposed Asynchronous Event Blob (AEB) tracker in order to construct small intensity patches of each candidate object. A novel learnt validation stage promotes or discards candidate objects based on classification of their intensity patches, with promoted objects having their position, velocity, size, and orientation estimated at their event rate. We evaluate AEMOT on a new Bee Swarm Dataset, where it tracks dozens of small bees with precision and recall performance exceeding that of alternative event-based detection and tracking algorithms by over 37%. Source code and the labelled event Bee Swarm Dataset will be open sourced. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>https://github.com/angus-apps/AEMOT

Occlusion, a pervasive challenge in Multi-Object Tracking (MOT) within complex scenes, severely degrades tracking performance. Current methods still face numerous problems when handling occlusions. Motion prediction struggles to accommodate diverse motion patterns, resulting in failure during short-term occlusions. Concurrently, appearance features possess insufficient discriminative power under occluded conditions, which frequently leads to identity switches following long-term occlusion. To enhance the performance of MOT under such challenging conditions, we propose an innovative Occlusion-Aware Multi-Object Tracking via Joint Diffusion Motion Prediction and Appearance Purification (OAMOT). For short-term occlusions, a Diffusion Motion Recovery Model (DMRM) is developed, which integrates residual and noise diffusion branches to recover trajectories precisely under varied motion patterns. For long-term occlusions, the SAM Appearance Purification for ReID (SAPR) module is proposed; the module leverages the mask generation mechanism of SAM as foreground attention to enhance feature discriminability. Furthermore, a Lightweight Attention Predictor (LAP) is integrated into the ReID network to achieve SAM-quality attention during inference without significant computational overhead. Experimental results on the MOT17, MOT20, and DanceTrack datasets demonstrate that the proposed OAMOT method outperforms current state-of-the-art multi-object tracking techniques across multiple evaluation metrics. The code can be available on https://github.com/wangtuo111/OAMOT.

Speed skating serves as a significant application domain for multiobject tracking (MOT), presenting unique challenges such as frequent occlusion, highly similar appearances, and motion blur. To address these challenges, this paper constructs an MOT dataset called SKT-MOT for speed skating and analyzes the shortcomings of existing datasets and methods. Accordingly, we propose a dynamic MOT method called DyTracker. The method builds upon the DeepSORT baseline and enhances three key modules. At the global level, we design the track dynamic management (TDM) algorithm. In the motion branch, a novel metric is proposed to evaluate occlusion and Kalman filter dynamic update (KFDU) is implemented. In the appearance branch, we account for the difference in human posture and propose the feature dynamic selection and updating (FDSU) strategy. This makes our DyTracker flexible and efficient to achieve a multiobject tracking accuracy (MOTA) of 93.70% and identification F1 (IDF1) score of 92.39% on SKT-MOT, which is a significant advantage over existing SOTA methods. To validate the generalization of our proposed module, two dynamic update modules are inserted into other methods and validated on the public dataset MOT17, and the accuracy is generally improved by 0.2%–0.6%.