NTrack: A Multiple-Object Tracker and Dataset for Infield Cotton Boll Counting

Multiple object tracking (MOT) in videos captured by unmanned aerial vehicle (UAV) is a fundamental aspect of computer vision. Recently, the one-shot tracking paradigm integrates the detection and re-identification (ReID) tasks, striking a balance between tracking accuracy and inference speed. This paradigm alleviates task conflicts and achieves remarkable results through various feature decoupling methods. However, in challenging scenarios like drone movements, lighting changes and object occlusion, it still encounters issues with detection failures and identity switches. In addition, traditional feature decoupling methods directly employ channel-based attention to decompose the detection and ReID branches, without a meticulous consideration of the specific requirements of each branch. To address the above problems, we introduce an asymmetric feature enhancement network with a global coordinate-aware enhancement (GCAE) module and an embedding feature aggregation (EFA) module, aiming to optimize the two branches independently. On the one hand, we develop the GCAE module for the detection branch, which effectively merges rich semantic information within the feature space to improve detection accuracy. On the other hand, we introduce the EFA module for the ReID branch, which highlights the significance of pixel-level features and acquires discriminative identity embedding through a local feature aggregation strategy. By efficiently incorporating the GCAE and EFA modules into the one-shot tracking pipeline, we present a novel MOT framework, named AsyUAV. Extensive experiments have demonstrated the effectiveness of our proposed AsyUAV. In particular, it achieves a MOTA of 38.3% and IDF1 of 51.7% on VisDrone2019, and a MOTA of 48.0% and IDF1 of 67.5% on UAVDT, outperforming existing state-of-the-art trackers.

With the increasing popularity of Unmanned Aerial Vehicles (UAVs) in computer vision-related applications, intelligent UAV video analysis has recently attracted the attention of an increasing number of researchers. To facilitate research in the UAV field, this paper presents a UAV dataset with 100 videos featuring approximately 2700 vehicles recorded under unconstrained conditions and 840k manually annotated bounding boxes. These UAV videos were recorded in complex real-world scenarios and pose significant new challenges, such as complex scenes, high density, small objects, and large camera motion, to the existing object detection and tracking methods. These challenges have encouraged us to define a benchmark for three fundamental computer vision tasks, namely, object detection, single object tracking (SOT) and multiple object tracking (MOT), on our UAV dataset. Specifically, our UAV benchmark facilitates evaluation and detailed analysis of state-of-the-art detection and tracking methods on the proposed UAV dataset. Furthermore, we propose a novel approach based on the so-called Context-aware Multi-task Siamese Network (CMSN) model that explores new cues in UAV videos by judging the consistency degree between objects and contexts and that can be used for SOT and MOT. The experimental results demonstrate that our model could make tracking results more robust in both SOT and MOT, showing that the current tracking and detection methods have limitations in dealing with the proposed UAV benchmark and that further research is indeed needed.

In the tracking‐by‐detection approach of online multiple object tracking (MOT), a major challenge is how to associate object detections on the new video frame with previously tracked objects. Two important aspects that directly influence the performance of MOT are quality of detection and accuracy in data association. The authors propose an efficient and unified MOT framework for improved object detection, followed by enhanced object tracking. The object detection and tracking are considered as two independent functions in the tracking‐by‐detection paradigm. In this study, object detection accuracy has been increased by employing a faster region‐based convolutional neural network (Faster R‐CNN) modified with the feedback region proposals from the tracker. Target association is performed by the correlation filter‐based Siamese CNN model, which finds the similarity score between the input image patches. The Siamese CNN is trained using a supervised hard sample mining strategy. An optical flow‐based motion model is employed to predict the next probable location of the targets from the tracker and these region proposals are fed back to the classifier module of Faster R‐CNN. The authors’ extensive analysis of publicly available MOT benchmark datasets and comparison with the state‐of‐the‐art tracking methods demonstrate competitive tracking performance of the proposed MOT framework.

We present a novel framework for multiple object tracking in which the problems of object detection and data association are expressed by a single objective function. The framework follows the Lagrange dual decomposition strategy, taking advantage of the often complementary nature of the two subproblems. Our coupling formulation avoids the problem of error propagation from which traditional “detection-tracking approaches” to multiple object tracking suffer. We also eschew common heuristics such as “nonmaximum suppression” of hypotheses by modeling the joint image likelihood as opposed to applying independent likelihood assumptions. Our coupling algorithm is guaranteed to converge and can handle partial or even complete occlusions. Furthermore, our method does not have any severe scalability issues but can process hundreds of frames at the same time. Our experiments involve challenging, notably distinct datasets and demonstrate that our method can achieve results comparable to those of state-of-art approaches, even without a heavily trained object detector.

ReMOT: A model-agnostic refinement for multiple object tracking

This study aimed to assess the impact of manipulating postural stability during the execution of the multiple objects tracking (MOT) task on gaze behaviour. Twenty-eight males performed the MOT task under three levels of postural stability while eye movements were recorded. Postural stability had a significant impact on MOT scores and gaze behaviour. Participants exhibited worse MOT performance (p < 0.001, η² = 0.20) in less stable conditions, showing a reduced fixation duration (p = 0.020, η² = 0.09) and increased saccade frequency (p = 0.013, η² = 0.10). As stability conditions decreased, participants shifted their gaze strategies by an average of 12.34% of the time distribution from maintaining fixation on a central location to adopting the target-looking strategy. Additionally, the decline in MOT scores with decreasing stability was correlated with a reduction in fixation duration (r = 0.409, p = 0.031). The findings evidence that gaze behaviour during MOT is dependent on postural stability and MOT performance is influenced by the chosen gaze strategy. Key points Reduced postural stability significantly impairs the ability to track multiple objects. Changes in postural stability during a multiple object tracking task affect gaze behaviour by alterations in fixation duration and saccadic frequency. Decrease in multiple object tracking task scores is associated with alterations in gaze behaviour, specifically a reduction in fixation duration. Participants shifted their gaze strategies from maintaining fixation on a central location to adopting a saccadic target-looking strategy as stability conditions decreased.

Traditional Multiple Object Tracking (MOT) methods in satellite videos mostly follow the Detection-Based Tracking (DBT) framework. However, the DBT framework assumes that all objects are correctly recognized and localized by the detector. In practice, the low resolution of satellite videos, small objects, and complex backgrounds inevitably leads to a decline in detector performance. To alleviate the impact of detector degradation on track, we propose Coarse-Fine Tracker, a framework that integrates the MOT framework with the Tracking Any Point (TAP) method CoTracker for the first time, leveraging TAP’s persistent point correspondence modeling to compensate for detector failures. In our Coarse-Fine Tracker, we divide the satellite video into sub-videos. For one sub-video, we first use ByteTrack to track the outputs of the detector, referred to as coarse tracking, which involves the Kalman filter and box-level motion features. Given the small size of objects in satellite videos, we treat each object as a point to be tracked. We then use CoTracker to track the center point of each object, referred to as fine tracking, by calculating the appearance feature similarity between each point and its neighboring points. Finally, the Consensus Fusion Strategy eliminates mismatched detections in coarse tracking results by checking their geometric consistency against fine tracking results and recovers missed objects via linear interpolation or linear fitting. This method is validated on the VISO and SAT-MTB datasets. Experimental results in VISO show that the tracker achieves a multi-object tracking accuracy (MOTA) of 66.9, a multi-object tracking precision (MOTP) of 64.1, and an IDF1 score of 77.8, surpassing the detector-only baseline by 11.1% in MOTA while reducing ID switches by 139. Comparative experiments with ByteTrack demonstrate the robustness of our tracking method when the performance of the detector deteriorates.

Introduction The ability to simultaneously track numerous moving objects in the presence of irrelevant stimuli is essential for successfully carrying out a variety of tasks. Sleep loss impairs neurocognitive functioning and, as a result, attentional processing capacity is reduced. The objective of the current study was to determine if performance on the multiple object tracking (MOT) task was adversely impacted by a week of chronic sleep restriction (CSR). Methods Twelve healthy participants (6 males, 6 females) kept a fixed sleep-wake schedule, with a constant waketime, at home for four weeks (actigraphy confirmed compliance). During weeks one and three, participants maintained 9 hours in bed. During weeks two and four, participants were randomly assigned to 5 and 9 hours of sleep. Following weeks two and four, participants completed a 13-hour laboratory visit under dim light (&amp;lt; 15 lux), where they maintained a constant posture and were provided with hourly isocaloric snacks. MOT was presented at approximately 6 and 8 hours after waking. Participants were required to track four, five, or six moving targets in the presence of identical distractors (always 12 total objects). Results Participants slept significantly less when assigned to 5 (M = 4.43 hours, SD = 0.33 hours), compared to 9 hours of sleep (M = 7.42 hours, SD = 0.42 hours; F (1, 22) = 206.89, p = 0.00). The proportion of correct MOT responses was significantly lower following 5 (M = 0.70, SD = 0.15) compared to 9 hours of sleep (M = 0.77, SD = 0.12; F (1, 22) = 10.29, p &amp;lt; .05). Conclusion A week of CSR adversely impacted MOT performance compared to a week of sleep satiation. These findings have implications for individuals, such as air traffic controllers and truck drivers, who must visually track multiple moving objects, often while chronically sleep deprived. Support Supported by the Force Health Protection Program of the Office of Naval Research (SAA2402925-1, Contract Award no. N0001418IP00050).

In this paper a multiple object detection, recognition, and tracking system for unmanned aerial vehicles (UAVs) has been studied. The system can be implemented on any UAVs platform, with the main requirement being that the UAV has a suitable onboard computational unit and a camera. It is intended to be used in a maritime object tracking system framework for UAVs, which enables a UAV to perform multiobject tracking and situational awareness of the sea surface, in real time, during a UAV operation. Using machine vision to automatically detect objects in the camera's image stream combined with the UAV's navigation data, the onboard computer is able to georeference each object detection to measure the location of the detected objects in a local North‐East (NE) coordinate frame. A tracking algorithm which uses a Kalman filter and a constant velocity motion model utilizes an object's position measurements, automatically found using the object detection algorithm, to track and estimate an object's position and velocity. Furthermore, a global‐nearest‐neighbor algorithm is applied for data association. This is achieved using a measure of distance that is based not only on the physical distance between an object's estimated position and the measured position, but also how similar the objects appear in the camera image. Four field tests were conducted at sea to verify the object detection and tracking system. One of the flight tests was a two‐object tracking scenario, which is also used in three scenarios with an additional two simulated objects. The tracking results demonstrate the effectiveness of using visual recognition for data association to avoid interchanging the two estimated object trajectories. Furthermore, real‐time computations performed on the gathered data show that the system is able to automatically detect and track the position and velocity of a boat. Given that the system had at least 100 georeferenced measurements of the boat's position, the position was estimated and tracked with an accuracy of 5–15 m from 400 m altitude while the boat was in the camera's field of view (FOV). The estimated speed and course would also converge to the object's true trajectories (measured by Global Positioning System, GPS) for the tested scenarios. This enables the system to track boats while they are outside the FOV of the camera for extended periods of time, with tracking results showing a drift in the boat's position estimate down to 1–5 m/min outside of the FOV of the camera.

Despite considerable similarities between multiple object tracking (MOT) and single object tracking (SOT) tasks, modern MOT methods have not benefited from the development of SOT ones to achieve satisfactory performance. The major reason for this situation is that it is inappropriate and inefficient to apply multiple SOT models directly to the MOT task, although advanced SOT methods are of the strong discriminative power and can run at fast speeds.In this paper, we propose a novel and end-to-end trainable MOT architecture that extends CenterNet by adding an SOT branch for tracking objects in parallel with the existing branch for object detection, allowing the MOT task to benefit from the strong discriminative power of SOT methods in an effective and efficient way. Unlike most existing SOT methods which learn to distinguish the target object from its local backgrounds, the added SOT branch trains a separate SOT model per target online to distinguish the target from its surrounding targets, assigning SOT models the novel discrimination. Moreover, similar to the detection branch, the SOT branch treats objects as points, making its online learning efficient even if multiple targets are processed simultaneously. Without tricks, the proposed tracker achieves MOTAs of 0.710 and 0.686, IDF1s of 0.719 and 0.714, on MOT17 and MOT20 benchmarks, respectively, while running at 16 FPS on MOT17.

A new integrated navigation system designed for small size Unmanned Aerial Vehicles (UAVs) is presented. The proposed system is based on a number of low-cost avionics sensors, including Global Navigation Satellite Systems (GNSS), Micro-Electro-Mechanical System (MEMS) based Inertial Measurement Unit (IMU) and Vision Based Sensors (VBS). The use of an Aircraft Dynamics Models (ADMs) to provide additional information to compensate for the shortcomings of Vision Based Navigation (VBN) and MEMS-IMU sensors in high-dynamics attitude determination tasks is also considered. Additionally, the research concentrates on the potential of carrier-phase GNSS for Attitude Determination (GAD) using interferometric techniques. The main objective is to design a compact, light and relatively inexpensive system capable of providing the required navigation performance (position and attitude data) in all phases of flight of small UAVs, with a special focus on precision approach and landing, where VBN techniques can be fully exploited in a multi-sensor data fusion architecture. An Extended Kalman Filter (EKF) is developed to integrate the information provided by the different sensors and to provide estimates of position, velocity and attitude of the UAV platform in real-time. Three different integrated navigation system architectures are implemented. The first architecture uses VBN at 20 Hz and GNSS at 1 Hz to augment the MEMS-IMU running at 100 Hz. The second mode also includes the ADM (computations performed at 100 Hz) to provide augmentation of the attitude channel. The third fusion architecture uses GNSS based attitude values. The simulations are carried out on the AEROSONDE UAV performing high-dynamics manoeuvres repre-sentative of the UAV operational flight envelope. Simulation of the VBN-IMU-GNSS (VIG) integrated navigation system shows that the system can attain position, velocity and attitude accuracies complying with Category Two (CAT II) precision approach requirements. Simulation of the VBN-IMU-GNSS-ADM (VIGA) system also shows promising results, since the achieved attitude accuracy is higher using the ADM-VBN-IMU than using VBN-IMU only. However, due to rapid divergence of the ADM virtual sensor, there is a need for frequent re-initialisation of the ADM data module, which is strongly dependent on the UAV flight dynamics and the specific manoeuvring transitions performed. In the simulation of the third integrated navigation system, the VIG system is augmented by employing the GAD, forming the VIG-GAD (VIGGA) system architecture. The performances achieved with the VIG, VIGA and VIGGA integrated Navigation and Guidance System (NGS) are presented and are in line with the International Civil Aviation Organization (ICAO) precision approach requirements.

Real-time object tracking for unmanned aerial vehicles (UAVs) is an essential and challenging research topic for computer vision. However, the scenarios that UAVs deal with are complicated, and the UAV tracking targets are small. Therefore, general trackers often fail to take full advantage of their performance in UAV scenarios. In this paper, we propose a tracking method for UAV scenes, which utilizes background cues and aberrances response suppression mechanism to track in 4 degrees-of-freedom. Firstly, we consider the tracking task as a similarity measurement problem. In this study, we decompose this problem into two subproblems for optimization. Secondly, to alleviate the problem of small targets in UAV scenes, we utilize background cues fully. Also, to reduce interference by background information, we employ an aberrance response suppression mechanism. Then, to obtain accurate target state information, we introduce a logarithmic polar coordinate system. We perform phase correlation calculations in logarithmic polar coordinates to obtain the rotation and scale changes of the target. Finally, target states are obtained through response fusion, which includes displacement, scale, and rotation angle. Our approach is carried out in a large number of experiments on various UAV datasets, such as UAV123, DBT70, and UAVDT2019. Compared with the current advanced trackers, our method is superior on UAV small target tracking.

End-to-end Multiple Object Tracking (MOT) frameworks integrate detection and tracking into a unified model, avoiding intermediate information loss and complicated post-processing. However, existing end-to-end MOT trackers rely on track queries of the previous frame to provide prior information. Their limited short-term temporal modeling struggle to cope with high dynamic tracking scenarios, where inter-frame target variations exhibit significant heterogeneity. To address these shortcomings, we propose a scene-perception MOT framework (SP-MOT) that encodes scene context understanding into long-term embedding and adaptively complements it with short-term cues, enabling discriminative and flexible instance representations. Specifically, SP-MOT introduces: (1) a learnable scene query that globally profiles foreground and background to capture short-term scene-level features; (2) a context understanding module to uncover long-term stable relationships across dynamic scenes based on multiple historical scene features; (3) scene-adaptive augmented decoding that leverages scene information as guidance, adaptively aggregating long-term and short-term information into object embeddings, improving the model's association ability and fault-tolerance. Extensive experiments on MOT benchmarks demonstrate that SP-MOT outperforms state-of-the-art end-to-end trackers across multiple metrics, particularly in challenging scenarios with high dynamics.

Abstract: In team sports, attention must be divided to multiple players in order to monitor their movements and to initiate the correct motor response at the right time. To solve such tasks, experienced athletes seem to make use of their peripheral vision by using a task-dependent gaze behavior (Vater, Williams, & Hossner, 2019). In a series of three experimental studies, this monitoring functionality has been investigated to propose ways for sport-specific investigations on peripheral vision. For the experimental studies, we chose the Multiple-Object Tracking (MOT) task, a task with high experimental control in which we could expect that peripheral vision would naturally be used for target monitoring (i.e., without the instruction to locate gaze on a fixation cross). The task requires one to monitor 4 initially highlighted targets moving together with 6 distractors (10 square objects in total). In this task, gaze seems predominantly not be located on individual targets, but rather on the virtual center of mass (“centroid” of the targets). Thus, it can be expected that peripheral vision is used for target monitoring. In Study 1, the validity of the MOT paradigm was tested to make sure that peripheral vision was indeed used for target monitoring as well as to examine the motion sensitivity of peripheral vision. In Study 2, we tested if this motion sensitivity holds true in dual task situations with a concurrent monitoring and detection tasks. In Study 3, we systematically manipulated visual and attentional demands to analyze the influence of these factors on gaze behavior during monitoring and change-detection performance. Considering the results of all three studies, the functionality of peripheral vision allows for the monitoring of multiple moving objects, the (simultaneous) detection of motion changes and the avoidance of saccade-related costs (because saccades would interrupt visual information processing). It could further be shown that capabilities of the visual and the attentional systems affect the gaze anchoring location and peripheral monitoring and detection performance. In a theoretical paper, we link our fundamental results to sports situations, for example, the monitoring of multiple players in soccer and the peripheral detection of a crucial player’s movement, and discuss how a sport-specific functionality could be tested in future experiments. In field- and laboratory experiments, we already tested this proposed functionality and found, for example in martial arts, that national athletes anchor their gaze on the opponent’s chest to detect attacks from arms and legs. In a beach-volleyball defense situations, experts seem to anchor their gaze between the ball and the attacking player (i.e., in free space and not on an actual visual cue). This suggests that peripheral vision is used to monitor both movements. In current studies, we make use of virtual reality applications to examine the functionality of peripheral vision with a higher degree of experimental control but at the same time a high ecological validity. Our final aim is to train expert’s peripheral vision in sport specific settings to improve visual information pick-up outside the fovea.

Walking while simultaneously engaged in another activity becomes more difficult as one grows older. Here, we address the issue of changes in dual-task behavior at different stages of life, particularly in the latter stages. We developed a dual task that combined walking along an 8-m walkway with a multiple object tracking (MOT) task of increasing difficulty. This secondary cognitive task imitates visuospatial daily activities and provides reliable quantitative measurements. Our dual-task paradigm was tested on 27 young adults (23.85 ± 2.09 years old) and two groups of older adults (18 young-old and 18 old-old adults, aged 63.89 ± 3.32 and 80.83 ± 3.84 years, respectively). Significant decrease in tracking performance with increasing complexity of the MOT task was found in all three groups. An age-related decrease in MOT and gait performance was also found. However, young-old adults performed as well as young adults under low attentional load conditions (in the MOT task and simple walking), whereas their performance was as impaired as those of old-old adults under high attentional load conditions (in the MOT task and walking under dual-task condition). These different profiles between the two groups of older participants could be explained in terms of compensation strategies and risk of falling.