Visual Object Tracking using Neural Networks for Embedded Systems

This paper considers a task of visual object tracking with application in embedded systems with limited computational power. The complexity of the task lies in finding the proper balance between algorithm result quality and its runtime efficiency. To select the tracking method to be used in embedded systems multiple machine learning approaches were analyzed: convolutional neural network-based methods, recurrent neural network-based methods, one-shot learning-based methods, siamese neural networks and transformers. The method nanotrack was selected for further research and analysis. To perform runtime performance and algorithm accuracy evaluation the test environment was developed using C++ programming language which was deployed on Raspberry Pi hardware. Performance analysis showed the method is suitable for real time object tracking on Raspberry Pi 5 as it allows to process more than 25 frames per second, additionally leaving the room for usage of a more complex backbone model if needed. Accuracy was evaluated using LaSOT dataset. The main types of challenging situations for the tracking algorithm were identified and corresponding videos from LaSOT dataset were selected. Based on visually observing tracking results on these videos, the conclusions about overall sufficient adaptive qualities of the algorithm were made. To evaluate overall algorithm quality the accuracy metric was calculated on LaSOT dataset and compared to other state-of-the-art tracking methods. Accuracy metrics were computed on LaSOT dataset, showing good performance for some object classes and worse scores for others. Based on the analysis, it is expected that training neural network on a data representing limited target set of object classes will lead to satisfactory accuracy for these classes while preserving original runtime performance. The further research directions are experimenting with different backbone models for the Nanotrack method and evaluating their influence on accuracy and runtime performance.

Transformer-based visual object tracking via fine–coarse concatenated attention and cross concatenated MLP

Visual object tracking has become one of the most active research topics in computer vision, which has been growing in commercial development as well as academic research. Many visual trackers have been proposed in the last two decades. Recent studies of computer vision for dynamic scenes include motion detection, object classification, environment modeling, tracking of moving objects, understanding of object behaviors, object identification, and data fusion from multiple sensors. This paper provides an in-depth overview of recent object tracking research. Object tracking tasks in realistic scenario often face challenging problems such as camera motion, occlusion, illumination effect, clutter, and similar appearance. A variety of tracker techniques have been published, which combine multiple techniques to solve multiple visual tracking sub-problems. This paper also reviews the latest research trend in object tracking based on convolutional neural networks, which is receiving growing attention. Finally, the paper discusses the future challenges and research directions for the object tracking problems that still need extensive studies in coming years.

Despite the extensive adoption of machine learning on the task of visual object tracking, recent learning-based approaches have largely overlooked the fact that visual tracking is a sequence-level task in its nature; they rely heavily on frame-level training, which inevitably induces inconsistency between training and testing in terms of both data distributions and task objectives. This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning and discusses how a sequence-level design of data sampling, learning objectives, and data augmentation can improve the accuracy and robustness of tracking algorithms. Our experiments on standard benchmarks including LaSOT, TrackingNet, and GOT-10k demonstrate that four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training without modifying architectures.KeywordsVisual trackingSequence-level trainingReinforcement learning

Subject of the research – visual object tracking using various image hashing algorithms for real-time tracking tasks. The goal of this study is to evaluate the tracking success and processing speed of existing and new hashing algorithms for object tracking and to identify the most suitable algorithms to be used under limited computational resources. The objectives of the research include: developing and implementing object tracking based on the aHash, dHash, pHash, mHash, LHash, and LDHash algorithms; comparing the processing speed and accuracy of these methods on the video sequences "OccludedFace2," "David," and "Sylvester"; determining the tracking success rate (TSR) and frames per second (FPS) metrics for each algorithm; analyzing the impact of the search window size, search strategy, and type of hashing on tracking quality, and providing recommendations for their use. The study also explores the trade-off between accuracy and processing speed for each algorithm considering the constraints of limited computational resources. The methods of this study involve testing and evaluating the accuracy and speed of image hashing algorithms on different test video sequences, as well as the use of metrics to determine object similarity using the Hamming distance. The results demonstrate that the aHash and mHash algorithms demonstrate the best accuracy indicators for all hash window sizes, aHash has a higher processing speed, and mHash offers better robustness to changes in lighting and object position. The dHash and pHash algorithms were less effective than the aHash and mHash algorithms due to their sensitivity to changes in scale and rotation. However, perceptual hashing-based methods, such as pHash, are more robust to contrast and blurring. Conclusions. The best hashing algorithms for real-time object-tracking tasks are aHash and mHash. This study underscores the significance of selecting suitable hashing algorithms and search strategies tailored to specific application scenarios and offers possibilities for further optimization.

There are two main trends in the development of unmanned aerial vehicle (UAV) technologies: miniaturization and intellectualization, in which realizing object tracking capabilities for a nano-scale UAV is one of the most challenging problems. In this paper, we present a visual object tracking and servoing control system utilizing a tailor-made 38 g nano-scale quadrotor. A lightweight visual module is integrated to enable object tracking capabilities, and a micro positioning deck is mounted to provide accurate pose estimation. In order to be robust against object appearance variations, a novel object tracking algorithm, denoted by RMCTer, is proposed, which integrates a powerful short-term tracking module and an efficient long-term processing module. In particular, the long-term processing module can provide additional object information and modify the short-term tracking model in a timely manner. Furthermore, a position-based visual servoing control method is proposed for the quadrotor, where an adaptive tracking controller is designed by leveraging backstepping and adaptive techniques. Stable and accurate object tracking is achieved even under disturbances. Experimental results are presented to demonstrate the high accuracy and stability of the whole tracking system.

Differences in motor control between the lower limbs may influence the risk of sports injury and recovery from rehabilitation. In this study, differences in the visual feedback ability of the left and right lower limbs were assessed using visual target tracking tasks. Thirty-four healthy young subjects (aged 20.4 ± 1.2 years) were asked to move their bodies back and forth while tracking a visual target displayed on a monitor in front of them for 30 s. The two target motions were sinusoidal (i.e., predictable patterns) and more complex (random) patterns. To assess the ability of the lower limbs to follow visual target tracking, antero-posterior CoP (right limb, CoPap-r; left limb, CoPap-l) and medio-lateral CoP (right limb, CoPml-r; left limb, CoPml-l) data were measured using a stabilometer. Tracking ability by visual feedback ability was calculated as the difference in displacement between the target signal and the trajectories of the right and left pressure centers as trapezoidal areas, and a smaller sum of area (SoA) over the entire measurement time was defined as a greater tracking ability. Regarding the SoA in the anterior-posterior CoP, the mean SoA in the sinusoidal and random tasks was significantly lower in the CoP-r data than in the CoP-l data, indicating that the right lower limb had a more remarkable ability to follow visual target tracking. Regarding the SoA in the medial-lateral direction (CoP), the mean SoA in the sinusoidal and random tasks did not significantly differ between the two legs. The right lower limb may have a tracking function activated by the target signal when responding to visual stimuli. Identifying the motor strategies of each lower limb in response to visual stimuli will not only help identify potential differences between each lower limb but also suggest the possibility of enhancing the role of each lower limb in balance control.

Recently, Gall & Lempitsky [6] and Okada [9] introduced Hough Forests (HF), which emerged as a powerful tool in object detection, tracking and several other vision applications. HFs are based on the generalized Hough transform [2] and are ensembles of randomized decision trees, consisting of both classification and regression nodes, which are trained recursively. Densly sampled patches of the target object {Pi = (Ai,yi,di)} represent the training data, where Ai is the appearance, yi the label, and di a vector pointing to the center of the object. Each node tries to find an optimal splitting function by either optimizing the information gain for classification nodes or the variance of offset vectors di for regression nodes. This yields quite clean leaf nodes according to both, appearance and offset. However, typically HFs are trained in off-line mode, which means that they assume having access to the entire training set at once. This limits their application in situations where the data arrives sequentially, e.g., in object tracking, in incremental, or large-scale learning. For all of these applications, on-line methods inherently can perform better. Thus, we propose in this paper an on-line learning scheme for Hough forests, which allows to extend their usage to further applications, such as the tracking of arbitrary target instances or large-scale learning of visual classifiers. Growing such a tree in an on-line fashion is a difficult task, as errors in the hard splitting rules cannot be corrected easily further down the tree. While Godec et al. [8] circumvent the recursive on-line update of classification trees by randomly growing the trees to their full size and just update the leaf node statistics, we integrate the ideas from [5, 10] that follow a tree-growing principle. The basic idea there is to start with a tree consisting of only one node, which is the root node and the only leaf at that time. Each node collects the data falling in it and decides on its own, based on a certain splitting criterion, whether to split this node or to further update the statistics. Although the splitting criteria in [5, 10] have strong theoretical support, we will show in the experiments that it even suffices to only count the number n of samples Pi that a node has already incorporated and split when n > γ , where γ is a predefined threshold. An overview of this procedure is given in Figure 1. This splitting criterion requires to find reasonable splitting functions with only a small subset of the data, which does not necessarily have to be a disadvantage when building random forests. As stated in Breiman [4], the upper bound for the generalization error of random forests can be optimized with a high strength of the individual trees but also a low correlation between them. To this end, we derive a new but simple splitting procedure for off-line HFs based on subsampling the input space on the node level, which can further decrease the correlation between the trees. That is, each node in a tree randomly samples a predefined number γ of data samples uniformly over all available data at the current node, which is then used for finding a good splitting function. In the first experiment, we demonstrate on three object detection data sets that both, our on-line formulation and subsample splitting scheme, can reach similar performance compared to the classical Hough forests and can even outperform them, see Figures 2(a)&(b). Additionally, during training both proposed methods are orders of magnitudes faster than the original approach (Figure 2(c)). In the second part of the experiments, we demonstrate the power of our method on visual object tracking. Especially, our focus lies on tracking objects of a priori unknown classes, as class-specific tracking with off-line forests has already been demonstrated before [7]. We present results on seven tracking data sets and show that our on-line HFs can outperform state-of-the-art tracking-by-detection methods. Figure 1: While labeled samples arrive on-line, each tree propagates the sample to the corresponding leaf node, which decides whether to split the current leaf or to update its statistics.

Object tracking is the main task in the field of computer vision. This chapter proposes and compares a new method for recognition and tracking by using CamShift and color detection. All the presented algorithms were programmed with Python programming language supported by OpenCV libraries, and executed with a credit card–size computer board called Raspberry Pi with attached external camera. A camera mimics and uses real-time digital videos for object recognition and tracking. The chapter also proposes a color detection algorithm in very low light conditions is proposed for the enhancement of target tracking. Mean shift is a nonparametric feature-space analysis technique for locating the maxima of a density function. The CamShift algorithm is dependent on an adaptation of mean shift algorithm. Histogram equalization is a nonlinear technique for adjusting the contrast of an image using its histogram.

Attention shake siamese network with auxiliary relocation branch for visual object tracking

Chapter 10 - Long-term deep object tracking

Visual object tracking technology is one of the key issues in computer vision. In this paper, we propose a visual object tracking algorithm based on cross-modality featuredeep learning using Gaussian-Bernoulli deep Boltzmann machines (DBM) with RGB-D sensors. First, a cross-modality featurelearning network based on aGaussian-Bernoulli DBM is constructed, which can extract cross-modality features of the samples in RGB-D video data. Second, the cross-modality features of the samples are input into the logistic regression classifier, andthe observation likelihood model is established according to the confidence score of the classifier. Finally, the object tracking results over RGB-D data are obtained using aBayesian maximum a posteriori (MAP) probability estimation algorithm. The experimental results show that the proposed method has strong robustness to abnormal changes (e.g., occlusion, rotation, illumination change, etc.). The algorithm can steadily track multiple targets and has higher accuracy.

Visual tracking of generic objects is one of the fundamental but challenging problems in computer vision. Here, we propose a novel fully convolutional Siamese network to solve visual tracking by directly predicting the target bounding box in an end-to-end manner. We first reformulate the visual tracking task as two subproblems: a classification problem for pixel category prediction and a regression task for object status estimation at this pixel. With this decomposition, we design a simple yet effective Siamese architecture based classification and regression framework, termed SiamCAR, which consists of two subnetworks: a Siamese subnetwork for feature extraction and a classification-regression subnetwork for direct bounding box prediction. Since the proposed framework is both proposal- and anchor-free, SiamCAR can avoid the tedious hyper-parameter tuning of anchors, considerably simplifying the training. To demonstrate that a much simpler tracking framework can achieve superior tracking results, we conduct extensive experiments and comparisons with state-of-the-art trackers on a few challenging benchmarks. Without bells and whistles, SiamCAR achieves leading performance with a real-time speed. Furthermore, the ablation study validates that the proposed framework is effective with various backbone networks, and can benefit from deeper networks. Code is available at https://github.com/ohhhyeahhh/SiamCAR.

In this paper, we present a novel visual object tracking model via multi-view semi-supervised learning. Instead of concatenating multiple views into a single view directly to adapt to conventional machine learning algorithms, the combination of views is learned by exploiting the consensus of distinct views in the entire tracking. Besides, semi-supervised learning alleviates the lack of sufficient labeled samples in the tracking task, resulting in significant improvement in generalization performance. By showing that the sample data is block-circulant, we diagonalize it with the Discrete Fourier Transform to keep the tracking at high speed. Using features extracted by the VGG-19 network and in a 1:1 ratio of the labeled samples to the unlabeled, the experiment results on the CVPR2013 Online Object Tracking Benchmark show the effectiveness of our multi-view semi-supervised tracking model.

Despite remarkable progress, visual object tracking is still a challenging task as objects usually suffer from significant appearance changes, fast motion, and serious occlusion. In this paper, we propose an anti-occlusion correlation filter-based tracking method (AO-CF) for robust visual tracking. We first propose an occlusion criterion based on continuous response values. Based on the criterion, objects are divided into four categories to adaptively identify the occlusion of objects. Then we propose a new detection condition for detecting proposals. When the occlusion criterion is triggered, the re-detection mechanism is executed and the tracker is commanded to stop, and then the re-detector selects the most reliable proposal to reinitialize the tracker. Experimental results show that our method outperforms other state-of-the-art trackers in terms of both precision rate and success rate on the widely used object tracking benchmark dataset. In addition, AO-CF is able to achieve real-time tracking speed.