Visual Tracking via Multi-view Semi-supervised Learning

Multi-view semi-supervised learning (SSL) has attracted great attention due to its effectiveness in information utilization of multiple views and labeled and unlabeled data to solve practical problems. However, most existing methods exhibit high computational complexity. Effective integration of the information on different views to achieve enhanced performance remains a challenging task. In this study, we combine an anchor-based approach with multi-view semi-supervised learning to address these problems. A novel multi-view SSL method called fast multi-view SSL (FMSSL) based on learned graph is proposed. Starting from the affinity graphs constructed by using an anchor-based strategy, FMSSL learns an optimal multi-view consensus graph by using feature and label information. The learned graph can jointly consider the relation of multiple views to approximate the manifold structure. The learned graph is then introduced into the SSL model as the weight matrix of a bipartite graph to simultaneously perform separate classification on the original samples and anchors. Accordingly, multi-view SSL can be efficiently performed, and the computational complexity can be significantly reduced. We propose an effective algorithm to optimize the objective function. Extensive experimental results on different real-world datasets demonstrate the effectiveness and efficiency of the proposed algorithm.

Multi-view multi-label learning has become a research focus for describing objects with rich expressions and annotations. However, real-world data often contains numerous unlabeled instances, due to the high cost and technical limitations of manual labeling. This crucial problem involves three main challenges: i) How to extract advanced semantics from available views? ii) How to build a refined classification framework with limited labeled space? iii) How to provide more high-quality supervisory information? To address these problems, we propose a Semi-Supervised Multi-View Multi-Label Learning Method with View-Specific Transformer and Enhanced Pseudo-Label named SMVTEP. Specifically, Generative Adversarial Networks are employed to extract informative shared and specific representations and their consistency and distinctiveness are ensured through the adversarial mechanism and information theory based contrastive learning. Then we build specific classifiers for each extracted feature and apply instance-level manifold constraints to reduce bias across classifiers. Moreover, we design a transformer-style fusion approach that simultaneously captures the imbalance of expressive power among views, mapping effects on specific labels, and label dependencies by incorporating confidence scores and category semantics into the self-attention mechanism. Furthermore, after using Mixup for data augmentation, category-enhanced pseudo-labels are leveraged to improve the reliability of additional annotations by aligning the label distribution of unlabeled samples with the true distribution. Finally, extensive experimental results validate the effectiveness of SMVTEP against state-of-the-art methods.

Contributions from the physical laboratory of the university of Pennsylvania. No. 1.—A new vertical-lantern galvanometer

Sparse representation has been exploited for visual object tracking to yield impressive performance in recent years. However, sparse models in tracking are commonly limited by two intrinsic challenges, i.e., the high computational cost that makes it impractical in real-world applications and few reliable target observations available in dictionary initialization that renders it hard to handle great target dynamics in complicated scenarios. In this paper, we propose a novel object tracking method to address these problems. Our method integrates fast ensemble proposal and sparse verification, and the two complementarily work in the following way: The target proposals are fast generated and filtered to reduce computational load and provide evidence reference for later sparse verification, and the spare model updates its dictionary dynamically with the target observations verified by fused evidences. Extensive experimental results demonstrate that the proposed method handles the aforementioned problems with impressive success and that it outperforms state-of-the-art methods in various challenging scenarios.

Object tracking is a long standing problem in vision. While great efforts have been spent to improve tracking performance, a simple yet reliable prior knowledge is left unexploited: the target object in tracking must be an object other than non-object. The recently proposed and popularized objectness measure provides a natural way to model such prior in visual tracking. Thus motivated, in this paper we propose to adapt objectness for visual object tracking. Instead of directly applying an existing objectness measure that is generic and handles various objects and environments, we adapt it to be compatible to the specific tracking sequence and object. More specifically, we use the newly proposed BING objectness as the base, and then train an object-adaptive objectness for each tracking task. The training is implemented by using an adaptive support vector machine that integrates information from the specific tracking target into the BING measure. We emphasize that the benefit of the proposed adaptive objectness, named ADOBING, is generic. To show this, we combine ADOBING with seven top performed trackers in recent evaluations. We run the ADOBING-enhanced trackers with their base trackers on two popular benchmarks, the CVPR2013 benchmark (50 sequences) and the Princeton Tracking Benchmark (100 sequences). On both benchmarks, our methods not only consistently improve the base trackers, but also achieve the best known performances. Noting that the way we integrate objectness in visual tracking is generic and straightforward, we expect even more improvement by using tracker-specific objectness.

Despite the extensive adoption of machine learning on the task of visual object tracking, recent learning-based approaches have largely overlooked the fact that visual tracking is a sequence-level task in its nature; they rely heavily on frame-level training, which inevitably induces inconsistency between training and testing in terms of both data distributions and task objectives. This work introduces a sequence-level training strategy for visual tracking based on reinforcement learning and discusses how a sequence-level design of data sampling, learning objectives, and data augmentation can improve the accuracy and robustness of tracking algorithms. Our experiments on standard benchmarks including LaSOT, TrackingNet, and GOT-10k demonstrate that four representative tracking models, SiamRPN++, SiamAttn, TransT, and TrDiMP, consistently improve by incorporating the proposed methods in training without modifying architectures.KeywordsVisual trackingSequence-level trainingReinforcement learning

There are mainly two components in almost every visual object tracking algorithm, which are the object presentation and the searching mechanism. In the literature, metaheuristic algorithms have been recently, widely used as searching method in visual tracking. In fact, the more discriminative is their combined object presentation the better is their precision. Salient background information is often inside the first detected region of the object, which may decrease considerably the tracking performance. In this work, we combine a powerful metaheuristic searching method, whale optimization algorithm (WOA), with the discriminative presentation named as corrected background weighted histogram (CBWH) to improve the tracking accuracy in difficult environments. WOA algorithm is used because of its strong searching ability, which enable it avoid being trapped in local optimum, and also its lowest computation cost compared to other metaheuristic algorithms. The WOA sensitivity parameters are studied experimentally to be adjusted in visual tracking. For a fair assessment of the proposed approach, its performance has been evaluated on 20 video sequences, from the famous object tracking benchmark OTB15, and compared with six (6) other state-of-art trackers. The evaluations were carried quantitatively and qualitatively on challenging situations and have provided satisfying results. The experimental results shows that WOA combined with CBWH can lead to further improvement, and better success rate in tracking.

Flexible multi-view semi-supervised learning with unified graph

Current hidden Markov acoustic modeling for large vocabulary continuous speech recognition (LVCSR) relies on the availability of abundant labeled transcriptions. Given that speech labeling is both expensive and time-consuming while there is a huge amount of unlabeled data easily available nowadays, semi-supervised learning (SSL) from both labeled and unlabeled data which aims to reduce the development cost for LVCSR becomes more important than ever. In this paper, we propose SSL for LVCSR by using the multiple views learned from different acoustic features and randomized decision trees. In addition, we develop the multi-objective learning of HMM-based acoustic models by optimizing a hybrid criterion which is established by the combination of the discriminative mutual information from labeled data and the entropy from unlabeled data. Experiments conducted on Broadcast News show the benefits of proposed methods.

Appearance change of moving targets is a challenging problem in visual tracking. In this paper, we present a novel visual object tracking algorithm based on the observation dependent hidden Markov model (OD-HMM) framework. The observation dependency is computed by structure complexity coefficients (SCC) which is defined to predict the target appearance change. Unlike conventional methods addressing the appearance change problem by investigating different online appearance models, we handle this problem by addressing the fundamental reason of motion -related appearance change during visual tracking. Based on the analysis of motion-related appearance change, we investigate the relationship between the structure of the object surface and the appearance stability. The appearance of complex structural regions is easier to change compared with that of smooth structural regions with object moving. Based on this, we define SCC to predict the appearance stability of moving objects. Different from the standard HMM-based tracking algorithms where observations between different frames are assumed to be independent, we consider the observation dependency between consecutive frames with the information provided by SCC. Moreover , we present a novel outlier removing method in appearance model updating which helps to avoid error accumulation. Experimental results on challenging video sequences demonstrate that the proposed visual tracking algorithm with OD-HMM and SCC achieves better performance than existing related tracking algorithms.

Multi-view learning machines aim to process multi-view data set which consists of instances with different views. In recent years, the scale of multi-view data set is more and more larger and traditional multi-view learning machines which should store the entire data set cannot process them with the limitation of computation and storage ability. In order to overcome such a disadvantage, one-pass multi-view (OPMV) framework has been proposed and it goes through instances only once without storing the entire data set. But most multi-view data sets are semi-supervised, i.e., some training instances of the data sets are labeled while others are unlabeled. Furthermore, for a real-world multi-view data set, labeling instances is a high-cost task. In this paper, we propose semi-supervised OPMV (SSOPMV) framework for solving this issue. In SSOPMV, we develop an approach to generate additional unlabeled instances which possess useful discriminant information and apply them into the original OPMV framework along with the labeled and original unlabeled training instances. We verify the effectiveness of the SSOPMV with some real-world semi-supervised multi-view data sets used theoretically and empirically.

Building behavior profiles of Android applications (apps) with holistic, rich and multi-view information (e.g., incorporating several semantic views of an app such as API sequences, system calls, etc.) would help catering downstream analytics tasks such as app categorization, recommendation and malware analysis significantly better. Towards this goal, we design a semi-supervised Representation Learning (RL) framework named apk2vec to automatically generate a compact representation (aka profile/embedding) for a given app. More specifically, apk2vec has the three following unique characteristics which make it an excellent choice for largescale app profiling: (1) it encompasses information from multiple semantic views such as API sequences, permissions, etc., (2) being a semi-supervised embedding technique, it can make use of labels associated with apps (e.g., malware family or app category labels) to build high quality app profiles, and (3) it combines RL and feature hashing which allows it to efficiently build profiles of apps that stream over time (i.e., online learning). The resulting semi-supervised multi-view hash embeddings of apps could then be used for a wide variety of downstream tasks such as the ones mentioned above. Our extensive evaluations with more than 42,000 apps demonstrate that apk2vec's app profiles could significantly outperform state-of-the-art techniques in four app analytics tasks namely, malware detection, familial clustering, app clone detection and app recommendation.

This paper considers a task of visual object tracking with application in embedded systems. Specifically, Raspberry Pi is considered as target hardware platform. The main challenge in this problem setting is to balance between algorithm result quality and its runtime efficiency. To select the method, the main object tracking approaches were analyzed and compared. Based on this, Nanotrack method was picked for deeper analysis and experiments. Method results accuracy was analyzed by already available benchmark scores and by running test on handpicked cases from LaSOT dataset highlighting different challenges for object tracking methods such as occlusion, scale variability, etc. The runtime performance was measured using target hardware. Based on the obtained results it was concluded that selected method is suitable to perform real time object tracking in the target setting, and further development will be based on this method.

As a fundamental task in computer vision, visual object tracking has received much attention in recent years. Most studies focus on short-term visual tracking which addresses shorter videos and always-visible targets. However, long-term visual tracking is much closer to practical applications with more complicated challenges. There exists a longer duration such as minute-level or even hour-level in the long-term tracking task, and the task also needs to handle more frequent target disappearance and reappearance. In this paper, we provide a thorough review of long-term tracking, summarizing long-term tracking algorithms from two perspectives: framework architectures and utilization of intermediate tracking results. Then we provide a detailed description of existing benchmarks and corresponding evaluation protocols. Furthermore, we conduct extensive experiments and analyse the performance of trackers on six benchmarks: VOTLT2018, VOTLT2019 (2020/2021), OxUvA, LaSOT, TLP and the long-term subset of VTUAV-V. Finally, we discuss the future prospects from multiple perspectives, including algorithm design and benchmark construction. To our knowledge, this is the first comprehensive survey for long-term visual object tracking. The relevant content is available at https://github.com/wangdong-dut/Long-term-Visual-Tracking.

We present a graph-based semi-supervised learning for the question-answering (QA) task for ranking candidate sentences. Using textual entailment analysis, we obtain entailment scores between a natural language question posed by the user and the candidate sentences returned from search engine. The textual entailment between two sentences is assessed via features representing high-level attributes of the entailment problem such as sentence structure matching, question-type named-entity matching based on a question-classifier, etc. We implement a semi-supervised learning (SSL) approach to demonstrate that utilization of more unlabeled data points can improve the answer-ranking task of QA. We create a graph for labeled and unlabeled data using match-scores of textual entailment features as similarity weights between data points. We apply a summarization method on the graph to make the computations feasible on large datasets. With a new representation of graph-based SSL on QA datasets using only a handful of features, and under limited amounts of labeled data, we show improvement in generalization performance over state-of-the-art QA models.