Learning Dual-Level Deep Representation for Thermal Infrared Tracking

Existing deep Thermal InfraRed (TIR) trackers usually use the feature models of RGB trackers for representation. However, these feature models learned on RGB images are neither effective in representing TIR objects nor taking fine-grained TIR information into consideration. To this end, we develop a multi-task framework to learn the TIR-specific discriminative features and fine-grained correlation features for TIR tracking. Specifically, we first use an auxiliary classification network to guide the generation of TIR-specific discriminative features for distinguishing the TIR objects belonging to different classes. Second, we design a fine-grained aware module to capture more subtle information for distinguishing the TIR objects belonging to the same class. These two kinds of features complement each other and recognize TIR objects in the levels of inter-class and intra-class respectively. These two feature models are learned using a multi-task matching framework and are jointly optimized on the TIR tracking task. In addition, we develop a large-scale TIR training dataset to train the network for adapting the model to the TIR domain. Extensive experimental results on three benchmarks show that the proposed algorithm achieves a relative gain of 10% over the baseline and performs favorably against the state-of-the-art methods. Codes and the proposed TIR dataset are available at https://github.com/QiaoLiuHit/MMNet.

Unlike visual object tracking, thermal infrared (TIR) object tracking methods can track the target of interest in poor visibility such as rain, snow, and fog, or even in total darkness. This feature brings a wide range of application prospects for TIR object-tracking methods. However, this field lacks a unified and large-scale training and evaluation benchmark, which has severely hindered its development. To this end, we present a large-scale and high-diversity unified TIR single object tracking benchmark, called LSOTB-TIR, which consists of a tracking evaluation dataset and a general training dataset with a total of 1416 TIR sequences and more than 643 K frames. We annotate the bounding box of objects in every frame of all sequences and generate over 770 K bounding boxes in total. To the best of our knowledge, LSOTB-TIR is the largest and most diverse TIR object tracking benchmark to date. We spilt the evaluation dataset into a short-term tracking subset and a long-term tracking subset to evaluate trackers using different paradigms. What's more, to evaluate a tracker on different attributes, we also define four scenario attributes and 12 challenge attributes in the short-term tracking evaluation subset. By releasing LSOTB-TIR, we encourage the community to develop deep learning-based TIR trackers and evaluate them fairly and comprehensively. We evaluate and analyze 40 trackers on LSOTB-TIR to provide a series of baselines and give some insights and future research directions in TIR object tracking. Furthermore, we retrain several representative deep trackers on LSOTB-TIR, and their results demonstrate that the proposed training dataset significantly improves the performance of deep TIR trackers. Codes and dataset are available at https://github.com/QiaoLiuHit/LSOTB-TIR.

The usage of both off-the-shelf and end-to-end trained deep networks have significantly improved the performance of visual tracking on RGB videos. However, the lack of large labeled datasets hampers the usage of convolutional neural networks for tracking in thermal infrared (TIR) images. Therefore, most state-of-the-art methods on tracking for TIR data are still based on handcrafted features. To address this problem, we propose to use image-to-image translation models. These models allow us to translate the abundantly available labeled RGB data to synthetic TIR data. We explore both the usage of paired and unpaired image translation models for this purpose. These methods provide us with a large labeled dataset of synthetic TIR sequences, on which we can train end-to-end optimal features for tracking. To the best of our knowledge, we are the first to train end-to-end features for TIR tracking. We perform extensive experiments on the VOT-TIR2017 dataset. We show that a network trained on a large dataset of synthetic TIR data obtains better performance than one trained on the available real TIR data. Combining both data sources leads to further improvement. In addition, when we combine the network with motion features, we outperform the state of the art with a relative gain of over 10%, clearly showing the efficiency of using synthetic data to train end-to-end TIR trackers.

Hierarchical spatial-aware Siamese network for thermal infrared object tracking

Scene images usually involve semantic correlations, particularly when considering large-scale image data sets. This paper proposes a novel generative image representation, correlated topic vector, to model such semantic correlations. Oriented from the correlated topic model, correlated topic vector intends to naturally utilize the correlations among topics, which are seldom considered in the conventional feature encoding, e.g., Fisher vector, but do exist in scene images. It is expected that the involvement of correlations can increase the discriminative capability of the learned generative model and consequently improve the recognition accuracy. Incorporated with the Fisher kernel method, correlated topic vector inherits the advantages of Fisher vector. The contributions to the topics of visual words have been further employed by incorporating the Fisher kernel framework to indicate the differences among scenes. Combined with the deep convolutional neural network (CNN) features and Gibbs sampling solution, correlated topic vector shows great potential when processing large-scale and complex scene image data sets. Experiments on two scene image data sets demonstrate that correlated topic vector improves significantly the deep CNN features, and outperforms existing Fisher kernel-based features.

If the training data sets are large-scale, the SVM models turn into large-scale quadratic programming problems and are hard to be effectively solved. That is the main reason why the learning efficiency of SVM is so low on large-scale data sets. According to the principle of SVM, the border support vectors play a decisive role on the SVM decision hyper-plane. The nest template ant clustering algorithm in kernel space is proposed. The algorithm is used to extract border support vectors from large-scale training datasets. When using the less border support vectors to train SVM, the scale of SVM is reduced and the training performance is improved. The algorithm has better adaptability than the kernel K-means algorithm. Experimental results on UCI datasets show that the algorithm is effective, and the classification accuracy of SVM is still maintained.

THINGS-data reflects three large-scale neuroimaging and behavioral datasets of object processing in humans, comprising densely sampled functional MRI and magnetoencephalographic recordings, as well as 4.70 million similarity judgments in response to thousands of photographic images for up to 1854 objects.

With the development of the Internet, more and more people like to share their daily food, which makes the Internet rich in food image data. Many researchers collect food images on the Internet, label them, and combine machine learning and other methods to achieve food recognition systems. These food recognition systems can quickly help users record their daily diet and analyze their eating habits. The advanced food identification system can currently accurately identify hundreds of foods. At the same time, it can explore the food nutrition content and give food reference suggestions by accessing the food nutrition database. In this paper, we collected and evaluated large-scale food image data sets and food recognition systems.

Space situational awareness (SSA) plays an imperative role in maintaining safe space operations, especially given the increasingly congested space traffic around the Earth. Space-based SSA offers a flexible and lightweight solution compared to traditional ground-based SSA. With advanced machine learning approaches, space-based SSA can extract features from high-resolution images in space to detect and track resident space objects (RSOs). However, existing spacecraft image datasets, such as SPARK, fall short of providing realistic camera observations, rendering the derived algorithms unsuitable for real SSA systems. In this work, we introduce SpaceDet, a large-scale realistic space-based image dataset for SSA. We consider accurate space orbit dynamics and a physical camera model with various noise distributions, generating images at the photon level. To extend the available observation window, four overlapping cameras are simulated with a fixed rotation angle. SpaceDet includes images of RSOs observed from 19 km to 63,000 km, captured by a tracker operating in LEO, MEO, and GEO orbits over a period of 5,000 seconds. Each image has a resolution of 4418 x 4418 pixels, providing detailed features for developing advanced SSA approaches. We split the dataset into three subsets: SpaceDet-100, SpaceDet-5000, and SpaceDet-full, catering to various image processing applications. The SpaceDet-full corpus includes a comprehensive dataloader with 781.5 GB of images and 25.9 MB of ground truth labels. Furthermore, we adapted detection and tracking algorithms on the collected dataset using a specified splitting method to accelerate the training process. The trained model can detect RSOs from real-world space observations with zero-shot capability.

Transparent objects are encountered frequently in our daily lives, yet recognizing them poses challenges for conventional vision sensors due to their unique material properties, not being well perceived from RGB or depth cameras. Overcoming this limitation, thermal infrared cameras have emerged as a solution, offering improved visibility and shape information for transparent objects. In this paper, we present TRansPose, the first large-scale multispectral dataset that combines stereo RGB-D, thermal infrared (TIR) images, and object poses to promote transparent object research. The dataset includes 99 transparent objects, encompassing 43 household items, 27 recyclable trashes, 29 chemical laboratory equivalents, and 12 non-transparent objects. It comprises a vast collection of 333,819 images and 4,000,056 annotations, providing instance-level segmentation masks, ground-truth poses, and completed depth information. The data was acquired using an FLIR A65 thermal infrared camera, two Intel RealSense L515 RGB-D cameras, and a Franka Emika Panda robot manipulator. Spanning 87 sequences, TRansPose covers various challenging real-life scenarios, including objects filled with water, diverse lighting conditions, heavy clutter, non-transparent or translucent containers, objects in plastic bags, and multi-stacked objects. Supplementary material can be accessed from the following link: https://sites.google.com/view/transpose-dataset.

Due to the complex scenarios and the limited feature information in a single image, a precise smoke detection is much more challenging in practice. Most of previous smoke detection methods either extract textural and spatiotemporal characteristics of smoke or separate the smoke and background components of the image. However, those methods often fail in detecting smoke positions because of the limited feature information within a single image. Moreover, the task of smoke detection can be better achieved if the extra information from collected training dataset is available. One key issue is how to build a training dataset of paired smoke images and ground-truth bounding box positions for end-to-end learning. This paper proposes a large-scale benchmark image dataset to train a smoke detector. With the built dataset, experimental results demonstrate that the discriminative models can be effectively trained as the smoke detector to detect the smoldering fires precisely.

Parsing the semantic structure of a web page is a key component of web information extraction. Successful extraction algorithms usually require large-scale training and evaluation datasets, which are difficult to acquire. Recently, crowdsourcing has proven to be an effective method of collecting large-scale training data in domains that do not require much domain knowledge. For more complex domains, researchers have proposed sophisticated quality control mechanisms to replicate tasks in parallel or sequential ways and then aggregate responses from multiple workers. Conventional annotation integration methods often put more trust in the workers with high historical performance; thus, they are called performance-based methods. Recently, Rzeszotarski and Kittur have demonstrated that behavioral features are also highly correlated with annotation quality in several crowdsourcing applications. In this article, we present a new crowdsourcing system, called Wernicke, to provide annotations for web information extraction. Wernicke collects a wide set of behavioral features and, based on these features, predicts annotation quality for a challenging task domain: annotating web page structure. We evaluate the effectiveness of quality control using behavioral features through a case study where 32 workers annotate 200 Q&A web pages from five popular websites. In doing so, we discover several things: (1) Many behavioral features are significant predictors for crowdsourcing quality. (2) The behavioral-feature-based method outperforms performance-based methods in recall prediction, while performing equally with precision prediction. In addition, using behavioral features is less vulnerable to the cold-start problem, and the corresponding prediction model is more generalizable for predicting recall than precision for cross-website quality analysis. (3) One can effectively combine workers’ behavioral information and historical performance information to further reduce prediction errors.

While deep learning (DL) has proven to be a powerful paradigm for the classification of the large-scale big image data set. Deep learning network such as CNN requires a large number of labeled samples for training the network. However, often times the labeled data are difficult, expensive, and time-consuming. In this paper, we propose a semi-supervised approach to fused fuzzy-rough C-mean clustering with convolutional neural networks (CNNs) to knowledge learning from simultaneously intra-model and inter-model relationships forming final data representation to be classified, which can be achieved by better performance. Conception behind this is to reduce the uncertainty in terms of vagueness and indiscernibility by using fuzzy-rough C-mean clustering and specifically removing noise samples by using CNN from raw data. The framework of our proposed semi-supervised approach uses all the training data with abundant unlabeled data with few labeled data to train FRCNN model. To show the effectiveness of our model, we used four benchmark large-scale image datasets and also compared it with the state-of-the-art supervised, unsupervised, and semi-supervised learning methods for image classification.

Many object re-identification (Re-ID) methods that depend on large-scale training datasets have been proposed in recent years. However, the performance of these methods degrades dramatically when insufficient training data are available. To address this challenging problem, we propose a few-shot object re-identification (FSOR) method that enhances the generalization and discrimination abilities of object Re-ID models trained on small datasets. This method applies two novel techniques: reparameterization for feature vectors and dual-distance metric learning. The reparameterization mechanism transforms the primary feature vector of each input image into a Gaussian distribution to enhance the robustness of the FSOR method when performing object Re-ID tasks. The dual-distance metric learning technique, called H&C learning, considers both the hard mining distance and the center-point distance between each query sample and each support set of different object identities. H&C learning extracts the characteristics of the entire training dataset more precisely than other approaches and thus improves the discriminative abilities of object Re-ID models. Extensive experiments on both person and vehicle Re-ID datasets, such as Market-1501, DukeMTMC-ReID, CUHK03, and VeRi-776, show that the FSOR method has improved performance and outperforms state-of-the-art methods when the amount of labeled training data is small.

Blind quality assessment for screen content images by combining local and global features