Multi-Task Driven Feature Models for Thermal Infrared Tracking

The feature models used by existing Thermal InfraRed (TIR) tracking methods are usually learned from RGB images due to the lack of a large-scale TIR image training dataset. However, these feature models are less effective in representing TIR objects and they are difficult to effectively distinguish distractors because they do not contain fine-grained discriminative information. To this end, we propose a dual-level feature model containing the TIR-specific discriminative feature and fine-grained correlation feature for robust TIR object tracking. Specifically, to distinguish inter-class TIR objects, we first design an auxiliary multi-classification network to learn the TIR-specific discriminative feature. Then, to recognize intra-class TIR objects, we propose a fine-grained aware module to learn the fine-grained correlation feature. These two kinds of features complement each other and represent TIR objects in the levels of inter-class and intra-class respectively. These two feature models are constructed using a multi-task matching framework and are jointly optimized on the TIR object tracking task. In addition, we develop a large-scale TIR image dataset to train the network for learning TIR-specific feature patterns. To the best of our knowledge, this is the largest TIR tracking training dataset with the richest object class and scenario. To verify the effectiveness of the proposed dual-level feature model, we propose an offline TIR tracker (MMNet) and an online TIR tracker (ECO-MM) based on the feature model and evaluate them on three TIR tracking benchmarks. Extensive experimental results on these benchmarks demonstrate that the proposed algorithms perform favorably against the state-of-the-art methods.

Abstract. This paper deals with two aspects of photogrammetric processing of thermal images: image quality and 3D reconstruction quality. The first aspect of the paper relates to the influence of day light on Thermal InfraRed (TIR) images captured by an Unmanned Aerial Vehicle (UAV). Environmental factors such as ambient temperature and lack of sun light affect TIR image quality. We acquire image sequences of the same object during day and night and compare the generated orthophotos according to different metrics like contrast and signal-to-noise ratio (SNR). Our experiments show that performing TIR image acquisition during night time provides a better thermal contrast, regardless of whether we compute contrast over the whole image or over small patches. The second aspect investigated in this work is the potential of using TIR images for photogrammetric tasks such as the automatic generation of Digital Surface Models (DSM) and orthophotos. Due to the low geometrical resolution of a TIR camera and the low image quality in terms of contrast and noise compared to RGB images, the TIR DSM suffers from reconstruction errors and an orthophoto generated using the TIR DSM and TIR images is visibly influenced by those errors. We therefore include measurements of the UAVs positions during image capturing provided by a Global Navigation Satellite System (GNSS) receiver to retrieve position and orientation of TIR and RGB images in the same world coordinate system. To generate an orthophoto from TIR images, they are projected onto the DSM reconstructed from RGB images. This procedure leads to a TIR orthophoto of much higher quality in terms of geometrical correctness.

Unlike visual object tracking, thermal infrared (TIR) object tracking methods can track the target of interest in poor visibility such as rain, snow, and fog, or even in total darkness. This feature brings a wide range of application prospects for TIR object-tracking methods. However, this field lacks a unified and large-scale training and evaluation benchmark, which has severely hindered its development. To this end, we present a large-scale and high-diversity unified TIR single object tracking benchmark, called LSOTB-TIR, which consists of a tracking evaluation dataset and a general training dataset with a total of 1416 TIR sequences and more than 643 K frames. We annotate the bounding box of objects in every frame of all sequences and generate over 770 K bounding boxes in total. To the best of our knowledge, LSOTB-TIR is the largest and most diverse TIR object tracking benchmark to date. We spilt the evaluation dataset into a short-term tracking subset and a long-term tracking subset to evaluate trackers using different paradigms. What's more, to evaluate a tracker on different attributes, we also define four scenario attributes and 12 challenge attributes in the short-term tracking evaluation subset. By releasing LSOTB-TIR, we encourage the community to develop deep learning-based TIR trackers and evaluate them fairly and comprehensively. We evaluate and analyze 40 trackers on LSOTB-TIR to provide a series of baselines and give some insights and future research directions in TIR object tracking. Furthermore, we retrain several representative deep trackers on LSOTB-TIR, and their results demonstrate that the proposed training dataset significantly improves the performance of deep TIR trackers. Codes and dataset are available at https://github.com/QiaoLiuHit/LSOTB-TIR.

Advances in drone technology have given rise to much interest in the use of drone-mounted thermal imagery in wildlife monitoring. This research tested the feasibility of monitoring large mammals in an urban environment and investigated the influence of drone flight parameters and environmental conditions on their successful detection using thermal infrared (TIR) and true-colour (RGB) imagery. We conducted 18 drone flights at different altitudes on the Sunshine Coast, Queensland, Australia. Eastern grey kangaroos (Macropus giganteus) were detected from TIR (n=39) and RGB orthomosaics (n=33) using manual image interpretation. Factors that predicted the detection of kangaroos from drone images were identified using unbiased recursive partitioning. Drone-mounted imagery achieved an overall 73.2% detection success rate using TIR imagery and 67.2% using RGB imagery when compared to on-ground counts of kangaroos. We showed that the successful detection of kangaroos using TIR images was influenced by vegetation type, whereas detection using RGB images was influenced by vegetation type, time of day that the drone was deployed, and weather conditions. Kangaroo detection was highest in grasslands, and kangaroos were not successfully detected in shrublands. Drone-mounted TIR and RGB imagery are effective at detecting large mammals in urban and peri-urban environments.

The stability of cemented backfill mass is important to keep miners and equipment safe in underground backfill miming. The stress-strain behavior, resistivity and thermal infrared (TIR) characteristics of backfill mass under uniaxial compression were investigated. The monitoring system consisted of a TIR observation system, a stress-strain monitoring system and a resistivity measurement system. Precursory information for impending failure of cemented backfill mass was collected, including TIR, strain and resistivity precursors. The sensitivity and difference of different monitoring information to the same failure event were compared. The results show that the time-space evolution process of the resistivity and TIR is basically the same as the whole process from compression deformation to failure of backfill mass, and the time variation of resistivity and TIR is obviously characterized by stage. The resistivity precursor turns out earlier than the TIR and the strain. The resistivity relation with loading compression is anti-symmetry, decreasing as the compression stress increases before the peak strength of backfill mass. However, when the backfill mass enters into the phase of failure, the resistivity starts to increase as the stress increases. The change of the resistivity growth direction can be regarded as the resistivity-caution-point for the failure of backfill mass under uniaxial compression. It is also indicated that the TIR information mainly represents the surface temperature evolution in the process of compression before the backfill enters into the plastic-yield state. It can be a valuable tool to obtain the precursors for failure of cemented backfill mass for backfill mines.

The usage of both off-the-shelf and end-to-end trained deep networks have significantly improved the performance of visual tracking on RGB videos. However, the lack of large labeled datasets hampers the usage of convolutional neural networks for tracking in thermal infrared (TIR) images. Therefore, most state-of-the-art methods on tracking for TIR data are still based on handcrafted features. To address this problem, we propose to use image-to-image translation models. These models allow us to translate the abundantly available labeled RGB data to synthetic TIR data. We explore both the usage of paired and unpaired image translation models for this purpose. These methods provide us with a large labeled dataset of synthetic TIR sequences, on which we can train end-to-end optimal features for tracking. To the best of our knowledge, we are the first to train end-to-end features for TIR tracking. We perform extensive experiments on the VOT-TIR2017 dataset. We show that a network trained on a large dataset of synthetic TIR data obtains better performance than one trained on the available real TIR data. Combining both data sources leads to further improvement. In addition, when we combine the network with motion features, we outperform the state of the art with a relative gain of over 10%, clearly showing the efficiency of using synthetic data to train end-to-end TIR trackers.

The photogrammetric processing of thermal infrared (TIR) images deals with several difficulties. TIR images ordinarily have low-resolution and the contrast of the images is very low. These factors strongly complicate the photogrammetric processing, especially when a modern structure from motion method is used. These factors can be avoided by a certain co-processing method of TIR and RGB images. Two of the solutions of co-processing were suggested by the authors and are presented in this article. Each solution requires a different type of transformation–plane transformation and spatial transformation. Both types of transformations are discussed in this paper. On the experiments that were performed, there are presented requirements, advantages, disadvantages, and results of the transformations. Both methods are evaluated mainly in terms of accuracy. The transformations are presented on suggested methods, but they can be easily applied to different kinds of methods of co-processing of TIR and RGB images.

Hierarchical spatial-aware Siamese network for thermal infrared object tracking

The angular distribution of thermal infrared (TIR) radiation emitted by vegetation covers can vary widely depending on environmental conditions and canopy structure. As an aid in the interpretation of TIR remotely sensed data from vegetated surface with incomplete canopies, we developed a three‐dimensional radiative transfer model in the thermal infrared domain. The model simulates the TIR radiative budget and upward spectral radiance of vegetation covers. The model is an extension to the TIR region of the DART (Discrete Anisotropic Radiative Transfer) model developed for the short wave domain. Radiative transfer simulation relies on discrete 3‐D scene representations, which include any distribution of trees and ground covers, possibly with topography. Propagation of emitted and scattered radiation is tracked with a ray‐tracing approach and the discrete ordinate method. The model was successfully tested against a physically based model for homogeneous canopies, and a partial validation was carried out with directional TIR measurements on a cotton row crop. Model simulations are presented to illustrate the influence of the canopy geometric structure on the directional apparent temperature of hypothetical vegetation covers.

Autonomous driving systems usually depend on RGB images but face difficulties in low-visibility conditions. Thermal infrared (TIR) imaging offers an alternative that performs well in such environments. However, TIR is hindered by low contrast, indistinct boundaries, and the absence of color, making it less compatible with standard algorithms. Recent advances in thermal infrared colorization are promising, but they face challenges such as semantic distortions, frame instability, and issues with small objects. We introduce the frame-based Weber-law-driven Generative Adversarial Network (FWGAN), a new model that converts thermal images into visible-spectrum images. FWGAN improves object recognition by maintaining textures, increasing temporal consistency, and enabling reliable detection in different scenarios. Tests on benchmark datasets like FLIR and KAIST show that FWGAN's enhancements in semantic accuracy, object detection, and visual quality significantly push forward autonomous driving technology.

Recent studies have shown the unique value of satellite-observed land surface thermal infrared (TIR) information (e.g., skin temperature) and the feasibility of assimilating land surface temperature (LST) into land surface models (LSMs) to improve the simulation of land-atmosphere water and energy exchanges. In this study, two different types of LST assimilation techniques are implemented and the benefits from the techniques are compared. One of the techniques is to directly assimilate LST using ensemble Kalman filter (EnKF) data assimilation (DA) utilities. The other is to use the Atmosphere-Land Exchange Inversion model (ALEXI) as an “observation operator” that converts LST retrievals into the soil moisture (SM) proxy based on the ratio of actual to potential evapotranspiration (fPET), which is then assimilated into an LSM. While most current studies have shown some success in both directly the assimilating LST and assimilating ALEXI SM proxy into offline LSMs, the potential impact of the assimilation of TIR information through coupled numerical weather prediction (NWP) models is unclear. In this study, a semi-coupled Land Information System (LIS) and Weather Research and Forecast (WRF) system is employed to assess the impact of the two different techniques for assimilating the TIR observations from NOAA GOES satellites on WRF model forecasts. The NASA LIS, equipped with a variety of LSMs and advanced data assimilation tools (e.g., the ensemble Kalman Filter (EnKF)), takes atmospheric forcing data from the WRF model run, generates updated initial land surface conditions with the assimilation of either LST- or TIR-based SM and returns them to WRF for initializing the forecasts. The WRF forecasts using the daily updated initializations with the TIR data assimilation are evaluated against ground weather observations and re-analysis products. It is found that WRF forecasts with the LST-based SM assimilation have better agreement with the ground weather observations than those with the direct LST assimilation or without the land TIR data assimilation.

Close-range remote sensing often fills the gap between in situ measurements and space-based observations. While most platforms are equipped with RGB cameras, there is a growing availability of thermal infrared (TIR) cameras. Both Uncrewed Aerial Vehicles (UAV) surveys in general and TIR remote sensing pose their individual challenges, especially in complex topography. In particular, TIR datasets are far from being ready-to-use upon acquisition, and thorough post-processing is required. However, they offer a great potential to monitor cryospheric landforms and assess their surface energy budget and related dynamics.In this contribution, we present two years of TIR and RGB UAV data in combination with multiple in situ measurements, both for calibration and validation, obtained for a creeping permafrost landform, rock glacier Murtèl in the Engadine, Switzerland. We highlight the challenges evoked by the complex topography in the alpine environment (e.g. irradiance distribution, wind) and shed light on varying correction possibilities (e.g. laboratory-, field-, camera-based) that allow for a more accurate retrieval of land surface temperature over middle-sized landforms, such as a rock glacier.In light of future thermal infrared satellite missions, an appropriate use of close-range remote sensing techniques, including survey protocols for calibration and validation, is urgently needed. This application study contributes to a better across-scale methodological understanding of sensors and methods, as well as the role of close-range remote sensing in complementing in situ and space-based observations, but also illustrates the potential of TIR datasets for cryospheric process understanding and long-term monitoring.

Mapping and quantifying hydrologic exchange flows (HEFs) is critical to environmental monitoring and remediation at contaminated sites; however, these objectives are challenging in highly dynamic systems, e.g., along dam-regulated rivers, where HEFs vary rapidly. Direct seepage measurements are labor-intensive and difficult to automate, whereas indirect (e.g., thermal) and remote sensing methods have potential to allow continuous monitoring with limited field effort. We present a preliminary assessment of a multi-scale temperature-based strategy for monitoring HEFs along the Hanford Reach of the Columbia River, in eastern WA, United States. Five thermal methods were assessed. First, a vertical temperature profile (VTP) was installed into the streambed. The VTP data were analyzed using a data assimilation algorithm designed for automated real-time estimation in dynamic systems. Second, a thermal infrared (TIR) camera was used in roving surveys to identify seeps. Third, a TIR camera was stationed at the VTP site to collect images at 1-h intervals. Together, the two TIR datasets provided a basis to assess the potential for drone-based TIR. Fourth, temperature was measured at the sediment/water interface to assess fiber-optic distributed temperature sensing. Fifth, imagery from the ECOSTRESS satellite mission was acquired to assess the potential of spaceborne thermal monitoring. Based on our preliminary assessment, VTP, TIR, and bed temperature measurements provide complementary spatial coverage, temporal sampling, and resolution; these methods have potential for long-term, automated monitoring of HEFs. The publicly available spaceborne imagery, however, proved inadequate because of insufficient spatial resolution and data gaps resulting from cloud cover and revisit frequency.

Recently, semantic segmentation based on RGB and thermal infrared (TIR) images has become a research hotspot because of its stability in the weak light environment. However, most of the current methods ignore the differences between the two modalities of data and do not use semantic information in multi-modal fusion. In this paper, we propose a novel Semantic-Guided Fusion Network (SGFNet) for RGB-Thermal semantic segmentation, which makes full use of semantic information in the multi-modal fusion. Our SGFNet consists of an asymmetric encoder with TIR branch and RGB branch and a decoder. We concentrate on enhancing the multi-modal feature representation in the encoder with a pattern of fusion and enhancement. Specifically, considering that TIR images are stable under weak light conditions, we first propose a Semantic Guidance Head to extract semantic information in the TIR branch. In the RGB branch, we propose a Multi-modal Coordination and Distillation Unit to fuse multi-modal features first. Then, we propose a Cross-level and Semantic-guided Enhancement Unit to enhance the fused features with cross-level information and semantic information. We arrange these two units at all stages of the RGB branch to generate features with strong representation abilities at different levels. For the decoder, to obtain large receptive fields and fine edges, we improve the Lawin ASPP decoder by introducing edge information extracted from the low-level features, proposing the edge-aware Lawin ASPP decoder. With our encoder and decoder working together, our SGFNet can identify objects accurately and segment objects finely. Extensive experiments on the MFNet dataset demonstrate the superior performance of the proposed SGFNet compared with state-of-the-art methods. The code and results of our method are available at https://github.com/kw717/SGFNet.

Thermal Infrared (TIR) imaging is a promising technology which can provide enhanced capabilities to current vision-based Sense-and-Avoid (SAA) systems. It allows operation under extreme illumination conditions, such as direct sun exposure and during nighttime. This paper presents a lightweight obstacle detection system for small UAVs that integrates a TIR camera and an Automatic Dependent Surveillance Broadcast (ADS-B) receiver. Algorithms for the detection of flying obstacles in TIR images were developed and TIR images were experimentally compared with synchronized RGB images for validation. Matching between aircraft detected in TIR images and those reported by an ADS-B receiver was performed to obtain distance information to the visually detected aircraft. We experimentally proved that our system is able to enhance individual ADS-B and TIR detection capabilities by detecting aircraft under challenging illumination conditions at real-time frame rates while providing distance estimations to visual detections.