Downstream Task-Aware Cloud Removal for Very-High-Resolution Remote Sensing Images: An Information Loss Perspective

Aerial Thermal Infrared (TIR) imagery has demonstrated tremendous potential to monitor active forest fires and acquire detailed information about fire behavior. However, aerial video is usually unstable and requires inter-frame registration before further processing. Measurement of image misalignment is an essential operation for video stabilization. Misalignment can usually be estimated through image similarity, although image similarity metrics are also sensitive to other factors such as changes in the scene and lighting conditions. Therefore, this article presents a thorough analysis of image similarity measurement techniques useful for inter-frame registration in wildfire thermal video. Image similarity metrics most commonly and successfully employed in other fields were surveyed, adapted, benchmarked and compared. We investigated their response to different camera movement components as well as recording frequency and natural variations in fire, background and ambient conditions. The study was conducted in real video from six fire experimental scenarios, ranging from laboratory tests to large-scale controlled burns. Both Global and Local Sensitivity Analyses (GSA and LSA, respectively) were performed using state-of-the-art techniques. Based on the obtained results, two different similarity metrics are proposed to satisfy two different needs. A normalized version of Mutual Information is recommended as cost function during registration, whereas 2D correlation performed the best as quality control metric after registration. These results provide a sound basis for image alignment measurement and open the door to further developments in image registration, motion estimation and video stabilization for aerial monitoring of active wildland fires.

Objective. The study aims to systematically characterize the effect of CT parameter variations on images and lung radiomic and deep features, and to evaluate the ability of different image harmonization methods to mitigate the observed variations.Approach. A retrospective in-house sinogram dataset of 100 low-dose chest CT scans was reconstructed by varying radiation dose (100%, 25%, 10%) and reconstruction kernels (smooth, medium, sharp). A set of image processing, convolutional neural network (CNNs), and generative adversarial network-based (GANs) methods were trained to harmonize all image conditions to a reference condition (100% dose, medium kernel). Harmonized scans were evaluated for image similarity using peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS), and for the reproducibility of radiomic and deep features using concordance correlation coefficient (CCC).Main Results. CNNs consistently yielded higher image similarity metrics amongst others; for Sharp/10%, which exhibited the poorest visual similarity, PSNR increased from a mean ± CI of 17.763 ± 0.492 to 31.925 ± 0.571, SSIM from 0.219 ± 0.009 to 0.754 ± 0.017, and LPIPS decreased from 0.490 ± 0.005 to 0.275 ± 0.016. Texture-based radiomic features exhibited a greater degree of variability across conditions, i.e. a CCC of 0.500 ± 0.332, compared to intensity-based features (0.972 ± 0.045). GANs achieved the highest CCC (0.969 ± 0.009 for radiomic and 0.841 ± 0.070 for deep features) amongst others. CNNs are suitable if downstream applications necessitate visual interpretation of images, whereas GANs are better alternatives for generating reproducible quantitative image features needed for machine learning applications.Significance. Understanding the efficacy of harmonization in addressing multi-parameter variability is crucial for optimizing diagnostic accuracy and a critical step toward building generalizable models suitable for clinical use.

In the paper the idea of the visual self-localization of mobile robots based on the estimation of image similarity is discussed. It is assumed that a rough position of the mobile robot is known e.g. from the built-in GPS device. Assuming known orientation of the robot, the main advantage of the application of image analysis algorithms is the increase of the self-localization accuracy. The basic idea of our approach is related to the use of image similarity metrics for the estimation of the distance to the known positions associated with the images stored in the database. In order to verify the the validity of the proposed approach for the images containing textures characteristic for urban areas, some experiments have been conducted in Java based Simbad environment. As the result of applying various image similarity metrics for this purpose, the noticeable increase of the localization accuracy has been obtained for the artificial model of the urban area developed in Simbad environment. Nevertheless, achieved results are encouraging for further experiments utilizing natural images captured by real cameras also in the presence of various image distortions.

A Self-structuring Data Learning Algorithm was introduced and has been implemented in our prior work. While the algorithm and the software package are advancing, it has been tested with both synthetic data and real-world data. After encouraging synthetic data test results, real-world data testing also shows promising outcomes while posing some challenges such as object occlusion, objects merging, and going into and emerging from under bridge. To resolve such problems, a multi-int solution is proposed. One of the key features in this solution is similarity measure. There are different types of similarity measures. In this paper, we primarily focus on aerial images similarity measure. The images we worked on presents unique challenge in similarity measure because of small object in distance and large area image, which consequently provides limited information. To deal with this difficulty, we have developed 14 different similarity metrics by employing Normalized Cross Correlation method, Sum of Squared Differences, and overlapping and colors of pixels. We used object tracking ability to evaluate the metrics. The simulation results show each metric has some advantages and disadvantages. In attempt to improve tracking capability, we imposed some metrics thresholds in addition to the image similarity metrics. Such metrics thresholds were learned from labeled data with valuation of tracking correctness. To further enhance tracking ability, speed similarity was incorporated on top of two features mentioned above. More improvement can be done by studying robustness of images similarity metrics and using tracks fusion.

Traditional image similarity metrics compare two images on a point-by-point basis. On the other hand, structural similarity metrics (SSIM) attempt to base image similarity on "structural" information. We evaluate the performance of SSIM metrics in the context of texture similarity, and propose new metrics that incorporate the best features of SSIM and eliminate the most serious drawbacks. We show that the proposed new texture similarity metrics outperform SSIM and its variations, as well as PSNR and other traditional metrics. We demonstrate the advantages of the new metrics on a carefully selected set of 39 texture pairs and comparisons with informal subjective test results.

Toward an Image Similarity Metric

Temporal subtraction images constructed from image registration can facilitate the visualization of pathologic changes. In this study, we propose a deformable image registration (DIR) framework for creating temporal subtraction images of chest radiographs. We developed a DIR methodology using two different image similarity metrics, varying flow (VF) and compressible flow (CF). The proposed registration method consists of block matching, filtering, and interpolation. Specifically, corresponding point pairs between reference and target images are initially determined by minimizing a nonlinear least squares formulation using grid-searching optimization. A two-step filtering process, including least median of squares filtering and backward matching filtering, is then applied to the estimated point matches in order to remove erroneous matches. Finally, moving least squares is used to generate a full displacement field from the filtered point pairs. We applied the proposed DIR method to 10 pairs of clinical chest radiographs and compared it with the demons and B-spline algorithms using the five-point rating score method. The average quality scores were 2.7 and 3 for the demons and B-spline methods, but 3.5 and 4.1 for the VF and CF methods. In addition, subtraction images improved the visual perception of abnormalities in the lungs by using the proposed method. The VF and CF models achieved a higher accuracy than the demons and the B-spline methods. Furthermore, the proposed methodology demonstrated the ability to create clinically acceptable temporal subtraction chest radiographs that enhance interval changes and can be used to detect abnormalities such as non-small cell lung cancer.

An image similarity comparison method for images with minor distortions is introduced in this paper. The proposed image similarity metrics is based on a new method to measure structure similarity for image quality comparisons. We make use of the fact that Dual-Tree wavelet Transform (DTWT) can provide direction selectivity and keep the structure features between the original and images with minor distortions. Despite the simplicity of our method, our experimental results demonstrate the effectiveness of the proposed method.

This paper examines infrared image registration applied to human breast. The main aspect of the paper is the use of specific characteristics of these examinations that are important on diagnosis and have not been applied properly yet. The approach considers that the subject presents 3D degrees of freedom for movement and its correction by 3 types of transformations are compared by several image similarity metrics (Mean Squares, Normalized Correlation, Mutual Information and Normalized Mutual Information). Moreover, this evaluation allows considering clinically relevant factors (i.e. the ability to detect abnormalities in image pairs and its physiological correctness). Furthermore, we study the number of iterations for convergence of each type of function used in the registration algorithms and a ground truth is published for validation purposes.

Cross-browser incompatibilities (XBIs) are defined as inconsistencies that can be observed in Web applications when they are rendered in a specific browser compared to others. These inconsistencies are associated with differences in the way each browser implements its capabilities and renders Web applications. The inconsistencies range from minor layout differences to lack of core functionalities of Web applications when rendered in specific browsers. The state of the art proposes different approaches for detecting XBIs and many of them are based on classification models, using features extracted from the document object model (DOM) structure (DOM-based approaches) and screenshots (computer vision approaches) of Web applications. To the best of our knowledge, a comparison between DOM-based and computer vision classification models has not yet been reported in the literature, and a combination between both approaches could possibly lead to increased accuracy of classification models. In this article, we extend the use of these classification models for detecting layout XBIs in responsive design Web applications, rendered on different browser viewport widths and devices (iPhone 12 mini, iPhone 12, iPhone 12 Pro Max, and Pixel XL). We investigate the use of state-of-the-art classification models (Browserbite, CrossCheck, and our previous work) for detecting layout cross-browser failures, which consist of layout XBIs that negatively affect the layout of responsive design Web applications. Furthermore, we propose an enhanced classification model that combines features from different state-of-the-art classification models (DOM based and computer vision) using feature selection. We built two datasets for evaluating the efficacy of classification models in separately detecting external and internal layout failures using data from 72 responsive design Web applications. The proposed classification model reported the highest F1-score for detecting external layout failures (0.65) and internal layout failures (0.35), and these results reported significant differences compared to Browserbite and CrossCheck classification models. Nevertheless, the experiment showed a lower accuracy in the classification of internal layout failures and suggests the use of other image similarity metrics or deep learning models for increasing the efficacy of classification models.

Automatic localization of pneumonia on chest X-rays (CXRs) is highly desirable both as an interpretive aid to the radiologist and for timely diagnosis of the disease. However, pneumonia's amorphous appearance on CXRs and complexity of normal anatomy in the chest present key challenges that hinder accurate localization. Existing studies in this area are either not optimized to preserve spatial information of abnormality or depend on expensive expert-annotated bounding boxes. We present a novel generative adversarial network (GAN)-based machine learning approach for this problem, which is weakly supervised (does not require any location annotations), was trained to retain spatial information, and can produce pixel-wise abnormality maps highlighting regions of abnormality (as opposed to bounding boxes around abnormality). Our method is based on the Wasserstein GAN framework and, to the best of our knowledge, the first application of GANs to this problem. Specifically, from an abnormal CXR as input, we generated the corresponding pseudo normal CXR image as output. The pseudo normal CXR is the "hypothetical" normal, if the same abnormal CXR were not to have any abnormalities. We surmise that the difference between the pseudo normal and the abnormal CXR highlights the pixels suspected to have pneumonia and hence is our output abnormality map. We trained our algorithm on an "unpaired" data set of abnormal and normal CXRs and did not require any location annotations such as bounding boxes/segmentations of abnormal regions. Furthermore, we incorporated additional prior knowledge/constraints into the model and showed that they help improve localization performance. We validated the model on a data set consisting of 14184 CXRs from the Radiological Society of North America pneumonia detectionchallenge. We evaluated our methods by comparing the generated abnormality maps with radiologist annotated bounding boxes using receiver operating characteristic (ROC) analysis, image similarity metrics such as normalized cross-correlation/mutual information, and abnormality detection rate.We also present visual examples of the abnormality maps, covering various scenarios of abnormality occurrence. Results demonstrate the ability to highlight regions of abnormality with the best method achieving an ROC area under the curve (AUC) of 0.77 and a detection rate of 85%.The GAN tended to perform better as prior knowledge/constraints were incorporated into themodel. We presented a novel GAN based approach for localizing pneumonia on CXRs that (1) does not require expensive hand annotated location ground truth; and (2) was trained to produce abnormality maps at the pixel level as opposed to bounding boxes. We demonstrated the efficacy of our methods via quantitative and qualitative results.

Motivation: In prostate brachytherapy, transrectal ultrasound (TRUS) is used to visualize the anatomy, while implanted seeds can be seen in C-arm fluoroscopy or CT. Intra-operative dosimetry optimization requires localization of the implants in TRUS relative to the anatomy. This could be achieved by registration of TRUS images and the implants reconstructed from fluoroscopy or CT. Methods: TRUS images are filtered, compounded, and registered on the reconstructed implants by using an intensity-based metric based on a 3D point-to-volume registration scheme. A phantom was implanted with 48 seeds, imaged with TRUS and CT/X-ray. Ground-truth registration was established between the two. Seeds were reconstructed from CT/X-ray. Seven TRUS filtering techniques and two image similarity metrics were analyzed as well. Results: For point-to-volume registration, noise reduction combined with beam profile filter and mean squares metrics yielded the best result: an average of 0.38 ± 0.19 mm seed localization error relative to the ground-truth. In human patient data C-arm fluoroscopy images showed 81 radioactive seeds implanted inside the prostate. A qualitative analysis showed clinically correct agreement between the seeds visible in TRUS and reconstructed from intra-operative fluoroscopy imaging. The measured registration error compared to the manually selected seed locations by the clinician was 2.86 ± 1.26 mm. Conclusion: Fully automated seed localization in TRUS performed excellently on ground-truth phantom, adequate in clinical data and was time efficient having an average runtime of 90 seconds.

Stylized text plays a crucial role in graphic design, digital art, and media, yet most existing stylizations are embedded in flattened raster images, limiting editability and reuse. This paper presents TextFX, a framework that reconstructs stylized text from raster images into editable, parameterized layers. TextFX decomposes complex visual effects—such as shadows, gradients, strokes, and embossing—into interpretable style parameters. Quantitative results show consistent improvement across image similarity metrics (SSIM, RMSE, SRE, SAnM), and user studies with designers and novices confirm high usability, quality, and adoption potential. Experiments also showed that the addition of human-in-the-loop corrections during refinement, substantially improves the accuracy and perceptual quality of the reconstruction. TextFX, therefore, bridges the gap between rasterized artwork and structured design, enabling flexible, editable typography and setting the stage for prompt-driven, generative design workflows.

Over the past decade, there have been significant developments in the mechanisms for examination of biological and material samples. These developments exploit techniques in light microscopy to elucidate specific parts of cells and tissues, as well as inorganic particles. In recent years, spectral microscopy has become more prevalent for characterization of samples. Simultaneously, sensor technology has progressed as well and modern charge-coupled devices (CCD) cameras are now capable of achieving high spatial resolution and high sensitivity measurements of signals in the optical microscope. One major impediment in obtaining absolute quantitative information of imaged samples is the lack of automated photometric calibration mechanisms for spectral microscopes. In this paper, we present a methodology for achieving photometric calibration of an automated spectral imaging system targeted towards examination of biological samples. By acquiring spatial and spectral data simultaneously, spectral imaging allows one to exploit physical connections between a particle's morphology and its characteristic response to the optical spectrum. In composite biological material, the interpretation of the spectra is a complicated problem. This is because any light source and charge-coupled device camera used for data acquisition does not have a uniform illumination spectra and quantum efficiency, respectively, across the emitted light spectra. To balance the spectral response across individual wavelengths, our method modulates the exposure duration for the charge-coupled device camera during image acquisition. We present an image similarity based method to calibrate the system. Experiments to test the effectiveness of the calibration method under the various image similarity metrics are presented along with results to show the calibrated system's ability to accurately measure spectra based on the measured transmission profiles of optical filters.

JPEG is the prevailing compression algorithm used for digital images. Compression ratio and quality depend on quantization tables that are matrixes of 64 integers. The quality of compression for many applications has to be determined not by human judgment, but by software systems that perform some processing on compressed images, based on successfulness of such processing. Since there are many such applications, there is not unique best quantization table but it has to be selected for each application. Quantization table selection is intractable combinatorial problem that can be successfully solved by swarm intelligence metaheuristics. In this paper we present framework for application of the recently introduced firefly algorithm to the quantization table selection problem for different image similarity metrics.