Automatic cloud-free image generation from high-resolution multitemporal imagery

Cloud-free remote sensing images are required for many applications, such as land cover classification, land surface temperature retrieval and agricultural-drought monitoring. Cloud cover in remote sensing images can be pervasive, dynamic and often unavoidable. Current techniques of cloud removal for the VNIR (visible and near-infrared) bands still encounters the problem of pixel values estimated for the cloudy area incomparable and inconsistent with the cloud-free region in the target image. In this paper, we proposed an efficient approach to remove thick clouds and their shadows in VNIR bands using multi-temporal images with good maintenance of DN (digital number) value consistency. We constructed the spectral similarity between the target image and reference one for DN value estimation of the cloudy pixels. The information reconstruction was done with 10 neighboring cloud-free pair-pixels with the highest similarity over a small window centering the cloudy pixel between target and reference images. Four Landsat5 TM images around Nanjing city of Jiangsu Province in Eastern China were used to validate the approach over four representative surface patterns (mountain, plain, water and city) for diverse sizes of cloud cover. Comparison with the conventional approaches indicates high accuracy of the approach in cloud removal for the VNIR bands. The approach was applied to the Landsat8 OLI (Operational Land Imager) image on 29 April 2016 in Nanjing area using two reference images. Very good consistency was achieved in the resulted images, which confirms that the proposed approach could be served as an alternative for cloud removal in the VNIR bands using multi-temporal images.

An important aspect of 2D object recognition is matching. Based on a generalized distance transform method, Borgefors introduced the concept of the Chamfer scheme to 2D object matching. In Chamfer matching, a successful match is obtained when a scene in a reference image is correctly matched to the corresponding scene in the target image. The successful match corresponds to the lowest matching measure computed from the sum of Chamfer distances. However, if the target image has not been correctly corrected for distortions, there may not be an ideal best fit between the reference and target images. Instead, matching measures corresponding to local minima exist, for which it is difficult to choose the correct matching measure that corresponds to the best fit. The match fit corresponding to the global minimum measure may not necessarily be the best fit as there are many sub-optimal measures that could provide equally good fits between the target and reference images. This paper advocates that aside from the Chamfer matching measure, additional considerations need to be factored in to improve on finding the true global minimum measure which corresponds to the optimal fit. Int his paper, a new method has been designed to find the optimal fit between poorly-rectified target and reference images. The new approach uses simulated annealing to improve on the matching results. Simulated annealing is a powerful stochastic optimization technique for nonlinear problems. In our novel technique, the energy function for simulated annealing comprises two terms: a smoothness constraint term and the Chamfer matching measure term. The high computational burden of simulated annealing is reduced by using edge information for the matching process. Results are presented to illustrate the new matching technique.© (1996) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

Change detection is usually treated as a problem of explicitly detecting land cover transitions in satellite images obtained at different times, and helps with emergency response and government management. This study presents an unsupervised change detection method based on the image fusion of multi-temporal images. The main objective of this study is to improve the accuracy of unsupervised change detection from high-resolution multi-temporal images. Our method effectively reduces change detection errors, since spatial displacement and spectral differences between multi-temporal images are evaluated. To this end, a total of four cross-fused images are generated with multi-temporal images, and the iteratively reweighted multivariate alteration detection (IR-MAD) method—a measure for the spectral distortion of change information—is applied to the fused images. In this experiment, the land cover change maps were extracted using multi-temporal IKONOS-2, WorldView-3, and GF-1 satellite images. The effectiveness of the proposed method compared with other unsupervised change detection methods is demonstrated through experimentation. The proposed method achieved an overall accuracy of 80.51% and 97.87% for cases 1 and 2, respectively. Moreover, the proposed method performed better when differentiating the water area from the vegetation area compared to the existing change detection methods. Although the water area beneath moderate and sparse vegetation canopy was captured, vegetation cover and paved regions of the water body were the main sources of omission error, and commission errors occurred primarily in pixels of mixed land use and along the water body edge. Nevertheless, the proposed method, in conjunction with high-resolution satellite imagery, offers a robust and flexible approach to land cover change mapping that requires no ancillary data for rapid implementation.

Magnetic resonance imaging plays a crucial role in clinical diagnosis due to its ability to provide information on soft tissue structure. At present, multi-contrast super-resolution (SR) methods for magnetic resonance (MR) images have been widely studied and have achieved good results. However, most studies overlook the impact of modal differences between reference and low-resolution (LR) image features on feature reconstruction, which may result in inaccurate reconstruction of detail features due to improper alignment of structural features. To address this issue, we propose a high-frequency feature transfer network for MR image SR task (HFTN-SR), which consists of two feature extraction branches and one high-frequency feature transfer branch. Considering the modal differences between the target and reference images, a high-low frequency decomposition method is used to decompose the reference image into high-frequency and low-frequency components, where the low-frequency components are used in the subsequent network to match LR target image features. A feature extraction block (FEB) is constructed to extract and integrate high-frequency and low-frequency features of the reference image, as well as features of the LR target image. In response to the modal differences between two image features, a feature transfer block (FTB) is designed to establish the correlation between the low-frequency features of the reference image and the target image features, and use the correlation matrix to transfer the high-frequency features of the reference image to the target image features. To further reduce the loss of shallow features caused by the increase in network depth, a standardized combined residual feature module (SCRFM) is constructed to supplement the shallow features of the target image into the final reconstructed features. Experiments on the public dataset FastMRI and the self-built dataset AXA show that the performance of HFTN-SR is superior to some state-of-the-art (SOTA) methods. Notably, HFTN-SR achieves the highest PSNR and SSIM scores across all tested datasets, with significant improvements in visual quality and detail reconstruction.

The spatial pattern and temporal variation of paddy rice fields based on remote sensing have strong effects on the allocation of agricultural water resources on a regional scale. Free Land Remote-Sensing Satellite (Landsat) data have been successfully used to map paddy rice. However, due to frequent clouds and a temporal gap of 16 days, the shortage of Landsat data availability poses a great challenge to long-term paddy rice mapping. This study proposed a decision tree algorithm based on the Enhanced Vegetation Index (EVI) to map multi-season paddy rice in southern China. The study area is located in the Ganfu Plain Irrigation System (GFPIS), where double-cropping rice (early and late rice) and single-cropping rice (middle rice) are mixed. First, we explored the effects of the number and temporal distribution of Landsat images on the classification accuracy. With available cloud-free images in 2017, ten image combinations were set up to map rice fields using the proposed algorithm separately. Then, the algorithm was applied to map historical paddy rice planting in this study area. The results indicated that with a cropland mask, a single-date image in early rice growing season can map the total early rice with an overall accuracy varying from 82.02% to 93.26%, and the accuracy was mainly influenced by the temporal distribution of the images. In addition, through post-processing, multi-temporal images could be used to recognize early rice only, late rice only, double-cropping rice and middle rice with an overall accuracy ranging from 71.83% to 85.81%. In contrast, images in the early or late growing season without obvious vegetation characteristics may result in confusion. Two peak growing season images with obvious differences in the vegetation index played a key role in detecting different rice cropping types. In addition, when the EVI of rice was within a certain range, one or more images could achieve a similar overall accuracy. The total area of middle rice and late rice remained constant before 2001, but it changed from 10.72 × 104 ha in 2001 to 7.55 × 104 ha in 2017. Additionally, there was a twofold increase in the mapped middle rice area.

This paper proposed a new land cover mapping algorithm for remote sensing images under the presence of clouds. Here, we modelled the observed image as the weighted sum between the reflectances from the cloud-free image and clouds where weights depend on the degree of cloud contamination called the ‘cloud thickness’ map. Next, we represented the cloud thickness map using a level set function. Similarly, we also represented the land cover map as functions of different level set functions whose values indicate the class label. Next, the maximum a posteriori (MAP) criterion is employed where the most likely land cover and cloud thickness maps are chosen. Under the MAP criterion, the corresponding energy function whose values depend on the level set functions can be derived where the minimum energy point corresponds to the optimum land cover and cloud thickness maps. Since the level set functions are continuous, the optimum solution can be obtained by applying the calculus of variation where the gradient descent algorithm is employed. From the observation model, the cloud-free data can only be reconstructed only if thin to medium clouds are present since thick clouds can completely block the reflectance from the land cover materials. Thus, our algorithm is more suitable for thin to medium than thick cloud covers. Our synthesis and real cloud-contamination examples support our statement since our algorithm achieves significantly higher overall accuracies over other classification techniques, especially, in the thin to medium cloud contamination, and fails to recover the true land cover classes over the thick clouds.

Multiplexed Superresolution CRISPR Imaging of Chromatin in Living Cells

Detection of unruptured aneurysms is a major task in magnetic resonance angiography (MRA). However, it is difficult for radiologists to detect small aneurysms on the maximum intensity projection (MIP) images because adjacent vessels may overlap with the aneurysms. Therefore, we proposed a method for making a new MIP image, the SelMIP image, with the interested vessels only, as opposed to all vessels, by manually selecting a cerebral artery from a list of cerebral arteries recognized automatically. By using our new SelMIP viewing technique, the selected vessel regions can also be observed from various directions and would further facilitate the radiologists in detecting small aneurysms. For automated classification of cerebral arteries, two 3D images, a target image and a reference image, are compared. Image registration is performed using the global matching and feature correspondence techniques. Segmentation of vessels in the target image is performed using the thresholding and region growing techniques. The segmented vessel regions were classified into eight cerebral arteries by calculating the Euclidean distance between a voxel in the target image and each of the voxels in the labeled eight vessel regions in the reference image. In applying the automated cerebral arteries recognization algorithm to thirteen MRA studies, results of 10 MRA studies were evaluated as clinically acceptable. Our new viewing technique would be useful in assisting radiologists for detection of aneurysms and for reducing the interpretation time.

Multiple cropping, a common practice of intensive agriculture that grows crops multiple times in the agricultural land in one growing season, is an effective way to fulfill the food demand given limited cropland areas. Deriving cropping cycles from satellite data provides the spatial distribution of cropping intensities that allows for monitoring of the multiple cropping activities over large areas. Although efforts have been made to map cropping cycles at 500 m or coarser resolution, producing cropping cycle maps at high resolution remain challenging because data from single satellite sensor do not provide sufficient spatiotemporal observations. In this paper, we generate dense time series of satellite data at 30 m resolution by fusion of Landsat and MODIS data, and derive the cropping cycles from the fused time series data. The method achieves overall accuracies of 92.5% and 89.2%, respectively, for two typical regions of multiple cropping in China using samples identified based on satellite time series data, and an overall accuracy of 81.2% for four subregions using all samples identified based on multi-temporal high resolution images. The mapped crop cycles show to be reasonable geographically and agree with the national census data. The fusion approach provides a feasible way to map cropping cycles at 30 m resolution and enables improved depiction of the spatial distribution of multiple cropping.

Optical satellite images are the main source of geophysical information which can be used for various land surface studies. But one of the most challenging issue faced by these images are the severe cloud contamination. As far as optical satellite images are of concern, thick clouds will completely obstructs the observation of landscape and uneven illumination caused by thin clouds will further limits the processing of optical satellite images. So a method to eliminate the impact of clouds in optical satellite images is necessary to improve the effectiveness and availability of remote sensing data for various land surface studies. In this paper, a novel methodology for removing the uneven illumination caused by thick and thin clouds is intro-duced. For this approach, multitemporal cloud contaminated optical satellite images are required with the assumption that in each image cloud cover change significantly over a short duration of time. A cloud detection algorithm in frequency domain is used to detect the cloud contaminated portions in multitemporal optical satellite images. Suitable cloud free patch from reference images for reconstructing cloud contaminated portions in target image are selected based on lower amount of cloud rate and quality assessment technique. Some experimental analysis is conducted on collected Aqua MODIS sensor captured multitemporalsatellite images. Experimental result shows that the proposed algorithm can effectively remove majority of cloud contamination within the optical satellite images. Finally quantitative measures are performed to evaluate the feasibility of the methodology.

Fourier phase retrieval problem is equivalent to the recovery of a two-dimensional image from its autocorrelation measurements. This problem is generally nonlinear and nonconvex. Good initialization and prior information about the support or sparsity of the target image are often critical for a robust recovery. In this paper, we show that the presence of a known reference image can help us solve the nonlinear phase retrieval problem as a sequence of small linear inverse problems. Instead of recovering the entire image at once, our sequential method recovers a small number of rows or columns by solving a linear deconvolution problem at every step. Existing methods for the reference-based (holographic) phase retrieval either assume that the reference and target images are sufficiently separated so that the recovery problem is linear or recover the image via nonlinear optimization. In contrast, our proposed method does not require the separation condition. We performed an extensive set of simulations to demonstrate that our proposed method can successfully recover images from autocorrelation data under different settings of reference placement and noise.

We present a novel approach to change detection between two brain MRI scans (reference and target.) The proposed method uses a single modality to find subtle changes; and does not require prior knowledge (learning) of the type of changes to be sought. The method is based on the computation of a local kernel from the reference image, which measures the likeness of a pixel to its surroundings. This kernel is then used as a feature and compared against analogous features from the target image. This comparison is made using cosine similarity. The overall algorithm yields a scalar dissimilarity map (DM), indicating the local statistical likelihood of dissimilarity between the reference and target images. DM values exceeding a threshold then identify meaningful and relevant changes. The proposed method is robust to various challenging conditions including unequal signal strength.

Image registration is a vital problem in medical imaging. It has many potential applications in clinical diagnosis (e.g. diagnosis of cardiac, retinal, pelvic, renal, abdomen, liver and tissue disorders). In this paper a new technique has been proposed based on feature based image registration using SURF (Speeded up robust features) and Hessian matrix feature extraction techniques for detecting, extracting and matching corresponding points in the reference (fixed) and source image (deformed). The source image is aligned using SURF and Hessian feature extraction techniques to align with the reference image. The proposed technique is based on multimodality of images, feature based classification and non-rigid images as the images taken are from different sources like MRI, CT scan and X-ray. The proposed technique uses feature-based methods to find correspondence between image features such as points, lines and contours. The non-rigid images use fixed (source) image and also deformable (target) image of the subject and aligns target image according to source/reference image.

This dissertation addresses the persistent challenge of missing or cloud-obscured optical satellite images, which hinder timely and complete Earth observation. Cloud cover disrupts remote sensing applications such as change detection and environmental monitoring, where full-scene visibility is essential. Since consistently cloud-free imagery is often unattainable, this research focuses on enhancing the effectiveness and reliability of reconstructed images. Effectiveness refers to the ability to restore cloud-covered regions for practical use, while reliability ensures the reconstructed data accurately reflects the Earth’s surface.<br/>To achieve this, the thesis proposes multi-source data fusion methods, integrating SAR and optical imagery to improve image quality in real-world applications like crop mapping and post-flood reconstruction. Four key objectives guide the research:<br/>1. Feature Enhancement Network (FENet) is introduced to overcome limitations in existing SAR-optical fusion methods, which often struggle with accuracy and scalability. FENet learns residual maps to preserve clear pixels and reconstruct obscured areas. It incorporates a Linear Attention Mechanism (LAM) to capture global context, enhancing reconstruction accuracy. Experiments show FENet delivers superior visual consistency and faster processing, making it suitable for large-scale deployment.<br/>2. To address the need for efficient processing of large datasets, the Feature Pyramid Network (FPNet) is developed. FPNet combines multi-scale feature extraction with a streamlined fusion design, significantly reducing computational load. It achieves inference speeds up to 96 FPS and shortens training time, demonstrating strong practical efficiency.<br/>3. The third objective evaluates how reconstructed cloud-free images impact downstream tasks, specifically crop type mapping. Results indicate that improved image quality modestly enhances classification accuracy and robustness, validating the utility of SAR-optical fusion in agricultural analysis.<br/>4. Finally, the Cloud-Resilient Generation Network (CRGenNet) is proposed to enable full-scene image recovery under heavy cloud conditions. Using dual-temporal and dual-modal inputs, CRGenNet integrates DownUpBlock and FusionAttention modules to effectively fuse SAR and optical data. It consistently performs well across diverse land cover types and environmental challenges.<br/>In summary, this thesis presents deep learning-based methods that significantly improve optical satellite image quality under cloud contamination, supporting reliable Earth observation when clear imagery is limited.<br/>

High/very-high-resolution (HR/VHR) multitemporal images are important in remote sensing to monitor the dynamics of the Earth's surface. Unsupervised object-based image analysis provides an effective solution to analyze such images. Image semantic segmentation assigns pixel labels from meaningful object groups and has been extensively studied in the context of single-image analysis, however not explored for multitemporal one. In this article, we propose to extend supervised semantic segmentation to the unsupervised joint semantic segmentation of multitemporal images. We propose a novel method that processes multitemporal images by separately feeding to a deep network comprising of trainable convolutional layers. The training process does not involve any external label, and segmentation labels are obtained from the argmax classification of the final layer. A novel loss function is used to detect object segments from individual images as well as establish a correspondence between distinct multitemporal segments. Multitemporal semantic labels and weights of the trainable layers are jointly optimized in iterations. We tested the method on three different HR/VHR data sets from Munich, Paris, and Trento, which shows the method to be effective. We further extended the proposed joint segmentation method for change detection (CD) and tested on a VHR multisensor data set from Trento.