Satellite SAR-optical image fusion for cloud-free image generation

Optical satellite images are the main source of geophysical information which can be used for various land surface studies. But one of the most challenging issue faced by these images are the severe cloud contamination. As far as optical satellite images are of concern, thick clouds will completely obstructs the observation of landscape and uneven illumination caused by thin clouds will further limits the processing of optical satellite images. So a method to eliminate the impact of clouds in optical satellite images is necessary to improve the effectiveness and availability of remote sensing data for various land surface studies. In this paper, a novel methodology for removing the uneven illumination caused by thick and thin clouds is intro-duced. For this approach, multitemporal cloud contaminated optical satellite images are required with the assumption that in each image cloud cover change significantly over a short duration of time. A cloud detection algorithm in frequency domain is used to detect the cloud contaminated portions in multitemporal optical satellite images. Suitable cloud free patch from reference images for reconstructing cloud contaminated portions in target image are selected based on lower amount of cloud rate and quality assessment technique. Some experimental analysis is conducted on collected Aqua MODIS sensor captured multitemporalsatellite images. Experimental result shows that the proposed algorithm can effectively remove majority of cloud contamination within the optical satellite images. Finally quantitative measures are performed to evaluate the feasibility of the methodology.

Cloud contamination impedes timely and accurate urban monitoring via optical remote sensing in numerous emergent cloudy situations, such as rainstorms and flood responses. Synthetic aperture radar (SAR) offers advantages because of its all-weather capability but has geometric backscattering constraints such as layover and foreshortening. The complementary information from these modalities highlights the importance of fusing optical and SAR data. However, most existing fusion methods are tailored for cloud-free scenarios, overlooking the challenge of cloud contamination. This study develops a novel fusion framework, SAR-optical dictionary learning (SODL), to synergize cloud-contaminated optical data and polarimetric SAR data, with an example of estimating urban impervious surfaces (UIS). SODL avoids cloud interference by constructing a coupled SAR-optical dictionary space with specifically designed constraints, effectively maximizing the discriminative capabilities of both data sources. The experimental results across subtropical China demonstrate the robust performance of SODL, with improvements in UIS estimation of up to 33.74% in overall accuracy (OA) compared with traditional and deep learning fusion methods. Furthermore, SAR and optical fusion with SODL significantly outperform the results using SAR or optical images alone, increasing the OA by up to 45.71%. These outcomes underscore the promising potential of SODL in urban monitoring in cloudy regions.

Sentinel-1, a pair of Synthetic Aperture Radar (SAR) sensors, provides valuable all-weather and day-night imaging capabilities, enabling continuous monitoring of vegetation dynamics even in the presence of cloud cover. SAR sensors excel at penetrating vegetation canopies and providing information on crucial factors like vegetation structure, biomass, and moisture content. However, most remote sensing vegetation studies have primarily relied on optical data, benefiting from longer historical datasets but facing challenges due to atmospheric interference, limited temporal resolution. Moreover, there is no research assessing the sensitivity of optical and radar data to the dynamics of vegetation in Mozambique. This study investigates the sensitivity of the Sentinel-1 (S-1) backscatter signal to vegetation dynamics over Mozambique. We compare it with the Normalized Difference Vegetation Index (NDVI) from MODIS data and explore its relationship with precipitation variability and droughts across different land covers, including forest, cropland, herbaceous vegetation, and herbaceous wetland in Mozambique.The Sentinel-1 VV and VH polarized images were used to calculate the Cross Ratio (CR=VH/VV). Temporal behaviors of CR S-1 and MODIS NDVI were analyzed from 2017 to 2022, examining seasonal patterns, inter-annual variability, trends, and outliers. NDVI anomalies were calculated to identify the spatial and temporal occurrence of agricultural droughts, while CHIRPS precipitation data was utilized to detect fluctuations in the CR S-1 and NDVI time series relative to precipitation. The analysis revealed distinct seasonality in the CR time series data across all land cover types. Notably, croplands, herbaceous vegetation, and herbaceous wetlands exhibited a consistent increase in CR during winter months, followed by a decline during summer months. In contrast, forests displayed an inverse trend, with CR decreasing in winter and increasing in summer. Furthermore, no pronounced CR patterns were observed in herbaceous wetlands during 2019, coinciding with the agricultural drought between 2018 and 2019. Additionally, the seasonality of MODIS NDVI time series remained consistent across all land cover types, with no noticeable differences. It was observed that fluctuations in NDVI time series preceded those in CR for these specific land cover types, suggesting a potential correlation with photosynthetic activity and subsequent biomass production. Significantly, this trend was found to be opposite in forested areas. Overall, the CR trend exhibited a clear correlation with rainfall seasonality across the various land cover types, except for forests where an inverse relationship was observed. On the other hand, NDVI demonstrated a higher sensitivity to changes in precipitation across the different land cover categories.These findings highlight the unique sensitivity of Sentinel-1 SAR data in capturing the intricate dynamics of vegetation across diverse land cover types in Mozambique, providing valuable complementary information to traditional optical data sources.

Abstract. Strategic response during disasters requires the swiftness of the flow of data and information. Simple change detections can be used from Optical Satellite Images to achieve fast results that can be used for analysis and planning. In response to the earthquake that devastated the Noto Peninsula of Japan on New Year’s Day, news reports and early optical satellite images were utilized to identify areas that were damaged by the earthquake. The changes in the reflectance values and indicators, such as NDBI and NDVI were analyzed to identify the areas with damages such as collapsed buildings and landslides. From the after-event satellite images of PlanetScope and Sentinel-2, three areas were identified as severely affected based on news reports namely, Wajima City, Niemachi, and Noto Town. To verify the results of the analysis, aerial images provided by the Geographical Survey Institute of Japan, and Google Earth Street View Images were used. Through change detection methods, the locations of damaged areas were pinpointed using Optical Satellite Images after the earthquake incident.

Modern change detection (CD) has achieved remarkable success by the powerful\ndiscriminative ability of deep convolutions. However, high-resolution remote\nsensing CD remains challenging due to the complexity of objects in the scene.\nObjects with the same semantic concept may show distinct spectral\ncharacteristics at different times and spatial locations. Most recent CD\npipelines using pure convolutions are still struggling to relate long-range\nconcepts in space-time. Non-local self-attention approaches show promising\nperformance via modeling dense relations among pixels, yet are computationally\ninefficient. Here, we propose a bitemporal image transformer (BIT) to\nefficiently and effectively model contexts within the spatial-temporal domain.\nOur intuition is that the high-level concepts of the change of interest can be\nrepresented by a few visual words, i.e., semantic tokens. To achieve this, we\nexpress the bitemporal image into a few tokens, and use a transformer encoder\nto model contexts in the compact token-based space-time. The learned\ncontext-rich tokens are then feedback to the pixel-space for refining the\noriginal features via a transformer decoder. We incorporate BIT in a deep\nfeature differencing-based CD framework. Extensive experiments on three CD\ndatasets demonstrate the effectiveness and efficiency of the proposed method.\nNotably, our BIT-based model significantly outperforms the purely convolutional\nbaseline using only 3 times lower computational costs and model parameters.\nBased on a naive backbone (ResNet18) without sophisticated structures (e.g.,\nFPN, UNet), our model surpasses several state-of-the-art CD methods, including\nbetter than four recent attention-based methods in terms of efficiency and\naccuracy. Our code is available at https://github.com/justchenhao/BIT\\_CD.\n

This paper proposed a new land cover mapping algorithm for remote sensing images under the presence of clouds. Here, we modelled the observed image as the weighted sum between the reflectances from the cloud-free image and clouds where weights depend on the degree of cloud contamination called the ‘cloud thickness’ map. Next, we represented the cloud thickness map using a level set function. Similarly, we also represented the land cover map as functions of different level set functions whose values indicate the class label. Next, the maximum a posteriori (MAP) criterion is employed where the most likely land cover and cloud thickness maps are chosen. Under the MAP criterion, the corresponding energy function whose values depend on the level set functions can be derived where the minimum energy point corresponds to the optimum land cover and cloud thickness maps. Since the level set functions are continuous, the optimum solution can be obtained by applying the calculus of variation where the gradient descent algorithm is employed. From the observation model, the cloud-free data can only be reconstructed only if thin to medium clouds are present since thick clouds can completely block the reflectance from the land cover materials. Thus, our algorithm is more suitable for thin to medium than thick cloud covers. Our synthesis and real cloud-contamination examples support our statement since our algorithm achieves significantly higher overall accuracies over other classification techniques, especially, in the thin to medium cloud contamination, and fails to recover the true land cover classes over the thick clouds.

In general object detection, scale variation is always a big challenge. At present, feature pyramid networks are employed in numerous methods to alleviate the problems caused by large scale range of objects in object detection, which makes use of multi-level features extracted from the backbone for top-down upsampling and fusion to acquire a set of multi-scale depth image features. However, the feature pyramid network proposed by Ghiasi et al. adopts a simple fusion method, which fails to consider the fusion feature context, and therefore, it is difficult to acquire good features. In addition, the fusion of multi-scale features directly by traditional upsampling is prone to feature misalignment and loss of details. In this paper, an adaptive feature pyramid network is proposed based on the feature pyramid network to alleviate the foregoing potential problems, which includes two major designs, i.e., adaptive feature upsampling and adaptive feature fusion. The adaptive feature upsampling aims to predict a group of sampling points of each pixel through some models, and constitute feature representation of the pixel by feature combination of sampling points, while adaptive feature fusion is to construct pixel-level fusion weights between fusion features through attention mechanism. The experimental results verified the effectiveness of the method proposed in this paper. On the public object detection dataset MS-COCO test-dev, Faster R-CNN model achieved performance improvement of 1.2 AP by virtue of the adaptive feature pyramid network, and FCOS model could achieve performance improvement of 1.0 AP. What's more, the experiments also validated that the adaptive feature pyramid network proposed herein was more accurate for object localization.

SIFNet: A self-attention interaction fusion network for multisource satellite imagery template matching

At the present time, a lot of problems in a sphere of fundamental sciences as well as technical and applied tasks can be solved only with the use of satellite images, since their usage reduces material, financial and time costs significantly in comparison with traditional methods. One of the modern integrated approach remote sensing processing is to join the measurements obtained from the various sources, such as optical and radar sensors, allowing to achieve a gain in comparison with independent processing due to the extension of the information volume and the opportunities of data acquisition (weather conditions, spectral ranges, etc.). However, methods of digital processing and interpretation of radar data, as well as qualitative and proven methods and algorithms for joint processing of optical and radar satellite images, has not sufficiently been well developed yet. Therefore, the development of new methods and information technology of joint analysis and interpretation of optical and radar data which are a major issue of the current paper, are certainly relevant. The paper presents an information technology for joint processing of optical and radar satellite imagery, based on training the processing procedure based on the reference values of data from sensors of the one type (optical data), followed by applying to both data types: optical and SAR data.

Exploring optimal integration schemes for Sentinel-1 SAR and Sentinel-2 multispectral data in land cover mapping across different atmospheric conditions

Cloud contamination of optical data is a constant and annoying feature of time series analyses, whether while using vegetation indices or surface temperatures, since it tends to decrease artificially the values taken by these parameters. Therefore, any time series analysis of optical data needs a previous step for gap-filling reconstruction of the time series. Numerous techniques have been presented in the literature to carry out this preliminary and mandatory step. However, the evaluation and comparison of these techniques is difficult, since no “truth” time series is available. We present here a probabilistic model (CloudSim) to provide global typical annual time series for NDVI (Normalized Difference Vegetation Index) and surface temperature (ST) time series (both over land and sea) with and without cloud contamination. This model takes advantage of the newly released Long Term Data Record Version 4 (LTDR-V4) dataset, by estimating at the pixel level daily probabilities of cloud presence, as well as mean and standard deviations for both cloud and clear acquisitions of daily NDVI and ST. Therefore, we can simulate real time series of cloud contamination influence on NDVI or ST parameters. These simulated time series allow for the assessment of the validity and usefulness of any time series reconstruction method, as well as an objective comparison of the efficiency of different methods.

Crop type inventory and within season estimates at moderate (<30 m) resolution have been elusive in many regions due to the lack of temporal frequency, clouds, and restrictive data policies. New opportunities exist from the operational fusion of Landsat-8 Operational Land Imager (OLI), Sentinel-2 (A & B), and Sentinel-1 (A & B) which provide more frequent open access observations now that these satellites are fully operating. The overarching goal of this research application was to compare Harmonized Landsat-8 Sentinel-2 (HLS), Sentinel-1 (S1), and combined radar and optical data in an operational, near-real-time (within 24 h) context. We evaluated the ability of these Earth observations (EO) across major crops in four case study regions in United States (US) production hot spots. Hindcast time series combinations of these EO were fed into random forest classifiers trained with crop cover type information from the Cropland Data Layer (CDL) and ancillary ground truth. The outcomes show HLS achieved high (>85%) accuracies and the ability to provide insight on crop location and extent within the crop season. HLS fused with S1 had, at times, a higher accuracy (5–10% relative overall accuracy and kappa increases) within season although the combination of fused data was minimal at times, crop dependent, and the accuracies tended to converge by harvest. In cloud prone regions and certain temporal periods, S1 performed well overall. The growth in the availability of time dense moderate resolution data streams and different sensitivities of optical and radar data provide a mechanism for within season crop mapping and area estimates that can help improve food security.

In the neck part of a two-stage object detection network, feature fusion is generally carried out in either a top-down or bottom-up manner. However, two types of imbalance may exist: feature imbalance in the neck of the model and gradient imbalance in the region of interest extraction layer due to the scale changes of objects. The deeper the network is, the more abstract the learned features are, that is to say, more semantic information can be extracted. However, the extracted image background, spatial location, and other resolution information are less. In contrast, the shallow part can learn little semantic information, but a lot of spatial location information. We propose the Both Ends to Centre to Multiple Layers (BEtM) feature fusion method to solve the feature imbalance problem in the neck and a Multi-level Region of Interest Feature Extraction (MRoIE) layer to solve the gradient imbalance problem. In combination with the Region-based Convolutional Neural Network (R-CNN) framework, our Balanced Feature Fusion (BFF) method offers significantly improved network performance compared with the Faster R-CNN architecture. On the MS COCO 2017 dataset, it achieves an average precision (AP) that is 1.9 points and 3.2 points higher than those of the Feature Pyramid Network (FPN) Faster R-CNN framework and the Generic Region of Interest Extractor (GRoIE) framework, respectively.

Satellite optical images and altimetry data are two major data sources used in Antarctic research. The integration use of these two datasets is expected to provide more accurate and higher quality products, during which data registration is the first issue that needs to be solved. This paper presents an alternative approach for the registration of high-resolution satellite optical images and ICESat (Ice, Cloud, and land Elevation Satellite) laser altimetry data. Due to the sparse distribution characteristic of the ICESat laser point data, it is difficult and even impossible to find same-type conjugate features between ICESat data and satellite optical images. The method is implemented in a direct way to correct the point-to-line inconsistency in image space through 2D transformation between the projected terrain feature points and the corresponding 2D image lines, which is simpler than discrepancy correction in object space that requires stereo images for 3D model construction, and easier than the indirect way of image orientation correction via photogrammetric bundle adjustment. The correction parameters are further incorporated into imaging model through RPCs (Rational Polynomial Coefficients) generation/regeneration for the convenience of photogrammetric applications. The experimental results by using the ASTER (Advanced Spaceborne Thermal Emission and Reflection Radiometer) images and ZY-3 (Ziyuan-3 satellite) images for registration with ICESat data showed that sub-pixel level registration accuracies were achieved after registration, which have validated the feasibility and effectiveness of the presented approach.

Abstract. Disaster events occur around the world and cause significant damage to human life and property. Earth observation (EO) data enables rapid and comprehensive building damage assessment, an essential capability crucial in the aftermath of a disaster to reduce human casualties and inform disaster relief efforts. Recent research focuses on developing artificial intelligence (AI) models to accurately map unseen disaster events, mostly using optical EO data. These solutions based on optical data are limited to clear skies and daylight hours, preventing a prompt response to disasters. Integrating multimodal EO data, particularly combining optical and synthetic aperture radar (SAR) imagery, makes it possible to provide all-weather, day-and-night disaster responses. Despite this potential, the lack of suitable benchmark datasets has constrained the development of robust multimodal AI models. In this paper, we present a Building damage assessment dataset using veRy-hIGH-resoluTion optical and SAR imagery (Bright) to support AI-based all-weather disaster response. To the best of our knowledge, Bright is the first open-access, globally distributed, event-diverse multimodal dataset specifically curated to support AI-based disaster response. It covers five types of natural disasters and two types of human-made disasters across 14 regions worldwide, focusing on developing countries where external assistance is most needed. The dataset's optical and SAR images with spatial resolutions between 0.3 and 1 m provide detailed representations of individual buildings, making it ideal for precise damage assessment. We train seven advanced AI models on Bright to validate transferability and robustness. Beyond that, it also serves as a challenging benchmark for a variety of tasks in real-world disaster scenarios, including unsupervised domain adaptation, semi-supervised learning, unsupervised multimodal change detection, and unsupervised multimodal image matching. The experimental results serve as baselines to inspire future research and model development. The dataset (https://doi.org/10.5281/zenodo.14619797, Chen et al., 2025a), along with the code and pretrained models, is available at https://github.com/ChenHongruixuan/BRIGHT (last access: 7 November 2025) and will be updated as and when a new disaster data is available. Bright also serves as the official dataset for the 2025 IEEE GRSS Data Fusion Contest Track II. We hope that this effort will promote the development of AI-driven methods in support of people in disaster-affected areas.