Pansharpening Scheme Using Bi-dimensional Empirical Mode Decomposition and Neural Network

This paper proposes an efficient and high-fidelity image fusion method based on adaptive smoothing filtering for panchromatic (PAN) and multispectral (MS) image fusion. The scale ratio reflects the ratio of spatial resolution between the panchromatic image and the multispectral image. When facing a multiscale fusion task, traditional methods are unable to simultaneously handle the problems of spectral resolution loss resulting from high scale ratios and the issue of reduced spatial resolution due to low scale ratios. To adapt to the fusion of panchromatic and multispectral satellite images of different scales, this paper improves the problem of the insufficient filtering of high-frequency information of remote sensing images of different scales by the classic filter-based intensity modulation (SFIM) model. It uses Gaussian convolution kernels instead of traditional mean convolution kernels and builds a Gaussian pyramid to adaptively construct convolution kernels of different scales to filter out high-frequency information of high-resolution images. It can adaptively process panchromatic multispectral images of different scales, iteratively filter the spatial information in panchromatic images, and ensure that the scale transformation is consistent with the definition of multispectral images. Using 15 common fusion methods, this paper compares the experimental results of ZY-3 with scale ratio 2.7 and SV-1 with scale ratio 4 data. The results show that the method proposed in this paper retains good spatial information for image fusion at different scales and has good spectral preservation.

In remote sensing, satellite images acquired from sensors provide either high spectral or high spatial resolution. The pansharpening framework is applied to remote-sensing systems to enhance the spatial quality of coarse-resolution multispectral (MS) images using information from panchromatic imagery. A multidecomposition pansharpening approach combining MS and panchromatic (PAN) images is proposed in this paper in order to bring the resolution of the low-resolution MS imagery up to that of the panchromatic images. In particular, multilevel wavelet decomposition is applied to the luminance-chrominance (YUV) space transformation (taking into account the red green and blue (RGB) bands) or extended-YUV transformation (taking into account the near infrared (NIR) band in addition to RGB) of the original MS channels, where geometrical details from the panchromatic image are introduced into the MS ones. Our approach contains a preprocessing step that consists of homogenizing the luminance, Y, and the panchromatic image reflectance, which are, respectively, a value integrated over a wavelength spectrum and simply a linear combination of some values in the same spectrum. Hence, as the panchromatic image reflectance and luminance reflectance correspond to different measurements, they do not correspond to the same physical information, which results in a difference between their histograms. Therefore, simple histogram matching is traditionally applied to panchromatic data to fit it to the luminance to avoid colour distortion after fusion. However, as the transformation concerns just the details of the panchromatic and MS images, a new scheme for matching the images which ignores the divergence between their approximations and maximizes the resemblance between their details is proposed in this work. After that, the fusion approach is applied, and in contrast to the original approach where the details of the fused MS luminance are set equal to the PAN luminance, we propose an adaptive approach in which just a part of the PAN details proportional to the similarity between the luminance and lowered PAN image is taken. Indeed, high-resolution geometrical details cannot be similar if the low-resolution details are not in good agreement. Besides, as the agreement between PAN and MS images depends on the occupation class, we have created a segmentation map and then computed separately the correlation in each region. Finally, the evaluation is done based on QuickBird and Pleiades-1A data sets showing rural and suburban areas. When compared to recent methods, our approach provides better results.

Remote sensing image fusion (or pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. In this paper, a deep convolutional neural network with two-stream inputs respectively for PAN and MS images is proposed for remote sensing image pan-sharpening. Firstly the network extracts features from PAN and MS images, then it fuses them to form compact feature maps that can represent both spatial and spectral information of PAN and MS images, simultaneously. Finally, the desired high spatial resolution MS image is recovered from the fused features using an encoding-decoding scheme. Experiments on Quickbird satellite images demonstrate that the proposed method can fuse the PAN and MS image effectively.

A new image fusion method based on hue-saturation-value (HSV) color space is proposed, the improved HSV fuses multi-spectral (MS) and panchromatic (PAN) images to improve spatial information and preserve spectral characteristics. The main advantage of the new fusion method is a simple and efficient way which can maximize the extraction of the spatial information and eliminate the disturbance of spectral information in PAN image. the difference of the low components information between PAN and the value component is filtered from the PAN image, and then value component are replaced by the new PAN image, the new HSV color space perform reverse transform to obtain a multispectral image with the high spatial resolution.SPOT-5 and QuickBird MS and PAN images were employed to execute the existing HSV and improved HSV fusion methods. Qualitative and quantitative analyses and classification accuracy assessment were conducted to evaluate the performance of the fusion methods. The results demonstrate that the improved HSV is better than traditional HSV methods. The new fusion method can achieve a wide range of balance between high spatial resolution retention and spectral characteristic preservation.

Superresolution land cover mapping (SRM) is a technique for generating land cover maps with a finer spatial resolution than the input image. In general, either the original multispectral (MS) images or the spectral unmixing results of the MS image are used as input for SRM models. Panchromatic (PAN) images are often captured together with MS images by many remote sensors and provide more spatial information due to their higher spatial resolution compared with the MS image. In this paper, a spatially adaptive spatial–spectral managed SRM model (SA_SSMSRM) that incorporates both MS and PAN images is proposed. SA_SSMSRM aims to better smooth homogeneous regions of objects (which represent a territory within which there is a uniformity in terms of land cover class) and preserve land cover class boundaries simultaneously by using the PAN image pixel photometric distance (i.e., gray-level distance or pixel value difference). Homogeneous regions in the PAN images are usually characterized by the photometric (pixel value) similarity, whereas class boundaries are usually characterized by photometric dissimilarity. The SA_SSMSRM smoothing parameter, which controls the contribution of the prior term (which encodes prior knowledge about land cover spatial patterns), is designed to be spatially adaptive, with its value decreasing if the photometric similarity of neighboring PAN image pixels decreases. SA_SSMSRM was examined on high-spatial-resolution QuickBird images, IKONOS images, and Advanced Land Observing Satellite (ALOS) images with both MS and PAN data. Results showed that the proposed SA_SSMSRM can generate more accurate superresolution maps than other SRM models.

As a branch of Internet of Things (IoT), network geographic information (WebGIS) has the task of enhancing remote sensing images, including correction, fusion, mapping, etc. Remote sensing images captured by different satellite platforms often present earth objects with various scales. Due to the different spatial resolutions of imaging sensors, the same object will appear in different sizes. Usually panchromatic (PAN) image, without spectral diversity, has high spatial resolution and multispectral (MS) image has lower spatial resolution but more spectral information. Fusing MS and PAN images, also known as pansharpening, will increase the spatial resolution of MS image without changing its spectral information. In the fusion process, inaccurate perception and extraction of the spatial details will lead to spectral and spatial distortions in the fused image. A new multi-scale structure perception based fusion method is proposed in this paper. Firstly, an improved structure-preserving filter is designed by using MS imagery as the reference to capture the structures in the PAN image. Then the structures having different scales are progressively perceived through a multi-scale decomposition scheme and the spatial details are accurately extracted from the PAN image. Finally, the details weighted by the band-dependent gains are injected into each band of MS images. Experimental results are compared with some recently proposed edge-preserving filtering based fusion methods. Visual analysis and objective evaluations demonstrate that the proposed method can produce high-quality fused output and achieve better performance than some state-of-the-arts.KeywordsFusionRemote sensingFilterMultispectral image

In this study, we propose Variation-net, an interpretable variation-inspired deep network for pansharpening, which aims to fuse panchromatic (PAN) and multispectral (MS) images for a high-resolution MS image. We first construct a novel variational pan-sharpening model with clear physical meanings. As the relationship between the PAN and MS images in the real situation is complex and nonlinear, we explore the similarity between PAN and MS images from the sparsity of nonlinear transforms in this variational pansharpening model. As a result, spatial details can be accurately transferred from PAN image to MS image. Furthermore, we build the Variation-net by unrolling the iterative shrinkage-thresholding algorithm to solve the proposed variational pansharpening model. Therefore, all modules in Variation-net have clear physical meanings and are easily observed, leading to good generalization capability. Meanwhile, nonlinear transforms and other parameters in the variational pansharpening model are learned end–to–end. The experiments demonstrate that Variation-net outperforms the state-of-the-art methods from the aspects of visual effect and objective quality analysis.

A novel image fusion method using IKONOS satellite images In satellite remote sensing, spatial resolutions of multispectral images over a particular region can be enhanced using better spatial resolution panchromatic images for the same region by a process called image fusion, or more generally data fusion. A fusion method is considered successful, if the spatial detail of the panchromatic image is transferred into the multispectral image and the spectral content of the original multispectral image is preserved in the fused product. This research proposes a novel image fusion algorithm which takes aim at producing both spatially enhanced and spectrally appealing fused multispectral images. In the proposed method, first an intermediary image is created using original panchromatic and multispectral images. This intermediary image contains the high frequency content of the panchromatic source image such that it is the one closest to the given multispectral source image (upsampled) by a natural semi inner product defined. The final fused image is obtained by applying a function which performs convex linear combination of the intermediary image and the upsampled multispectral image. The function used depends on the local standard deviations of the source images.To test the performance of the method, the images from IKONOS sensor are fused using the Brovey, IHS, PCA, wavelet transform based methods, and the proposed method. Both visual and quantitative evaluation results indicate that the proposed method yields to both spectrally and spatially appealing results as the wavelet transform based method, and it gives a better performance when both spatial detail enhancement and spectral content preservation in the fused products are considered. It is also obvious that the method has a potential to get better results if a better fitting, more complex function is found.

Most modern satellites can provide two types of images: 1) panchromatic (PAN) image and 2) multispectral (MS) image. The former has high spatial resolution and low spectral resolution, while the latter has high spectral resolution and low spatial resolution. To obtain images with both high spectral and spatial resolution, pansharpening has emerged to fuse the spatial information of the PAN image and the spectral information of the MS image. However, most pansharpening methods fail to preserve spatial and spectral information simultaneously. In this article, we propose a framelet-based convolutional neural network (CNN) for pansharpening which makes it possible to pursue both high spectral and high spatial resolution. Our network consists of three subnetworks: 1) feature embedding net; 2) feature fusion net; and 3) framelet prediction net. Different from conventional CNN methods directly inferring high-resolution MS images, our approach learns to predict their framelet coefficients from available PAN and MS images. The introduction of multilevel feature aggregation and hybrid residual connection makes full use of spatial information of PAN image and spectral information of MS image. Quantitative and qualitative experiments at reduced- and full-resolution demonstrate that the proposed method achieves more appealing results than other state-of-the-art pansharpening methods. The source code and trained models are available at https://github.com/TingMAC/FrMLNet.

In this article, we present a new pansharpening method, a zero-reference generative adversarial network (ZeRGAN), which fuses low spatial resolution multispectral (LR MS) and high spatial resolution panchromatic (PAN) images. In the proposed method, zero-reference indicates that it does not require paired reduced-scale images or unpaired full-scale images for training. To obtain accurate fusion results, we establish an adversarial game between a set of multiscale generators and their corresponding discriminators. Through multiscale generators, the fused high spatial resolution MS (HR MS) images are progressively produced from LR MS and PAN images, while the discriminators aim to distinguish the differences of spatial information between the HR MS images and the PAN images. In other words, the HR MS images are generated from LR MS and PAN images after the optimization of ZeRGAN. Furthermore, we construct a nonreference loss function, including an adversarial loss, spatial and spectral reconstruction losses, a spatial enhancement loss, and an average constancy loss. Through the minimization of the total loss, the spatial details in the HR MS images can be enhanced efficiently. Extensive experiments are implemented on datasets acquired by different satellites. The results demonstrate that the effectiveness of the proposed method compared with the state-of-the-art methods. The source code is publicly available at https://github.com/RSMagneto/ZeRGAN.

This paper presents a near real-time automatic sub-pixel registration method of high-resolution panchromatic (PAN) and multispectral (MS) images using a graphics processing unit (GPU). In the first step, the method uses differential geo-registration to enable accurate geographic registration of PAN and MS images. Differential geo-registration normalizes PAN and MS images to the same direction and scale. There are also some residual misalignments due to the geometrical configuration of the acquisition instruments. These residual misalignments mean the PAN and MS images still have deviations after differential geo-registration. The second step is to use differential rectification with tiny facet primitive to eliminate possible residual misalignments. Differential rectification corrects the relative internal geometric distortion between PAN and MS images. The computational burden of these two steps is large, and traditional central processing unit (CPU) processing takes a long time. Due to the natural parallelism of the differential methods, these two steps are very suitable for mapping to a GPU for processing, to achieve near real-time processing while ensuring processing accuracy. This paper used GaoFen-6, GaoFen-7, ZiYuan3-02 and SuperView-1 satellite data to conduct an experiment. The experiment showed that our method’s processing accuracy is within 0.5 pixels. The automatic processing time of this method is about 2.5 s for 1 GB output data in the NVIDIA GeForce RTX 2080Ti, which can meet the near real-time processing requirements for most satellites. The method in this paper can quickly achieve high-precision registration of PAN and MS images. It is suitable for different scenes and different sensors. It is extremely robust to registration errors between PAN and MS.

Pansharpening is a remote sensing image fusion technique that combines a high-resolution (HR) panchromatic (PAN) image with a low-resolution (LR) multispectral (MS) image to produce an HR MS image. The primary challenge in pansharpening lies in preserving the spatial details of the PAN image while maintaining the spectral integrity of the MS image. To address this, this article presents a generative adversarial network (GAN)-based approach to pansharpening. The GAN discriminator facilitated matching the generated image’s intensity to the HR PAN image and preserving the spectral characteristics of the LR MS image. The performance in generating images was evaluated using the peak signal-to-noise ratio (PSNR). For the experiment, original LR MS and HR PAN satellite images were partitioned into smaller patches, and the GAN model was validated using an 80:20 training-to-testing data ratio. The results illustrated that the super-resolution images generated by the SRGAN model achieved a PSNR of 31 dB. These results demonstrated the developed model’s ability to reconstruct the geometric, textural, and spectral information from the images.

International audience

Pansharpening is a process of fusing a high-resolution panchromatic (PAN) image with a low-resolution multispectral (LRMS) image to obtain a high-resolution multispectral (HRMS) image. Convolutional neural networks (CNNs) have been commonly utilized in this field because of their remarkable learning capabilities. However, their convolutional operators limit the long-range feature extraction ability of CNN. Meanwhile, the Transformer models have exhibited strong capabilities in modeling long-range representations, but there are shortcomings in modeling local-range feature dependencies. To this end, we propose a novel synergistic transformer and CNN for pansharpening (STCP). First, a parallel U-shaped feature extraction module (PUFEM) is constructed for extracting the features of the LRMS and PAN images, which improves the feature representation ability for the two source images. In the PUFEM, combining the different feature learning capabilities of the CNN and transformer, we design a long-short-range feature integration block (LSFIB) to extract the short-range features and long-range features at different scales in parallel. Then, a channel attention module (CAM)-based feature fusion module (CFFM) is constructed to integrate the features extracted by the PUFEM. Finally, the shallow features from the PAN image are reused to provide detailed features, which are integrated with the fused features from the CFFM to achieve the final pansharpened results. Numerous experiments show that our STCP outperforms some state-of-the-art approaches both subjectively and objectively.

High spatial resolution multispectral (HMS) images can provide sufficient information for researchers to analyze the potential disasters in the living environment. However, an original multispectral (MS) image is with low-space-resolution and high-spectrum-resolution, while an original panchromatic (PAN) image has the opposite property. Pansharpening aims at obtaining HMS image by retaining spectrum of the MS image and injecting details of the PAN image simultaneously. In this paper, we present a new pansharpening method. First, we use a bilateral filter (BF) to obtain the low-frequency-component (LFC) of PAN and MS images, respectively. Then the high-frequency-component (HFC) of PAN and MS images are readily obtained. Second, an adaptive intensity-hue-saturation (AIHS) based method is applied to generate the HFC of intensity. Finally, a dual-scale guided image filter (GIF) is utilized to calculate the difference between HFC of intensity and PAN to get the detail images. And then, these detail images are injected into the original MS image to achieve the HMS image. The proposed method is applied into testing various satellite data sets, and performs better effect on both visual quality and objective indictors than the existing methods.