A novel method for multispectral image pansharpening based on high dimensional model representation

Pansharpening usually improves the resolution of low-resolution multispectral (LRMS) images with spatial information from corresponding high-resolution panchromatic (HRPAN) images to produce high-resolution MS (HRMS) images. Traditional pansharpening methods use various domain transformations to make the fused image suffer varying degrees of spatial or spectral distortion because the information in the LRMS and PAN images is heterogeneous and distributed in different domains. The motivation of our proposed work is to develop a balanced and robust pansharpening method named cross-domain correspondence intensity modulation, which is based on Bayesian decision-making for remote sensing image pansharpening. First, the intensity component of the MS image is obtained via the intensity hue saturation (IHS) transform. Second, a fusion rule based on the Bayesian probabilistic model is designed to fuse the intensity component and the corresponding PAN image to obtain an intermediate component. Third, a cross-domain correspondence intensity modulation algorithm is proposed to modulate the intensity information in the intermediate component to produce the desired intensity component. Finally, an inverse IHS transformation is performed to obtain the pansharpened MS image by replacing the original intensity component with the modulated intensity component. The results on different satellite datasets show that the proposed method can effectively enhance the spatial and spectral fidelity of the fused image.

In this paper, a model-based approach to multiresolution fusion of remotely sensed images is presented. Given a high spatial resolution panchromatic (Pan) image and a lowspatial resolution multispectral (MS) image acquired on the same geographical area, the presented method aims to enhance the spatial resolution of the MS image to the resolution of the Pan observation. The proposed fusion technique utilizes the spatial correlation of each of the high-resolution MS channels by using an autoregressive (AR) model, whose parameters are learnt from the analysis of the Pan data. Under the assumption that the parameters of the AR model for the Pan image are the same as those that represent the MS images due to spectral correlation, the proposed technique exploits the learnt parameter values in the context of a proper regularization technique to estimate the high spatial resolution fields for the MS bands. This results in a combination of the spectral characteristics of the low-resolution MS data with the high spatial resolution of the Pan image. The main advantages of the proposed technique are: 1) unlike standard methods proposed in the literature, it requires no registration between the Pan and the MS images; 2) it models effectively the texture of the scene during the fusion process; 3) it shows very small spectral distortion (as it is less affected, compared to standard methods, by the specific digital numbers of pixels in the Pan image, since it exploits the learnt parameters from the Pan image rather than the actual Pan digital numbers for fusion); and 4) it can be used in critical situations in which the Pan and the MS images are acquired (also by different sensors) in slightly different areas. Quantitative experimental results obtained using Landsat-7 Enhanced Thematic Mapper Plus (ETM+) and Quickbird images point out the effectiveness of the proposed method

Most modern satellites can provide two types of images: 1) panchromatic (PAN) image and 2) multispectral (MS) image. The former has high spatial resolution and low spectral resolution, while the latter has high spectral resolution and low spatial resolution. To obtain images with both high spectral and spatial resolution, pansharpening has emerged to fuse the spatial information of the PAN image and the spectral information of the MS image. However, most pansharpening methods fail to preserve spatial and spectral information simultaneously. In this article, we propose a framelet-based convolutional neural network (CNN) for pansharpening which makes it possible to pursue both high spectral and high spatial resolution. Our network consists of three subnetworks: 1) feature embedding net; 2) feature fusion net; and 3) framelet prediction net. Different from conventional CNN methods directly inferring high-resolution MS images, our approach learns to predict their framelet coefficients from available PAN and MS images. The introduction of multilevel feature aggregation and hybrid residual connection makes full use of spatial information of PAN image and spectral information of MS image. Quantitative and qualitative experiments at reduced- and full-resolution demonstrate that the proposed method achieves more appealing results than other state-of-the-art pansharpening methods. The source code and trained models are available at https://github.com/TingMAC/FrMLNet.

A new IHS and wavelet based pansharpening algorithm for high spatial resolution satellite imagery

Many pan-sharpening techniques have been developed to synthesize a multispectral (MS) image at high resolution by fusing MS images and panchromatic (Pan) images. Most existing pan-sharpening methods can achieve results with high spatial resolution, but the spectral distortion in the fused results is still a problem that needs to be solved. In this paper, an adaptive linear model is proposed to reduce the spectral distortion by weakening the dependence on the correlation between Pan and MS. The difference between a Pan image and the combination of MS images is estimated by least square optimization, and embedded into the proposed model as a virtual band. According to the adaptive model, an iterative pan-sharpening algorithm is proposed based on the steepest descent method, in which the virtual band is used as a local adaptive constraint to the optimized solution. The proposed method is tested on datasets acquired by IKONOS, QuickBird, and Landsat 7 ETM+ and compared with the existing methods. The quality measures and the visual impressions show that the proposed method is an efficient approach to preserving spectral information and represents strong robustness against various scenes and sensors.

<p>Earth observation satellites capture panchromatic images at high spatial resolution and multispectral images at lower resolution to optimize the use of their onboard energy sources. This results in a technical necessity to synthesize high-resolution multispectral images from these data. Pansharpening techniques aim to combine the spatial detail of panchromatic images with the spectral information of multispectral images. However, due to the discrete nature of these images and their varying local statistical properties, many pansharpening methods suffer from numerical artifacts such as chromatic and spatial distortions. This paper introduces the L0-Norm-based pansharpening method (L0pan), which addressed these challenges by maximizing the number of similar pixels between the synthesized pansharpened image and the original panchromatic and multispectral images. L0pan was optimized using a population-based colony search algorithm, enabling it to effectively balance both chromatic fidelity and spatial resolution. Extensive experiments across nine different datasets and comparison with nine other pansharpening methods using ten quality metrics demonstrated that L0pan significantly outperformed its counterparts. Notably, the colony search algorithm yielded the best overall results, highlighting the algorithm's strength in refining pansharpening accuracy. This study contributed to the advancement of pansharpening techniques, offering a method that preserved both chromatic and spatial details more effectively than existing approaches.</p>

This paper proposes a pan-sharpening method based on multi-direction subbands and deep neural networks. First, by utilizing the multi-scale and multi-direction properties of the nonsubsampled contourlet transform (NSCT), panchromatic (PAN) image is decomposed into the low frequency subbands in different resolutions and the high frequency subbands in different directions. Pan-sharpening method aims to fuse the high frequency subband coefficients of PAN image and the low frequency subband coefficients of multispectral (MS) image. Second, in order to better extract the feature of the high frequency subbands in different directions of PAN image, the deep neural network (DNN) is trained using the image patches of high frequency subbands of PAN image. Third, in the fusion stage, we exploit NSCT on the principal component of resampled low resolution (LR) MS image. The high frequency subbands of output high resolution (HR) MS image is obtained by forward propagation of the trained DNN, which input is the high frequency subbands of LR MS image. Finally, a new subband set is obtained by fusing the reconstructed high frequency subband and the original low frequency subband of LR MS image. The HR MS image is produced by executing the inverse transform of NSCT and adaptive PCA (A-PCA) on the new subband set. The experimental results show the proposed method outperforms other well-known methods in terms of both objective measurements and visual evaluation.

Sparse tensor neighbor embedding based pan-sharpening via N-way block pursuit

The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods.

High spatial resolution multispectral (HMS) images can provide sufficient information for researchers to analyze the potential disasters in the living environment. However, an original multispectral (MS) image is with low-space-resolution and high-spectrum-resolution, while an original panchromatic (PAN) image has the opposite property. Pansharpening aims at obtaining HMS image by retaining spectrum of the MS image and injecting details of the PAN image simultaneously. In this paper, we present a new pansharpening method. First, we use a bilateral filter (BF) to obtain the low-frequency-component (LFC) of PAN and MS images, respectively. Then the high-frequency-component (HFC) of PAN and MS images are readily obtained. Second, an adaptive intensity-hue-saturation (AIHS) based method is applied to generate the HFC of intensity. Finally, a dual-scale guided image filter (GIF) is utilized to calculate the difference between HFC of intensity and PAN to get the detail images. And then, these detail images are injected into the original MS image to achieve the HMS image. The proposed method is applied into testing various satellite data sets, and performs better effect on both visual quality and objective indictors than the existing methods.

In this paper, we propose a novel method to enhance the pan-sharpening result of low-resolution multispectral images (MS) and high-resolution panchromatic images (Pan) by minimizing the spectral distortion engendered by the fusion process. In fact, spectral distortion is the most significant problem in many pan-sharpening techniques, due to the non linearity between Pan and MS images. In this method, an improvement of the Pan image is performed in order to enhance the correlation between Pan and MS images before pan-sharpening process. The proposed method is applied as a preprocessing by fusing the intensity image derived from MS image with the original Pan to get an improved Pan image which could be more correlated with MS image. And later, the pan-sharpening is applied on both MS and the improved Pan using any pan-sharpening technique. Simulation results of proposed method are compared in four different techniques, such as: Generalized IHS, DWT, Brovey and HPF. It has been observed that simulation results of this method preserves more spectral information and gets better visual quality compared with earlier reported techniques using original Pan.KeywordsPan-sharpeningFusionPanchromaticMultispectralSpectral distortion

In order to overcome the incompatibility of spatial characteristic enhancement and spectral information preservation for remote sensing image fusion, we present a new wavelet fusion algorithm based on color compensation (CC) rule and intensity-hue-saturation (IHS) transform in this paper. First, the RGB representation of the multispectral (MS) image is transformed by IHS into the intensity, hue and saturation components. Then, the panchromatic (PAN) image and the intensity component of multispectral image are decomposed based on the wavelet transform (WT). For detail wavelet coefficients, we introduce CC into local deviation (LD) fusion, and propose a local compensation (LC) fusion rule; and for approximate wavelet coefficients, we combine CC with high pass filtering (HPF), to propose a high-pass-filtering compensation (HC) fusion rule. Finally, perform inverse IHS transform to the wavelet reconstructed new intensity component and the hue and saturation components from the original MS image to reconstruct the fused image. The visual evaluation and statistical analysis both demonstrate that the proposed algorithm not only enhances the details of result image greatly, but also preserves the spectral information of original multispectral image effectively

Hyperspectral images (HSI) have high spectral and low spatial resolutions. However, multispectral images (MSI) usually have low spectral and high spatial resolutions. In various applications HSI with high spectral and spatial resolutions are required. In this paper, a new method for spatial resolution enhancement of HSI using high resolution MSI based on sparse coding and linear spectral unmixing (SCLSU) is introduced. In the proposed method (SCLSU), high spectral resolution features of HSI and high spatial resolution features of MSI are fused. In this case, the sparse representation of some high resolution MSI and linear spectral unmixing (LSU) model of HSI and MSI is simultaneously used in order to construct high resolution HSI (HRHSI). The fusion process of HSI and MSI is formulated as an ill-posed inverse problem. It is solved by the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) and an orthogonal matching pursuit (OMP) algorithm. Finally, the proposed algorithm is applied to the Hyperion and ALI datasets. Compared with the other state-of-the-art algorithms such as Coupled Nonnegative Matrix Factorization (CNMF) and local spectral unmixing, the SCLSU has significantly increased the spatial resolution and in addition the spectral content of HSI is well maintained.

There are several situations that simultaneously require both high spatial and spectral resolution in a single image. In literature a certain number of methods have been proposed to merge panchromatic and multispectral data. Classical processes involve a simple linear transformation of the combined data spaces (Brovey, HSV, PCA). However standard data fusion methods may not be satisfactory to merge a high-resolution panchromatic image and a low-resolution multispectral image because they distort the spectral characteristics of the original multispectral data. It has been shown that an image fusion technique based on multiresolution wavelet decomposition (MWD) seems to be a better trade-off between spectral and spatial information. In this paper, we develop some techniques, based on MWD, for the data fusion of such images. The methods presented for reducing the fusion imperfections don't consist simply of replacing some wavelet coefficients of the multispectral image by the corresponding coefficients of the high-resolution image but tend to carry out a real. integration between two pyramidal decomposition. In fact, it is possible to fuse the wavelet coefficients of two decompositions in each scale level and with some appropriate rules. Moreover, some ad hoc algorithms have been implemented to correct spectral distortions in the synthesized high resolution multispectral image. We tested the methods to merge IKONOS panchromatic (1 m spatial resolution) and IKONOS multispectral (4 m spatial resolution) images in urban areas. The techniques presented are clearly better than classical fusion methods in preserving both spectral and spatial information on urban areas.

In order to decrease the spectral distortion using the IHS-based fusion method and more effectively inject the spatial detail information of high spatial resolution images into the fused image, this paper presents a new image fusion method that combines the IHS transform with dual-tree complex wavelet transform. The proposed method can overcome the distortion of spectrum. Firstly, intensity-hue-saturation (IHS) transform of multispectral(MS) image is implemented. Then panchromatic (PAN) image and the intensity component of MS images are decomposed into three scales using dual-tree complex wavelet transform respectively. After that, the region-based fusion operator is adopted to select the wavelet coefficients by fusing the wavelet coefficients of PAN image and the intensity component of MS at each scale in wavelet domain. Further, the fused intensity component is obtained by inverse dual-tree complex wavelet transform. Finally, the fused image is obtained by inverse IHS transform. The experiment results demonstrate our proposed method is effective.