Distributed compressed sensing-based pan-sharpening with hybrid dictionary

This paper proposes a pan-sharpening method based on multi-direction subbands and deep neural networks. First, by utilizing the multi-scale and multi-direction properties of the nonsubsampled contourlet transform (NSCT), panchromatic (PAN) image is decomposed into the low frequency subbands in different resolutions and the high frequency subbands in different directions. Pan-sharpening method aims to fuse the high frequency subband coefficients of PAN image and the low frequency subband coefficients of multispectral (MS) image. Second, in order to better extract the feature of the high frequency subbands in different directions of PAN image, the deep neural network (DNN) is trained using the image patches of high frequency subbands of PAN image. Third, in the fusion stage, we exploit NSCT on the principal component of resampled low resolution (LR) MS image. The high frequency subbands of output high resolution (HR) MS image is obtained by forward propagation of the trained DNN, which input is the high frequency subbands of LR MS image. Finally, a new subband set is obtained by fusing the reconstructed high frequency subband and the original low frequency subband of LR MS image. The HR MS image is produced by executing the inverse transform of NSCT and adaptive PCA (A-PCA) on the new subband set. The experimental results show the proposed method outperforms other well-known methods in terms of both objective measurements and visual evaluation.

Pansharpening technology plays a crucial role in remote sensing image processing by integrating low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images to generate high-resolution multispectral (HRMS) images. This process addresses the limitations of satellite sensors, which cannot directly capture HRMS images. Despite significant developments achieved by deep learning-based pansharpening methods over traditional approaches, most existing techniques either fail to account for the modal differences between LRMS and PAN images, relying on direct concatenation, or use similar network structures to extract spectral and spatial information. Additionally, many methods neglect the extraction of common features between LRMS and PAN images and lack network architectures specifically designed to extract spectral features. To address these limitations, this study proposed a novel three-branch pansharpening network that leverages both spatial and frequency domain interactions, resulting in improved spectral and spatial fidelity in the fusion outputs. The proposed method was validated on three datasets, including IKONOS, WorldView-3 (WV3), and WorldView-4 (WV4). The results demonstrate that the proposed method surpasses several leading techniques, achieving superior performance in both visual quality and quantitative metrics.

Pan-sharpening is an approach that fuse low resolution multi-spectral (LRMS) images with a high spatial detail of panchromatic (PAN) image to obtain the high resolution multispectral (HRMS) images. In this paper, we present a compressed sensing-based pan-sharpening method that include joint data fidelity and blind blurring kernel estimation. The joint data fidelity contain following three fidelity terms: (1) the LRMS images could be the decimated form of the HRMS images by convolving a blurring kernel, (2) the gradient of HRMS images in the spectrum direction could be proximity to those of the LRMS images, (3) the high frequency part of linear combination of HRMS image bands is approximate to the corresponding parts of the PAN image. Different from other methods which simply apply average blurring kernel for pan-sharpening, a blind deconvolution algorithm is introduced to estimate the blurring kernel from different satellites respectively. We also include a novel anisotropic total variation (TV) prior term to better reconstruct the image edges. The alternating direction method of multipliers (ADMM) is used to solve the proposed model efficiently. Finally, a Pleiades satellite image is employed to demonstrate that the proposed method achieve effective and efficient results simultaneously compared with other existing methods.

Pansharpening usually improves the resolution of low-resolution multispectral (LRMS) images with spatial information from corresponding high-resolution panchromatic (HRPAN) images to produce high-resolution MS (HRMS) images. Traditional pansharpening methods use various domain transformations to make the fused image suffer varying degrees of spatial or spectral distortion because the information in the LRMS and PAN images is heterogeneous and distributed in different domains. The motivation of our proposed work is to develop a balanced and robust pansharpening method named cross-domain correspondence intensity modulation, which is based on Bayesian decision-making for remote sensing image pansharpening. First, the intensity component of the MS image is obtained via the intensity hue saturation (IHS) transform. Second, a fusion rule based on the Bayesian probabilistic model is designed to fuse the intensity component and the corresponding PAN image to obtain an intermediate component. Third, a cross-domain correspondence intensity modulation algorithm is proposed to modulate the intensity information in the intermediate component to produce the desired intensity component. Finally, an inverse IHS transformation is performed to obtain the pansharpened MS image by replacing the original intensity component with the modulated intensity component. The results on different satellite datasets show that the proposed method can effectively enhance the spatial and spectral fidelity of the fused image.

Pansharpening is to fuse high-resolution panchromatic (PAN) images with low-resolution multispectral (LR-MS) images to generate high-resolution multispectral (HR-MS) images. Most of the deep learning-based pansharpening methods did not consider the inconsistency of the PAN and LR-MS images and used simple concatenation to fuse the source images, which may cause spectral and spatial distortion in the fused results. To address this problem, a multi-scale spatial-spectral attention guided fusion network for pansharpening is proposed. First, the spatial features from the PAN image and spectral features from the LR-MS image are independently extracted to obtain the shallow features. Then, a spatial-spectral attention feature fusion module (SAFFM) is constructed to guide the reconstruction of spatial-spectral features by generating a guidance map to achieve the fusion of reconstructed features at different scales. In SAFFM, the guidance map is designed to ensure the spatial-spectral consistency of the reconstructed features. Finally, considering the difference between multiply scale features, a multi-level feature integration scheme is proposed to progressively achieve fusion of multi-scale features from different SAFFMs. Extensive experiments validate the effectiveness of the proposed network against other state-of-the-art (SOTA) pansharpening methods in both quantitative and qualitative assessments. The source code will be released at https://github.com/MELiMZ/ssaff.

Pansharpening aims to generate high-resolution multispectral (HRMS) images by fusing high-resolution panchromatic (PAN) images with low-resolution multispectral (LRMS) images for various practical tasks such as ground-object classification and change detection. Currently, Transformers have been widely utilized in the pansharpening field because of their capability to model global representations. Transformer-based pansharpening methods capture a global representation of local patch sequences in the spatial domain, but they do not aggregate each pixel in the image, which hinders their ability to capture the global context and dependencies. To solve this issue, a novel pansharpening network based on Fourier with dropout in global modeling (FDGM) is proposed, which can better model global representations and obtain higher-quality fusion images without increasing computational complexity. First, an efficient Fourier with dropout block (FDB) is designed by transforming the features from the spatial domain to the frequency domain to capture the global information from the panchromatic (PAN) and low-resolution multispectral (LRMS) images. This study represents a novel attempt to explore the generalization performance of dropout in the Fourier frequency domain. Furthermore, a residual spatial-frequency fusion module (RSFM) based on frequency channel attention is designed to enhance feature representation capability and effectively facilitate the fusion of the spatial and frequency features. Finally, the shallow features extracted from the source images are combined with the RSFM output to generate the final fusion results. The results of numerous experiments demonstrate that FDGM surpasses other state-of-the-art approaches in visual perception and quantitative evaluation.

Pansharpening is an important technology for obtaining high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images. Although many pansharpening models have emerged by taking advantage of deep learning (DL) technology, there remains a pressing need to further assess pansharpening accuracy and stability when LRMS images with complex land-cover types. What’s more, these models often overlook the exploitation of PAN images’ inherent high-frequency information. To address these issues, we propose a pansharpening model combining multi-level and multi-scale network architectures. The multi-level network architecture is used to build spatial-spectral dependence on LRMS-PAN pairs, and strengthen the network’s feature capture capability by keeping the multi-level texture details. The multi-scale architecture is subsequently used to extract the spatial structure and deep texture of the PAN images at different scales. Downsampled experiments and real experiments in four standard datasets show that the proposed model achieves a state-of-the-art performance.

Hyperspectral-multispectral image fusion aims to generate high-resolution hyperspectral (HRHS) images with rich spectral details, by mining the complementary information between low-resolution hyperspectral (LRHS) images and high resolution multispectral (HRMS) images. However, in practical scenarios, LRHS images are susceptible to noise interference due to their narrow spectral bandwidth, resulting in limited energy absorption of surface reflectance in each band. Various noise disrupt the correlations of LRHS and HRMS images, thus leading to the spatial details loss or spectral distortion. To address these challenges, we propose an anti-noise hyperspectral and multispectral image fusion network based on State Space Model (SSM), namely AHMFNet. To mitigate noise in LRHS images, the spectral denoising and refinement module is designed by combining low-rank representation with the discrete wavelet transform, while simultaneously extracting key spectral features. Then, the SSM is utilized for global modeling of spectral information to enhance essential features. Finally, although spectral and spatial features have differences, the spatial-spectral cross Mamba is designed to fuse them by mining their relationships, thus improving the fusion quality. Experiments on four public datasets show that, compared to existing state-of-the-art methods, the proposed AHMFNet generates images with superior spatial quality and more accurate spectral characteristics under various noise conditions.

Pansharpening aims to produce a higher resolution multi-spectral (HRMS) image by fusing the spectral information in lower resolution multispectral (LRMS) image and the spatial information in corresponding high resolution panchromatic (PAN) image. In this work, we propose a multi-scale densely deep learning based pansharpening method. Following an end-to-end learning architecture, the proposed deep neural network contains three modules: 1) a parallel multi-scale convolutional layer is used to extract multiscale features of PAN image; 2) a global identity branch structure is adopted to preserve spectral structures; and 3) a dense learning block is integrated to improve the spectral-spatial expressive power. Compared with other state-of-the-art methods, experimental results obtained with our proposed method achieve high pansharpening quality in visualization and quantification.

Pansharpening is a process of fusing a high-resolution panchromatic (PAN) image with a low-resolution multispectral (LRMS) image to obtain a high-resolution multispectral (HRMS) image. Convolutional neural networks (CNNs) have been commonly utilized in this field because of their remarkable learning capabilities. However, their convolutional operators limit the long-range feature extraction ability of CNN. Meanwhile, the Transformer models have exhibited strong capabilities in modeling long-range representations, but there are shortcomings in modeling local-range feature dependencies. To this end, we propose a novel synergistic transformer and CNN for pansharpening (STCP). First, a parallel U-shaped feature extraction module (PUFEM) is constructed for extracting the features of the LRMS and PAN images, which improves the feature representation ability for the two source images. In the PUFEM, combining the different feature learning capabilities of the CNN and transformer, we design a long-short-range feature integration block (LSFIB) to extract the short-range features and long-range features at different scales in parallel. Then, a channel attention module (CAM)-based feature fusion module (CFFM) is constructed to integrate the features extracted by the PUFEM. Finally, the shallow features from the PAN image are reused to provide detailed features, which are integrated with the fused features from the CFFM to achieve the final pansharpened results. Numerous experiments show that our STCP outperforms some state-of-the-art approaches both subjectively and objectively.

This paper addresses the remote sensing image pan-sharpening problem from the perspective of compressed sensing (CS) theory which ensures that with the sparsity regularization, a compressible signal can be correctly recovered from the global linear sampled data. First, the degradation model from a high- to low-resolution multispectral (MS) image and high-resolution panchromatic (PAN) image is constructed as a linear sampling process which is formulated as a matrix. Then, the model matrix is considered as the measurement matrix in CS, so pan-sharpening is converted into signal restoration problem with sparsity regularization. Finally, the basis pursuit (BP) algorithm is used to resolve the restoration problem, which can recover the high-resolution MS image effectively. The QuickBird and IKONOS satellite images are used to test the proposed method. The experimental results show that the proposed method can well preserve spectral and spatial details of the source images. The pan-sharpened high-resolution MS image by the proposed method is competitive or even superior to those images fused by other well-known methods.

Remote sensing image fusion (or pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. In this paper, a deep convolutional neural network with two-stream inputs respectively for PAN and MS images is proposed for remote sensing image pan-sharpening. Firstly the network extracts features from PAN and MS images, then it fuses them to form compact feature maps that can represent both spatial and spectral information of PAN and MS images, simultaneously. Finally, the desired high spatial resolution MS image is recovered from the fused features using an encoding-decoding scheme. Experiments on Quickbird satellite images demonstrate that the proposed method can fuse the PAN and MS image effectively.

The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods.

Pansharpening is a remote sensing image fusion technique that combines a high-resolution (HR) panchromatic (PAN) image with a low-resolution (LR) multispectral (MS) image to produce an HR MS image. The primary challenge in pansharpening lies in preserving the spatial details of the PAN image while maintaining the spectral integrity of the MS image. To address this, this article presents a generative adversarial network (GAN)-based approach to pansharpening. The GAN discriminator facilitated matching the generated image’s intensity to the HR PAN image and preserving the spectral characteristics of the LR MS image. The performance in generating images was evaluated using the peak signal-to-noise ratio (PSNR). For the experiment, original LR MS and HR PAN satellite images were partitioned into smaller patches, and the GAN model was validated using an 80:20 training-to-testing data ratio. The results illustrated that the super-resolution images generated by the SRGAN model achieved a PSNR of 31 dB. These results demonstrated the developed model’s ability to reconstruct the geometric, textural, and spectral information from the images.

Pan-sharpening is a fundamental task for remote sensing image processing. It aims at creating a high-resolution multispectral (HRMS) image from a multispectral (MS) image and a panchromatic (PAN) image. In this article, a new band-independent encoder-decoder network is proposed for pan-sharpening. The network takes a single band of the MS (BMS) image, the PAN image, and the low-resolution PAN (LRPAN) image as inputs. The output of the network is the corresponding band of high-resolution MS (HRBMS) image. In this way, the network can process MS images with any number of bands. The overall structure of the network consists of two encoder-decoder modules at low-resolution and high-resolution, respectively. An auxiliary LRPAN image is used to speed up the training and improve the performance. The partly shared network and hierarchical structure for low-resolution and high-resolution enable a better fusion of features extracted from different scales. With a fast fine-tuning strategy, the trained model can be applied to images from different sensors. Experiments performed on different data sets demonstrate that the proposed method outperforms several state-of-the-art pan-sharpening methods in both visual appearance and objective indexes, and the single-band evaluation results further verify the superiority of the proposed method.