Fusion of Multi-spectral Image Using Non-separable Additive Wavelets for High Spatial Resolution Enhancement

This paper proposes an efficient and high-fidelity image fusion method based on adaptive smoothing filtering for panchromatic (PAN) and multispectral (MS) image fusion. The scale ratio reflects the ratio of spatial resolution between the panchromatic image and the multispectral image. When facing a multiscale fusion task, traditional methods are unable to simultaneously handle the problems of spectral resolution loss resulting from high scale ratios and the issue of reduced spatial resolution due to low scale ratios. To adapt to the fusion of panchromatic and multispectral satellite images of different scales, this paper improves the problem of the insufficient filtering of high-frequency information of remote sensing images of different scales by the classic filter-based intensity modulation (SFIM) model. It uses Gaussian convolution kernels instead of traditional mean convolution kernels and builds a Gaussian pyramid to adaptively construct convolution kernels of different scales to filter out high-frequency information of high-resolution images. It can adaptively process panchromatic multispectral images of different scales, iteratively filter the spatial information in panchromatic images, and ensure that the scale transformation is consistent with the definition of multispectral images. Using 15 common fusion methods, this paper compares the experimental results of ZY-3 with scale ratio 2.7 and SV-1 with scale ratio 4 data. The results show that the method proposed in this paper retains good spatial information for image fusion at different scales and has good spectral preservation.

There are various sensors invented to capture the images of earth surface which are used to analyze the earth surface. The images captured by different sensors have different features. To analyze the earth surface images captured at different spectrums are used. In urban area characteristic of urban object are determined by not only their spectra but also by their structure. Various sensors captures the image at various spectrums knows as multispectral image but is with low spatial resolution. Some sensors captures the high resolution image also known as panchromatic image. Therefore it becomes very necessary and useful to fuse panchromatic and multispectral image to detect different area from the image easily. The fused image is multispectral image with high spatial resolution. The various methods of image fusion studied in this paper. Comparison of them is done is done statistically and visually and it is found that fused image is of greater quality and clarity than the original one and shows the features of that area more accurately.

The intent of this study is to remotely monitor the status of revegetation growth in a reclaimed, previously mined region on the island of Milos, Greece. Quickbird multispectral (spatial resolution 2.4m×2.4m) and panchromatic (spatial resolution 0.6m×0.6m) images have been fused to obtain an optimal combination of the initial spatial and spectral resolution. The Blue (450nm-520nm), Green (520nm-600nm) and Near Infrared (760nm-900nm) bands of the multispectral image have been used for vegetation monitoring. Different fusion methods, like the Principal Component Analysis, the Intensity-Hue-Saturation transform and the Wavelet The goal of this study is to remotely monitor the status of re-vegetation growth in a reclaimed, previously mined region on the island of Milos, Greece. Quickbird multispectral (spatial resolution 2.4m×2.4m) and panchromatic (spatial resolution 0.6m×0.6m) images have been fused to obtain an optimal combination of the initial spatial and spectral resolution. The Blue (450nm-520nm), Green (520nm-600nm) and Near Infrared (760nm-900nm) bands of the multispectral image have been used for vegetation monitoring. Different fusion methods, like the Principal Component Analysis, the Intensity-Hue-Saturation transform and the Wavelet Analysis have been applied to Quickbird images. Both statistical (correlation coefficient, accuracy measures, etc.) and subjective (i.e., visual) measures have been used to evaluate the produced fused images. The degree to which each of the fused images retains the spectral and spatial features of the initial images has been thus estimated. Based on statistical measures, it has been found that the Additive Wavelet Principal Component "A Trous" and the Additive Wavelet Intensity Mallat methods effectively preserve most of the spectral information of the original multispectral image. On the other hand, the "A Trous" and Intensity-Hue-Saturation fusion techniques retain most of the spatial information of the panchromatic image. Additionally, the IHS transform offers a compromise between the spectral and spatial content of the fused image. Since the spectral content in the NIR band is of primary importance for monitoring re-vegetation growth, the Additive Wavelet Mallat and the IHS transforms are the most suitable choices.

A novel image fusion method using IKONOS satellite images In satellite remote sensing, spatial resolutions of multispectral images over a particular region can be enhanced using better spatial resolution panchromatic images for the same region by a process called image fusion, or more generally data fusion. A fusion method is considered successful, if the spatial detail of the panchromatic image is transferred into the multispectral image and the spectral content of the original multispectral image is preserved in the fused product. This research proposes a novel image fusion algorithm which takes aim at producing both spatially enhanced and spectrally appealing fused multispectral images. In the proposed method, first an intermediary image is created using original panchromatic and multispectral images. This intermediary image contains the high frequency content of the panchromatic source image such that it is the one closest to the given multispectral source image (upsampled) by a natural semi inner product defined. The final fused image is obtained by applying a function which performs convex linear combination of the intermediary image and the upsampled multispectral image. The function used depends on the local standard deviations of the source images.To test the performance of the method, the images from IKONOS sensor are fused using the Brovey, IHS, PCA, wavelet transform based methods, and the proposed method. Both visual and quantitative evaluation results indicate that the proposed method yields to both spectrally and spatially appealing results as the wavelet transform based method, and it gives a better performance when both spatial detail enhancement and spectral content preservation in the fused products are considered. It is also obvious that the method has a potential to get better results if a better fitting, more complex function is found.

In this article, we present a new pansharpening method, a zero-reference generative adversarial network (ZeRGAN), which fuses low spatial resolution multispectral (LR MS) and high spatial resolution panchromatic (PAN) images. In the proposed method, zero-reference indicates that it does not require paired reduced-scale images or unpaired full-scale images for training. To obtain accurate fusion results, we establish an adversarial game between a set of multiscale generators and their corresponding discriminators. Through multiscale generators, the fused high spatial resolution MS (HR MS) images are progressively produced from LR MS and PAN images, while the discriminators aim to distinguish the differences of spatial information between the HR MS images and the PAN images. In other words, the HR MS images are generated from LR MS and PAN images after the optimization of ZeRGAN. Furthermore, we construct a nonreference loss function, including an adversarial loss, spatial and spectral reconstruction losses, a spatial enhancement loss, and an average constancy loss. Through the minimization of the total loss, the spatial details in the HR MS images can be enhanced efficiently. Extensive experiments are implemented on datasets acquired by different satellites. The results demonstrate that the effectiveness of the proposed method compared with the state-of-the-art methods. The source code is publicly available at https://github.com/RSMagneto/ZeRGAN.

There are several situations that simultaneously require both high spatial and spectral resolution in a single image. In literature a certain number of methods have been proposed to merge panchromatic and multispectral data. Classical processes involve a simple linear transformation of the combined data spaces (Brovey, HSV, PCA). However standard data fusion methods may not be satisfactory to merge a high-resolution panchromatic image and a low-resolution multispectral image because they distort the spectral characteristics of the original multispectral data. It has been shown that an image fusion technique based on multiresolution wavelet decomposition (MWD) seems to be a better trade-off between spectral and spatial information. In this paper, we develop some techniques, based on MWD, for the data fusion of such images. The methods presented for reducing the fusion imperfections don't consist simply of replacing some wavelet coefficients of the multispectral image by the corresponding coefficients of the high-resolution image but tend to carry out a real. integration between two pyramidal decomposition. In fact, it is possible to fuse the wavelet coefficients of two decompositions in each scale level and with some appropriate rules. Moreover, some ad hoc algorithms have been implemented to correct spectral distortions in the synthesized high resolution multispectral image. We tested the methods to merge IKONOS panchromatic (1 m spatial resolution) and IKONOS multispectral (4 m spatial resolution) images in urban areas. The techniques presented are clearly better than classical fusion methods in preserving both spectral and spatial information on urban areas.

A new image fusion method based on hue-saturation-value (HSV) color space is proposed, the improved HSV fuses multi-spectral (MS) and panchromatic (PAN) images to improve spatial information and preserve spectral characteristics. The main advantage of the new fusion method is a simple and efficient way which can maximize the extraction of the spatial information and eliminate the disturbance of spectral information in PAN image. the difference of the low components information between PAN and the value component is filtered from the PAN image, and then value component are replaced by the new PAN image, the new HSV color space perform reverse transform to obtain a multispectral image with the high spatial resolution.SPOT-5 and QuickBird MS and PAN images were employed to execute the existing HSV and improved HSV fusion methods. Qualitative and quantitative analyses and classification accuracy assessment were conducted to evaluate the performance of the fusion methods. The results demonstrate that the improved HSV is better than traditional HSV methods. The new fusion method can achieve a wide range of balance between high spatial resolution retention and spectral characteristic preservation.

Simultaneously fusing hyperspectral (HS), multispectral (MS), and panchromatic (PAN) images brings a new paradigm to generate a high-resolution HS (HRHS) image. In this study, we propose an interpretable model-driven deep network for HS, MS, and PAN image fusion, called HMPNet. We first propose a new fusion model that utilizes a deep before describing the complicated relationship between the HRHS and PAN images owing to their large resolution difference. Consequently, the difficulty of traditional model-based approaches in designing suitable hand-crafted priors can be alleviated because this deep prior is learned from data. We further solve the optimization problem of this fusion model based on the proximal gradient descent (PGD) algorithm, achieved by a series of iterative steps. By unrolling these iterative steps into several network modules, we finally obtain the HMPNet. Therefore, all parameters besides the deep prior are learned in the deep network, simplifying the selection of optimal parameters in the fusion and achieving a favorable equilibrium between the spatial and spectral qualities. Meanwhile, all modules contained in the HMPNet have explainable physical meanings, which can improve its generalization capability. In the experiment, we exhibit the advantages of the HMPNet over other state-of-the-art methods from the aspects of visual comparison and quantitative analysis, where a series of simulated as well as real datasets are utilized for validation.

Convolutional neural networks (CNNs) have recently achieved outstanding performance for hyperspectral (HS) and multispectral (MS) image fusion. However, CNNs cannot explore the long-range dependence for HS and MS image fusion because of their local receptive fields. To overcome this limitation, a transformer is proposed to leverage the long-range dependence from the network inputs. Because of the ability of long-range modeling, the transformer overcomes the sole CNN on many tasks, whereas its use for HS and MS image fusion is still unexplored. In this article, we propose a spectral-spatial transformer (SST) to show the potentiality of transformers for HS and MS image fusion. We devise first two branches to extract spectral and spatial features in the HS and MS images by SST blocks, which can explore the spectral and spatial long-range dependence, respectively. Afterward, spectral and spatial features are fused feeding the result back to spectral and spatial branches for information interaction. Finally, the high-resolution (HR) HS image is reconstructed by dense links from all the fused features to make full use of them. The experimental analysis demonstrates the high performance of the proposed approach compared with some state-of-the-art (SOTA) methods.

In this paper, we propose a fuzzy-based multi-spectral (MS) and panchromatic (PAN) image fusion approach which provides a tradeoff solution between spectral and spatial fidelity and is able to preserve more detail in terms of spectral and spatial information. First, we perform principal component analysis on the multi-spectral images and utilise the first principal component to extract matched low and high frequency coefficients. We then apply fuzzy-based image fusion rules to fuse the first principal component with the PAN image, followed by fusing the approximation coefficients. The proposed approach is tested on several satellite images and shown to provide a feasible and effective approach.

Recent advances and new guidelines on hyperspectral and multispectral image fusion

The hyperspectral and multispectral image fusion process, commonly known as the multisharpening process (also known as the hypersharpening process), results in the creation of unobservable high spectral and spatial resolution hyperspectral remote sensing data, by combining (or fusing) a hyperspectral image, characterized by a high spectral resolution, with, especially, a high spatial resolution multispectral one. There are several approaches in the literature performing such a process, in particular, those based on the spectral unmixing concept. The latter fusion methods aim to extract pure-material spectra from the high spectral resolution hyperspectral image and combine them with their corresponding proportions extracted from the high spatial resolution multispectral image. However, most of these methods, which mainly consider the standard linear mixing model, do not address the spectral variability phenomenon, precisely due to the considered data mixing model, which does not handle variations in illumination and atmospheric conditions or material composition inducing spectral variability. Neglecting such a phenomenon may lead to errors in the fusion process, thus decreasing the spatial and spectral qualities of sharpened products. In this paper, a new hyperspectral and multispectral data fusion method, based on the linear spectral unmixing concept that deals with spectral variability, is proposed. This approach, based on a multiplicative Nonnegative Matrix Factorization (NMF) technique, substantially extends the multiplicative Coupled NMF (CNMF) algorithm by considering a linear data mixing model that addresses the considered phenomenon. Indeed, the introduced approach minimizes new cost functions involving matrices with specific structures in order to deal with the spectral variability phenomenon. The proposed method is applied to synthetic and real data, and its effectiveness is evaluated with established performance criteria. Experimental results show that the designed algorithm globally yields sharpened hyperspectral data with fine spectral and spatial fidelities. Also, these results demonstrate that the proposed technique globally outperforms the tested literature approaches.

Computer based diagnosis of Alzheimer's disease can be performed by dint of the analysis of the functional and structural changes in the brain. Multispectral image fusion deliberates upon fusion of the complementary information while discarding the surplus information to achieve a solitary image which encloses both spatial and spectral details. This paper presents a Non-Sub-sampled Contourlet Transform (NSCT) based multispectral image fusion model for computer-aided diagnosis of Alzheimer's disease. The proposed fusion methodology involves color transformation of the input multispectral image. The multispectral image in YIQ color space is decomposed using NSCT followed by dimensionality reduction using modified Principal Component Analysis algorithm on the low frequency coefficients. Further, the high frequency coefficients are enhanced using non-linear enhancement function. Two different fusion rules are then applied to the low-pass and high-pass sub-bands: Phase congruency is applied to low frequency coefficients and a combination of directive contrast and normalized Shannon entropy is applied to high frequency coefficients. The superiority of the fusion response is depicted by the comparisons made with the other state-of-the-art fusion approaches (in terms of various fusion metrics).

In order to decrease the spectral distortion using the IHS-based fusion method and more effectively inject the spatial detail information of high spatial resolution images into the fused image, this paper presents a new image fusion method that combines the IHS transform with dual-tree complex wavelet transform. The proposed method can overcome the distortion of spectrum. Firstly, intensity-hue-saturation (IHS) transform of multispectral(MS) image is implemented. Then panchromatic (PAN) image and the intensity component of MS images are decomposed into three scales using dual-tree complex wavelet transform respectively. After that, the region-based fusion operator is adopted to select the wavelet coefficients by fusing the wavelet coefficients of PAN image and the intensity component of MS at each scale in wavelet domain. Further, the fused intensity component is obtained by inverse dual-tree complex wavelet transform. Finally, the fused image is obtained by inverse IHS transform. The experiment results demonstrate our proposed method is effective.

In this paper, a new method for Panchromatic and Multispectral satellite image fusion is proposed. The major challenge of a fusion algorithm is to improve the spatial and spectral qualities of the fused image. But the spatial and spectral qualities are inversely proportional; we cannot improve either quality above particular range without losing visual quality, and most of the current methods do not take into consideration about visual quality. The proposed method tries to improve the spatial and visual quality with reduced spectral distortion using a Bi-Level Intensity Hue Saturation transform. Proposed method is rigorously tested over QuickBird and IKONOS satellite images and the experimental results shows that our method produces high visual quality fused images with a good spatial and spectral quality levels compared with existing methods.