Three phase segmentation algorithm for high resolution satellite images

This paper proposes a pan-sharpening method based on multi-direction subbands and deep neural networks. First, by utilizing the multi-scale and multi-direction properties of the nonsubsampled contourlet transform (NSCT), panchromatic (PAN) image is decomposed into the low frequency subbands in different resolutions and the high frequency subbands in different directions. Pan-sharpening method aims to fuse the high frequency subband coefficients of PAN image and the low frequency subband coefficients of multispectral (MS) image. Second, in order to better extract the feature of the high frequency subbands in different directions of PAN image, the deep neural network (DNN) is trained using the image patches of high frequency subbands of PAN image. Third, in the fusion stage, we exploit NSCT on the principal component of resampled low resolution (LR) MS image. The high frequency subbands of output high resolution (HR) MS image is obtained by forward propagation of the trained DNN, which input is the high frequency subbands of LR MS image. Finally, a new subband set is obtained by fusing the reconstructed high frequency subband and the original low frequency subband of LR MS image. The HR MS image is produced by executing the inverse transform of NSCT and adaptive PCA (A-PCA) on the new subband set. The experimental results show the proposed method outperforms other well-known methods in terms of both objective measurements and visual evaluation.

Recently, many convolutional networks have been built to fuse a low spatial resolution (LR) hyperspectral image (HSI) and a high spatial resolution (HR) multispectral image (MSI) to obtain HR HSIs. However, most deep learning-based methods are supervised methods, which require sufficient HR HSIs for supervised training. Collecting plenty of HR HSIs is laborious and time-consuming. In this paper, a self-supervised spectral-spatial residual network (SSRN) is proposed to alleviate dependence on a mass of HR HSIs. In SSRN, the fusion of HR MSIs and LR HSIs is considered a pixel-wise spectral mapping problem. Firstly, this paper assumes that the spectral mapping between HR MSIs and HR HSIs can be approximated by the spectral mapping between LR MSIs (derived from HR MSIs) and LR HSIs. Secondly, the spectral mapping between LR MSIs and LR HSIs is explored by SSRN. Finally, a self-supervised fine-tuning strategy is proposed to transfer the learned spectral mapping to generate HR HSIs. SSRN does not require HR HSIs as the supervised information in training. Simulated and real hyperspectral databases are utilized to verify the performance of SSRN.

Remote sensing image fusion (or pan-sharpening) aims at generating high resolution multi-spectral (MS) image from inputs of a high spatial resolution single band panchromatic (PAN) image and a low spatial resolution multi-spectral image. In this paper, a deep convolutional neural network with two-stream inputs respectively for PAN and MS images is proposed for remote sensing image pan-sharpening. Firstly the network extracts features from PAN and MS images, then it fuses them to form compact feature maps that can represent both spatial and spectral information of PAN and MS images, simultaneously. Finally, the desired high spatial resolution MS image is recovered from the fused features using an encoding-decoding scheme. Experiments on Quickbird satellite images demonstrate that the proposed method can fuse the PAN and MS image effectively.

Hyperspectral-multispectral image fusion aims to generate high-resolution hyperspectral (HRHS) images with rich spectral details, by mining the complementary information between low-resolution hyperspectral (LRHS) images and high resolution multispectral (HRMS) images. However, in practical scenarios, LRHS images are susceptible to noise interference due to their narrow spectral bandwidth, resulting in limited energy absorption of surface reflectance in each band. Various noise disrupt the correlations of LRHS and HRMS images, thus leading to the spatial details loss or spectral distortion. To address these challenges, we propose an anti-noise hyperspectral and multispectral image fusion network based on State Space Model (SSM), namely AHMFNet. To mitigate noise in LRHS images, the spectral denoising and refinement module is designed by combining low-rank representation with the discrete wavelet transform, while simultaneously extracting key spectral features. Then, the SSM is utilized for global modeling of spectral information to enhance essential features. Finally, although spectral and spatial features have differences, the spatial-spectral cross Mamba is designed to fuse them by mining their relationships, thus improving the fusion quality. Experiments on four public datasets show that, compared to existing state-of-the-art methods, the proposed AHMFNet generates images with superior spatial quality and more accurate spectral characteristics under various noise conditions.

High spatial resolution multispectral (HMS) images can provide sufficient information for researchers to analyze the potential disasters in the living environment. However, an original multispectral (MS) image is with low-space-resolution and high-spectrum-resolution, while an original panchromatic (PAN) image has the opposite property. Pansharpening aims at obtaining HMS image by retaining spectrum of the MS image and injecting details of the PAN image simultaneously. In this paper, we present a new pansharpening method. First, we use a bilateral filter (BF) to obtain the low-frequency-component (LFC) of PAN and MS images, respectively. Then the high-frequency-component (HFC) of PAN and MS images are readily obtained. Second, an adaptive intensity-hue-saturation (AIHS) based method is applied to generate the HFC of intensity. Finally, a dual-scale guided image filter (GIF) is utilized to calculate the difference between HFC of intensity and PAN to get the detail images. And then, these detail images are injected into the original MS image to achieve the HMS image. The proposed method is applied into testing various satellite data sets, and performs better effect on both visual quality and objective indictors than the existing methods.

Hyperspectral images (HSI) have high spectral and low spatial resolutions. However, multispectral images (MSI) usually have low spectral and high spatial resolutions. In various applications HSI with high spectral and spatial resolutions are required. In this paper, a new method for spatial resolution enhancement of HSI using high resolution MSI based on sparse coding and linear spectral unmixing (SCLSU) is introduced. In the proposed method (SCLSU), high spectral resolution features of HSI and high spatial resolution features of MSI are fused. In this case, the sparse representation of some high resolution MSI and linear spectral unmixing (LSU) model of HSI and MSI is simultaneously used in order to construct high resolution HSI (HRHSI). The fusion process of HSI and MSI is formulated as an ill-posed inverse problem. It is solved by the Split Augmented Lagrangian Shrinkage Algorithm (SALSA) and an orthogonal matching pursuit (OMP) algorithm. Finally, the proposed algorithm is applied to the Hyperion and ALI datasets. Compared with the other state-of-the-art algorithms such as Coupled Nonnegative Matrix Factorization (CNMF) and local spectral unmixing, the SCLSU has significantly increased the spatial resolution and in addition the spectral content of HSI is well maintained.

in vivo multispectral imaging (MSI) increases the sensitivity and usefulness of small animal models of development and disease, including the most popular and promising experimental systems. In mice, MSI can essentially eliminate autofluorescence that otherwise limits sensitivity, reducing detection limits by as much as 100-fold. Fluorescence imaging is a key technology for determining the where, the when and the how much of gene expression. However, the ability to image and quantitate fluorescently labeled markers in vivo has typically been limited by the bright autofluorescence of animal tissues. Small-animal imaging, which often means the imaging of mouse models, encounters autofluorescence primarily from components in skin (collagen, which fluoresces green) and food (chlorophyll, which fluoresces red). However, other small animal model systems, such as nematodes (C. elegans), zebrafish (D. rerio) embryos or adult fish, and fruit flies (D. melanogaster) pose similar problems due to the presence of autofluorescent structures that may render common labeling strategies, including the use of green fluorescent protein (GFP) labels, problematic. Various solutions have been proposed for the reduction or elimination of autofluorescence. The most prevalent is simply to use narrow bandpass emission filters in an effort to isolate the desired fluorescence signal [1]. More recently, there has been a move to use infrared-emitting labels that are excited at wavelengths that are much less likely to induce autofluorescence signals. However, autofluorescence still limits sensitivity. Fortunately, the low signal-to-background contrast problem created by autofluorescence can be substantially eliminated using MSI techniques. Moreover, MSI permits the simultaneous use of multiple labels, such as GFP and RFP, even when they spectrally overlap with themselves or autofluorescence [2]. Presented here are the results of applying an MSI strategy to the whole-animal in vivo imaging of mice, C. elegans, D. rerio and D. melanogaster at a variety of magnifications, ranging from the imaging of three whole mice simultaneously down to microscopic images of whole animals and embryos. This is accomplished using a liquid crystal tunable filter (LCTF)-based imaging system that has been adapted to either a C-mount on a fluorescence microscope or macroscope, or incorporated into a whole-animal imaging system. In each case, the sensitivity of the fluorescence measurement was greatly improved by using a MSI approach. Rapid, quantitative, imaging of GFP signals of up to a few hundred flies simultaneously in a few seconds can be easily achieved, as can high-sensitivity microscopic imaging of GFP in nematode worms. These types of imaging experiments are of value to anyone with whole-animal fluorescence-based models. [1] Niswender KD et al, J Microsc 180, 109-116 (1995) [2] Levenson, et al, Cytometry A. 2006 Aug; 69(8):748-58 Citation Format: {Authors}. {Abstract title} [abstract]. In: Proceedings of the 101st Annual Meeting of the American Association for Cancer Research; 2010 Apr 17-21; Washington, DC. Philadelphia (PA): AACR; Cancer Res 2010;70(8 Suppl):Abstract nr 3236.

The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods.

Fusing a low spatial resolution hyperspectral image (HSI) with a high spatial resolution multispectral image (MSI), aiming to produce a super-resolution hyperspectral image, has recently attracted increasing research interest. In this paper, a novel approach based on coupled non-negative tensor decomposition is proposed. The proposed method performs a tucker tensor factorization of a low resolution hyperspectral image and a high resolution multispectral image under the constraint of non-negative tensor decomposition (NTD). The conventional matrix factorization methods essentially lose spatio-spectral structure information when stacking the 3D data structure of a hyperspectral image into a matrix form. Moreover, the spectral, spatial, or their joint structural features have to be imposed from the outside as a constraint to well pose the matrix factorization problem. The proposed method has the advantage of preserving the spatio-spectral structure of hyperspectral images. In this paper, the NTD is directly imposed on the coupled tensors of the HSI and MSI. Hence, the intrinsic spatio-spectral structure of the HSI is represented without loss, and spatial and spectral information can be interdependently exploited. Furthermore, multilinear interactions of different modes of the HSIs can be exactly modeled with the core tensor of the Tucker tensor decomposition. The proposed method is straightforward and easy to implement. Unlike other state-of-the-art approaches, the complexity of the proposed approach is linear with the size of the HSI cube. Experiments on two well-known datasets give promising results when compared with some recent methods from the literature.

Fusing a low-resolution (LR) hyperspectral image (HSI) with a high-resolution (HR) multispectral image (MSI) has emerged as a promising strategy for reconstructing high-quality HSIs that combine rich spectral and fine spatial information. However, most existing HSI fusion methods operate under the restrictive assumption that the LR HSI and HR MSI are spatially aligned and fully consistent on the field-of-view (FoV), which significantly limits their applicability in real-world scenarios when such alignment is unavailable. To overcome these limitations, we propose IrregFusion, a generalized HSI fusion framework capable of handling both FoV-consistent and inconsistent fusion scenarios. Specifically, IrregFusion incorporates a Transformer-based reconstruction module that captures both intra- and inter-modal correlations between the diverse spectra data and the MSI, enhancing the model’s ability to perceive and reconstruct non-local spectral–spatial structures. To further address the challenges posed by FoV inconsistencies, we introduce a spectral propagation strategy that diffuses observed spectral information into adjacent spectral-blank regions, thereby easing the reconstruction of missing spectral content. Additionally, a self-supervised adaptation mechanism is integrated into the framework, enabling robust spectral–spatial representation learning and enhancing generalization across diverse and challenging conditions. Extensive experiments conducted on benchmark datasets demonstrate that IrregFusion effectively addresses the challenges of diverse spectral data fusion and consistently outperforms state-of-the-art methods in both reconstruction accuracy and visual fidelity. The source code will be released in https://github.com/JiangtaoNie/IrregFusion.git.

Distributed compressed sensing-based pan-sharpening with hybrid dictionary

Promoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we propose a new hyperspectral image super-resolution method from a low-resolution (LR) hyperspectral image and a high resolution (HR) multispectral image of the same scene. The reconstruction of HR hyperspectral image is formulated as a joint estimation of the hyperspectral dictionary and the sparse codes based on the spatial-spectral sparsity of the hyperspectral image. The hyperspectral dictionary is learned from the LR hyperspectral image. The sparse codes with respect to the learned dictionary are estimated from LR hyperspectral image and the corresponding HR multispectral image. To improve the accuracy, both spectral dictionary learning and sparse coefficients estimation exploit the spatial correlation of the HR hyperspectral image. Experiments show that the proposed method outperforms several state-of-art hyperspectral image super-resolution methods in objective quality metrics and visual performance.

This paper presents an efficient and accurate automated system based on the hybrid XGBoost with Random forest (XGBRF) ensemble model in order to classify the Glioma (type of mostly diagnosed brain tumor) into low grade and high grade Glioma. In this approach initially global thresholding is employed on various MRI sequence and their fusion combinations in order to perform the accurate segmentation. Then uses a proposed Enhanced wavelet binary pattern run length matrix method (EWBPRL) for textural features extraction from the region of interest or segmented Glioma tumor region. This proposed feature extraction method is based on the Discrete wavelet transform (DWT), Local Binary pattern (LBP) and Gray level run length Matrix (GLRLM) methods to extract texture features from the segmented region. Some morphological features are also computed from the segmented region along the textural features. Finally both these extracted features are employed in order to train a hybrid XGBoost with Random forest ensemble model for the first time. The proposed automated system apart from accurately detecting and segmenting the tumors region from the fused MRI sequences, also tends to determine the grading of Glioma in terms of severity. The proposed system is evaluated on the large size balance local dataset and as well as on the popular global datasets like BRATS 2013 and BRATS 2015. This approach offers an encouraging accuracy of 99.25% on the local dataset with the fusion of T1C+T2+Flair MRI sequence as compare to 96.75% accuracy, which is achieved utilizing the fusion of T1+T1C+T2+Flair MRI sequence.

In remote sensing image fusion, pansharpening is a type of remote sensing image fusion method that aims to fuse panchromatic (PAN) images and multispectral (MS) images to produce high-resolution multispectral (HRMS) images. Deep learning based pansharpening technology offers a series of advanced unsupervised algorithms. However, there are several challenges: (1) The existing unsupervised pansharpening methods only consider the fusion of single-scale features; (2) for the fusion of MS and PAN image feature branches, the existing pansharpening methods are implemented directly by concatenation and summation, without paying attention to critical features or suppressing redundant features; (3) the semantic gap in the long skip connections of the network architecture will create unexpected results. In this paper, we design a multiscale unsupervised architecture based on generative adversarial networks (GANs) for remote sensing image pansharpening (Mun-GAN), which consists of a generator and two discriminators. The generator includes a multi-scale feature extractor (MFE), a self-adaptation weighted fusion (SWF) module, and a nest feature aggregation (NFA) module. First, the MFE is utilized to extract multiscale feature information from the input images and to then pass this information to the SWF module for adaptive weight fusion. Then, multiscale features are reconstructed by the NFA module to obtain HRMS images. The two discriminators are spectral and spatial discriminators used against the generator. Moreover, we design a hybrid loss function to aggregate the multiscale spectral and spatial feature information. Compared with other state-of-the-art methods using QuickBird, GaoFen-2 and WorldView-3 images, which demonstrate that the Mun-GAN yields better fusion results.

Detection of spinal cord injury (SCI) is one of the major problems in MRI images to detect the affected portion of spinal cord regions using feature sets. Automatic detection of spinal cord atrophy is complex due to change in structure, size, and white matter. Delineating gray matter and white matter are the essential factors that influence the detection of spinal cord atrophy and its severity. Automatic segmentation and classification are accurate methods for detecting the severity of the SCI. Hierarchical segmentation, partitioning segmentation, graph, and watershed segmentation methods are used to find the SCI segments in static fixed positions. Also, these segmentation models result in a high false positive rate due to over segmentation features and noise in the segmented regions. Furthermore, these classification methods fail to segment and detect the severity level in the affected region due to over segmentation. In order to overcome these issues, a novel segment-based classification model is required to find the severity of the injury and to predict the disease patterns on the over segmented regions and features. In the present model, a hybrid image threshold technique is used to segment the spinal cord regions for non-linear SVM classification approach. Among the traditional feature segmentation-based classification models, the proposed threshold-based non-linear SVM has better accuracy for SCI detection. The results proved that the present model is more efficient than the earlier approaches in terms of true positive rate ( $\text {TP}=0.9783$ ) and accuracy (0.9683).