Efficient super resolution-based detail injection network for multispectral pan-sharpening

Synthetic aperture radar (SAR) image target recognition has important application values in security reconnaissance and disaster monitoring. However, due to speckle noise and target orientation sensitivity in SAR images, traditional spatial domain recognition methods face challenges in accuracy and robustness. To effectively address these challenges, we propose a dual-branch spatial-frequency domain fusion recognition method with cross-attention, achieving deep fusion of spatial and frequency domain features. In the spatial domain, we propose an enhanced multi-scale feature extraction module (EMFE), which adopts a multi-branch parallel structure to effectively enhance the network’s multi-scale feature representation capability. Combining frequency domain guided attention, the model focuses on key regional features in the spatial domain. In the frequency domain, we design a hybrid frequency domain transformation module (HFDT) that extracts real and imaginary features through Fourier transform to capture the global structure of the image. Meanwhile, we introduce a spatially guided frequency domain attention to enhance the discriminative capability of frequency domain features. Finally, we propose a cross-domain feature fusion (CDFF) module, which achieves bidirectional interaction and optimal fusion of spatial-frequency domain features through cross attention and adaptive feature fusion. Experimental results demonstrate that our method achieves significantly superior recognition accuracy compared to existing methods on the MSTAR dataset.

Conventional spectral image demosaicing algorithms rely on pixels' spatial or spectral correlations for reconstruction. Due to the missing data in the multispectral filter array (MSFA), the estimation of spatial or spectral correlations is inaccurate, leading to poor reconstruction results, and these algorithms are time-consuming. Deep learning-based spectral image demosaicing methods directly learn the nonlinear mapping relationship between 2D spectral mosaic images and 3D multispectral images. However, these learning-based methods focused only on learning the mapping relationship in the spatial domain, but neglected valuable image information in the frequency domain, resulting in limited reconstruction quality. To address the above issues, this paper proposes a novel lightweight spectral image demosaicing method based on joint spatial and frequency domain information learning. First, a novel parameter-free spectral image initialization strategy based on the Fourier transform is proposed, which leads to better initialized spectral images and eases the difficulty of subsequent spectral image reconstruction. Furthermore, an efficient spatial-frequency transformer network is proposed, which jointly learns the spatial correlations and the frequency domain characteristics. Compared to existing learning-based spectral image demosaicing methods, the proposed method significantly reduces the number of model parameters and computational complexity. Extensive experiments on simulated and real-world data show that the proposed method notably outperforms existing spectral image demosaicing methods.

Pansharpening aims to generate high-resolution multispectral (HRMS) images by fusing high-resolution panchromatic (PAN) images with low-resolution multispectral (LRMS) images for various practical tasks such as ground-object classification and change detection. Currently, Transformers have been widely utilized in the pansharpening field because of their capability to model global representations. Transformer-based pansharpening methods capture a global representation of local patch sequences in the spatial domain, but they do not aggregate each pixel in the image, which hinders their ability to capture the global context and dependencies. To solve this issue, a novel pansharpening network based on Fourier with dropout in global modeling (FDGM) is proposed, which can better model global representations and obtain higher-quality fusion images without increasing computational complexity. First, an efficient Fourier with dropout block (FDB) is designed by transforming the features from the spatial domain to the frequency domain to capture the global information from the panchromatic (PAN) and low-resolution multispectral (LRMS) images. This study represents a novel attempt to explore the generalization performance of dropout in the Fourier frequency domain. Furthermore, a residual spatial-frequency fusion module (RSFM) based on frequency channel attention is designed to enhance feature representation capability and effectively facilitate the fusion of the spatial and frequency features. Finally, the shallow features extracted from the source images are combined with the RSFM output to generate the final fusion results. The results of numerous experiments demonstrate that FDGM surpasses other state-of-the-art approaches in visual perception and quantitative evaluation.

Emotion recognition from macroexpression and microexpression has been widely used in applications such as human–computer interaction, learning status evaluation, and mental disorder diagnosis. However, due to the complexity of human macroexpressions, recognizing macroexpressions with high accuracy is a challenging task. Moreover, the short duration and low movement intensity of microexpressions make its recognition more difficult. For MM-FER (macro and microfacial expression recognition), the key information can be more efficiently expressed by a graph. In this article, a novel framework based on graph neural network named SSGNN (spatial and spectral domain features based on a graph neural network) is designed to extract spatial and spectral domain features from facial images for MM-FER, which can efficiently recognize both macroexpressions and microexpressions under the same model. SSGNN consists of two parts, SPAGNN and SPEGNN, which are used to extract spectral and spatial domain features, respectively. Experiments proved that jointly using the spectral and spatial information extracted by SSGNN can largely improve the performance of MM-FER when the training sample is limited. First, the influences of different neighbors and samples to the model performance was analyzed. Then, the contribution of SPAGNN and SPEGNN were evaluated. It was discovered that fusing the result of SPAGNN and SPEGNN at decision level further improved the performance of MM-FER. Experiment proved that SSGNN can recognize microexpression acquired by various sensors with higher accuracy under different image resolutions and image formats than the compared state-of-the-art methods in most cases. A cross-dataset experiment demonstrated the generalization ability of SSGNN.

Pansharpening technology plays a crucial role in remote sensing image processing by integrating low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images to generate high-resolution multispectral (HRMS) images. This process addresses the limitations of satellite sensors, which cannot directly capture HRMS images. Despite significant developments achieved by deep learning-based pansharpening methods over traditional approaches, most existing techniques either fail to account for the modal differences between LRMS and PAN images, relying on direct concatenation, or use similar network structures to extract spectral and spatial information. Additionally, many methods neglect the extraction of common features between LRMS and PAN images and lack network architectures specifically designed to extract spectral features. To address these limitations, this study proposed a novel three-branch pansharpening network that leverages both spatial and frequency domain interactions, resulting in improved spectral and spatial fidelity in the fusion outputs. The proposed method was validated on three datasets, including IKONOS, WorldView-3 (WV3), and WorldView-4 (WV4). The results demonstrate that the proposed method surpasses several leading techniques, achieving superior performance in both visual quality and quantitative metrics.

The image data hiding technique can embed the secret message into the digital image. For military or medical applications the reversible data hiding technique is preferred because it offers the advantage to recover the original image after the message extraction, yet it still maintains the reversibility. In order to achieve reversibility, we propose a reversible data hiding algorithm that explores the spatial and frequency domains. Predictive coding is employed to embed the secret message in the spatial domain. The proposed algorithm applies an integer-to-integer transformation to convert the image originating in the spatial domain into the frequency domain. We then modify the high frequency coefficients only when embedding the secret message. Consequently, the low frequency coefficients are unchanged enabling the embedded image to preserve the image quality due to the hidden message. In addition, the algorithm further increases the embedding capacity because it takes advantage of both the spatial and frequency domains. Keywords-Data hiding; reversible data embedding; predictive coding; wavelet transform

Single image super-resolution (SISR) with deep convolutional neural networks has recently attracted increasing attention due to its potentials to generate rich details. To obtain better fidelity and visual quality, most of existing methods are of heavy design with the depth of network. However, the lack of high-frequency information in the deep network and the limit of non-local context relations of pixels remain the core issue. In addition, there are many arbitrary-scale image super-resolution tasks, so it is a focus that provides an effective and efficient model for super resolution with arbitrary-scale. To this end, we propose a Feature Preserving and Enhancing Network (FPEN) based on implicit representation, which is aims to preserving high-frequency information in deep network and enhancing non-local contextual features of pixels to output more realistic images at arbitrary scales. In particular, our proposed High Frequency Preserving Block (HFPB) can divide the features into high-frequency components and low-frequency components, and allocate more operations to high-frequency components to ensure that high-frequency components can be preserved in the network. Since high-frequency information contains the details and texture of the image, it can restore the finer details of the image. Moreover, Pixel Continuity Attention (PiCA) module our proposed utilizes visual cues observed from pixels to adaptively recalibrate the pixel range that needs attention in the image to better achieve feature enhancement and generate smoother images. Extensive experiments conducted on benchmark SISR models and datasets show that Feature Preserving and Enhancing Network can be employed for various SISR tasks with arbitrary-scale to obtain the better visual quality than other state-of-the-art SR algorithms.

To address the issues of insufficient restoration of texture details in deblurred images and inadequate learning of frequency domain features, an image deblurring algorithm based on frequency domain feature enhancement and convolutional neural networks is proposed. In this architecture, firstly, a Fourier residual module with a parallel structure is constructed to achieve collaborative learning and modeling of spatial and frequency domain features, aiming to improve frequency domain feature learning capability and the restoration effect of the texture details; secondly, a gated controlled feed-forward unit acts on the Fourier residual module to further enhance the nonlinear expression ability of the algorithm; thirdly, a supervised attention module is improved and added to the decoder to promote more effective capture of key features for image reconstruction; finally, the weighted sum of spatial domain Charbonnier loss function and frequency domain loss function is defined as a novel total loss function. In addition, to verify the performance of our proposed algorithm, we conducted experiments on the GOPRO and HIDE datasets. Through experiments on the GOPRO, we obtained an SSIM and an LPIPS of 0.961 and 0.0278, respectively. With regard to the experiments on the HIDE datasets, we obtained an SSIM and an LPIPS of 0.941 and 0.0286, respectively. As for parameter count and running time, their values were 1.197 and 9.15 × 106, respectively, obtained by the experiments on the GOPRO. In all algorithms, the values of our proposed algorithm are optimal. However, the PSNR of our proposed algorithm is very close to that of the latest comparison algorithm and is suboptimal. In a word, experimental results have demonstrated that our proposed algorithm effectively removes blur while better preserving the details and edges of the image. Therefore, it has more practical value and prospects in computer vision tasks.

The success of modern single image super-resolution (SISR) algorithms is inspired by the development of deep convolutional neural networks (CNNs). However, these CNN-based methods require considerable computation and complexity, making it impossible for these methods to perform real-time calculations in edge devices. Thus, lightweight model design has become a development trend in the super-resolution field, including pruning, quantization, and other methods. The 1-bit quantization is an extreme lightweight method which can reduce the calculation amount of the model in an extreme manner and is friendly to hardware such as edge devices. Most existing binary quantization approaches lead to a large information loss during forward propagation, especially in detailed color information (e.g., edge, texture, and contrast). The loss of color information makes modern binary methods unsuitable for SISR tasks. We think the loss occurs because these methods typically utilize a uniform threshold to quantize the weights and activations. Thus, in this article, we thoroughly analyze the difference between normal classification tasks and SISR tasks, and present a binarization scheme based on local means. The proposed method can maintain more detailed information in feature maps using dynamic thresholds during quantization. Specifically, each value in the full precision activations has a corresponding threshold during the quantization process, and those thresholds are determined by the full precision values of the surroundings. In addition, a gradient approximator is introduced to adaptively optimize the gradient for updating binary weights. We then verify the effectiveness of our method for training binary networks on several SISR benchmarks including VDSR and SRResNet. Experimental results show that the proposed method can outperform the state-of-the-art algorithms to obtain binary networks for image super-resolution with better peak signal-to-noise ratio (PSNR) values and visual quality.

In this paper, we proposed improved single image super resolution using multiple re-evaluation Process model for use in various image processing fields. The proposed method generates the first super resolution using the input image, and analyzes the change for each region by comparing the features of previous image and super resolution result. According to the analyzed features, the feature map for generate n-th super resolution was selected for improved detail. After then, next generate super resolution using previous super resolution result as input image. This process is repeated for final result. The existing single image super resolution method has two areas to be improved. First, it minimizes artifacts or staircases, which are unnecessary details that can be created during the super resolution process. Second, it is necessary to consider the input image because it affects the result depending on the quality of input image used in the super resolution process. Therefore, in order to minimize unnecessary details, the proposed method analyzed the feature map from the generated super resolution result and applied it according to the amount of change. In addition, aimed to gradually improve the input image to be used in the super resolution process by using the super resolution generated in the previous step. By comparing and evaluating the proposed method with the conventional single image super resolution method with PSNR and SSIM, it is improved by about 3%.

Remote sensing road extraction based on deep learning is an important method for road extraction. However, in complex remote sensing images, different road information often exhibits varying frequency distributions and texture characteristics, and it is usually difficult to express the comprehensive characteristics of roads effectively from a single spatial domain perspective. To address the aforementioned issues, this article proposes a road extraction method that couples global spatial learning with Fourier frequency domain learning. This method first utilizes a transformer to capture global road features and then applies Fourier transform to separate and enhance high-frequency and low-frequency information. Finally, it integrates spatial and frequency domain features to express road characteristics comprehensively and overcome the effects of intra-class differences and occlusions. Experimental results on HF, MS, and DeepGlobe road datasets show that our method can more comprehensively express road features compared with other deep learning models (e.g., Unet, D-Linknet, DeepLab-v3, DCSwin, SGCN) and extract road boundaries more accurately and coherently. The IOU accuracy of the extracted results also achieved 72.54%, 55.35%, and 71.87%.

An efficient algorithm for pattern detection using combined classifiers and data fusion

Single image super-resolution (SISR) reconstruction for magnetic resonance imaging (MRI) has generated significant interest because of its potential to not only speed up imaging but to improve quantitative processing and analysis of available image data. Generative Adversarial Networks (GAN) have proven to perform well in image recovery tasks. In this work, we followed the GAN framework and developed a generator coupled with discriminator to tackle the task of 3D SISR on T1 brain MRI images. We developed a novel 3D memory-efficient residual-dense block generator (MRDG) that achieves state-of-the-art performance in terms of SSIM (Structural Similarity), PSNR (Peak Signal to Noise Ratio) and NRMSE (Normalized Root Mean Squared Error) metrics. We also designed a pyramid pooling discriminator (PPD) to recover details on different size scales simultaneously. Finally, we introduced model blending, a simple and computational efficient method to balance between image and texture quality in the final output, to the task of SISR on 3D images.

We proposed a novel method to separate static and dynamic speckles based on spatial frequency domain filtering. First, the raw speckle image sequence is processed frame by frame through 2D Fourier transform, low-pass and high-pass filtering in the spatial frequency domain, and inverse Fourier transform. Then, we can obtain low- and high-frequency image sequences in the spatial domain. Second, we averaged both sequences in the time domain. After the above processing, we obtain the mean intensities of the dynamic and static speckle components in the spatial domain. Finally, we calculated the time-averaged modulation depth to map the 2-D blood flow distribution. Both phantom and vivo experiments demonstrated that the proposed method could effectively suppress the background non-uniformity and has the advantage of high computational efficiency. It also can effectively improve image contrast, contrast-to-noise ratio, and imaging dynamic range.

In this paper, a generalized statistical tool is introduced to estimate key frames in a video sequence. The tool works based on the inter-relationship between different features of image frames in a video. The image feature vectors are plotted in feature space as points and a randomness measure is determined from the distribution of these points. The randomness measure of the feature vectors is defined with respect to simulated random point patterns and expressed as a probability value of a frame being a key frame. Since, depending on the video content more than one inter-relationship of features can be used to determine a single key frame, different probability values are derived to support a frame as a key frame. To integrate these probability values a combiner model is designed to uniquely decide the status of a key frame. The combiner model is based on the Dempster-Shafer theory of evidence. To demonstrate the idea, randomness measures, and consequently the probabilities of a frame to be a key frame, are obtained separately from spatial domain and frequency domain features. The combined probability value enhances the confidence in selecting a frame as a key frame. The result is tested on a number of standard video sequences and it outperforms the related approach