DDFusion: An efficient multi-exposure fusion network with dense pyramidal convolution and de-correlation fusion

Emission computed tomography provides information on the growth and metabolic status of brain gliomas, but its use of radiopharmaceuticals brings radiation risks and financial costs to patients. These problems will be improved by reducing the dose of radiopharmaceuticals, but it will reduce the quality of brain emission computed tomography (ECT) images, which will lead to low resolution, unclear lesion areas, and blurred boundaries of parts of the lesion areas in the magnetic resonance imaging-emission computed tomography (MRI-ECT) fusion image of gliomas. Therefore, a high-definition fusion of multi-modal images of brain glioma based on a super-resolution reconstruction network based on the adaptive kernel selection (AKS-SRNet) is proposed. First, to solve the problem of loss of detailed structural information in the disease region of fusion images, a correlation-driven feature decomposition fusion network based on multi-scale and fine-grained feature extraction was constructed. The multi-scale feature extraction module and the outlook attention module are introduced into the fusion network to extract multi-scale features and retain more detailed features. Second, to solve the problems of low resolution and unclear lesion areas of MRI and ECT fusion images, a super-resolution reconstruction network based on adaptive kernel selection was constructed. A selective kernel attention module is added to the generator of PAUP-ESRGAN to improve the efficiency of the super-resolution reconstruction network generator. Finally, the experimental results show that the fusion image generated by the proposed high-definition fusion network has a less burr-prone and a clearer contour of the lesion area, which is helpful for doctors in analyzing and diagnosing the glioma accurately.

Infrared and visible image fusion improves the visual representation of scenes. Current deep learning-based fusion methods typically rely on either convolution operations for local feature extraction or Transformers for global feature extraction, often neglecting the contribution of multi-scale features to fusion performance. To address this limitation, we propose MRMAFusion, a nested connection model that relies on the multi-scale restoration-Transformer (Restormer) and multi-dimensional attention. We construct an encoder–decoder architecture on UNet++ network with multi-scale local and global feature extraction using convolution blocks and Restormer. Restormer can provide global dependency and more comprehensive attention to texture details of the target region along the vertical dimension, compared to extracting features by convolution operations. Along the horizontal dimension, we enhance MRMAFusion’s multi-scale feature extraction and reconstruction capability by incorporating multi-dimensional attention into the encoder’s convolutional blocks. We perform extensive experiments on the public datasets TNO, NIR and RoadScene and compare with other state-of-the-art methods for both objective and subjective evaluation.

Gait recognition is a distinctive biometric technique that can identify pedestrians by their walking patterns from considerable distances. A critical challenge in gait recognition lies in effectively acquiring discriminative spatial-temporal representations from silhouettes that exhibit invariance to disturbances. In this paper, we present a novel gait recognition network by aggregating features in the spatial-temporal and view domains, which consists of two-path spatial-temporal feature fusion module and view embedding module. Specifically, two-path spatial-temporal feature fusion module firstly utilizes multi-scale feature extraction (MSFE) to enrich the input features with multiple convolution kernels of various sizes. Then, frame-level spatial feature extraction (FLSFE) and multi-scale temporal feature extraction (MSTFE) are parallelly constructed to capture spatial and temporal gait features of different granularities and these features are fused together to obtain muti-scale spatial-temporal features. FLSFE is designed to extract both global and local gait features by employing a specially designed residual operation. Simultaneously, MSTFE is applied to adaptively interact multi-scale temporal features and produce suitable motion representations in temporal domain. Taking into account the view information, we introduce a view embedding module to reduce the impact of differing viewpoints. Through the extensive experimentation over CASIA-B and OU-MVLP datasets, the proposed method has achieved superior performance to the other state-of-the-art gait recognition approaches.

This paper proposes an effective method to improve the spatial resolution of FengYun-2 (FY-2) infrared cloud images via deep convolutional neural networks. The proposed model consists of four parts: shallow feature representation block, stacked multi-scale fusion blocks, global feature fusion block, and feature reconstruction block. The multi-scale fusion block combines dilated convolution, local feature fusion and local residual learning to extract multi-scale local features from the original low-resolution image directly. Then these local features are all merged by the global feature fusion block to reconstruct the residual representations in high-resolution space. For training and testing, we have specially built a dataset of infrared cloud images. We evaluated the proposed method both on simulated and real data. Experimental results demonstrate that the proposed approach achieves improved reconstruction accuracy than the state-of-the-art methods. Besides, the concise structure of the proposed model enables it to be applicable in practice.

Most of the impulse denoisers are either median filter‐based or fuzzy filter‐based, which can only perform well in low noise conditions. This study presents an efficient convolutional neural network (CNN) with particle swarm optimisation (PSO) model for high‐density impulse noise removal. The proposed high‐density impulse noise detection and removal model mainly consists of two parts: the impulse noise removal and impulse noisy pixel detection for restoration. The authors’ model initially leverages the powerful ability of deep CNN architecture to separate noise from the noisy image, then adopts PSO to pinpoint the most optimised threshold values for detecting impulse noisy pixels. An ensemble of these algorithms is an intelligent and adaptive solution, producing a clean output while preserving significant pixel information. Targeting to solve high‐density impulse noise problems, the authors have trained their model with a massive collection of natural images and 14 standard testing images are used for validation purposes. In order to validate the robustness of the proposed method, different levels of high‐density impulse noise are considered. Based on the final denoised images, their model has proven its reliability, in terms of both visual quality and quantitative evaluation, on greyscale and colour images.

In remote sensing image fusion, pansharpening is a type of remote sensing image fusion method that aims to fuse panchromatic (PAN) images and multispectral (MS) images to produce high-resolution multispectral (HRMS) images. Deep learning based pansharpening technology offers a series of advanced unsupervised algorithms. However, there are several challenges: (1) The existing unsupervised pansharpening methods only consider the fusion of single-scale features; (2) for the fusion of MS and PAN image feature branches, the existing pansharpening methods are implemented directly by concatenation and summation, without paying attention to critical features or suppressing redundant features; (3) the semantic gap in the long skip connections of the network architecture will create unexpected results. In this paper, we design a multiscale unsupervised architecture based on generative adversarial networks (GANs) for remote sensing image pansharpening (Mun-GAN), which consists of a generator and two discriminators. The generator includes a multi-scale feature extractor (MFE), a self-adaptation weighted fusion (SWF) module, and a nest feature aggregation (NFA) module. First, the MFE is utilized to extract multiscale feature information from the input images and to then pass this information to the SWF module for adaptive weight fusion. Then, multiscale features are reconstructed by the NFA module to obtain HRMS images. The two discriminators are spectral and spatial discriminators used against the generator. Moreover, we design a hybrid loss function to aggregate the multiscale spectral and spatial feature information. Compared with other state-of-the-art methods using QuickBird, GaoFen-2 and WorldView-3 images, which demonstrate that the Mun-GAN yields better fusion results.

In medical imaging diagnosis, accurate segmentation of the knee joint can help doctors better observe and diagnose lesions, thereby improving diagnostic accuracy and treatment effectiveness. Vision Mamba mainly relies on the State Space Model (SSM) for feature modeling, which excels at capturing global contextual information but cannot capture local texture features. Moreover, features of different scales are not effectively integrated, resulting in the model's weak segmentation ability on small-scale tissues (such as cartilage areas). To this end, this study proposed a novel multi-scale Vision Mamba Unet (VM-Unet) framework named MSPF-VM-Unet to perform the segmentation on the femur, tibia, femoral cartilage, and tibial cartilage in knee MRI images. The proposed MSPF-VM-Unet extends VM-Unet by introducing a designed multi-scale pyramid feature extraction network named MPSK, which synergizes multi-resolution feature extraction with channel-space attention. MPSK network enhances multi-scale local feature extraction through Selective Kernel (SK) convolution and pyramid pooling. The network merges the overall context information extracted by the Vision Mamba encoder to achieve the coordinated optimization of a multi-scale hierarchical feature fusion mechanism and global long-range dependency modeling. The results of the comparative experiments on the OAI-ZIB dataset indicate that MSPF-VM-Unet significantly improves the boundary accuracy and regional consistency of the MRI tibiofemoral joint tissue structure.

To realize the multi-focus image fusion task, an end-to-end deep convolutional neural network (DCNN) model that produces the final fused image directly from the source images is presented in this paper. In order to promote the fusion precision, the innovative multi-focus fusion DCNN introduces a multi-scale feature extraction (MFE) unit to collect more complementary features from different spatial scales and fuse them to excavate more spatial information. Moreover, a visual attention unit is designed to help the network locate the focused region more accurately and pick more useful features for perfectly splicing the details in the fusion process. Experimental results illustrate that the proposed method is superior to several existing multi-focus image fusion methods in both of the subjective visual effects and objective quality metrics.

Wind turbine rolling bearings are crucial components for ensuring the reliability and stability of wind power systems. Their failure can lead to significant economic losses and equipment downtime. Therefore, the accurate diagnosis of bearing faults is of great importance. Although existing deep learning fault diagnosis methods have achieved certain results, they still face limitations such as inadequate feature extraction capabilities, insufficient generalization to complex working conditions, and ineffective multi-scale feature capture. To address these issues, this paper proposes an advanced fault diagnosis method named the two-stream feature fusion convolutional neural network (TSFFResNet-Net). Firstly, the proposed method combines the advantages of one-dimensional convolutional neural networks (1D-ResNet) and two-dimensional convolutional neural networks (2D-ResNet). It transforms one-dimensional vibration signals into two-dimensional images through the empirical wavelet transform (EWT) method. Then, parallel convolutional kernels in 1D-ResNet and 2D-ResNet are used to extract multi-scale features, respectively. Next, the Convolutional Block Attention Module (CBAM) is introduced to enhance the network’s ability to capture key features by focusing on important features in specific channels or spatial areas. After feature fusion, CBAM is introduced again to further enhance the effect of feature fusion, ensuring that the features extracted by different network branches can be effectively integrated, ultimately providing more accurate input features for the classification task of the fully connected layer. The experimental results demonstrate that the proposed method outperforms other traditional methods and advanced convolutional neural network models on different datasets. Compared with convolutional neural network models such as LeNet-5, AlexNet, and ResNet, the proposed method achieves a significantly higher accuracy on the test set, with a stable accuracy of over 99%. Compared with other models, it shows better generalization and stability, effectively improving the overall performance of rolling bearing vibration signal fault diagnosis. The method provides an effective solution for the intelligent fault diagnosis of wind turbine rolling bearings.

A new framework for dynamic visual landscape quality evaluation of greenways based on perceived sensory dimensions: A case study of Chengdu urban greenway

Multimodal medical image fusion based on visual saliency map and multichannel dynamic threshold neural P systems in sub-window variance filter domain

Hyperspectral image classification is a critical task in remote sensing, but existing methods often employ fixed feature fusion strategies, making it difficult to adapt to the data characteristics of different scenarios. Additionally, there is a lack of effective synergy between multi-scale feature extraction and attention mechanisms. To address this issue, this paper proposes a dynamic gated fusion network with hierarchical multi-scale attention (DGFNet). This method comprises three core modules: the multi-scale feature aggregator (MSFA), which uses a pyramid expansion convolution structure to concurrently extract spatial features with different receptive fields, achieving comprehensive scale coverage from local texture to global context; the enhanced channel-spatial attention (ECSA) module, which employs multi-pooling strategies and a cascaded structure to achieve deep interaction between channel and spatial attention, thereby adaptively enhancing discriminative features; and the dynamic gated fusion module, which learns input-related fusion weights to adaptively adjust the contribution ratios of multi-scale features and attention features based on data characteristics. Experimental results on four benchmark datasets—Pavia University, Houston, Indian Pines, and WHU-HongHu—show that DGFNet achieves overall accuracy rates of 96.91, 97.12, 94.05, and 94.46%, respectively, representing significant improvements over existing state-of-the-art methods. Ablation experiments thoroughly validate the effectiveness and necessity of each module. Additionally, this paper systematically compares five different fusion strategies (cross-attention, hierarchical fusion, parallel fusion, recurrent fusion, and sequential fusion). Experimental results demonstrate that dynamic gated fusion outperforms other fusion methods in terms of classification accuracy, computational efficiency, and model stability. This method provides an efficient, accurate, and robust solution for hyperspectral image classification. The code will be published on https://github.com/willianbilledu-alt/DGFNet.

In this paper, a novel fusion network for infrared and visible images is proposed, named Image Pair Learning Fusion Network (IPLF). At present, most of the released deep learning-based fusion models are trained using unsupervised learning. This learning method lacks ground truth and cannot guide network model learning in a targeted manner. First, we propose the use of supervised learning to guide the training of fusion models (IPLF). The generated image pairs are used as ground truth. Second, we propose a model learning strategy using paired images. This learning strategy enhances the complementary constraints of the network model so that the final fused image possesses the rich features of both infrared and visible images. Third, we designed the structure measurement loss function and the edge preservation loss function to ensure that the generated fusion image has rich edge information and comfortable visual effects. In addition, we have introduced a spatial attention module, which can make the final fusion image highlight the target information. Finally, the size of the convolution kernel we use in the first convolution block is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$7\ \times\ 7$ </tex-math></inline-formula> , the main purpose is to increase the receptive field. Experiments on the TNO and CVC-14 datasets prove that our proposed method is superior to the existing state-of-the-art methods in terms of qualitative and quantitative evaluation. The source code can be found in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/depeng6/IPLF</uri> .

The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.

In the latest years, videoconferencing has taken a fundamental role in interpersonal relations, both for personal and business purposes. Lossy video compression algorithms are the enabling technology for videoconferencing, as they reduce the bandwidth required for real-time video streaming. However, lossy video compression decreases the perceived visual quality. Thus, many techniques for reducing compression artifacts and improving video visual quality have been proposed in recent years. In this work, we propose a novel GAN-based method for compression artifacts reduction in videoconferencing. Given that, in this context, the speaker is typically in front of the camera and remains the same for the entire duration of the transmission, we can maintain a set of reference keyframes of the person from the higher-quality I-frames that are transmitted within the video stream and exploit them to guide the visual quality improvement; a novel aspect of this approach is the update policy that maintains and updates a compact and effective set of reference keyframes. First, we extract multi-scale features from the compressed and reference frames. Then, our architecture combines these features in a progressive manner according to facial landmarks. This allows the restoration of the high-frequency details lost after the video compression. Experiments show that the proposed approach improves visual quality and generates photo-realistic results even with high compression rates. Code and pre-trained networks are publicly available at https://github.com/LorenzoAgnolucci/Keyframes-GAN.