Infrared and Visible Image Fusion: Methods, Datasets, Applications, and Prospects

Development of an online prediction system for soil organic matter and soil moisture content based on multi-modal fusion

In response to the limitations of current infrared and visible light image fusion algorithms-namely insufficient feature extraction, loss of detailed texture information, underutilization of differential and shared information, and the high number of model parameters-this paper proposes a novel multi-scale infrared and visible image fusion method with two-branch feature interaction. The proposed method introduces a lightweight multi-scale group convolution, based on GS convolution, which enhances multi-scale information interaction while reducing network parameters by incorporating group convolution and stacking multiple small convolutional kernels. Furthermore, the multi-level attention module is improved by integrating edge-enhanced branches and depthwise separable convolutions to preserve detailed texture information. Additionally, a lightweight cross-attention fusion module is introduced, optimizing the use of differential and shared features while minimizing computational complexity. Lastly, the efficiency of local attention is enhanced by adding a multi-dimensional fusion branch, which bolsters the interaction of information across multiple dimensions and facilitates comprehensive spatial information extraction from multimodal images. The proposed algorithm, along with seven others, was tested extensively on public datasets such as TNO and Roadscene. The experimental results demonstrate that the proposed method outperforms other algorithms in both subjective and objective evaluation results. Additionally, it demonstrates good performance in terms of operational efficiency. Moreover, target detection performance experiments conducted on the [Formula: see text] dataset confirm the superior performance of the proposed algorithm.

In low light conditions, visible (VIS) images are of a low dynamic range (low contrast) with severe noise and color, while near-infrared (NIR) images contain clear textures without noise and color. Multispectral fusion of VIS and NIR images produces color images of high quality, rich textures, and little noise by taking both advantages of VIS and NIR images. In this article, we propose the deep selective fusion of VIS and NIR images using unsupervised U-Net. Existing image fusion methods are afflicted with the low contrast in VIS images and flash-like effect in NIR images. Thus, we adopt unsupervised U-Net to achieve deep selective fusion of multiple scale features. Due to the absence of the ground truth, we use unsupervised learning by formulating an energy function as a loss function. To deal with insufficient training data, we perform data augmentation by rotating images and adjusting their intensity. We synthesize training data by degrading clean VIS images and masking clean NIR images using a circle. First, we utilize pretrained visual geometry group (VGG) to extract features from VIS images. Second, we build an encoding network to obtain edge information from NIR images. Finally, we combine all features and feed them into a decoding network for fusion. Experimental results demonstrate that the proposed fusion network produces visually pleasing results with fine details, little noise, and natural color and it is superior to state-of-the-art methods in terms of visual quality and quantitative measurements.

In this paper, an infrared (IR) and visible (VIS) image fusion algorithm is designed for the injection of the IR objects into the VIS background in a perceptual manner. It consists of four parts: image decomposition, layer fusion, image reconstruction, and image refinement. An edge-preserving filter is constructed for image decomposition, in which an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$L_1$</tex-math></inline-formula> regularization term and a fractional gradient are newly introduced. The resulting filter is capable of not only preserving edges, but also attenuating the influence of the IR background. A two-layer fusion rule is adopted, which consists of a routine weighted-average fusion rule and an injected fusion rule. It ensures that the fused image is with both rich background information of the VIS image and the salient features of the IR image. After image reconstruction, the guided filter is applied again to the IR image to refine the fused image, such that the final version of the fused image is with satisfactory human visual perception under even dim lights. The effectiveness and superiority of our fusion algorithm are illustrated by the results of ablation studies and comparative experiments.

Infrared (IR) and visible (VIS) image fusion technology combines the complementary information of the same scene from IR and VIS imaging sensors to generate a composite image, which is beneficial to post image-processing tasks. In order to achieve good fusion performance, a method by combining rolling guidance filter (RGF) and convolutional sparse representation (CSR) is proposed. In the proposed method, RGF is performed on every pre-registered IR and VIS source images to obtain their detail layers and base layer. Then, the detail layers are fused with a serious of weighted coefficients produced by joint bilateral filer (JBF). The base layer is decomposed into a sub-detail-layer and a sub-base-layer. CSR is applied to fuse the sub-detail-layer and averaging strategy is used to fuse the sub-base-layer. Finally, the fused image is reconstructed by adding the fused detail layer and base layer. Experimental results demonstrate the superiority of our proposed method both in subjective and objective assessment.

Fusing infrared and visible images taken by an unmanned aerial vehicle (UAV) is a challenging task, since infrared images distinguish the target from the background by the difference in infrared radiation, while the low resolution also produces a less pronounced effect. Conversely, the visible light spectrum has a high spatial resolution and rich texture; however, it is easily affected by harsh weather conditions like low light. Therefore, the fusion of infrared and visible light has the potential to provide complementary advantages. In this paper, we propose a multi-scale dense feature-aware network via integrated attention for infrared and visible image fusion, namely DFA-Net. Firstly, we construct a dual-channel encoder to extract the deep features of infrared and visible images. Secondly, we adopt a nested decoder to adequately integrate the features of various scales of the encoder so as to realize the multi-scale feature representation of visible image detail texture and infrared image salient target. Then, we present a feature-aware network via integrated attention to further fuse the feature information of different scales, which can focus on specific advantage features of infrared and visible images. Finally, we use unsupervised gradient estimation and intensity loss to learn significant fusion features of infrared and visible images. In addition, our proposed DFA-Net approach addresses the challenges of fusing infrared and visible images captured by a UAV. The results show that DFA-Net achieved excellent image fusion performance in nine quantitative evaluation indexes under a low-light environment.

Aiming at solving the problems of part edges loss and blurred texture information in the image fusion process of infrared light image and visible light image, an improved algorithm based on regional texture smoothness and gradation mutation degree for the infrared light and visible light image fusion is proposed. Firstly, Non-subsampled Contourlet transform(NSCT) is adopted. The multi-scale and multi-direction sparse decomposition is performed for the infrared light source image and visible light source image. Then the low-frequency component reflecting the image approximate contents and the every bandpass directional subband components reflecting the image detail characteristics are obtained respectively. The fusion method based on the gradation mutation weighted average for the low-frequency subband component is adopted, and the fusion method based on the composition of regional texture smoothness and Laplace energy sum for the high-frequency subband is adopted. Finally, the fusion image is reconstructed through the NSCT reversed transform. The experimental results show that in the condition of reserving the source image information, the fusion image clarity is raised, the detail information and brightness contrast are enhanced by this fusion method.

A Comparison of Prostate Cancer Detection between Visual Estimation (Cognitive Registration) and Image Fusion (Software Registration) Targeted Transperineal Prostate Biopsy.

Infrared and visible image fusion (IVIF) is an essential branch of image fusion, and enhancing the visible image of IVIF can significantly improve the fusion performance. However, many existing low‐light enhancement methods are unsuitable for the visible image enhancement of IVIF. In order to solve this problem, this paper proposes a new visible image enhancement method for IVIF. Firstly, the colour balance and contrast enhancement‐based self‐calibrated illumination estimation (CCSCE) is proposed to improve the input image's brightness, contrast, and colour information. Then, the method based on Mutually Guided Image Filtering (muGIF) is adopted to design a strategy to extract details adaptively from the original visible image, which can keep details without introducing additional noise effectively. Finally, the proposed visible image enhancement technique is used for IVIF tasks. In addition, the proposed method can be used for the visible image enhancement of IVIF and other low‐light images. Experiment results on different public datasets and IVIF demonstrate the authors’ method's superiority from both qualitative and quantitative comparisons. The authors’ code will be publicly available at https://github.com/yiqiao666/low‐light‐enhancement‐for‐IVIF/tree/master .

The goal of infrared and visible image fusion is to generate an integrated image which can simultaneously preserve more visible detail information and prominent information from the input images. In order to achieve this goal, a novel infrared and visible fusion method based on visual saliency sparse representation and detail injection model (DIM) is presented. The proposed fusion method contains four steps. The first step is to decompose the source images into base layers and detail layers through the proposed multiscale decomposition method, which has the advantages of scale awareness and high-edge-preservation efficiency. Then, we design a novel subfusion rule called visual saliency sparse representation to get the fused base layer. Third, a DIM is developed to fuse the detail layers. In this model, we first employ a "max-absolute" scheme to obtain prefused images, which are then used as the model items for participating in the fusion process of the detail layers. Finally, the target-merged image is achieved through an inverse multiscale decomposition method. The experimental results demonstrate that the proposed method can not only preserve the details and significant information of source images but also enhance the brightness of the fused image, when compared to other state-of-the-art fusion methods.

Feature transfer method for infrared and visible image fusion via fuzzy lifting scheme

Multi-scale convolutional neural networks and saliency weight maps for infrared and visible image fusion

Aiming at the problem that the current infrared and visible image fusion based on deep learning has no labels, this paper proposes an infrared and visible image fusion algorithm based on unsupervised learning. This method utilizes the characteristics of unsupervised learning, and introduces infrared image information with high gray value into the visible image to obtain the fusion image. The deep learning network proposed in this paper is composed of 6 layers of convolution blocks, and a dual attention module is also designed to make the fusion image pay more attention to the high gray value area in the infrared image. By introducing skip connections, the shallow features are fused with the deep features, so that the details of the entire fused image are richer and the appearance of halos is reduced. A large number of experimental results show that the fusion method proposed in this paper can accurately highlight the target object while maintaining the visible texture details, enhance the visual effect of the human eye, and improve the target recognition. At the same time, the quantitative experimental results show that the fusion algorithm proposed in this paper has obvious advantages in multiple indicators.

The purpose of the infrared and visible image fusion is to generate a fused image with rich information. Although most fusion methods can achieve good performance, there are still shortcomings in extracting feature information from source images, which make it difficult to balance the thermal radiation region information and texture detail information in the fused image. To address the above issues, an expectation maximization (EM) learning framework based on adversarial generative networks (GAN) for infrared and visible image fusion is proposed. The EM algorithm (EMA) can obtain maximum likelihood estimation for problems with potential variables, which is helpful in solving the problem of lack of labels in infrared and visible image fusion. The axial‐corner attention mechanism is designed to capture long‐range semantic information and texture information of the visible image. The multifrequency attention mechanism digs the relationships between features at different scales to highlight target information of infrared images in the fused result. Meanwhile, two discriminators are used to balance two different features, and a new loss function is designed to maximize the likelihood estimate of the data with soft class label assignments, which is obtained from the expectation network. Extensive experiments demonstrate the superiority of EMA‐GAN over the state‐of‐the‐art.

Infrared and visible image fusion using dual discriminators generative adversarial networks with Wasserstein distance