Fusion2Void: Unsupervised Multi-Focus Image Fusion Based on Image Inpainting

Traditional and deep learning-based fusion methods generated the intermediate decision map to obtain the fusion image through a series of post-processing procedures. However, the fusion results generated by these methods are easy to lose some source image details or results in artifacts. Inspired by the image reconstruction techniques based on deep learning, we propose a multi-focus image fusion network framework without any post-processing to solve these problems in the end-to-end and supervised learning way. To sufficiently train the fusion model, we have generated a large-scale multi-focus image dataset with ground-truth fusion images. What's more, to obtain a more informative fusion image, we further designed a novel fusion strategy based on unity fusion attention, which is composed of a channel attention module and a spatial attention module. Specifically, the proposed fusion approach mainly comprises three key components: feature extraction, feature fusion and image reconstruction. We firstly utilize seven convolutional blocks to extract the image features from source images. Then, the extracted convolutional features are fused by the proposed fusion strategy in the feature fusion layer. Finally, the fused image features are reconstructed by four convolutional blocks. Experimental results demonstrate that the proposed approach for multi-focus image fusion achieves remarkable fusion performance compared to 19 state-of-the-art fusion methods.

To overcome the artifact phenomenon caused by the incomplete registration of the source images, a new multi-focus image fusion approach is proposed based on sparse representation and non-subsampled shearlet transform (NSST). Firstly, the source images are decomposed to low- and high-frequency coefficients by NSST. The sparse representation is then adopted to fuse the low-frequency coefficients. For the high-frequency coefficients, a maximum sum-modified-Laplacian (SML) rule is put forward to merge them. Finally, the resultant image is obtained by the inverse NSST on the fused coefficients. Experimental results indicate that the proposed method can achieve satisfied effect compared with various existing image fusion methods.

All existing image inpainting methods based on neural network models are affected by structural distortions and blurred textures on visible connectivity, such that overfitting and overlearning phenomena can easily emerge in the image inpainting processing procedure. Accordingly, in an attempt to address the defects of image inpainting algorithm, such as long iteration time, poor adaptability and unsatisfactory repairing effects, the image inpainting algorithm of improved Generative Adversarial Networks based on deep learning method of Two-Discriminations Network has been proposed in the paper. The proposed method uses image inpainting network, global discrimination network and local discrimination network to create a fusion network to apply computational images. In the training procedure of proposed algorithm, the network of image inpainting algorithm uses similar patching method to fill the broken area in image and set it as input training objects, which greatly improves the speed and quality of image inpainting. The global discrimination network uses global structure with marginal information and feature information to judge the completed image, meaning that it comprehensively achieves visible connectivity. As local discrimination network can judge the computational images, it has also been trained with assisted feature patches found on multiple images. Furthermore, the proposed method can enhance the discriminant capability and solve the problem that the image inpainting network has easily been overfitting when the features are too concentrated and limited in number to process. Our results of designed experiments demonstrate that proposed algorithm has better adaptive capability on several image categories than those state-of-the-arts.

To realize the multi-focus image fusion task, an end-to-end deep convolutional neural network (DCNN) model that produces the final fused image directly from the source images is presented in this paper. In order to promote the fusion precision, the innovative multi-focus fusion DCNN introduces a multi-scale feature extraction (MFE) unit to collect more complementary features from different spatial scales and fuse them to excavate more spatial information. Moreover, a visual attention unit is designed to help the network locate the focused region more accurately and pick more useful features for perfectly splicing the details in the fusion process. Experimental results illustrate that the proposed method is superior to several existing multi-focus image fusion methods in both of the subjective visual effects and objective quality metrics.

This paper develops a Coupled Convolution Model (CCM) for Error Concealment (EC), Image Super Resolution (ISR) and Multifocus Image Fusion (MIF) used in unmanned aerial vehicles. The key points of this paper are: i) for EC, to efficiently exploit the inherent correlation relationship between the lost region and its known adjacent region, this paper introduces a special CCM in order that their convolution model results via different convolution filters are close to the counterpart on the convolution space; ii) for ISR, we develop CCM to train two convolution filters and map the original low-resolution(LR) and high-resolution(HR) image patches onto a unified convolution subspace. iii) as for MIF, since image fusion depends on the local information of source images resulting from the same scene, this paper constructs a special Coupled Two-Dimensional Convolution Model (2D-CCM) for the image patches from different source images. Experimental results show that the proposed CCM is very effective for EC, ISR and MIF.

Multi-scale convolutional neural network for multi-focus image fusion

This paper proposes a new multi-focus image fusion method named AMGW, and it is based on algebraic multi-grid (AMG) algorithm and watershed segmentation method. In the implementation, the coarse grids of the source images are first extracted with the affinity matrix, and with a spatial interpolation function the approximation of the source image can be reconstructed from the coarse grids. A considerable amount of edge and textural information is still preserved in such approximation. The two source images are compared with their corresponding approximation block by block respectively by employing the mean square error (MSE) as a sharpness criterion. The MSE values are then used to identify the blocks of higher fidelity from the source images. The watershed segmentation is applied to those uncertain blocks in one source image. The two source images are compared again with the MSEs of the segmented regions. The fused image is obtained by reserving the blocks and regions with higher MSEs and applying a post-processing operation. Experimental results demonstrate that the AMG-based method outperforms the state-of-the-art fusion approaches in terms of selected objective image quality assessments. The details of the source images are well preserved in the fused image.

Combined with the shared similarity among multiple source images and depth of field in a camera, a new image fusion algorithm based on nonsubsampled shearlet transform (NSST) domain is proposed. First, NSST is utilized for decomposition of the source images, the low frequency coefficients is fused by weight votes in the structure-driven regions by using shared similarity and depth of field (SSSID), and then apply larger sum-modified-Laplacian (SML) with depth of field to the high frequency coefficients, finally the fusion image is gained after we do the inverse NSST to the fused coefficients. The algorithm can not only preserve the information of the source images well, but also suppress pixel distortion due to nonlinear operations in transform domain. Experimental results demonstrate that the proposed method outperforms the state-of-the-art transform domain fusion methods on image quality and objective fusion criteria.

In recent years, sparse representation approaches have been integrated into multi-focus image fusion methods. The fused images of sparse-representation-based image fusion methods show great performance. Constructing an informative dictionary is a key step for sparsity-based image fusion method. In order to ensure sufficient number of useful bases for sparse representation in the process of informative dictionary construction, image patches from all source images are classified into different groups based on geometric similarities. The key information of each image-patch group is extracted by principle component analysis (PCA) to build dictionary. According to the constructed dictionary, image patches are converted to sparse coefficients by simultaneous orthogonal matching pursuit (SOMP) algorithm for representing the source multi-focus images. At last the sparse coefficients are fused by Max-L1 fusion rule and inverted to fused image. Due to the limitation of microscope, the fluorescence image cannot be fully focused. The proposed multi-focus image fusion solution is applied to fluorescence imaging area for generating all-in-focus images. The comparison experimentation results confirm the feasibility and effectiveness of the proposed multi-focus image fusion solution.

Multi-focus image fusion is a process of combining a set of images that have been captured from the same scene but with different focuses in order to construct an additional sharper image. This process plays an important role in the image processing and machine vision fields. Various algorithms have been developed for this task. The key challenge in the design of multi-focus image fusion algorithms is how to evaluate the local content information of each image from the source images. Simple, but effective, block-based techniques at pixel level are widely used for multi-focus image fusion. However, a fixed block size may not be applicable to every application. A block size that is too small or too large is also not desirable. Hence, optimization of the block size is necessary in order to obtain a fused image that comprises the sharper parts of the source images. Recently, a number of techniques based on evolutionary computation have been applied to block-based multi-focus image fusion. The artificial bee colony (ABC) algorithm is one of the more popular evolutionary computational approaches used to find an optimal solution. In this paper, an efficient and robust block-based multi-focus image fusion method based on the optimal selection of sharper image blocks from source images using best-so-far ABC strategies is proposed. Experiment results show that the proposed method is able to provide good results and outperforms other conventional methods, both visually and quantitatively.

As a powerful assistance technique for computer vision, multi-focus images fusion has attracted extensive attention in recent years. However, how to accurately detect the focused region from the focused source image is a key problem. To address the weak performance of current fusion methods, in this paper, we proposed a new multi-focus images fusion method. Firstly, the encoder based on VGG-16 aims to extract multi-scale features of the source images. Then, the cross features attention fusion module (CFAFM) integrates multi-scale features for subsequent decoding. Further, the refinement module (RM) aims to refine the predicted coarse decision map. Finally, an all-in focus image is obtained by using the decision map and source images according to the designed fusion rules. Extensive experiments on ‘Lytro’ dataset demonstrate that the proposed method significantly outperforms 5 typical image fusion models and achieves the new state-of-the-art.

The multi-focus image fusion technique is to extract the focus regions from source images and compose them together to form a clear image in the full field of view. In order to further improve the accuracy of focus region detection and ensure its efficiency, a novel multi-focus image fusion method in spatial domain, based on guided filter and mixed focus measure, is proposed in this paper. Firstly, a guided filter is employed as an edge-preserving smoothing operator to process the source images, and the difference operator is used between the filtered images and the source images to extract salient feature. Subsequently, the salient feature maps are measured by the mixed focus measure, combining the sum of energy of edge (SEOE) and the sum of local variance (SLV), to detect the focus regions, and the initial decision map is obtained. For holes of different sizes in the initial decision map, the closing operation and the small area removal strategy are used to fill and connect the truncated regions, and then the opening operation and the guide filter are used to optimize the decision map boundary to obtain the final decision map. Finally, the multi-focus fusion image is obtained by the pixel-wise weighted-averaging rule according to the final decision map. Simulation results demonstrate that the method is superior to some existing fusion methods on both subjective visual perception and objective evaluation metrics.

Pixel-level image fusion, which is widely used in remote sensing, medical imaging, surveillance and etc., directly combines the original information in the source images. As a pixel-level method, multi-focus image fusion is designed to combine the partially focused images into one fully fused single image, which is expected to be more informative for human or machine perception. To achieve this purpose, an algorithm using spatial frequency (SF) measure and discrete wavelet transform (DWT) for multi-focus image fusion is proposed. In this work, the source images are decomposed into low frequency components and high frequency components by using DWT. Then the spatial frequency of the low frequency components is calculated. The spatial frequency is used to judge the focused regions, followed by the morphological filter and median filter. The fused low frequency can be obtained. And the high frequency components are fused using traditional method. Finally, the fused image is obtained by doing inverse discrete wavelet transform. To do the comparison, the proposed algorithm is compared with several existing fusion algorithms in qualitative and quantitative ways. Experimental results demonstrate that our method can be competitive or even outperforms the methods in comparison.

DCKN: Multi-focus image fusion via dynamic convolutional kernel network

Nowadays, image inpainting methods based on deep learning would lead to information loss when acquiring deep features, which is not conducive to the restoration of texture details and ignores the inpainting of semantic features. Besides, great majority of them generate inpainting results with unreasonable structures. In response to the above problems, an image inpainting network based on a multi-scale feature joint attention model is proposed. First of all, an image inpainting network using multi-scale feature joint attention model is advanced. When acquiring image depth features, multi-scale feature fusion is used to reduce the loss of information in the convolution process. Afterwards, a joint attention mechanism not only strengthens the model’s ability to repair image semantics, but also ensures the model could generate inpainting results with distinct texture. Last but not least, the style loss and perceptual loss are introduced into the network for the purpose of ensuring the consistency of the detail and style of the inpainting results. The qualitative experimental results on the CelebA-HQ and Places2 datasets and commonly used evaluation indicators such as PSNR and SSIM verify the method is superior to the existing image inpainting methods. Compared with the comparison methods, the proposed method improves PSNR and SSIM by 0.4%−6% and 0.4%−3% respectively.