Similar Patch Selection in Embedding Space for Multi-View Image Denoising

Blind image deconvolution, i.e., estimating both the latent image and the blur kernel from the only observed blurry image, is a severely ill-posed inverse problem. In this paper, we propose a blur kernel estimation method for blind motion deblurring using sparse representation and cross-scale self-similarity of image patches as priors to recover the latent sharp image from a single blurry image. Sparse representation indicates that image patches can always be represented well as a sparse linear combination of atoms in an appropriate dictionary. Cross-scale self-similarity results in that any image patch can in some way be well approximated by a number of other similar patches across different image scales. Our method is based on the observations that almost any image patch in a natural image has multiple similar patches in down-sampled versions of the image, and down-sampling produces image patches that are sharper than those in the blurry image itself. In our method, the dictionary for sparse representation is trained adaptively from sharper patches sampled from the down-sampled latent image estimate to make the similar patches of the latent sharp image well represented sparsely, and meanwhile, all patches from the latent image estimate are optimized to be as close to the sharper similar patches searched from the down-sampled version to enforce the sharp recovery of the latent image by constructing a non-local regularization. Experimental results on both simulated and real blurry images demonstrate that our method outperforms state-of-the-art blind deblurring methods.

With the increased availability of multi-view satellite images, the number of investigations on 3D urban scene reconstruction from multiple satellite images is also increasing. Conventional Multi-View Stereo (MVS) pipelines require the calibrated pose information of the satellite cameras to determine the epipolar geometry and the 3D structure of the stereo correspondences. In this study, we propose a novel Monocular Height estimation and Fusion (MHF) method for 3D reconstruction from uncalibrated multi-view satellite images. By employing a learned monocular depth network, the proposed method first obtains the height map of each satellite image. Second, all height maps obtained from the multi-view images are fused to a refined height map in each image plane. To fuse the height maps, all maps are affine transformed to a virtual reference coordinate system and the transformed maps are then projected to the image plane of each camera coordinate system. The monocular depth network was trained and evaluated on the Data Fusion Contest 2019 (DFC19) dataset including Jacksonville, FL, and Omaha, NE. We also evaluate the ATL-SN4 dataset covering Atlanta, GA to test on untrained new urban scenes.

Multi-view satellite images can form multiple stereo image pairs (MSIPs). Each pair allows generating two types of surface information: surface disparity maps (SDMs) and digital elevation models (DEMs). Both types are important for 3D assisted information extraction applications. Since multi-view images can be used to generate more accurate stereo-based DEMs than using one stereo-pair, it is more computationally cost-effective to generate enhanced SDMs from MSIPs. However, the disparity values from different stereo-pairs are unequal. Hence, redundant data cannot be generated to enhance the SDMs. Therefore, our research focuses on the generation of inter-proportional disparity maps from multi-view very high resolution satellite images. The method uses the sensor model information of the stereo images to construct multiple epipolar images in the object-space. After implementation, the disparity values of the constructed images were found to be proportional. This property allows the generation of SDMs that may replace the DEMs to some extent.

The key of image denoising algorithms is to preserve the details of the original image while denoising the noise in the image. The existing algorithms use the external information to better preserve the details of the image, but the use of external information needs the support of similar images or image patches. In this paper, an edge-aware image denoising algorithm is proposed to achieve the goal of preserving the details of original image while denoising and using only the characteristics of the noisy image. In general, image denoising algorithms use the noise prior to set parameters todenoise the noisy image. In this paper, it is found that the details of original image can be better preserved by combining the prior information of noise and the image edge features to set denoising parameters. The experimental results show that the proposed edge-aware image denoising algorithm can effectively improve the performance of block-matching and 3D filtering and patch group prior-based denoising algorithms and obtain higher peak signal-to-noise ratio and structural similarity values.

Non-local patch-based methods were until recently the state of the art for image denoising but are now outperformed by CNNs. In video denoising, however, they are still competitive with CNNs, as they can effectively exploit the video temporal redundancy, which is a key factor to attain high denoising performance. The problem is that CNN architectures are not compatible with the search for self-similarities. In this work, we propose a simple, yet efficient way to feed video self-similarities to a CNN. The non-locality is incorporated into the network via a first non-trainable layer which finds for each patch in the input image its most similar patches in a search region. The central values of these patches are then gathered in a feature vector which is assigned to each image pixel. This information is presented to a CNN which is trained to predict the clean image. We apply the proposed method to image and video denoising. In the case of video, the patches are searched for in a 3D spatiotemporal volume. The proposed method achieves state-of-the-art results.

The widely used wavelet-thresholding techniques (DWT-H and DWT-S) have a near-optimal behavior that cannot be enhanced by any local denoising filter, but they cannot utilize the similarity of small-size image patches to enhance the denoising performance. Two of the latest improvements (WNLM and NLMW) introduced the Euclidean distance to measure the similarity of image patches, and then used the non-local meaning of similar patches for further denoising. Since the Euclidean distance is not a good similarity measurement, these two improvements are limited. In this study, we introduced the earth mover’s distance (EMD) as the similarity measure of small-scale patches within the wavelet sub-bands of noisy images. Moreover, at higher noise levels, we further incorporated joint bilateral filtering, which can filter both the spatial domain and the intensity domain of images. Denoising simulation experiments on BSDS500 demonstrated that our algorithm outperformed the DWT-H, DWT-S, WNLM, and NLMW algorithms by 4.197 dB, 3.326 dB, 2.097 dB, and 1.162 dB in terms of the average PSNR, and by 0.230, 0.213, 0.132, and 0.085 in terms of the average SSIM.

We introduce an regularized reconstruction scheme to recover dynamic imaging datasets with significant inter frame motion from undersampled Fourier data. The proposed non-local regularization penalty is an unweighted sum of distances between image patch pairs in the 3-D dataset. We use robust distance metrics to compute the distance between image patches; these metrics encourage the smoothing between similar patches, while discouraging the averaging of dissimilar patches. Hence, this algorithm is capable of exploiting the similarities between patch pairs in adjacent frames even when they are well separated due to motion, eventhough it does not perform explicit motion estimation. Unlike current non-local regularization schemes, the proposed penalty does not need good initial guesses to estimate the weights. Hence, this approach is readily applicable to accelerated dynamic imaging problems, where good initial guesses are challenging to obtain. The validation of the proposed scheme on numerical phantoms and dynamic MRI datasets demonstrate the superior performance of the proposed scheme over current dynamic imaging schemes.

Recent compressed sensing (CS) approaches utilize the similarity and redundancy of image patches of the magnetic resonance (MR) image to enable reconstruction from highly undersampled k-space measurements. In this paper, the patches similarity and redundancy is exploited by assuming that the image patches sample a low-dimensional manifold embedded in a high dimensional space. The MR image is then reconstructed by keeping the dimension of the patch manifold as small as possible. This is achieved by using the dimension of the patch manifold of a MR image as a regularizer in the objective function. Results show that the proposed method is superior to state-of-the-art patch-based MRI reconstruction methods. The algorithm is evaluated on two datasets containing 100 MR images each. The reconstruction quality of the algorithm, gauged using the three quality metrics: peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) and normalized root mean square error (NRMSE), is better than the comparison methods.

Deep reinforcement learning-based patch selection for illuminant estimation

Multiple-view change caused by small unmanned aerial vehicles (UAVs) monitoring the ground, resulting in image distortion, multiview transformation, and low overlap. Thus, such change has a strong effect on the accuracy of image registration. In this study, we utilize a Siamese network to deal with the complexity registration of low-altitude remote-sensing images. A robust neighbor-guided patch representation is designed to describe feature points based on neighborhood relation reconstruction, and patch selection. The network is trained based on rotation-invariant layer to solve the inevitable rotation, and nonrigid deformation caused by multiview images in low-altitude remote-sensing images. With only three training images involving 4500 putative matches, the experiment results demonstrated that the learned network can process the scenarios of yaw rotation, pitch rotation, mixture, and extreme (e.g., mixture, scaling, and distortion occur simultaneously) of UAV better than other six state-of-the-art methods.

In computer vision, the bag-of-words (BoW) model has been widely applied to image related tasks, such as large scale image retrieval, image classification, and object categorization. The sparse coding (SC) method which leverages SC as a means of feature coding can guarantee both sparsity of coding vector and lower reconstruction error in the BoW model. Thus it can achieve better performance than the traditional vector quantization method. However, it suffers from the side effect introduced by the non-smooth sparsity regularizer that quite different words may be selected for similar patches to favor sparsity, resulting in the loss of correlation between the corresponding coding vectors. To address this problem, in this paper, we propose a novel soft assignment method based on index combination of top-2 large sparse codes of local descriptors to make the SC-based BoW tolerate the case of different word selection for similar patches. To further ensure similar patches select same words to generate similar coding vectors, we propose a collaborative dictionary learning method through imposing the sparse code similarity regularization factor along with the row sparsity regularization across data instances on top of group sparse coding. Experiments on the well-known public Oxford dataset demonstrate the effectiveness of our proposed methods.

Image denoising is a fundamental problem in computer vision and image processing that holds considerable practical importance for real-world applications. The traditional patch-based and sparse coding-driven image denoising methods convert 2D image patches into 1D vectors for further processing. Thus, these methods inevitably break down the inherent 2D geometric structure of natural images. To overcome this limitation pertaining to the previous image denoising methods, we propose a 2D image denoising model, namely, the dictionary pair learning (DPL) model, and we design a corresponding algorithm called the DPL on the Grassmann-manifold (DPLG) algorithm. The DPLG algorithm first learns an initial dictionary pair (i.e., the left and right dictionaries) by employing a subspace partition technique on the Grassmann manifold, wherein the refined dictionary pair is obtained through a sub-dictionary pair merging. The DPLG obtains a sparse representation by encoding each image patch only with the selected sub-dictionary pair. The non-zero elements of the sparse representation are further smoothed by the graph Laplacian operator to remove the noise. Consequently, the DPLG algorithm not only preserves the inherent 2D geometric structure of natural images but also performs manifold smoothing in the 2D sparse coding space. We demonstrate that the DPLG algorithm also improves the structural SIMilarity values of the perceptual visual quality for denoised images using the experimental evaluations on the benchmark images and Berkeley segmentation data sets. Moreover, the DPLG also produces the competitive peak signal-to-noise ratio values from popular image denoising algorithms.

The paper presents a new Convolutional Neural Network (CNN) architecture, called stacked stereo CNN, for computing disparity map from stereo images. In stacked stereo CNN, left and right image patches are stacked back-to-back and fed to a single tower CNN. This is in contrast to Siamese network where two towers are used, one for the left patch and other for the right patch. The proposed network is trained on a large set of similar and dissimilar image patches, which are generated from stereo images and their ground truth images from Middlebury stereo datasets. The network returns a dissimilarity score for a pair of image patch which is used to compute the cost volume. The cost volume is further refined using post processing steps before generating the final disparity map. The proposed network is evaluated on Middlebury datasets and achieves comparable results to the state-of-art algorithms.

The goal of pan-sharpening tasks is to fuse panchromatic (PAN) images and low-spatial-resolution (LR) multispectral (MS) images for the purpose of aggregating texture and spectral information. Although traditional embedding-based pan-sharpening methods achieve competitive results, they are limited by the shallow network and not suitable for large-scale datasets. In this study, we design a novel multiscale locally linear embedding residual network (LLERN) that consists of two phases: the spectral preservation phase and the structural preservation phase. As the pretreatment of the structural preservation network, the spectral preservation network aims to upscale the LR MS image while retaining spectral information. The proposed locally linear embedding residual block (LLERB) in the structural preservation phase can search for similar sparse patches from the PAN image space and embed the corresponding local geometric relationship into the residual space to enhance the MS image. Extensive experiments suggest that the proposed LLERN outperforms state-of-the-art methods from visual and quantitative perspectives, and confirm the assumption that LR image patches and residual image patches in a local region share a similar manifold structure, which can be used to guide deep-learning modeling with improved interpretability. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/jiaming-wang/LLERN</uri> .

Shape modeling is a very important issue for many studies, for example, object recognition for robot vision, virtual environment construction, and so on. In this paper, a new method for obtaining polyhedral model from multiview images using genetic algorithms (GAs) is proposed. In this method, a similarity between model and every input image is calculated, and then the model which has the maximum similarity is found. For finding the model of maximum similarity, genetic algorithms are used as the optimization method. In the genetic algorithm, the sharing scheme is employed for efficient detection of multiple solution, because some shape may be represented by multiple shape models. Some results of modeling experiments from real multiple images demonstrate that the proposed method can robustly generate model by using the GA.