Depth Image Reconstruction Using Low Rank and Total Variation Representations

Accurate depth reconstruction is vital for numerous applications including autonomous vehicles, virtual reality, and robot perception. However, the depth imaging is challenging because of limited hardware operations, resource-constrained limitations, and incomplete data measurements. To address such shortcomings, this paper introduces an imaging model for efficient depth image estimation from incomplete depth pixels using non-local low-rank (NLLR) and total variation (TV) representations. The motivation is that NLLR is used to model global similar structure among depth patches, and the TV is incorporated to capture the correlations among local depth pixels. We reformulate the problem of depth reconstruction as a regularized least squares minimization problem with the non-local LR and TV regularizers. Furthermore, this paper proposes an iterative algorithm using the alternating direction method of multipliers (ADMM) to solve the optimization model, yielding an estimate of the depth map from far reduced data points. Experimental results on benchmark datasets validate the efficiency of the proposed approach.

This paper presents a new algorithm for the joint restoration of depth and intensity images constructed from the time-correlated single-photon counting (TCSPC) measurement in the limit of very few photon counts [1]. Under some justified approximations, the restoration problem (regularized likelihood) reduces to a convex formulation with respect to the parameters of interest. The first advantage of this formulation is that it only processes the corrupted depth and intensity images obtained from preliminary estimation, without the need for the use of full TCSPC waveforms. The second advantage is its flexibility in being able to use different convex regularization terms such as: total variation (TV); and sparsity of the discrete cosine transform (DCT) coefficients. The estimation problems are efficiently solved using the alternating direction method of multipliers (ADMM) that presents good convergence properties and thus a reduced computational cost. Results on single photon depth data from field trials show the benefit of the proposed strategy that improves the quality of the estimated depth and intensity images.

A two-step low rank matrices approach for constrained MR image reconstruction

Attention reweighted sparse subspace clustering

This paper introduces a variational method for destriping data acquired by pushbroom-type satellite imaging systems. The model leverages sparsity in signals and is based on current research in sparse optimization and compressed sensing. It is based on the basic principles of regularization and data fidelity with certain constraints using modern methods in variational optimization, namely, total variation (TV), L 1 fidelity, and the alternating direction method of multipliers (ADMM). The proposed algorithm, TV– L 1 , uses sparsity-promoting energy functionals to achieve two important imaging effects. The TV term maintains boundary sharpness of the content in the underlying clean image, while the L 1 fidelity allows for the equitable removal of stripes without over- or under-penalization, providing a more accurate model of presumably independent sensors with an unspecified and unrestricted bias distribution. A comparison is made between the TV– L 2 model and the proposed TV– L 1 model to exemplify the qualitative efficacy of an L 1 striping penalty. The model makes use of novel minimization splittings and proximal mapping operators, successfully yielding more realistic destriped images in very few iterations.

Subspace clustering based on latent low rank representation with Frobenius norm minimization

Regularization plays a crucial role in reliably utilizing imaging systems for scientific and medical investigations. It helps to stabilize the process of computationally undoing any degradation caused by physical limitations of the imaging process. In the past decades, total variation regularization played a dominant role in the literature. Two forms of total variation regularizations, namely the first-order and the second-order total variation (TV-1 and TV-2) have been widely used. TV-1 has a disadvantage: it reconstructs images in the form of piece-wise constants when the noise and/or under-sampling is severe, while TV-2 reconstructs natural-looking images under such scenarios. On the other hand, TV-1 can recover sharp jumps better than TV-2. Two forms of generalizations, namely Hessian-Schatten norm (HSN) regularization, and total generalized variation (TGV) regularization, have been proposed and have become significant developments in the area of regularization for imaging inverse problems owing to their performance. While the strength of TGV is that it can combine the advantages of TV-1 and TV-2, HSN has better structure-preserving property. Here, we develop a novel regularization for image recovery that combines the strengths of TGV and HSN. We achieve this by restricting the maximization space in the dual form of HSN in the same way that TGV is obtained from TV-2. We call the new regularization the generalized Hessian-Schatten norm regularization (GHSN). We develop a novel computational method for image reconstruction using the new form of regularization based on the well-known framework called the alternating direction method of multipliers (ADMM). We demonstrate the strength of the GHSN using some reconstruction examples.

This paper investigates the application of unsupervised learning methods for computed tomography reconstruction. To motivate our work, we review several existing priors, namely the truncated Gaussian prior, the prior, the total variation prior, and the deep image prior (DIP). We find that DIP outperforms the other three priors in terms of representational capability and visual performance. However, the performance of DIP deteriorates when the number of iterations exceeds a certain threshold due to overfitting. To address this issue, we propose a novel method (MCDIP‐ADMM) based on multi‐code deep image prior (MCDIP) and plug‐and‐play alternative direction method of multipliers (ADMM). Specifically, MCDIP utilizes multiple latent codes to generate a series of feature maps at an intermediate layer within a generator model. These maps are then composed with trainable weights, representing the complete image prior. Experimental results demonstrate the superior performance of the proposed MCDIP‐ADMM compared to three existing competitors. In the case of parallel beam projection with Gaussian noise, MCDIP‐ADMM achieves an average improvement of 4.3 dB over DIP, 1.7 dB over ADMM DIP‐weighted total variation (WTV) and 1.2 dB over PnP‐DIP in terms of peak‐signal‐to‐noise ratio (PSNR). Similarly, for fan‐beam projection with Poisson noise, MCDIP‐ADMM achieves an average improvement of 3.09 dB over DIP, 1.86 dB over ADMM DIP‐WTV and 0.84 dB over PnP‐DIP in terms of PSNR.

In the last decades, unsupervised deep learning based methods have caught researchers' attention, since in many real applications, such as medical imaging, collecting a large amount of training examples is not always feasible. Moreover, the construction of a good training set is time consuming and hard because the selected data have to be enough representative for the task. In this paper, we focus on the Deep Image Prior (DIP) framework and we propose to combine it with a space-variant Total Variation regularizer with an automatic estimation of the local regularization parameters. Differently from other existing approaches, we solve the arising minimization problem via the flexible Alternating Direction Method of Multipliers (ADMM). Furthermore, we provide a specific implementation also for the standard isotropic Total Variation. The promising performances of the proposed approach, in terms of PSNR and SSIM values, are addressed through several experiments on simulated as well as real natural and medical corrupted images.

Hyperspectral and multispectral image fusion plays an important role in the fields such as image restoration, space robotics, remote sensing, computer vision, etc. The fused images take advantages of the high spectral resolutions of hyperspectral images and the high spatial resolutions of multispectral images. So there has been a recent outgrowth of interests of research in this field. However, the local structural image regularity and non-local image self-similarity are not yet explored in existing fusion methods. This paper develops a novel method for fusing hyperspectral and multispectral images with non-local structure tensor total variation regularization. Specifically, the weight function to measure self-similarity in non-local domain is introduced on the basis of the definition of structure tensor. Therefore, the proposed method can exploit the regularity and self-similarity of spatial-spectral image’s non-local structures simultaneously. In addition to the general formulation of non-local structure tensor variation, this paper presents the discrete non-local structure tensor variation for hyperspectral and multispectral image fusion whose measurements are discrete. This method can provide an alternative way to measure the image variation of multi-band image. The resulting problem is solved by alternating direction method of multipliers (ADMM), a simple but powerful algorithm for convex optimization problems. The optimization problem is handled by a splitting variables strategy, which is implemented by introducing four auxiliary variables regarding to the required fused image. The iteration related to the proposed non-local structure tensor regularization term is solved by the proximal mapping method. The solutions of other iterations are given through fast Fourier transforms. Extensive computational experiments are performed on various datasets. The results of index evaluation demonstrate the effectiveness of the proposed method. These results also indicate non-local structure tensor total variation regularization has obvious advantages over the state-of-the-art fusion methods.KeywordsImage fusionNon-local tensor regularizationOptimization method

We address the task of single depth image inpainting. Without the corresponding color images, previous or next frames, depth image inpainting is quite challenging. One natural solution is to regard the image as a matrix and adopt the low rank regularization just as color image inpainting. However, the low rank assumption does not make full use of the properties of depth images. A shallow observation inspires us to penalize the nonzero gradients by sparse gradient regularization. However, statistics show that though most pixels have zero gradients, there is still a non-ignorable part of pixels, whose gradients are small but nonzero. Based on this property of depth images, we propose a low gradient regularization method in which we reduce the penalty for small gradients while penalizing the nonzero gradients to allow for gradual depth changes. The proposed low gradient regularization is integrated with the low rank regularization into the low rank low gradient approach for depth image inpainting. We compare our proposed low gradient regularization with the sparse gradient regularization. The experimental results show the effectiveness of our proposed approach.

Recently, low-rank representation (LRR) methods have been widely applied for hyperspectral anomaly detection, due to their potentials in separating the backgrounds and anomalies. However, existing LRR models generally convert 3-D hyperspectral images (HSIs) into 2-D matrices, inevitably leading to the destruction of intrinsic 3-D structure properties in HSIs. To this end, we propose a novel tensor low-rank and sparse representation (TLRSR) method for hyperspectral anomaly detection. A 3-D TLR model is expanded to separate the LR background part represented by a tensorial background dictionary and corresponding coefficients. This representation characterizes the multiple subspace property of the complex LR background. Based on the weighted tensor nuclear norm and the LF,1 sparse norm, a dictionary is designed to make its atoms more relevant to the background. Moreover, a principal component analysis (PCA) method can be assigned as one preprocessing step to exact a subset of HSI bands, retaining enough the HSI object information and reducing computational time of the postprocessing tensorial operations. The proposed model is efficiently solved by the well-designed alternating direction method of multipliers (ADMMs). A comparison with the existing algorithms via experiments establishes the competitiveness of the proposed method with the state-of-the-art competitors in the hyperspectral anomaly detection task.

It is required that input features are represented as vectors or scalars in machine learning for classification, e.g. support vector machine (SVM). However, real world data such as 2D images is naturally represented as matrices or tensors with higher dimensions. Thus, structural information of the data whose dimensions are more than two is not successfully considered. One typical structural information which is useful for the classification task is the spatial relationship of the nearby data points. In this paper, to leverage this kind of structural information, we propose a novel classification method which combines total variational (TV) regularization with SVM, called TV-SVM. Since TV achieves a local smoothing property by penalizing the local discontinuity of data, TV-SVM preserves better local structure than the original SVM due to TV regularization. We solve the objective function of TV-SVM via the alternating direction method of multipliers (ADMM) algorithm. Experimental results on image classification show that TV-SVM is competitive to the state-of-the-art learning method in both classification accuracy and computational complexity.

Total variation (TV) regularization method is a typical method to preserve the discontinuities structure in EIT. Isotropic TV and anisotropic TV are two well-known variants of TV. The main differences between them are that the latter tends to distort the reconstructed internal inhomogeneities along the coordinate axis. In this article, we adopt the alternating direction method of multipliers (ADMM) to overcome the non-differentiability of the anisotropic TV and verify the characteristics of anisotropic TV regularization by the tank experiments.

Total variation (TV) based sparsity and non local self-similarity have been shown to be powerful tools for the reconstruction of magnetic resonance (MR) images. However, due to the uniform regularization of gradient sparsity, standard TV approaches often over-smooth edges in the image, resulting in the loss of important details. This paper presents a novel compressed sensing method for the reconstruction of MRI data, which uses a regularization strategy based on re-weighted TV to preserve image edges. This method also leverages the redundancy of non local image patches through the use of a sparse regression model. An efficient strategy based on the Alternating Direction Method of Multipliers (ADMM) algorithm is used to recover images with the proposed model. Experimental results on a simulated phantom and real brain MR data show our method to outperform state-of-the-art compressed sensing approaches, by better preserving edges and removing artifacts in the image.