Restoration of intensity and depth images constructed using sparse single-photon data

This paper presents a new algorithm for the joint restoration of depth and intensity (DI) images constructed using a gated SPAD-array imaging system. The three dimensional (3D) data consists of two spatial dimensions and one temporal dimension, and contains photon counts (i.e., histograms). The algorithm is based on two steps: (i) construction of a graph connecting patches of pixels with similar temporal responses, and (ii) estimation of the DI values for pixels belonging to homogeneous spatial classes. The first step is achieved by building a graph representation of the 3D data, while giving a special attention to the computational complexity of the algorithm. The second step is achieved using a Fisher scoring gradient descent algorithm while accounting for the data statistics and the Laplacian regularization term. Results on laboratory data show the benefit of the proposed strategy that improves the quality of the estimated DI images.

This paper presents a new algorithm for the non-local restoration of single-photon 3-Dimensional Lidar images acquired in the photon starved regime or with a reduced number of scanned spatial points (pixels). The algorithm alternates between two steps: evaluation of the spatial correlations between pixels using a graph, then restore the depth and reflectivity images by their spatial correlations. To reduce the computational cost associated with the graph, we adopt a non-uniform sampling approach, where bigger patches are assigned to homogeneous regions and smaller ones to heterogeneous regions. The restoration of 3D images is achieved by minimizing a cost function accounting for the data Poisson statistics and the non-local spatial correlations between patches. This minimization problem is efficiently solved using the alternating direction method of multipliers (ADMM) that presents fast convergence properties. Results on real Lidar data show the benefits of the proposed algorithm in improving the quality of the estimated depth images, especially in photon starved cases, which can contain a reduced number of photons.

Rapid advancement and active research in computer vision applications and 3D imaging have made a high demand for efficient depth image estimation techniques. The depth image acquisition, however, is typically challenged due to poor hardware performance and high computation cost. To tackle such limitations, this paper proposes an efficient approach for depth image reconstruction using low rank (LR) and total variation (TV) regularizations. The key idea is LR incorporates non-local depth information and TV takes into account the local spatial consistency. The proposed model reformulates the task of depth image estimate as a joint LR-TV regularized minimization problem, in which LR is used to approximate the low-dimensional structure of the depth image, and TV is employed to promote the depth sparsity in the gradient domain. Furthermore, this paper introduces an algorithm based on alternating direction method of multipliers (ADMM) for solving the minimization problem, whose solution provides an estimate of the depth map from incomplete pixels. Experimental results are conducted and the results show that the proposed approach is very effective at estimating high-quality depth images and is robust to different types of data missing models.

Registering a single intensity image to a 3D geometric model represented by a set of depth images is still a challenge. Since depth images represent only the shape of the objects, in turn, the intensity image is relative to viewpoint, texture and lighting condition. Thus, it is essential to firstly bring 2D and 3D representations to common features and then match them to find the correct view. In this paper, we used the concept of curvilinear saliency, related to curvature estimation, for extracting the shape information of both modalities. However, matching the features extracted from an intensity image to thousand(s) of depth images rendered from a 3D model is an exhausting process. Consequently, we propose to cluster the depth images into groups based on Clustering Rule-based Algorithm (CRA). In order to reduce the matching space between the intensity and depth images, a 2D/3D registration framework based on multi-class Support Vector Machine (SVM) is then used. SVM predicts the closest class (i.e., a set of depth images) to the input image. Finally, the closest view is refined and verified by using RANSAC. The effectiveness of the proposed registration approach has been evaluated by using the public PASCAL3D+ dataset. The obtaining results show that the proposed algorithm provides a high precision with an average of 88%.

The expectation-maximization (EM) algorithm for maximum-likelihood image recovery is guaranteed to converge, but it converges slowly. Its ordered-subset version (OS-EM) is used widely in tomographic image reconstruction because of its order-of-magnitude acceleration compared with the EM algorithm, but it does not guarantee convergence. Recently the ordered-subset, separable-paraboloidal-surrogate (OS-SPS) algorithm with relaxation has been shown to converge to the optimal point while providing fast convergence. We adapt the relaxed OS-SPS algorithm to the problem of image restoration. Because data acquisition in image restoration is different from that in tomography, we employ a different strategy for choosing subsets, using pixel locations rather than projection angles. Simulation results show that the relaxed OS-SPS algorithm can provide an order-of-magnitude acceleration over the EM algorithm for image restoration. This new algorithm now provides the speed and guaranteed convergence necessary for efficient image restoration.

This paper presents a new algorithm for the learning of spatial correlation and non-local restoration of single-photon 3-Dimensional Lidar images acquired in the photon starved regime (fewer or less than one photon per pixel) or with a reduced number of scanned spatial points (pixels). The algorithm alternates between three steps: (i) extract multi-scale information, (ii) build a robust graph of non-local spatial correlations between pixels, and (iii) the restoration of depth and reflectivity images. A non-uniform sampling approach, which assigns larger patches to homogeneous regions and smaller ones to heterogeneous regions, is adopted to reduce the computational cost associated with the graph. The restoration of the 3D images is achieved by minimizing a cost function accounting for the multi-scale information and the non-local spatial correlation between patches. This minimization problem is efficiently solved using the alternating direction method of multipliers (ADMM) that presents fast convergence properties. Various results based on simulated and real Lidar data show the benefits of the proposed algorithm that improves the quality of the estimated depth and reflectivity images, especially in the photon-starved regime or when containing a reduced number of spatial points.

This paper deals with the problem of depth recovery and image restoration from sparse and noisy image data. The image is modeled as a Markov random field and a new energy function is developed to effectively detect discontinuities in highly sparse and noisy images. The model provides an alternative to the use of a line process. Interpolation over missing data sites is first done using local characteristics to obtain initial estimates and then simulated annealing is used to compute the maximum a posteriori (MAP) estimate. A threshold on energy reduction per iteration is used to speed up simulated annealing by avoiding computation that contributes little to the energy minimization. Moreover, a minor modification of the posterior energy function gives improved results for random as well as structured sparsing problems. Results of simulations carried out on real range and intensity images along with details of the simulations are presented.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Accurate depth reconstruction is vital for numerous applications including autonomous vehicles, virtual reality, and robot perception. However, the depth imaging is challenging because of limited hardware operations, resource-constrained limitations, and incomplete data measurements. To address such shortcomings, this paper introduces an imaging model for efficient depth image estimation from incomplete depth pixels using non-local low-rank (NLLR) and total variation (TV) representations. The motivation is that NLLR is used to model global similar structure among depth patches, and the TV is incorporated to capture the correlations among local depth pixels. We reformulate the problem of depth reconstruction as a regularized least squares minimization problem with the non-local LR and TV regularizers. Furthermore, this paper proposes an iterative algorithm using the alternating direction method of multipliers (ADMM) to solve the optimization model, yielding an estimate of the depth map from far reduced data points. Experimental results on benchmark datasets validate the efficiency of the proposed approach.

This paper introduces an expectation-maximization (EM) algorithm for image restoration (deconvolution) based on a penalized likelihood formulated in the wavelet domain. Regularization is achieved by promoting a reconstruction with low-complexity, expressed in the wavelet coefficients, taking advantage of the well known sparsity of wavelet representations. Previous works have investigated wavelet-based restoration but, except for certain special cases, the resulting criteria are solved approximately or require demanding optimization methods. The EM algorithm herein proposed combines the efficient image representation offered by the discrete wavelet transform (DWT) with the diagonalization of the convolution operator obtained in the Fourier domain. Thus, it is a general-purpose approach to wavelet-based image restoration with computational complexity comparable to that of standard wavelet denoising schemes or of frequency domain deconvolution methods. The algorithm alternates between an E-step based on the fast Fourier transform (FFT) and a DWT-based M-step, resulting in an efficient iterative process requiring O(N log N) operations per iteration. The convergence behavior of the algorithm is investigated, and it is shown that under mild conditions the algorithm converges to a globally optimal restoration. Moreover, our new approach performs competitively with, in some cases better than, the best existing methods in benchmark tests.

Poisson noise is an important type of electronic noise that is present in a variety of photon-limited imaging systems. Different from the Gaussian noise, Poisson noise depends on the image intensity, which makes image restoration very challenging. Moreover, complex geometry of images desires a regularization that is capable of preserving piecewise smoothness. In this paper, we propose a Poisson denoising model based on the fractional-order total variation (FOTV). The existence and uniqueness of a solution to the model are established. To solve the problem efficiently, we propose three numerical algorithms based on the Chambolle-Pock primal-dual method, a forward-backward splitting scheme, and the alternating direction method of multipliers (ADMM), each with guaranteed convergence. Various experimental results are provided to demonstrate the effectiveness and efficiency of our proposed methods over the state-of-the-art in Poisson denoising.

Color image restoration is a fundamental task in computer vision and image processing, with extensive real‐world applications. In practice, color images often suffer from degradations caused by sensor noise, optical blur, compression artifacts, and data loss during the acquisition, transmission, or storage. Unlike grayscale images, color images exhibit high correlations among their RGB channels. Directly extending grayscale restoration methods to color images often leads to issues such as color distortion and structural artifacts. To address these challenges, this paper proposes a novel quaternion‐based color image restoration framework. The method integrates low‐rank pseudo‐norm constraints with saturation‐value total variation (SVTV) regularization, effectively enhancing restoration quality in tasks including denoising, deblurring, and inpainting of degraded color images. The proposed algorithm is efficiently solved using the alternating direction method of multipliers (ADMM), and restoration performance is rigorously evaluated through quantitative metrics including peak signal‐to‐noise ratio (PSNR), structural similarity index measure (SSIM), and S‐CIELAB error. Extensive experimental results demonstrate the superior performance of our method compared to existing approaches.

We propose a novel approach to localize a 3D object from the intensity and depth information images provided by a Time-of-Flight (ToF) sensor. Our method builds on two convolutional neural networks (CNNs). The first one uses raw depth and intensity images as input, to segment the floor pixels, from which the extrinsic parameters of the camera are estimated. The second CNN is in charge of segmenting the object-of-interest so as to align its point cloud with a reference model. As a main innovation, the object segmentation exploits the calibration estimated from the prediction of the first CNN to represent the geometric depth information in a coordinate system that is attached to the ground, and is thus independent of the camera elevation. In practice, both the height of pixels with respect to the ground, and the orientation of normals to the point cloud are provided as input to the second CNN.

Depth image super-resolution is a challenging problem, since normally high upscaling factors are required (e.g., 16×), and depth images are often noisy. In order to achieve large upscaling factors and resilience to noise, we propose a Robust Algorithm for Depth imAge super Resolution (RADAR) that combines the power of finite rate of innovation (FRI) theory with multimodal dictionary learning. Given a low-resolution (LR) depth image, we first model its rows and columns as piece-wise polynomials and propose an FRI-based depth upscaling (FDU) algorithm to super-resolve the image. Then, the upscaled moderate quality (MQ) depth image is further enhanced with the guidance of a registered high-resolution (HR) intensity image. This is achieved by learning multimodal mappings from the joint MQ depth and HR intensity pairs to the HR depth, through a recently proposed triple dictionary learning (TDL) algorithm. Moreover, to speed up the super-resolution process, we introduce a new projection-based rapid upscaling (PRU) technique that pre-calculates the projections from the joint MQ depth and HR intensity pairs to the HR depth. Compared with the state-of-the-art deep learning-based methods, our approach has two distinct advantages: we need a fraction of training data but can achieve the best performance, and we are resilient to mismatches between training and testing datasets. The extensive numerical results show that the proposed method outperforms other state-of-the-art methods on either noise-free or noisy datasets with large upscaling factors up to 16× and can handle unknown blurring kernels well.

Total variation regularization is well-known for recovering sharp edges; however, it usually produces staircase artifacts. In this paper, in order to overcome the shortcoming of total variation regularization, we propose a new variational model combining high-order total variation regularization and l1 regularization. The new model has separable structure which enables us to solve the involved subproblems more efficiently. We propose a fast alternating method by employing the fast iterative shrinkage-thresholding algorithm (FISTA) and the alternating direction method of multipliers (ADMM). Compared with some current state-of-the-art methods, numerical experiments show that our proposed model can significantly improve the quality of restored images and obtain higher SNR and SSIM values.

High-order TVL1-based images restoration and spatially adapted regularization parameter selection