Hybrid Multi-level Regularizations with Sparse Representation for Single Depth Map Super-Resolution

Consumer depth sensors such as time-of-flight camera or Kinect have gained significant popularity in recently. However, the captured depth maps suffer from limited spatial resolution and a variety of noise, making such depth maps difficult to be directly applied in related applications. In this paper, we present a novel single depth map super-resolution method, aiming to reconstruct high-resolution depth map from its associated low-resolution depth map without any auxiliary information. Particularly, we exploit the depth local self-similarity to assist in constructing patch pairs in terms of high-resolution and low-resolution depth edge patches, and then deduce a high-resolution depth edge map via Markov model. Finally, we implement a joint bilateral filter to reconstruct the high-resolution depth map. Experimental results show that our method overcomes existing methods on the benchmark database as well as Kinect captured depth maps.

In this paper, we propose a simple and effective depth upsampling technique using self-guided residual interpolation. The original residual interpolation requires guidance information such as high-resolution RGB color image. However, self-guided residual interpolation requires only a single depth map. In the proposed algorithm, a tentative estimation of a high-resolution depth map is first generated from an input low-resolution depth map. Then, re-interpolation is applied to the residual domain, which is defined by differences between the input depth map and the tentative estimate. A precise high-resolution depth map is obtainable by interpolating in the residual domain. Experimental results demonstrate that our algorithm can outperform state-of-the-art depth map upsampling algorithms.

Most of the existing deep learning-based methods for Mark-less human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of human body joints via 2D convolutional neural networks (CNNs). But depth map is intrinsically 3D data, treat it as 2D images will distort the shape of the actual object through projection from 3D to 2D space, and compels the network to perform perspective distortion-invariant estimation. Moreover, directly regressing 3D coordinates from a 2D image is a highly nonlinear mapping, which causes difficulty in learning procedure. To overcome these problems, a module called Supervised Endecoder is proposed to process 3D convolution data, which can also be stacked through series connection to adapt different size of dataset. Based on the module, a network called Supervised High Dimension Endecoder Network is designed, which can be used to predict key points of markless human in a single depth map in 3D space. Experiments show improved prediction accuracy compared to the state-of-the-art approaches.

In this letter, a novel single depth map super‐resolution algorithm is proposed, which combines the non‐local prior and local smoothness prior. Unlike the color‐guided methods, the proposed method does not need a corresponding color image to aid the depth map super‐resolution. To explore the non‐local self‐similarity in the depth map, a shape‐adaptive adjusted non‐local regression is constructed using the shape‐adaptive similar patch groups. This prior can make full use of the non‐local information of the depth map and alleviate the effect of irrelevant pixels. To construct different local structures, the direction‐based local smoothness prior to model the different directional information is proposed, which can preserve the different structures in the depth map. Compared with the state‐of‐the‐art methods, experimental results indicate that the proposed approach can achieve better reconstruction performance.

Abstract. This article considers the problem of foreground detection on depth maps. The problem of finding objects of interest on images appears in many object detection, recognition and tracking applications as one of the first steps. However, this problem becomes too complicated for RGB images with multicolored or constantly changing background and in presence of occlusions. Depth maps provide valuable information about distance to the camera for each point of the scene, making it possible to explore object detection methods, based on depth features. We define foreground as a set of objects silhouettes, nearest to the camera relative to the local background. We propose a method of foreground detection on depth maps based on medial representation of objects silhouettes which does not require any machine learning procedures and is able to detect foreground in near real-time in complex scenes with occlusions, using a single depth map. Proposed method is implemented to depth maps, obtained from Kinect sensor.

With the enormous advances of the acquisition technology over the last years, fast processing and high-quality visualization of large point clouds have gained increasing attention. Commonly, a mesh surface is reconstructed from the point cloud and a high-resolution texture is generated over the mesh from the images taken at the site to represent surface materials. However, this global reconstruction and texturing approach becomes impractical with increasing data sizes. Recently, due to its potential for scalability and extensibility, a method for texturing a set of depth maps in a preprocessing and stitching them at runtime has been proposed to represent large scenes. However, the rendering performance of this method is strongly dependent on the number of depth maps and their resolution. Moreover, for the proposed scene representation, every single depth map has to be textured by the images, which in practice heavily increases processing costs. In this paper, we present a novel method to break these dependencies by introducing an efficient raytracing of multiple depth maps. In a preprocessing phase, we first generate high-resolution textured depth maps by rendering the input points from image cameras and then perform a graph-cut based optimization to assign a small subset of these points to the images. At runtime, we use the resulting point-to-image assignments (1) to identify for each view ray which depth map contains the closest ray-surface intersection and (2) to efficiently compute this intersection point. The resulting algorithm accelerates both the texturing and the rendering of the depth maps by an order of magnitude.

Time-of-flight (TOF) cameras are becoming increasingly popular for many mobile applications. To obtain accurate depth maps, TOF cameras must emit many pulses of light, which consumes a lot of power and lowers the battery life of mobile devices. However, lowering the number of emitted pulses results in noisy depth maps. To obtain accurate depth maps while reducing the overall number of emitted pulses, we propose an algorithm that adaptively varies the number of pulses to infrequently obtain high power depth maps and uses them to help estimate subsequent low power ones. To estimate these depth maps, our technique uses the previous frame by accounting for the 3D motion in the scene. We assume that the scene contains independently moving rigid objects and show that we can efficiently estimate the motions using just the data from a TOF camera. The resulting algorithm estimates 640 × 480 depth maps at 30 frames per second on an embedded processor. We evaluate our approach on data collected with a pulsed TOF camera and show that we can reduce the mean relative error of the low power depth maps by up to 64% and the number of emitted pulses by up to 81%.

Although RGBD cameras can provide depth information in real scenes, the captured depth map is often of low resolution and insufficient quality compared to the color image. Typically, most of the existing methods work by assuming that the edges in depth map and its corresponding color image are more likely to occur simultaneously. However, when the color image is rich in detail, the high-frequency information which is non-existent in the depth map will be introduced into the depth map. In this paper, we propose a CNN-based method to detect the co-occurrent structural edge for color-guided depth map super-resolution. Firstly, we design an edge detection convolutional neural network (CNN) to obtain the co-occurrent structural edge in depth map and its corresponding color image. Then we pack the obtained co-occurrent structural edges and the interpolated low-resolution depth maps into another customized CNN for depth map super-resolution. The presented scheme can effectively interpret and exploit the structural correlation between the depth map and the color image. Additionally, recursive learning is adopted to reduce the parameters of the customized CNN for depth map super-resolution and avoid overfitting. Experimental results demonstrate the effectiveness and reliability of our proposed approach by comparing with the state-of-the-art methods.

This paper presents a novel method for depth map enhancement. Given a low resolution and noisy depth map, we improve its spatial resolution and decrease noise with a registered high resolution color image. Different from the previous window-based local filter methods, this approach is based on geodesic distance weighted nonlocal filter, which does not require the optimal support window and more natural pixel similarity measure metric is given. By quantitative and qualitative experiments on publicly available test sequences, we demonstrate the effectiveness and efficiency of our proposed method on the depth enhancement task, compared with the existing methods.

Depth maps are widely used in 3D imaging techniques because of the appearance of the consumer depth cameras. However, the practical application of the depth map is limited by the poor image quality. In this paper, we propose a novel framework for the single depth map super-resolution via joint the local and non-local constraints simultaneously in the depth map. For the non-local constraint, we use the group-based sparse representation to explore the non-local self-similarity of the depth map. For the local constraint, we first estimate gradient images in different directions of the desired high-resolution (HR) depth map, and then build a multi-directional gradient guided regularizer using these estimated gradient images to describe depth gradients with different orientations. Finally, the two complementary regularizers are cast into a unified optimization framework to obtain the desired HR image. The experimental results show that the proposed method can achieve better depth super-resolution performance than state-of-the-art methods.

The emergence of Microsoft Kinect has attracted the attention not only from consumers but also from researchers in the field of computer vision. It facilitates the possibility to capture the depth map of the scene in real time and with low cost. Nonetheless, due to the limitations of structured light measurements used by Kinect, the captured depth map suffers random depth missing in the occlusion or smooth regions, which affects the accuracy of many Kinect based applications. In order to fill in the holes existing in Kinect depth map, some approaches that adopted color image guided in-painting or joint bilateral filter have been proposed to represent the missing depth pixel by available depth pixels. However, they are not able to obtain the optimal weights, thus the obtained missing depth values are not best. In this paper, we propose a color image guided locality regularized representation (CGLRR) to reconstruct the missing depth pixels by comprehensively determining the optimal weights of the available depth pixels from collocated patches in color image. Experimental results demonstrate that the proposed algorithm can better fill in the holes of depth map both in smooth and edge region than previous works.

Camera motion introduces spatially varying blur due to the depth changes in the 3D world. This work investigates scene configurations where such blur is produced under parallax camera motion. We present a simple, yet accurate, Image Compositing Blur (ICB) model for depth-dependent spatially varying blur. The (forward) model produces realistic motion blur from a single image, depth map, and camera trajectory. Furthermore, we utilize the ICB model, combined with a coordinate-based MLP, to learn a sharp neural representation from the blurred input. Experimental results are reported for synthetic and real examples. The results verify that the ICB forward model is computationally efficient and produces realistic blur, despite the lack of occlusion information. Additionally, our method for restoring a sharp representation proves to be a competitive approach for the deblurring task.

Depth information is being widely used in many real-world applications. However, due to the limitation of depth sensing technology, the captured depth map in practice usually has much lower resolution than that of color image counterpart. In this paper, we propose to joint exploit the internal smoothness prior and external gradient consistency constraint in graph domain for depth super-resolution. On one hand, a new graph Laplacian regularizer is proposed to the preserve the inherent piecewise smooth characteristic of depth, which has desirable filtering properties. On the other hand, inspired by an observation that the gradient of depth is zero except at edge separating regions, we introduce a graph gradient consistency constraint to enforce that the graph gradient of depth is close to the thresholded gradient of guidance. Finally, the internal and external regularizations are casted into a unified optimization framework, which can be efficiently addressed by ADMM. Experiments results demonstrate that our method outperforms the state-of-the-art with respect to both objective and subjective quality evaluations.

Depth-aware salient object detection using anisotropic center-surround difference

In this paper, we present a system for automatic object detection and pose estimation from a single depth map containing multiple objects for bin-picking applications. The proposed object detection algorithm is based on matching the keypoints extracted from the depth image by using the RANSAC algorithm with the spin image descriptor. In the proposed system, we combine the keypoint detection and the RANSAC algorithm to detect the objects, followed by the ICP algorithm to refine the 3D pose estimation. In addition, we implement the proposed algorithm on the GPGPU platform to speed-up the computation. Experimental results on simulated depth data are shown to demonstrate the proposed system.