Multi-Depth-Map Raytracing for Efficient Large-Scene Reconstruction.

In this paper, we propose a simple and effective depth upsampling technique using self-guided residual interpolation. The original residual interpolation requires guidance information such as high-resolution RGB color image. However, self-guided residual interpolation requires only a single depth map. In the proposed algorithm, a tentative estimation of a high-resolution depth map is first generated from an input low-resolution depth map. Then, re-interpolation is applied to the residual domain, which is defined by differences between the input depth map and the tentative estimate. A precise high-resolution depth map is obtainable by interpolating in the residual domain. Experimental results demonstrate that our algorithm can outperform state-of-the-art depth map upsampling algorithms.

This paper addresses the generation and visualization of noisy 3D point clouds. The goal is to extract 3D information from image sequences in real-time and to transform the resulting noisy point clouds into a representation with dense surfaces and an intuitive character. Therefore a bridge is built from image sequence analysis to computer graphics. A point cloud is calculated from a single camera by using state of the art algorithms for egomotion estimation and dense 3D reconstruction. Due to inaccuracies in the measurement process and hence of the resulting depth maps, a degree of uncertainty of the 3D data is defined. This additional information can be used to filter the point cloud in space and time. We present an approach to filter and polygonize weighted point clouds and visualize it by mapping image texture on the resulting surface.

In this letter, a novel single depth map super‐resolution algorithm is proposed, which combines the non‐local prior and local smoothness prior. Unlike the color‐guided methods, the proposed method does not need a corresponding color image to aid the depth map super‐resolution. To explore the non‐local self‐similarity in the depth map, a shape‐adaptive adjusted non‐local regression is constructed using the shape‐adaptive similar patch groups. This prior can make full use of the non‐local information of the depth map and alleviate the effect of irrelevant pixels. To construct different local structures, the direction‐based local smoothness prior to model the different directional information is proposed, which can preserve the different structures in the depth map. Compared with the state‐of‐the‐art methods, experimental results indicate that the proposed approach can achieve better reconstruction performance.

Abstract. This article considers the problem of foreground detection on depth maps. The problem of finding objects of interest on images appears in many object detection, recognition and tracking applications as one of the first steps. However, this problem becomes too complicated for RGB images with multicolored or constantly changing background and in presence of occlusions. Depth maps provide valuable information about distance to the camera for each point of the scene, making it possible to explore object detection methods, based on depth features. We define foreground as a set of objects silhouettes, nearest to the camera relative to the local background. We propose a method of foreground detection on depth maps based on medial representation of objects silhouettes which does not require any machine learning procedures and is able to detect foreground in near real-time in complex scenes with occlusions, using a single depth map. Proposed method is implemented to depth maps, obtained from Kinect sensor.

Most of the existing deep learning-based methods for Mark-less human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of human body joints via 2D convolutional neural networks (CNNs). But depth map is intrinsically 3D data, treat it as 2D images will distort the shape of the actual object through projection from 3D to 2D space, and compels the network to perform perspective distortion-invariant estimation. Moreover, directly regressing 3D coordinates from a 2D image is a highly nonlinear mapping, which causes difficulty in learning procedure. To overcome these problems, a module called Supervised Endecoder is proposed to process 3D convolution data, which can also be stacked through series connection to adapt different size of dataset. Based on the module, a network called Supervised High Dimension Endecoder Network is designed, which can be used to predict key points of markless human in a single depth map in 3D space. Experiments show improved prediction accuracy compared to the state-of-the-art approaches.

Depth maps are widely used in 3D imaging techniques because of the appearance of the consumer depth cameras. However, the practical application of the depth map is limited by the poor image quality. In this paper, we propose a novel framework for the single depth map super-resolution via joint the local and non-local constraints simultaneously in the depth map. For the non-local constraint, we use the group-based sparse representation to explore the non-local self-similarity of the depth map. For the local constraint, we first estimate gradient images in different directions of the desired high-resolution (HR) depth map, and then build a multi-directional gradient guided regularizer using these estimated gradient images to describe depth gradients with different orientations. Finally, the two complementary regularizers are cast into a unified optimization framework to obtain the desired HR image. The experimental results show that the proposed method can achieve better depth super-resolution performance than state-of-the-art methods.

Limited spatial resolution and varieties of degradations are the main restrictions of today’s captured depth map by active 3D sensing devices. Typical restrictions limit the direct use of the obtained depth maps in most of 3D applications. In this paper, we present a single depth map upsampling approach in contrast to the common work of using the corresponding combined color image to guide the upsampling process. The proposed approach employs a multi-level decomposition to convert the depth upsampling process to a classification-based problem via a multi-level classification-based learning algorithm. Hence, the lost high frequency details can be better preserved at different levels. The adopted multi-level decomposition algorithm utilizes $$l_{1} ,$$ and $$l_{0}$$ sparse regularization with total-variation regularization to keep structure- and edge-preserving smoothing with robustness to noisy degradations. In addition, the proposed classification-based learning algorithm supports the accuracy of discrimination by learning discriminative dictionaries that carry original features about each class and learning common shared dictionaries that represent the shared features between classes. The proposed algorithm has been validated via different experiments under variety of degradations using different datasets from different sensing devices. Results show superiority to the state of the art, especially in case of upsampling noisy low-resolution depth maps.

Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging issue for guided DSR is how to correctly select consistent structures and propagate them, and properly handle inconsistent ones. In this paper, we propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided DSR. Specifically, to effectively extract and combine relevant information from LR depth and HR guidance, we propose a multi-modal attention based fusion (MMAF) strategy for hierarchical convolutional layers, including a feature enhancement block to select valuable features and a feature recalibration block to unify the similarity metrics of modalities with different appearance characteristics. Furthermore, we propose a bi-directional hierarchical feature collaboration (BHFC) module to fully leverage low-level spatial information and high-level structure information among multi-scale features. Experimental results show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.

Consumer depth sensors such as time-of-flight camera or Kinect have gained significant popularity in recently. However, the captured depth maps suffer from limited spatial resolution and a variety of noise, making such depth maps difficult to be directly applied in related applications. In this paper, we present a novel single depth map super-resolution method, aiming to reconstruct high-resolution depth map from its associated low-resolution depth map without any auxiliary information. Particularly, we exploit the depth local self-similarity to assist in constructing patch pairs in terms of high-resolution and low-resolution depth edge patches, and then deduce a high-resolution depth edge map via Markov model. Finally, we implement a joint bilateral filter to reconstruct the high-resolution depth map. Experimental results show that our method overcomes existing methods on the benchmark database as well as Kinect captured depth maps.

Depth estimation and camera calibration of a focused plenoptic camera for visual odometry

In the recent past, vast amounts of stereo and augmented reality based applications are being developed for hand-held devices. In most of these applications depth map is a key ingredient for acceptable user experience. Accuracy and high density of depth map are important along with meeting real-time constraints on an embedded system. There is an inherent tradeoff between depth map quality and speed and invariably performance is usually important for competing in todays high-definition video marketplace. In this paper we present a method that addresses depth map quality while still maintaining performance at video frame-rates. Specifically, we discuss a technique to enhance a low-quality depth map for 3D point cloud generation on an embedded platform. We provide performance metrics and estimates on a Texas Instruments (TI) OMAP embedded platform and show that using simple pre and post-processing techniques one can achieve both quality and performance. A preliminary version of our point cloud application developed has a frame rate of about 15fps, majority being display and rendering related overheads. The core algorithms including pre and post processing have a much higher frame rate of about 23-25fps. We estimate that with adequate mapping of the algorithms to various cores and accelerated kernels, the frame rate could reach real-time performance of 30fps.

The use of virtual reality technologies for data visualization and analysis has been an emerging topic of research in the past years. However, one type of data has been left neglected, the point cloud. While some strides have been made in the visualization and analysis of point clouds in immersive environments, these have yet to be used for direct manipulation interactions. It is hypothesized that as with other types of data, bringing direct interactions and 3D visualization to point clouds may increase the ease of performing basic handling tasks. An immersive application for virtual reality HMDs was developed in Unity to help research this hypothesis. It is capable of parsing classified point cloud files with extracted objects and representing them in a virtual environment. Several editing tools were also developed, designed with the HMD controllers in mind. The end result allows the user to perform basic transformative tasks to the point cloud with an ease of use and intuitive feeling unmatched by the traditional desktop-based tools.

Airborne LIDAR is one of the quickest way to collect digital surface information, and point cloud dataset is widely used for natural disaster emergency and route planning. However, little research has been conducted on massive point cloud dataset transmission and visualization based on Internet. The paper discussed the characteristics of point cloud data, existing organization and visualization methods. We report our experience on the development of a prototype visualization system based on Quad-tree. Experiments show that our methodology is applicable and efficient.

A depth map upsampling method using depth local features is proposed. Depth discontinuity and depth variance are extracted from a low-resolution depth map and a colour image as the depth local features. They are incorporated into an energy function for the Markov random field (MRF)-based depth upsampling. The high-resolution depth map is obtained by optimising the energy function using belief propagation. The experimental results show that the proposed method outperforms other depth upsampling approaches in terms of the bad pixel rate.

Depth information captured by affordable depth sensors is characterized by low spatial resolution, which limits potential applications. Several methods have recently been proposed for guided super-resolution of depth maps using convolutional neural networks to overcome this limitation. In a guided super-resolution scheme, high-resolution depth maps are inferred from low-resolution ones with the additional guidance of a corresponding high-resolution intensity image. However, these methods are still prone to texture copying issues due to improper guidance by the intensity image. We propose a multi-scale residual deep network for depth map super-resolution. A cascaded transformer module incorporates high-resolution structural information from the intensity image into the depth upsampling process. The proposed cascaded transformer module achieves linear complexity in image resolution, making it applicable to high-resolution images. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art techniques for guided depth super-resolution.