Depth map generation based on scene categories

Computer generated 2D plus Depth (2D+Z) images are common input data for 3D display with depth image-based rendering (DIBR) technique. Due to their simplicity, linear interpolation methods are usually used to convert low-resolution images into high-resolution images for not only depth maps but also 2D RGB images. However linear methods suffer from zigzag artifacts in both depth map and RGB images, which severely affects the 3D visual experience. In this paper, spatial distance-based interpolation algorithm for computer generated 2D+Z images is proposed. The method interpolates RGB images with the help of depth and edge information from depth maps. Spatial distance from interpolated pixel to surrounding available pixels is utilized to obtain the weight factors of surrounding pixels. Experiment results show that such spatial distance-based interpolation can achieve sharp edges and less artifacts for 2D RGB images. Naturally, it can improve the performance of 3D display. Since bilinear interpolation is used in homogenous areas, the proposed algorithm keeps low computational complexity.

This paper proposes a simplified method to generate elemental images using the image pair of the scene for integral imaging stereo display. Firstly, the generalized superpixel matching algorithm is designed for integral imaging to obtain the depth map. We define the discrete boundary according to the characteristics of the low-texture region. A kind of consistency check method by calculating the inclination degree of the discrete boundary is proposed. The remaining texture regions are handled by assisted growth matching method. The precise depth map is obtained through the above two steps. Secondly, the depth map and color image are used to generate multiple elemental images according to the reverse mapping principle. Therefore the reconstruction quality depends on the accuracy of the depth map. The experiments verify that the depth map has a small error range and prominent details which can improve the accuracy of the reconstructed light field and is suitable for integral imaging.

Depth map coding plays a crucial role in 3D Video communication systems based on the “Multi-view Video plus Depth” representation as view synthesis performance is strongly affected by the accuracy of depth information, especially at edges in the depth map image. In this paper an efficient algorithm for edge-preserving intra depth compression based on H.264/AVC is presented. The proposed method introduces a new Intra mode specifically targeted to depth macroblocks with arbitrarily shaped edges, which are typically not efficiently represented by DCT. Edge macroblocks are partitioned into two regions each approximated by a flat surface. Edge information is encoded by means of contextcoding with an adaptive template. As a novel element, the proposed method allows exploiting the edge structure of previously encoded edge macroblocks during the context-coding step to further increase compression performance. Experiments show that the proposed Intra mode can improve view synthesis performance: average Bjøntegaard bit rate savings of 25% have been reported over a standard H.264/AVC Intra coder.

Depth completion involves recovering a dense depth map from a sparse map and an RGB image. Recent approaches focus on utilizing color images as guidance images to recover depth at invalid pixels. However, color images alone are not enough to provide the necessary semantic understanding of the scene. Consequently, the depth completion task suffers from sudden illumination changes in RGB images (e.g., shadows). In this paper, we propose a novel three-branch backbone comprising color-guided, semantic-guided, and depth-guided branches. Specifically, the color-guided branch takes a sparse depth map and RGB image as an input and generates color depth which includes color cues (e.g., object boundaries) of the scene. The predicted dense depth map of color-guided branch along-with semantic image and sparse depth map is passed as input to semantic-guided branch for estimating semantic depth. The depth-guided branch takes sparse, color, and semantic depths to generate the dense depth map. The color depth, semantic depth, and guided depth are adaptively fused to produce the output of our proposed three-branch backbone. In addition, we also propose to apply semantic-aware multi-modal attention-based fusion block (SAMMAFB) to fuse features between all three branches. We further use CSPN++ with Atrous convolutions to refine the dense depth map produced by our three-branch backbone. Extensive experiments show that our model achieves state-of-the-art performance in the KITTI depth completion benchmark at the time of submission.

Three-dimensional modeling of a scene from the automatic analysis of 2D image sequences is a big challenge for future interactive audiovisual services based on 3D content manipulation such as virtual vests, 3D teleconferencing and interactive television. We propose a scheme that computes 3D objects models from stereo analysis of image triplets shot by calibrated cameras. After matching the different views with a correlation based algorithm, a depth map referring to a given view is built by using a fusion criterion taking into account depth coherency, visibility constraints and correlation scores. Because luminance segmentation helps to compute accurate object borders and to detect and improve the unreliable depth values, a two steps segmentation algorithm using both depth map and graylevel image is applied to extract the objects masks. First an edge detection segments the luminance image in regions and a multimodal thresholding method selects depth classes from the depth map. Then the regions are merged and labelled with the different depth classes numbers by using a coherence test on depth values according to the rate of reliable and dominant depth values and the size of the regions. The structures of the segmented objects are obtained with a constrained Delaunay triangulation followed by a refining stage. Finally, texture mapping is performed using open inventor or VRML1.0 tools.© (1997) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.

In this paper, we propose a convex optimization framework for simultaneous estimation of super-resolved depth map and images from a single moving camera. The pixel measurement error in 3D reconstruction is directly related to the resolution of the images at hand. In turn, even a small measurement error can cause significant errors in reconstructing 3D scene structure or camera pose. Therefore, enhancing image resolution can be an effective solution for securing the accuracy as well as the resolution of 3D reconstruction. In the proposed method, depth map estimation and image super-resolution are formulated in a single energy minimization framework with a convex function and solved efficiently by a first-order primal-dual algorithm. Explicit inter-frame pixel correspondences are not required for our super-resolution procedure, thus we can avoid a huge computation time and obtain improved depth map in the accuracy and resolution as well as high-resolution images with reasonable time. The superiority of our algorithm is demonstrated by presenting the improved depth map accuracy, image super-resolution results, and camera pose estimation.

In this paper, we propose a 3D display method combining a pickup process using axially recorded multiple images and an integral imaging display process. First, we extract the color and depth information of 3D objects for displaying 3D images from axially recorded multiple 2D images. Next, using the extracted depth map and color images, elemental images are computationally synthesized based on a ray mapping model between 3D space and an elemental image plane. Finally, we display 3D images optically by an integral imaging system with a lenslet array. To show the usefulness of the proposed system, we carry out optical experiments for 3D objects and present the experimental results.

To support an efficient media consumption in a wearable device and IoT (Internet of Things) environment, the standardization of IoMTW (Internet of Media-Things and Wearables) is in the progress in MPEG (Moving Picture Experts Group). In this paper, we present a hand gesture detection and recognition algorithm to generate hand gesture-based commands for controlling the media consumption in smart glasses. In the proposed method, we use depth map and color image together to extract more accurate hand contour. We are going to present representation of the detected hand contour based on Bezier curve as metadata to provide an interoperable interface between a detection module and a recognition module. In a recognition module, the detected hand contour is reconstructed by parsing the delivered metadata. In the proposed recognition method, a set of hand gestures featured with diverse combination of open fingers and rotational angles can be recognized with quite stable performance in the proposed method. Finally, the recognized hand gesture is mapped into one of the pre-defined gesture commands.

This paper presents stereo vision-based vehicle detection approach on the road using a road feature and disparity histogram. It is not easy to detect only vehicles robustly on the road in various traffic situations, for example, a nonflat road or a multiple-obstacle situation. This paper focuses on the improvement of vehicle detection performance in various real traffic situations. The approach consists of three steps, namely obstacle localization, obstacle segmentation, and vehicle verification. First, we extract a road feature from v-disparity maps binarized using the most frequent values in each row and column, and adopt the extracted road feature as an obstacle criterion in column detection. However, many obstacles still coexist in each localized obstacle area. Thus, we divide the localized obstacle area into multiple obstacles using a disparity histogram and remerge the divided obstacles using four criteria parameters, namely the obstacle size, distance, and angle between the divided obstacles, and the difference of disparity values. Finally, we verify the vehicles using a depth map and gray image to improve the performance. We verify the performance of our proposed method by conducting experiments in various real traffic situations. The average recall rate of vehicle detection is 95.5%.

This paper presents a low complexity 3D image depth map generation algorithm for embedded stereo applications. The proposed algorithm generates depth information based on a single view 2D image automatically. Owing to different scene characteristics of image, we propose a mechanism to classify images to “Scenery”, “Normal” and “Close-up” types first and generate the associated depth map according to the proposed techniques. In addition, we propose a human detection method for strengthening the depth information in images with humans and post-processing for refining depth map. With good quality in the generated depth map, the proposed algorithm achieves about 93% in complexity reduction as compared to the traditional algorithm, which is suitable for realization in both the hardware and embedded systems for portable stereo applications.

A technique to improve the rendering quality of novel views for colour plus depth based 3D video is proposed. Most depth discontinuities occur around the edges of depth map objects. If information around edges of both colour and depth map images is lost during transmission, this will affect the quality of the rendered views. Therefore this work proposes a technique to categorize edge and surrounding areas into two different regions (Region Of Interests (ROIs)) and later protect them separately to provide Unequal Error Protection (UEP) during transmission. In this way the most important edge areas (vital for novel view rendering) will be more protected than other surrounding areas. This method is tested over a H.264/AVC based simulcast encoding and transmission setup. The results show improved rendered quality with the proposed ROI-based UEP method compared to Equal Error Protection (EEP) method.

We present a novel closed-form solution for the joint self-calibration of video and range sensors. The approach single assumption is the availability of synchronous time of flight (i.e., range distances) measurements and visual position of the target on images acquired by a set of cameras. In such case, we make explicit a rank constraint that is valid for both image and range data. This rank property is used to find an initial and affine solution via bilinear factorization, which is then corrected by enforcing the metric constraints characteristic for both sensor modalities (i.e., camera and anchors constraints). The output of the algorithm is the identification of the target/range sensor position and the calibration of the cameras. The application extent of our approach is broad and versatile. In fact, with the same framework, we can deal with, but not restricted to, two very different applications. The first is aimed at calibrating cameras and microphones deployed in an unknown environment. The second uses a RGB-D device to reconstruct the 3D position of a set of keypoints using the camera and depth map images. Synthetic and real tests show the algorithm performance under different levels of noise and configurations of target locations, number of sensors and cameras.

Depth maps acquired by either physical sensors or learning methods are often seriously distorted due to boundary distortion problems, including missing, fake, and misaligned boundaries (compared with RGB images). An RGB-guided depth map recovery method is proposed in this paper to recover true boundaries in seriously distorted depth maps. Therefore, a unified model is first developed to observe all these kinds of distorted boundaries in depth maps. Observing distorted boundaries is equivalent to identifying erroneous regions in distorted depth maps, because depth boundaries are essentially formed by contiguous regions with different intensities. Then, erroneous regions are identified by separately extracting local structures of RGB image and depth map with Gaussian kernels and comparing their similarity on the basis of the SSIM index. A depth map recovery method is then proposed on the basis of the unified model. This method recovers true depth boundaries by iteratively identifying and correcting erroneous regions in recovered depth map based on the unified model and a weighted median filter. Because RGB image generally includes additional textural contents compared with depth maps, texture-copy artifacts problem is further addressed in the proposed method by restricting the model works around depth boundaries in each iteration. Extensive experiments are conducted on five RGB-depth datasets including depth map recovery, depth super-resolution, depth estimation enhancement, and depth completion enhancement. The results demonstrate that the proposed method considerably improves both the quantitative and visual qualities of recovered depth maps in comparison with fifteen competitive methods. Most object boundaries in recovered depth maps are corrected accurately, and kept sharply and well aligned with the ones in RGB images.

Hazy image obtained in the low‐light environment has the characteristics of low contrast, non‐uniform illumination, color cast and much noise. In this paper, a method is put forward which can be properly applied to recover low‐light hazy images. The original image is first decomposed into glow layer and haze layer with a modified color channel transformation for glow artifacts and color balanced. A new light segmentation function is proposed next by using gamma correction of channel difference and setting threshold levels to determine if the pixel belongs to light source regions. Then the ambient illuminance map is estimated using maximum reflectance prior to computing the atmosphere light in the light and non‐light regions. Finally, a novel nonlinear image depth estimation model is established to build the relationship between the image depth map and three image features including luminance, saturation and gradient map for the light areas. The experimental results prove that the dehazing algorithm is reliable for removing haze and glow artifacts of active light sources, reducing much noise and improving the visibility.

A persistent problem in the three-dimensional (3-D) reconstruction technique is to eliminate blank areas in the 3-D map, which commonly emerges after removing undesired objects, such as dynamic targets or occluded areas. This task is challenging for it is difficult to acquire the coherence between color and depth information, which are both lost for each pixel in the target region. Moreover, creating novel artifacts also needs to be avoided during the completion process. To address these problems, in this paper, we propose a collaborative method to complete the lost area in the color image and its corresponding depth map. In the proposed method, an examplar-based image inpainting technique combined with planarity knowledge is adopted to iteratively repair the texture and structure information. The color completion process provides candidates for plane-fitting on segmented superpixels. The depth filling step extracts image areas requiring reinpainting, which is defined by their gradient differences. The final 3-D map can then be reconstructed from both completed color and depth images. Experiments on real-world scenes demonstrate the success of our method in the completion of 3-D maps.