Multi-View Photometric Stereo Revisited

We propose a novel method for 3d modeling using a fusion of a laser range sensor, a camera, and a flashlight. This combination provides dense normals and surface colors that can be mapped on a 3d model, whereas conventional sensors only output point clouds of the 3d geometry. Furthermore, the fusion enables formulations to be made simply and practically. Multi-view photometric stereo is used for estimating the fine normal distribution with a basic shape measured by the laser range sensor. Our photometric stereo can easily handle near-light formulation and specularity. Detailed surfaces can be shown by applying the normal map as bump mapping to the basic shape. Robust estimation and clustering are used for estimating reflection parameters. Results demonstrate that our method can estimate highly accurate reflection parameters and provide fine surface appearances using only a small amount of data. The effectiveness of our method is shown with an application of 3d content.

Motivated by a Bayesian vision of the 3D multi-view reconstruction from images problem, we propose a dense 3D reconstruction technique that jointly refines the shape and the camera parameters of a scene by minimizing the photometric reprojection error between a generated model and the observed images, hence considering all pixels in the original images. The minimization is performed using a gradient descent scheme coherent with the shape representation (here a triangular mesh), where we derive evolution equations in order to optimize both the shape and the camera parameters. This can be used at a last refinement step in 3D reconstruction pipelines and helps improving the 3D reconstruction's quality by estimating the 3D shape and camera calibration more accurately. Examples are shown for multi-view stereo where the texture is also jointly optimized and improved, but could be used for any generative approaches dealing with multi-view reconstruction settings (ie depth map fusion, multi-view photometric stereo).

While conventional calibrated photometric stereo methods assume that light intensities and sensor exposures are known or unknown but identical across observed images, this assumption easily breaks down in practical settings due to individual light bulb's characteristics and limited control over sensors. This paper studies the effect of unknown and possibly non-uniform light intensities and sensor exposures among observed images on the shape recovery based on photometric stereo. This leads to the development of a "semi-calibrated" photometric stereo method, where the light directions are known but light intensities (and sensor exposures) are unknown. We show that the semi-calibrated photometric stereo becomes a bilinear problem, whose general form is difficult to solve, but in the photometric stereo context, there exists a unique solution for the surface normal and light intensities (or sensor exposures). We further show that there exists a linear solution method for the problem, and develop efficient and stable solution methods. The semi-calibrated photometric stereo is advantageous over conventional calibrated photometric stereo in accurate determination of surface normal, because it relaxes the assumption of known light intensity ratios/sensor exposures. The experimental results show superior accuracy of the semi-calibrated photometric stereo in comparison to conventional methods in practical settings.

We present a technique that enables capture of detailed body shapes for clothing and footwear design using off-the-shelf equipment. The main components of our acquisition system are a standard flash-equipped digital camera and a calibration plate. Detailed 3D shapes are recovered from flash photographs using multi-view version of photometric stereo. The key idea is to obtain lighting variation by moving the camera itself widely such that photometric stereo can be performed with a built-in flash, which is not achievable with traditional methods. To recover a body model from photographs, we employ a deformable model. Starting from a template mesh, we first deform this to fit a sparse set of markers and then silhouettes in the images. Finally, we refine the mesh to agree with normals estimated from shading using a novel efficient multi-view photometric stereo algorithm based on the deformation gradient fitting. Experimental results obtained using real and synthetic images show that the method provides an accurate model for clothing and footwear design merely using a digital camera. We also present a method for creating a customized shoe-last model from a captured foot model.

This paper studies the effects of non-uniform light intensities and sensor exposures across observed images in photometric stereo. While conventional photometric stereo methods typically assume that light intensities are identical and sensor exposure is constant across observed images taken under varying lightings, these assumptions easily break down in practical settings due to individual light bulb’s characteristics and limited control over sensors. Our method explicitly models these non-uniformities and develops a method for accurately determining surface normal without affected by these factors. In addition, we show that our method is advantageous for general photometric stereo settings, where auto-exposure control is desirable. We compare our method with conventional least-squares and robust photometric stereo methods, and the experimental result shows superior accuracy of our method in this practical circumstance.

In optical 3D shape measurement, stereo vision with structured light can measure 3D scan data with high accuracy and is used in many applications, but fine surface detail is difficult to obtain. On the other hand, photometric stereo can capture surface details but has disadvantages, in that its 3D data accuracy drops and it requires multiple light sources. When the two measurement methods are combined, more accurate 3D scan data and detailed surface features can be obtained at the same time. In this paper, we present a 3D optical measurement technique that uses re-projection of images to implement photometric stereo without an external light source. 3D scan data is enhanced by combining normal vector from this photometric stereo method, and the result is evaluated with the ground truth.

Chapter 1 Photometric Stereo: An Overview

In this paper, we propose the estimating method of reflectance properties and the recovery of shape by using Normal Sampler and Indirect Diffuse Illumination Method(IDIM). Photometric Stereo Method(PSM) is generally based on the direct illumination. PSM in this paper is modified with indirect diffuse illumination(IDI) and then applied to hybrid reflectance model which consists of two main components; Lambertian and specular reflectance. Under the indirect diffuse illumination, the reflectance properties of natural objects can be determined by using Normal Sampler that has almost all the normal components in the observed direction. The estimated reflectance properties are used to construct reference table. Also, 3-D shape of an object can be recovered from intensity distribution of a pixel and a reference table. In this paper, the reference table is used to recover the 3-D shape of an object and IDI simplifies the limited conditions of reflectance analysis for prior studies without any loss in performance. The proposed method can be applied to various types of surfaces which can be defined by hybrid reflectance.

Line structured light (LSL) measurement systems can obtain high accuracy profiles, but the overall clarity relies greatly on the sampling interval of the scanning process. Photometric stereo (PS), on the other hand, is sensitive to tiny features but has poor geometrical accuracy. Cooperative measurement with these two methods is an effective way to ensure precision and clarity results. In this paper, an LSL-PS cooperative measurement system is brought out. The calibration methods used in the LSL and PS measurement system are given. Then, a data fusion algorithm with adaptive weights is proposed, where an error function that contains the 3D point cloud matching error and normal vector error is established. The weights, which are based on the angles of adjacent normal vectors, are also added to the error function. Afterward, the fusion results can be obtained by solving linear equations. From the experimental results, it can be seen that the proposed method has the advantages of both the LSL and PS methods. The 3D reconstruction results have the merits of high accuracy and high clarity.

Compared with geometric stereo vision based on triangulation principle, photometric stereo method has advantages in recovering per-pixel surface details. In this paper, we present a practical 3D imaging system by combining the near-light photometric stereo and the speckle-based stereo matching method. The system is compact in structure and suitable for multi-albedo targets. The parameters (including position and intensity) of the light sources can be self-calibrated. To realize the auto-calibration, we first use the distant lighting model to estimate the initial surface albedo map, and then with the estimated albedo map and the normal vector field fixed, the parameters of the near lighting model are optimized. Next, with the optimized lighting model, we use the near-light photometric stereo method to re-compute the surface normal and fuse it with the coarse depth map from stereo vision to achieve high-quality depth map. Experimental results show that our system can realize high-quality reconstruction in general indoor environments.

The acquisition of geometric 3-D information is crucial for ensuring quality standards and monitoring procedures in various manufacturing applications. Photometric stereo is an established technique in computer vision to recover 3-D surfaces of objects. However, existing photometric stereo methods mainly focus on normal estimation of objects, without considering the depth estimation. On the other hand, current methods tend to prioritize accuracy while overlooking the confidence of predictions, which holds valuable information within the industry. In this article, we propose a deep learning-based photometric stereo system, consisting of hardware implementation, dataset generation, and algorithm design, to reconstruct 3-D information of physical objects, represented by normal and depth maps. In terms of the proposed algorithm, a coarse-to-fine network is introduced to improve the performance by exploiting the relationship between initial normal and depth predictions. Furthermore, the pixel-wise confidence associated with predictions is also estimated without requiring the ground truth, making a contribution to enhancing both performance and practicality. The experimental results on our synthetic dataset and real samples demonstrate the effectiveness of the proposed method on both normal/depth and confidence estimation.

This paper tackles a new photometric stereo task, named universal photometric stereo. Unlike existing tasks that assumed specific physical lighting models; hence, drastically limited their usability, a solution algorithm of this task is supposed to work for objects with diverse shapes and materials under arbitrary lighting variations without assum-ig any specific models. To solve this extremely challenging task, we present a purely data-driven method, which eliminates the prior assumption of lighting by replacing the recovery of physical lighting parameters with the extraction of the generic lighting representation, named global lighting contexts. We use them like lighting parameters in a calibrated photometric stereo network to recover surface normal vectors pixelwisely. To adapt our network to a wide variety of shapes, materials and lightings, it is trained on a new synthetic dataset which simulates the appearance of objects in the wild. Our method is compared with other state-of-the-art uncalibrated photometric stereo methods on our test data to demonstrate the significance of our method.

We present a technique for creating realistic facial animation from a set of high-resolution static scans of an actor's face driven by passive video of the actor from one or more viewpoints. We capture high-resolution static geometry using multi-view stereo and gradient-based photometric stereo [Ghosh et al. 2011]. The scan set includes around 30 expressions largely inspired by the Facial Action Coding System (FACS). Examples of the input scan geometry can be seen in Figure 1 (a). The base topology is defined by an artist for the neutral scan of each subject. The dynamic performance can be shot under existing environmental illumination using one or more off-the shelf HD video cameras.

Computer vision systems can be used to determine the shapes of real three-dimensional objects for purposes of object recognition and pose estimation or for CAD applications. One method that has been developed is photometric stereo. This method uses several images taken from the same viewpoint, but with different lightings, to determine the three-dimensional shape of an object. Most previous work in photometric stereo has been with gray-tone images; color images have only been used for dielectric materials. In this paper we describe a procedure for color photometric stereo, which recovers the shape of a colored object from two or more color images of the object under white illumination. This method can handle different types of materials, such as composites and metals, and can employ various reflection models such as the Lambertian, dichromatic, and Torrance-Sparrow models. For composite materials, colored metals, and dielectrics, there are two advantages of utilizing color information: at each pixel, there are more constraints on the orientation, and the result is less sensitive to noise. Consequently, the shape can be found more accurately. The method has been tested on both artificial and real images of objects of various materials, and on real images of a multi-colored object.

Multi-view stereo based on deep learning is mostly dedicated to improving the accuracy of point clouds, whlile complex scenes, occlusion, and other factors limit their reconstruction completeness, especially in the area with drastic changes in depth direction. In this paper, we propose a multi-view stereo network based on depth edge flow (DEF-MVSNet), using the reference image as a guide to dynamically infer the edge coordinates to improve reconstruction completeness. First, we ignore the boundaries in the depth prediction stage to generate better initial depth inference results. Then, we use a EdgeDetect module to extract the obviously features of the reference image and predict the pixel offset of the depth map. Finally, the EdgeFlow module modifies the initial depth map coordinates according to the offset and uses multiple iterations to dynamically update the depth map. The experimental results prove that our method has a great improvement in the completeness of reconstruction compared with MVSNet and R-MVSNet without increasing the memory and time overhead. Code and models are publicly available at https://github.com/linkuizzZ/EF-MVSNet.