Shape Estimation Using Polarization and Shading from Two Views

In this paper, we propose a geometric neural network with edge-aware refinement (GeoNet++) to jointly predict both depth and surface normal maps from a single image. Building on top of two-stream CNNs, GeoNet++ captures the geometric relationships between depth and surface normals with the proposed depth-to-normal and normal-to-depth modules. In particular, the "depth-to-normal" module exploits the least square solution of estimating surface normals from depth to improve their quality, while the "normal-to-depth" module refines the depth map based on the constraints on surface normals through kernel regression. Boundary information is exploited via an edge-aware refinement module. GeoNet++ effectively predicts depth and surface normals with high 3D consistency and sharp boundaries resulting in better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used in other depth/normal prediction frameworks to improve 3D reconstruction quality and pixel-wise accuracy of depth and surface normals. Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth prediction in 3D. In contrast to current metrics that focus on evaluating pixel-wise error/accuracy, 3DGM measures whether the predicted depth can reconstruct high quality 3D surface normals. This is a more natural metric for many 3D application domains. Our experiments on NYUD-V2 [1] and KITTI [2] datasets verify that GeoNet++ produces fine boundary details and the predicted depth can be used to reconstruct high quality 3D surfaces.

Deep-learning-based polarization 3D imaging techniques, which train networks in a data-driven manner, are capable of estimating a target’s surface normal distribution under passive lighting conditions. However, existing methods have limitations in restoring target texture details and accurately estimating surface normals. Information loss can occur in the fine-textured areas of the target during the reconstruction process, which can result in inaccurate normal estimation and reduce the overall reconstruction accuracy. The proposed method enables extraction of more comprehensive information, mitigates the loss of texture information during object reconstruction, enhances the accuracy of surface normal estimation, and facilitates more comprehensive and precise reconstruction of objects. The proposed networks optimize the polarization representation input by utilizing the Stokes-vector-based parameter, in addition to separated specular and diffuse reflection components. This approach reduces the impact of background noise, extracts more relevant polarization features of the target, and provides more accurate cues for restoration of surface normals. Experiments are performed using both the DeepSfP dataset and newly collected data. The results show that the proposed model can provide more accurate surface normal estimates. Compared to the UNet architecture-based method, the mean angular error is reduced by 19%, calculation time is reduced by 62%, and the model size is reduced by 11%.

Reillumination-driven shape from shading

In the past few years, convolutional neural nets (CNN) have shown incredible promise for learning visual representations. In this paper, we use CNNs for the task of predicting surface normals from a single image. But what is the right architecture we should use? We propose to build upon the decades of hard work in 3D scene understanding, to design new CNN architecture for the task of surface normal estimation. We show by incorporating several constraints (man-made, manhattan world) and meaningful intermediate representations (room layout, edge labels) in the architecture leads to state of the art performance on surface normal estimation. We also show that our network is quite robust and show state of the art results on other datasets as well without any fine-tuning.

<p><strong>This thesis explores innovative methods for enhancing the manipulation and geometric understanding of 360° images in virtual reality (VR) applications, with a focus on depth estimation, surface normal prediction, and multi-task learning. The research presents a comprehensive approach to improving the manipulation of stereo 360° images, particularly in the context of image composition. A novel method is proposed to seamlessly integrate new visual elements into stereo 360° scenes, thereby enhancing user interaction and immersion in VR environments.</strong></p><p>To support such advanced editing applications, a new technique for estimating depth maps from monocular 360° images is introduced. This method leverages both local and global scene information, significantly improving the accuracy of depth predictions. This is particularly crucial for applications that require spatial awareness, such as virtual and augmented reality, where accurate depth perception is key to realistic interaction with the virtual environment.</p><p>Furthermore, the thesis introduces a hybrid approach that combines convolutional neural networks (CNNs) and Vision Transformers (ViTs) to improve surface normal estimation. This method takes advantage of CNNs' ability to capture fine-grained details and ViTs' strength in modeling global context, resulting in more precise surface geometry analysis from 360° imagery. This enhanced surface normal estimation plays a vital role in better understanding the spatial structure of the scene.</p><p>In addition, the research demonstrates the effectiveness of multi-task learning (MTL) for comprehensive scene geometry understanding. By predicting both depth and surface normals simultaneously from monocular 360° images, the proposed MTL framework delivers a more detailed and coherent geometric representation, further contributing to the realism and immersion of VR used scenarios.</p><p>Through these contributions, the thesis significantly advances the field of 360° image-based manipulations and scene understanding, offering new tools and methodologies that pave the way for more immersive, interactive, and spatially aware experiences for VR applications.</p>

Photometric stereo is a method to obtain surface normals of an object using its images captured under varying illumination directions. The existing deep learning-based methods require multiple images of an object captured using complex image acquisition systems. In this work, we propose a deep learning framework to perform three tasks jointly: (i) lighting estimation, (ii) image relighting, and (iii) surface normal estimation, all from a single input image of an object with non-Lambertian surface and general reflectance. The network explicitly segregates global geometric features and local lighting-specific features of the object from a single image. The local features resemble attached shadows, shadings, and specular highlights, providing valuable lighting estimation and relighting cues. The global features capture the lighting-independent geometric attributes that effectively guide the surface normal estimation. The joint training transfers valuable insights to achieve significant improvements across all three tasks. We show that the proposed single-image-based relighting framework outperforms several existing photometric stereo methods which require multiple images of a static object.

Photometric stereo (PST) is a widely used technique of estimating surface normals from an image set. However, it often produces inaccurate results for non-Lambertian surface reflectance. In this study, PST is reformulated as a sparse recovery problem where non-Lambertian errors are explicitly identified and corrected. We show that such a problem can be accurately solved via a greedy algorithm called orthogonal matching pursuit (OMP). The performance of OMP is evaluated on synthesized and real-world datasets: we found that the greedy algorithm is overall more robust to non-Lambertian errors than other state-of-the-art sparse approaches with little loss of efficiency. Along with providing an overview of current methods, novel contributions in this paper are as follows: we propose an alternative sparse formulation for PST; in previous PST studies (Wu et al., Robust photometric stereo via low-rank matrix completion and recovery, 2010), (S. Ikehata et al., Robust photometric stereo using sparse regression, 2012), the surface normal vector and the error vector are treated as two entities and are solved independently. In this study, we convert their formulation into a new canonical form of the sparse recovery problem by combining the two vectors into one large vector in a new “stacked” formulation in this domain. This allows for a large repertoire of existing sparse recovery algorithms to be more straightforwardly applied to the PST problem. In our application of the OMP greedy algorithm, we show that greedy solvers can indeed be applied, with this study supplying the first of such attempt at employing greedy approaches to estimate surface normals within the framework of PST. We numerically compare the performance of several normal vector recovery methods. Most notably, this is the first detailed test on complex images of the normal estimation accuracy of our previously proposed method, least median of squares (LMS).

We present a single-capture photometric stereo method using a hyperspectral camera. A spectrally and spatially designed illumination enables a point-wise estimation of reflectance spectra and surface normals from a single hyperspectral image. The illumination works as a reflectance probe in wide spectral regions where reflectance spectra are measured, and the full spectra are estimated by interpolation. It also works as the resource for shadings in other spectral regions. The accuracy of estimation is evaluated in a simulation. Also, we prepare an experimental setup and demonstrate a surface reconstruction against a real scene.

One of the challenging problems in geometric modeling and computer graphics is the construction of realistic human facial geometry. Such geometry are essential for a wide range of applications, such as 3D face recognition, virtual reality applications, facial expression simulation and computer based plastic surgery application. This paper addresses a method for the construction of 3D geometry of human faces based on the use of Elliptic Partial Differential Equations (PDE). Here the geometry corresponding to a human face is treated as a set of surface patches, whereby each surface patch is represented using four boundary curves in the 3-space that formulate the appropriate boundary conditions for the chosen PDE. These boundary curves are extracted automatically using 3D data of human faces obtained using a 3D scanner. The solution of the PDE generates a continuous single surface patch describing the geometry of the original scanned data. In this study, through a number of experimental verifications we have shown the efficiency of the PDE based method for 3D facial surface reconstruction using scan data. In addition to this, we also show that our approach provides an efficient way of facial representation using a small set of parameters that could be utilized for efficient facial data storage and verification purposes.

Real-time 3D surface reconstruction is a widely interested technique in the manufacturing industry. In this paper, an effective 3D surface reconstruction method using the active stereo with the projection of straight line patterns is proposed. For the purpose of real-time applications, 3D surface reconstruction is achieved by using two simple line projections. A depth-map is obtained by searching the projected lines. The final 3D surface is constructed by applying a spatial Kalman filter to the measured depth-map. Because it uses very simple line projections and searches the robust line patterns, it can be a fast, reliable and cost saving 3D measurement in comparison with the sinusoidal fringes projection methods.KeywordsKalman FilterReference PlaneLine PatternGray CodeHeight EstimateThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Three-dimensional (3D) medical images are prone to overlap, and there are some problems, such as low detection efficiency and inconsistent with the actual situation. Therefore, a 3D medical image surface reconstruction method based on data mining and machine learning is proposed. The 3D medical images were classified according to different ways, the information frame of 3D medical images was established and the surface overlapping information model of 3D images was given. Based on this information framework, the nonlinear function of overlapping area information of 3D medical images was constructed. The weight of the nonlinear function was used to calculate the input and output results of overlapping area information. Combined with the input mode of 3D medical image information, the error between the information output and the expected output was set. The nonlinear function weight of the overlapping area information of 3D medical images was modified by using the learning rate and the use time of the overlapping area information, and the influence factors of the overlapping information detection were obtained by increasing the situation terms, so as to complete the detection of the surface reconstruction information of 3D medical images. The experimental results show that the information detection results of the proposed method fit well with the actual situation, and the information detection efficiency is high.

We describe a technique for surface recovery of a rotating object illuminated under a collinear light source (where the light source lies on or near the optical axis). We show that the surface reflectance function can be directly estimated from the image sequence without any assumption on the reflectance property of the object surface. From the image sequence, the 3D locations of some singular surface points are calculated and their brightness values are extracted for the estimation of the reflectance function. We also show that the surface can be recovered by using shading information in two images of the rotating object. Iteratively using the first-order Taylor series approximation and the estimated reflectance function, the depth and orientation of the surface can be recovered simultaneously. The experimental results on real image sequences of both matte and specular surfaces demonstrate that the technique is feasible and robust. >

In this paper, we propose a novel energy optimization framework to accurately estimate surface normal and reflectance of an object from an input image sequence. Input images are captured from a fixed viewpoint under varying lighting conditions. In the proposed approach we combine photometric stereo and Retinex constraints into our energy function. To formulate inter-image constraints, shading information is added to the Lambertian model to account for shadows. For intra-image constraints, we moderate the strength of shading smoothness according to shadow mask and normal variations. By minimizing this energy function we are able to recover accurate surface normals and reflectance. Experimental results show that our approach yields more realistic normal map and accurate albedo map than the state-of-the-art uncalibrated photometric stereo algorithms. As for intrinsic image decomposition, results on the real and synthetic scenes show that the proposed approach outperforms previous ones.

Learning-based surface reconstruction methods have received considerable attention in recent years due to their excellent expressiveness. However, existing learning-based methods lack scalability in processing large-scale point clouds. This paper proposes a novel scalable learning-based 3D surface reconstruction method based on octree, called SSRNet. SSRNet works in a scalable reconstruction pipeline, which divides oriented point clouds into different local parts and then processes them in parallel. Accommodating this scalable design pattern, SSRNet constructs local geometric features for octree vertices. Such features comprise the relation between the vertices and the implicit surface, ensuring geometric perception. Focusing on local geometric information also enables the network to avoid the overfitting problem and generalize well on different datasets. Finally, as a learning-based method, SSRNet can process large-scale point clouds in a short time. And to further solve the efficiency problem, we provide a lightweight and efficient version that is about five times faster while maintaining reconstruction performance. Experiments show that our methods achieve state-of-the-art performance with outstanding efficiency.

The characteristics of the point cloud obtained with laser scanning system are discussed in the paper, and a new algorithm of surface reconstruction based on point cloud is put forward. Based on the theory of B-spline curve and surface interpolation, the freeform curve and surface reconstruction method - we called the surface as loft surface - are introduced in detail. This method of surface reconstruction is adequately utilized the characteristics of the point cloud from laser scanning system. The experimental results testify that the method is effectual and feasible.