Reconstruction of transparent objects using phase shifting profilometry based on diffusion models

Three-dimensional (3D) shape measurement systems based on diffuse reflection of projected structured light do not deliver reliable data when measuring glossy, transparent, absorbent, or translucent objects. In recent years, we have developed a method based on stereo recording with infrared cameras and projection of areal aperiodic sinusoidal thermal patterns to detect such objects. However, the measurements took longer than 10 s, up to minutes; moreover, the measurement accuracy was improvable. Now, we have succeeded in both drastically reducing measurement time and significantly increasing measurement quality. This finally provides a technique for reliably measuring transparent objects, e.g., in series production. We demonstrate measurement examples achieved within 1 s and with 3D standard deviations less than 10 µm.

Structured light is a common tool to observe the 3D shape of objects. However, most of these sensor systems assume that the objects scatter light diffusely and that there is only a moderate amount of ambient light. They are not able to deal with specular or transparent objects or with the ambient lighting from direct sunlight. We have built a scanning laser line striper that works even in bright sunlight and is able to observe optically challenging objects. In this paper we will show how the sensor can be used to measure the 3D shape of reflective objects and objects behind a semi-transparent plane.

Reconstructing the geometry of transparent objects has been a long-standing challenge. Existing methods rely on complex setups, such as manual annotation or darkroom conditions, to obtain object silhouettes and usually require controlled environments with designed patterns to infer ray-background correspondence. However, these intricate arrangements limit the practical application for common users. In this paper, we significantly simplify the setups and present a novel method that reconstructs transparent objects in unknown natural scenes without manual assistance. Our method incorporates two key technologies. Firstly, we introduce a volume rendering-based method that estimates object silhouettes by projecting the 3D neural field onto 2D images. This automated process yields highly accurate multi-view object silhouettes from images captured in natural scenes. Secondly, we propose transparent object optimization through differentiable refraction rendering with the neural SDF field, enabling us to optimize the refraction ray based on color rather than explicit ray-background correspondence. Additionally, our optimization includes a ray sampling method to supervise the object silhouette at a low computational cost. Extensive experiments and comparisons demonstrate that our method produces high-quality results while offering much more convenient setups.

With the development of science and technology and industrial production, the application of three-dimensional shape measurement technology in the fields of machine vision, industrial monitoring, mechanical engineering and medical testing has become increasingly widespread. The optical three-dimensional shape measurement technology is a method that uses a series of optical methods to obtain the three-dimensional information of the measured object, and the structured light measurement method is a widely used measurement method. In the structured light measurement method, the generation of fringe patterns is a very important link. Here, we use holographic method to generate a fringe pattern whose period and phase can be easily modulated. First, a black and white fringe pattern with a certain spatial frequency is generated according to the cosine structured light period and phase. The spatial frequency of the black and white fringe is determined by the spatial frequency of the structured light to be generated and the magnification of the projection system. A prism phase that can cause lateral movement is applied to the black part of the black and white stripes, so that the light incident on the black area deviates from the optical axis to the first order of diffraction, while the light incident on other parts (without being modulated) continues to travel along the optical axis. In this way, the deviated beam is bright and dark, and finally structured illumination is obtained. Theories and experiments verify the effectiveness of the method. The method to generate fringe patterns is simple, fast, accurate and can be conveniently controlled, and can be well applied to three-dimensional shape measurement technology.

Deflectometry with structured light for high-precision measurement of optical specular surfaces has attracted considerable attention. However, conventional deflectometry schemes often encounter challenges when they deal with lens-like transparent objects due to parasitic reflections arising from the rear side of objects. To address this issue, we propose a novel stereo deflectometry method that replaces traditional phase-shift patterns with Fourier-based patterns inspired by single-pixel imaging. This method enables the acquisition of Fourier spectra of light response coefficients for each camera pixel. Specifically, through inverse discrete Fourier transform and the Fourier slice theorem, we obtain the light response coefficients of the light source to the camera pixel. Subsequently, a geometry optical analysis method is proposed to separate the light response coefficients components associated with the front surface given the lack of epipolar constraint between the camera and LCD screen. An iterative algorithm integrated with Particle swarm optimization is proposed to accelerate the reconstruction process and obtain the surface normal field of the upper surface of the lens-like transparent object. The wavefront reconstruction algorithm for gradient integration is used on the normal vector field to enables high-precision reconstruction of the object's surface shape. Experiments on actual optical components demonstrate that the proposed method can accurately reconstruct the front surface of transparent lens by eliminating the interference from parasitic reflection, and an error of less than 200 nm is obtained.

The measurement of transparent objects has always been regarded as one of the difficult problems in 3D reconstruction technology. For transparent objects conforming to the birefringence model, we propose a method for measuring the parameters of transparent objects based on multi-wavelength incident light. This paper analyzes the transparent objects where light enters the birefringence model. Because the transparent objects refract light of different wavelengths differently, when the lights with different wavelengths emitted from the same point and same direction passes through the transparent objects, the position and direction of light with different wavelength will come out differently. The position and direction of these outgoing rays are related to the refractive index and surface topography of the transparent objects. By obtaining the equations of these outgoing rays, we can derive the equations of the refractive index and surface topography of the transparent objects. In this paper, the incident light obtained by computer simulation is used as the initial data, and the simulated annealing algorithm is used to verify the model. The final results show that within the range of random error values given by the computer, the surface shape, curvature, thickness and refractive index of transparent objects with small error values can be obtained. At the same time, this paper introduces the Ceres Solver C++ nonlinear optimization library into the simulated annealing algorithm, which improves the speed and accuracy of the algorithm and greatly enhances the robustness of the algorithm.

The practical, rapid, and accurate optical 3D reconstruction of transparent objects with contemporary non-contact optical techniques, has been an open challenge in the field of optical metrology. The combination of refraction, reflection, and transmission in transparent objects makes it very hard to use common off-the-shelf 3D reconstruction solutions to accurately reconstruct transparent objects in three dimensions without completely coating the object with an opaque material. We demonstrate in this work that a specific class of transparent objects can indeed be reconstructed without the use of opaque spray coatings, via Optical Projection Tomography (OPT). Particularly, the 3D reconstruction of large thin-walled hollow transparent objects can be achieved via OPT, without the use of refractive-index-matching liquid, accurately enough for use in both cultural heritage and beverage packaging industry applications. We compare 3D reconstructions of our proposed OPT method to those achieved by an industrial-grade 3D scanner and report average shape differences of ±0.34 mm for 'shelled' hollow objects and ±0.92 mm for 'non-shelled' hollow objects. A disadvantage of using OPT, which was noticed on the thicker 'non-shelled' hollow objects, as opposed to the 'shelled' hollow objects, was that it induced partial filling of hollow areas and the deformation of embossed features.

The visual 3D reconstruction of transparent objects in unstructured scenes is challenging due to the complex image formation principles underlying their visual appearance. Most state-of-the-art reconstruction methods ignore this problem and assume Lambertian reflection. Yet, transparent objects are relevant scene information for applications in intelligent robotics (such as grasping) or virtual reality. In this work, we present an approach to detect non-planar transparent objects, like bottles or glasses, by specifically searching for geometry inconsistencies caused by refraction or reflection. Depth information is acquired using a Kinect sensor, which is moved within the scene in order to acquire multiple views. The individual measurements are combined into a 3D volume, yielding the objects' location and a rough shape estimate. Results are presented using various household objects made of glass or plastic.

Recovering 3D shapes of deformable objects from single 2D images is an extremely challenging and ill-posed problem. Most existing approaches are based on structure-from-motion or graph inference, where a 3D shape is solved by fitting 2D keypoints/mask instead of directly using the vital cue in the original 2D image. These methods usually require multiple views of an object instance and rely on accurate labeling, detection, and matching of 2D keypoints/mask across multiple images. To overcome these limitations, we make effort to reconstruct 3D deformable object shapes directly from the given unconstrained 2D images. In training, instead of using multiple images per object instance, our approach relaxes the constraint to use images from the same object category (with one 2D image per object instance). The key is to disentangle the category-specific representation of the 3D shape identity and the instance-specific representation of the 3D shape displacement from the 2D training images. In testing, the 3D shape of an object can be reconstructed from the given image by deforming the 3D shape identity according to the 3D shape displacement. To achieve this goal, we propose a novel convolutional encoder-decoder network—the Disentangling Deep Network (DisDN). To demonstrate the effectiveness of the proposed approach, we implement comprehensive experiments on the challenging PASCAL VOC benchmark and use different 3D shape ground-truth in training and testing to avoiding overfitting. The obtained experimental results show that DisDN outperforms other state-of-the-art and baseline methods.

Projector defocusing technique can eliminate projector nonlinearity in real-time three-dimensional shape measurement. However, the high-frequency harmonics brought by defocusing technique weakens the sinusoidal feature of the defocused fringe patterns, which will bring obvious measurement errors. The dithering algorithm is proposed to generate binary fringe, the projector is defocused properly to achieve approximate sinusoidal fringes. This method improves the sinusoidal property of the defocused fringe patterns. Absolute phase is retrieved by applying this method and phase-shifting algorithm. Simulations and experiments are carried out to verify the proposed method, and results show that phase quality has been improved greatly. Compared with existing methods, the proposed method achieves higher precision and can generate fringe patterns more sinusoidal, which is more proper for projector defocusing technique.

Light guide plate (LGP) is one of the essential components of a back light module (BLM). Warpage may be produced during the manufacturing process due to the lower structural strength of the LGP in a thinner thin film transistor liquid crystal display (TFT-LCD) as required by the market. Basically, the fringe pattern produced by the digital phase-shifting shadow moire (DPSSM) method is generated from interference between the reference grating and its projected grating on the test surface. For a transparent object such as a LGP, reflective painting is generally sprayed on the object’s surface to make the projected grating sufficiently visible. However, the LGP may no longer be re-used or may be damaged during the process of painting or de-painting. To investigate the feasibility of obtaining the surface topography of a transparent object by employing the DPSSM method without painting, fringe patterns generated from different capture angles with the same incident angle were examined. With the help of the image correction program, it was found that skewed images obtained from different capture angles can be successfully recovered. The surface topography of a LGP can then be correctly determined.

This paper introduces a novel 3D measurement scheme based on UV laser triangulation to ascertain the shape of transparent objects. Transparent objects are extremely difficult to scan with traditional 3D scanners because of the refraction problem observed in the visible range. Therefore, the object surface needs to be preliminary powdered before being digitized with commercial scanners. Our approach consists of using non contact measurement scheme while dealing with the refraction problem in visible environment. The object shape is computed by classical triangulation method based on stereovision constraint. The proposed acquisition system is composed of two classical visible range cameras and a UV laser source. The exploitation of the UV laser for triangulation system characterizes the novelty of the proposed approach. The fluorescence generated by the UV radiation enables to acquire 3D data of transparent surface with a classical stereovision scheme.

‘Ghosted’ fringe patterns simultaneously reflected from both the upper and lower sides of a transparent target in the fringe reflection technique are captured for transparent surface 3D shape measurement, but the phase retrieval from the captured ‘ghosted’ fringe patterns is still not solved. A novel method is proposed to solve this issue by using two sets of phase-shifted fringe patterns with slightly different frequencies. The nonlinear least-squares method is used to estimate the fringe phase and modulation from both front and rear interfaces. Several simulations are done to show the feasibility of the proposed method. The influence of fringe noise on the algorithm is studied as well, which indicates that the proposed method is able to retrieve the phase from double-sided reflective fringe patterns with fringe noise equivalent to that in practical measurements. The merits and limitations of the method are discussed and recommendations for future studies are made.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, the transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated, and therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

The scanning of 3D geometry has become a popular way of capturing the shape of real-world objects. Transparent objects, however, pose problems for traditional scanning methods. We present a visible light tomographic reconstruction method for recovering the shape of transparent objects, such as glass. Our setup is relatively simple to implement, and accounts for refraction, which can be a significant problem in visible light tomography.