Observing Optically Challenging Objects with Structured Light

Recovering 3D shapes of deformable objects from single 2D images is an extremely challenging and ill-posed problem. Most existing approaches are based on structure-from-motion or graph inference, where a 3D shape is solved by fitting 2D keypoints/mask instead of directly using the vital cue in the original 2D image. These methods usually require multiple views of an object instance and rely on accurate labeling, detection, and matching of 2D keypoints/mask across multiple images. To overcome these limitations, we make effort to reconstruct 3D deformable object shapes directly from the given unconstrained 2D images. In training, instead of using multiple images per object instance, our approach relaxes the constraint to use images from the same object category (with one 2D image per object instance). The key is to disentangle the category-specific representation of the 3D shape identity and the instance-specific representation of the 3D shape displacement from the 2D training images. In testing, the 3D shape of an object can be reconstructed from the given image by deforming the 3D shape identity according to the 3D shape displacement. To achieve this goal, we propose a novel convolutional encoder-decoder network—the Disentangling Deep Network (DisDN). To demonstrate the effectiveness of the proposed approach, we implement comprehensive experiments on the challenging PASCAL VOC benchmark and use different 3D shape ground-truth in training and testing to avoiding overfitting. The obtained experimental results show that DisDN outperforms other state-of-the-art and baseline methods.

Phase shifting profilometry is an important technique for reconstructing the three-dimensional (3D) geometry of objects with purely diffuse surfaces. However, it is challenging to measure the transparent objects due to the pattern aliasing caused by light refraction and multiple reflections inside the object. In this work, we analyze the aliasing fringe pattern formation for transparent objects and then, propose to learn the front surface light intensity distribution based on the formation principle by using the diffusion models for generating the non-aliased fringe patterns reflected from the front surface only. With the generated fringe patterns, the 3D shape of the transparent objects can be reconstructed via the conventional structured light. We show the feasibility and performance of the proposed method on the data of purely transparent objects that are not seen in the training stage. Moreover, we found it could be generalized to other cases with local-transparent and translucent objects, showing the potential capability of the diffusion based learnable framework in tackling the problems of transparent object reconstruction.

Anterior Regions of Monkey Parietal Cortex Process Visual 3D Shape

This paper proposes a technique for recovering 3D shape of remote non-rigid objects employing mobile cameras. Since the proposed technique allows independent move of the employed cameras, 3D recovery under various environments such as remote places can be realized. Our technique has an advantage over others in that camera calibration before taking images is not prerequisite to the recovery. A system is described that captures images of remote objects by the proposed motion capture technique, transfers the images by analog airwaves, and recovers 3D shape of the objects from the images. It is expected that this technique will improve the efficiency of image information transfer. In the performed experiment, a human walking motion captured at a remote place recovered its 3D motion successfully at a laboratory by image transfer.

Neural computations underlying depth perception

Multispectral imaging and photometric stereo are common in 3D imaging but rarely have been combined. Reconstructing a 3D object's shape using photometric stereo is challenging owing to indirect illumination, specular reflection, and self-shadows, and removing interreflection in photometric stereo is problematic. A new multispectral photometric-stereo method removes interreflection on diffuse materials using multispectral-reflectance information and reconstructs 3D shapes with high accuracy. You can integrate this method into photometric-stereo systems by simply substituting the original camera with a multispectral camera.

The measurement of transparent objects has always been regarded as one of the difficult problems in 3D reconstruction technology. For transparent objects conforming to the birefringence model, we propose a method for measuring the parameters of transparent objects based on multi-wavelength incident light. This paper analyzes the transparent objects where light enters the birefringence model. Because the transparent objects refract light of different wavelengths differently, when the lights with different wavelengths emitted from the same point and same direction passes through the transparent objects, the position and direction of light with different wavelength will come out differently. The position and direction of these outgoing rays are related to the refractive index and surface topography of the transparent objects. By obtaining the equations of these outgoing rays, we can derive the equations of the refractive index and surface topography of the transparent objects. In this paper, the incident light obtained by computer simulation is used as the initial data, and the simulated annealing algorithm is used to verify the model. The final results show that within the range of random error values given by the computer, the surface shape, curvature, thickness and refractive index of transparent objects with small error values can be obtained. At the same time, this paper introduces the Ceres Solver C++ nonlinear optimization library into the simulated annealing algorithm, which improves the speed and accuracy of the algorithm and greatly enhances the robustness of the algorithm.

In this paper, a practical method using phase tracking algorithm and ray tracing algorithm is proposed for measuring the 3D shape of an underwater object. A sinusoidal fringe is projected through the water to modulate the information of the tested object's 3D shape. Firstly, according to the binocular stereovision principle and regarding the projector as a special camera, the phase tracking algorithm is taken to identify the homologous points in the phase distribution of the deformed fringe captured by the camera and that of the fringe pattern projected by the projector. The relationship between those two coordinate systems will be fixed on. Secondly, the ray tracing algorithm is used to trace the propagation path of each ray and define the 3D coordinates of each point on the tested object's surface. Finally, the whole shape of the tested object can be reconstructed. Without any scanning device, the method proposed in this paper can speed up the measurement process and reduce the cost. An actual measurement for a model and a hand in the water was done, the prefect results of the reconstructed shapes prove this method correctly and feasibility.

Understanding 3D object shapes necessitates shape representation by object parts abstracted from results of instance and semantic segmentation. Promising shape representations enable computers to interpret a shape with meaningful parts and identify their repeatability. However, supervised shape representations depend on costly annotation efforts, while current unsupervised methods work under strong semantic priors and involve multi-stage training, thereby limiting their generalization and deployment in shape reasoning and understanding. Driven by the tendency of high-dimensional semantically similar features to lie in or near low-dimensional subspaces, we introduce a one-stage, fully unsupervised framework towards semantic-aware shape representation. This framework produces joint instance segmentation, semantic segmentation, and shape abstraction through sparse representation and feature alignment of object parts in a high-dimensional space. For sparse representation, we devise a sparse latent membership pursuit method that models each object part feature as a sparse convex combination of point features at either the semantic or instance level, promoting part features in the same subspace to exhibit similar semantics. For feature alignment, we customize an attention-based strategy in the feature space to align instance- and semantic-level object part features and reconstruct the input shape using both of them, ensuring geometric reusability and semantic consistency of object parts. To firm up semantic disambiguation, we construct cascade unfrozen learning on geometric parameters of object parts. Experiments conducted on benchmark datasets confirm that our approach results in instance- and semantic-level joint segmentation and shape abstraction with repeatable primitives, providing coherent semantic interpretations of 3D object shapes across categories in a one-stage, fully unsupervised manner, without relying on annotations or heuristic semantic priors.

There are various structures in the underground space, and the structures are managed as 3D data in underground geospatial information map system (UGIMS). However, there is a problem that takes a long time to load the data into a mobile app from UGIMS because the data are very large. In order to reduce the loading time of 3D data, the number of data vertices should be reduced within a range that can keep the shape of 3D object in a mobile app. In this paper, we try to reduce the number of vertices using a 3D Mesh Simplification method to reduce the loading time of data in a mobile app. The proposed method applies simplification by extracting refined vertex feature information using a deep learning encoder‐decoder based model and performing vertex clustering by grouping similar vertices together with the feature information using K‐means clustering. We were able to reduce the number of vertices by 30% compared with the previous data. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Estimating the complete 3D point cloud from an incomplete one lies at the core of many vision and robotics applications. Existing methods typically predict the complete point cloud based on the global shape representation extracted from the incomplete input. Although they could predict the overall shape of 3D objects, they are incapable of generating structure details of objects. Moreover, the partial input point sets obtained from range scans are often sparse, noisy and non-uniform, which largely hinder shape completion. In this paper, we propose an adaptive sampling and hierarchical folding network (ASHF-Net) for robust 3D point cloud completion. Our main contributions are two-fold. First, we propose a denoising auto-encoder with an adaptive sampling module, aiming at learning robust local region features that are insensitive to noise. Second, we propose a hierarchical folding decoder with the gated skip-attention and multi-resolution completion goal to effectively exploit the local structure details of partial inputs. We also design a KL regularization term to evenly distribute the generated points. Extensive experiments demonstrate that our method outperforms existing state-of-the-art methods on multiple 3D point cloud completion benchmarks.

The authors propose an object shape detection system using a force/torque sensor and a flexible/inflexible insensitive probe. A tactile sensing method which makes it possible to estimate a contact position on the probe only by measuring contact force has been developed. Though the probe does not have any sensitive devices. the method derives the contact position from force information measured by the force/torque sensor. When the probe bends according to the applied force, the method compensates for the probe deflection by estimating its transformation based on force/torque sensor information. It has been applied to a manipulator system, and object shape detection experiments have been conducted using a cylindrical probe. Shapes of 3D objects have successfully been reproduced.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

We present a technique for performing three-dimensional (3D) pattern recognition by use of in-line digital holography. The complex amplitude distribution generated by a 3D object at an arbitrary plane located in the Fresnel diffraction region is recorded by phase-shifting interferometry. The digital hologram contains information about the 3D object's shape, location, and orientation. This information allows us to perform 3D pattern-recognition techniques with high discrimination and to measure 3D orientation changes. Experimental results are presented.

A favoured sensor for mapping is a 3D laser scanner since it allows a wide scanning range, precise measurements, and is usable indoor and outdoor. Hence, a mapping module delivers detailed and high resolution maps which makes it possible to navigate safely. Difficulties result from transparent and specular reflective objects which cause erroneous and dubious measurements. At such objects, based on the incident angle, measurements result from the object surface, an object behind the transparent surface, or an object mirrored with respect to the reflective surface. This paper describes an enhanced Pre-Filter-Module to distinguish between these cases. Two experiments demonstrate the usability and show that for single scans the identification of mentioned objects in 3D is possible. The first experiment was made in an empty room with a mirror. The second experiment was made in a stairway which contains a glass door. Further, results show that a discrimination between a specular reflective and a transparent object is possible. Especially for transparent objects the detected size is restricted to the incident angle. That is why future work concentrates on implementing a post-filter module. Gained experience shows that collecting the data of multiple scans and postprocess them as soon as the object was bypassed will improve the map.

3D shape determines an object's physical properties to a large degree. In this article, we introduce an autonomous learning system for categorizing 3D shape of simulated objects from single views. The system extends an unsupervised bottom-up learning architecture based on the slowness principle with top-down information derived from the physical behavior of objects. The unsupervised bottom-up learning leads to pose invariant representations. Shape specificity is then integrated as top-down information from the movement trajectories of the objects. As a result, the system can categorize 3D object shape from a single static object view without supervised postprocessing.