3D Shape Recovery with Registration Assisted Stereo Matching

In this paper, a new 3D recognition method for intelligent assembly system is presented. In this method neural network technology is used to provide new methodologies for solving difficult computational problems in 3D recognition processes. The method can be divided into two parts. In the first part, phase based stereo matching techniques are used to find the correspondence between left and right image in stereo image pair. The Hopfield neural network is established, so that the computation can be implemented efficiently in parallel. A 3D object reconstruction neural network is constructed by using BP neural network. With the results of stereo matching, the 3D configuration and shape can be reconstructed. In the second part, the feature vector of 3D object is constructed by using 3D moment and its invariant. With the results obtained in first parts, ART2 neural network is adopted for neural network classifier. With the ART2 neural network classifier, the 3D objects can be recognized and classified. The method is tested with both synthetic and real parts in intelligent assembly system. Good results are obtained. It is proved through the experiments and actual applications that the method presented in this paper is correct and reliable. It is very suitable for intelligent assembly system.

Over the last two decades, research community of computer vision has developed various techniques suitable for underwater applications using intensity images. This chapter will explore 3D reconstruction of underwater natural scenes and objects based on stereo vision, which will be helpful in mine detection, inspection of shipwrecks, detection of telecommunication cables and pipelines. The general steps involved in 3D reconstruction using stereo vision are provided. The brief summary of papers for 3D reconstruction of underwater environment is presented. 3D reconstruction of underwater natural scenes and objects is challenging problem due to light propagation in underwater. In contrast to light propagation in the air, the light rays are attenuated and scattered, having a great effect on image quality. We have proposed preprocessing technique to enhance degraded underwater images. At the end of the chapter, we have presented the proposed stereo vision based 3D reconstruction technique to reconstruct 3D surface of underwater objects. Ultimately, this chapter intends to give an overview of the 3D reconstruction technique using stereo vision in order to help a reader in understanding stereo vision and its benefits for underwater applications.

This paper proposes a novel real-time stereo matching algorithm paired with an online auto-rectification framework to solve environment sensing problems in autonomous driving, such as road, curb and small object detection etc. The algorithm treats disparities of stereo images as hidden states and conducts a Viterbi process at 4 bi-directional paths to estimate them. Structural similarity, total variation constraint, and a specific hierarchical merging strategy are combined with the Viterbi process to improve the robustness and accuracy. Based on the results of Viterbi, a convex optimization equation is derived to estimate epipolar line distortion. The estimated distortion information is used for the online compensation of Viterbi process at an auto-rectification framework. The stereo matching result is used to generate the histogram map. A Radon transform and a Viterbi process are applied to the histogram map to detect roads and curbs. Disparities near the road are mapped to 3D space to detect small objects on road. Real-word experiments show that this method can help sensing the surrounding 3D environment for driving vehicles robustly.

Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding

Time-of-Flight (ToF) sensors and stereo vision systems are both widely used for capturing depth data. They have some complementary strengths and limitations, which have been exploited in prior research to produce more accurate depth maps by fusing data from the two sources. However, among these diverse data fusion approaches, none of them provides an end-to-end neural network solution. In this work, we propose the first end-to-end ToF and stereo data fusion network using the coarse-to-fine matching framework, where the prior of ToF depth is integrated into the stereo matching process by constraining the search range of stereo matching within an interval around the ToF camera depth measurement. We adopt a dynamic search range for each pixel according to an estimated ToF error map, which is more efficient and effective than a constant one when handling various errors. The ToF error map is estimated by the ToF error estimator branching out from the stereo matching network. Both ToF error estimation and stereo matching are performed in a joint framework, with the two tasks assisting each other mutually. We also propose an upsampling module to replace the naive bilinear upsampling in the coarse-to-fine stereo matching network, which reduces the error caused by the upsampling. The proposed deep network is trained end-to-end on synthetic datasets and generalizable to real-world datasets without further fine-tuning. Experimental results show that our fusion method achieves higher accuracy than either ToF or stereo alone, and outperforms state-of-the-art fusion methods on both synthetic and real data.

In order to obtain the phenotypic parameters of apple quickly and accurately, which were commonly used as the basis of fruit sorting, a fast estimation method of apple phenotypic parameters based on three-dimensional (3D) reconstruction was proposed in this study. In this study, a three-dimensional model was constructed to estimate the phenotypic parameters of apple, such as volume, height, diameter, and fruit shape index. Firstly, an image acquisition system was built to capture sequence images of fruit with a binocular stereo vision system, and the images were extracted and matched using the Accelerated-KAZE algorithm to create the point cloud data. Secondly, the point cloud data were matched with the algorithm of Iterative Closest Point to establish a whole model of apple, and the surface reconstruction model of fruit was obtained by constructing irregular triangulation network. Finally, the apple phenotypic parameters were calculated by means of segmentation, surface complement and integral of the fruit model. Total of 200 apples were used as samples in the experiment. By this method, the phenotypic parameters of the apples were estimated based on their 3D reconstruction model, and the linear regression analysis was carried out between the estimated values and the real values. The results showed that R2 of the linear regression fitting of each parameter was higher than 0.90. Among them, the fitting of volume was the best with R2 of 0.97. In addition, the average errors of apple volume, height, fruit shape index, maximum diameter D and minimum diameter d were 8.73 cm3, 1.43 mm, 1.28%, 0.90 mm, and 1.23 mm, respectively. According to the Chinese national standard of “fresh Apple”, the average error of the estimated result is within the range of allowable error. It indicates that the method of apple phenotypic parameter estimation based on 3D reconstruction has a high accuracy and practicability, and it can be used as the support for fruit sorting. Keywords: apple, 3D reconstruction, phenotypic parameter, stereo vision system, sequence image, sorting DOI: 10.25165/j.ijabe.20211405.6258 Citation: Ma H, Zhu X, Ji J T, Wang H, Jin X, Zhao K X. Rapid estimation of apple phenotypic parameters based on 3D reconstruction. Int J Agric & Biol Eng, 2021; 14(5): 180–188.

This work deals with the automatic 3D reconstruction of objects from frontal RGB images. This aims at a better understanding of the reconstruction of 3D objects from RGB images and their use in immersive virtual environments. We propose a complete workflow that can be easily adapted to almost any other family of rigid objects. To explain and validate our method, we focus on guitars. First, we detect and segment the guitars present in the image using semantic segmentation methods based on convolutional neural networks. In a second step, we perform the final 3D reconstruction of the guitar by warping the rendered depth maps of a fitted 3D template in 2D image space to match the input silhouette. We validated our method by obtaining guitar reconstructions from real input images and renders of all guitar models available in the ShapeNet database. Numerical results for different object families were obtained by computing standard mesh evaluation metrics such as Intersection over Union, Chamfer Distance, and the F-score. The results of this study show that our method can automatically generate high-quality 3D object reconstructions from frontal images using various segmentation and 3D reconstruction techniques.

Warping-based spectral translation network for unsupervised cross-spectral stereo matching

Binocular stereo vision is an important branch of the research area in computer vision. Stereo matching is the most important process in binocular vision. In this paper, a new stereo matching scheme using shape-based matching (SBM) is presented to improve the depth reconstruction method of binocular stereo vision systems. The method works in two steps. First, an operator registers the pattern including the key features of an object to be measured. Then during the operation stage, the stereo camera snaps stereo images and finds the patterns in right and left images separately by means of the SBM. The 3D positions of the object are calculated by using the corresponding points of the stereo images and the projection matrices of the stereo camera. Since we apply robust image processing algorithms, such as the SBM, the proposed method becomes more reliable than the conventional stereo vision systems.

A stereo vision system provides important support for underwater robots to achieve autonomous navigation, obstacle avoidance, and precise operation in complex underwater environments. This article proposes an unsupervised underwater stereo matching method based on semantic attention. By combining deep learning and semantic information, it fills the challenge of insufficient training data, enhances the intelligence level of underwater robots, and promotes the progress of underwater scientific research and marine resource development. This article proposes an underwater unsupervised stereo matching method based on semantic attention, targeting the missing training supervised dataset for underwater stereo matching. An adaptive double quadtree semantic attention model for the initial estimation of semantic disparity is designed, and an unsupervised AWLED semantic loss function is proposed, which is more robust to noise and textureless regions. Through quantitative and qualitative evaluations in the underwater stereo matching dataset, it was found that D1 all decreased by 0.222, EPE decreased by 2.57, 3px error decreased by 1.53, and the runtime decreased by 7 ms. This article obtained advanced results.

In this paper, the reconstruction of faces in the form of a 3D model from unconstrained images (different in pose) has been studied. Three-dimensional Morphable Model(3DMM) Fitting method has been used in 3D reconstruction techniques. In this paper propose improvement of the 3DMM fitting procedure to get accurate 3D model by taking best 2D landmark of all images rather than taking only first 2D image landmark (traditional method which proposed by A.Bas et al). The results of the proposed algorithm show its very encouraging as far as execution time and quality of reconstruction as shown in compare the improvement fitting process with the traditional method.

Current surgical planning relies on conventional two-dimensional (2D) imaging, but this is limited by its inability to fully represent three-dimensional (3D) anatomical relationships. This limitation is particularly evident in thoracic surgery, where accurate visualization of the pulmonary bronchovascular anatomy is essential for planning lung resections in a surgical field that has high anatomical variations and dynamic deformation during surgery. Although 3D reconstruction techniques have been explored to address these limitations, their adoption into routine clinical practice has been constrained by reliance on expensive proprietary software and a lack of detailed, efficient, reproducible workflows. Here, we describe a novel, practical imaging-based approach for generating patient-specific 3D reconstructions of pulmonary anatomy from computed tomography (CT) data using 3D Slicer, an open-source platform for imaging segmentation and 3D modelling. A reproducible step-by-step workflow is presented, detailing segmentation of pulmonary arteries, veins, bronchi, tumour localization, and resection margin assessment, together with common pitfalls, troubleshooting strategies, and factors influencing accuracy and efficiency. The workflow is designed to be intuitive and time-feasible, with model generation achievable within routine pre-operative planning timeframes. The process is demonstrated through two representative cases in which 3D reconstruction altered surgical planning by clarifying ambiguous anatomy, enabling precise pulmonary tumour localization and accurate resection. With the use of 3D reconstruction technology, anatomical understanding and operative precision can be enhanced, supporting more informed surgical decision-making and potentially improving patient outcomes.

The utilization of customized three-dimentional (3D) models based on patient's computed tomography (CT) scan data and by assistance of additive manufacturing/rapid prototyping (AM/RP) techniques for 3D reconstruction is one of the applicable trends for reducing the errors and time saving during surgeries. In the present study, the methodology of the fabrication of a custom-made 3D model based on converting CT scan data to standard triangle language (STL) format for a 33-years old male patient who was suffering from an accident trauma was described. The 3D model of the patient’s skull was fabricated and applied in preoperative planning. It was used for designing a comprehensive plan for rehabilitation of the damaged orbit to restore the appearance and bone reconstruction of the patient. Before fabricating the model, the accuracy of protocols used in converting CT scan data into STL file was evaluated. Then, the model was fabricated by a fused deposition modeling (FDM) machine. Using this procedure led to a maximum of 1.4% difference between the virtual model in the software and the fabricated 3D model in the fracture site. The present technique reduced operation time significantly. In addition, following eight months from the operation, the treatment approach ensured the patient's fractures healing process.

Fractal generated turbulence is highly anisotropy and inhomogeneous, particularly, within its unique turbulence production regime. This study aims to develop an experimental-based detection method which allows the details of fluid-structure-interaction (FSI) via grid-net interplays to better express the strength of fractal grid induced flow dynamics. Hence, a novel approach is initiated to perform nets photogrammetry for 3D-reconstruction through an innovative means that permits the historically difficult task of reconstructing nets to be achieved. A keypoint extraction algorithm designed specifically for nets dubbed Neighborhood OCCupancy (NOCC), and a complementary stereo matching algorithm named Propagative Stereo Matching (PSM) are presented. The former identifies the center of intersections in the net by determining the percentage of an ellipse filled at each pixel. PSM works on the NOCC keypoints via matching pairs close to a known match and using the newly matched pairs as reference for subsequent matches in a way reminiscence to how humans would perform the stereo matching task manually. The effectiveness, accuracy and robustness of the detection are carefully validated using three different net geometries, which are individually subjected to three synthetic configurations. Consequently, the system repeatability, utilisation, stereo matching accuracy and reconstruction accuracy, of which, with the mean of all cases outstandingly deduced to 99.0%, 87.0%, 100% and 0.12mm, respectively. The presented method is then employed to elucidate the windward flow straightener influence on grid-net fluid flow interplays. Overall, the current framework may be an enabling factor in providing critical FSI-features to realize the mechanical strength of fractal generated turbulence.

BackgroundTwo-dimensional, unidirectionally encoded, cardiovascular magnetic resonance (CMR) velocity mapping is an established technique for the quantification of blood flow in large vessels. However, it requires an operator to correctly align the planes of acquisition. If all three directional components of velocity are measured for each voxel of a 3D volume through the phases of the cardiac cycle, blood flow through any chosen plane can potentially be calculated retrospectively. The initial acquisition is then more time consuming but relatively operator independent.AimsTo compare the curves and volumes of flow derived from conventional 2D and comprehensive 3D flow acquisitions in a steady state flow model, and in vivo through planes transecting the ascending aorta and pulmonary trunk in 10 healthy volunteers.MethodsUsing a 1.5 T Phillips Intera CMR system, 3D acquisitions used an anisotropic 3D segmented k-space phase contrast gradient echo sequence with a short EPI readout, with prospective ECG and diaphragm navigator gating. The 2D acquisitions used segmented k-space phase contrast with prospective ECG and diaphragm navigator gating. Quantitative flow analyses were performed retrospectively with dedicated software for both the in vivo and in vitro acquisitions.ResultsAnalysis of in vitro data found the 3D technique to have overestimated the continuous flow rate by approximately 5% across the entire applied flow range. In vivo, the 2D and the 3D techniques yielded similar volumetric flow curves and measurements. Aortic flow: (mean ± SD), 2D = 89.5 ± 13.5 ml & 3D = 92.7 ± 17.5 ml. Pulmonary flow: 2D = 98.8 ± 18.4 ml & 3D = 94.9 ± 19.0 ml). Each in vivo 3D acquisition took about 8 minutes or more.ConclusionFlow measurements derived from the 3D and 2D acquisitions were comparable. Although time consuming, comprehensive 3D velocity acquisition could be relatively operator independent, and could potentially yield information on flow through several retrospectively chosen planes, for example in patients with congenital or valvular heart disease.