Pose estimation and integration for complete 3D model reconstruction

We propose a fast and simple application system of 3D model reconstruction. We acquire range images by using a combination of a regular camera and two types of depth sensors. The reconstruction of a 3D model consists of four key steps: (i) Initial alignment either feature tracking or the 4-points congruent sets algorithm is used to align surfaces captured at different frames. (ii)The iterative closest point (ICP) method is applied to further align the piecewise surfaces from the last step. (iii) The surfaces are merged into a whole 3D model by the volumetric method. (iv) In the refinement step, we fill holes and produce a complete 3D model that approximates the original model with robust repair of polygonal models. At last, we present the experimental results which show that the errors between our reconstructed model and the ground truth are less than 1%.

A new technique is proposed for calibrating a 3D modeling system with variable zoom based on multi-view stereo image analysis. The 3D modeling system uses a stereo camera with variable zoom setting and a turntable for rotating an object. Given an object whose complete 3D model (mesh and texture-map) needs to be generated, the object is placed on the turntable and stereo images of the object are captured from multiple views by rotating the turntable. Partial 3D models generated from different views are integrated to obtain a complete 3D model of the object. Changing the zoom to accommodate objects of different sizes and at different distances from the stereo camera changes several internal camera parameters such as focal length and image center. Also, the parameters of the rotation axis of the turntable changes. We present camera calibration techniques for estimating the camera parameters and the rotation axis for different zoom settings. The Perspective Projection Matrices (PPM) of the cameras are calibrated at a selected set of zoom settings. The PPM is decomposed into intrinsic parameters, orientation angles, and translation vectors. Camera parameters at an arbitrary intermediate zoom setting are estimated from the nearest calibrated zoom positions through interpolation. A performance evaluation of this technique is presented with experimental results. We also present a refinement technique for stereo rectification that improves partial shape recovery. And the rotation axis of multi-view at different zoom setting is estimated without further calibration. Complete 3D models obtained with our techniques are presented.

3D human face model reconstruction is essential to the generation of facial animations that is widely used in the field of virtual reality (VR). The main issues of 3D facial model reconstruction based on images by vision technologies are in twofold: one is to select and match the corresponding features of face from two images with minimal interaction and the other is to generate the realistic-looking human face model. In this paper, a new algorithm for realistic-looking face reconstruction is presented based on stereo vision. Firstly, a pattern is printed and attached to a planar surface for camera calibration, and corners generation and corners matching between two images are performed by integrating modified image pyramid Lucas-Kanade (PLK) algorithm and local adjustment algorithm, and then 3D coordinates of corners are obtained by 3D reconstruction. Individual face model is generated by the deformation of general 3D model and interpolation of the features. Finally, realistic-looking human face model is obtained after texture mapping and eyes modeling. In addition, some application examples in the field of VR are given. Experimental result shows that the proposed algorithm is robust and the 3D model is photo-realistic.

The digitization of the complete shape of real objects has essential applications in fields of intelligent manufacturing, industrial detection, and reverse modeling. In order to build the full geometric models of rigid objects, the object must be moved relative to the measurement system (or the scanner must be moved relative to the object) to obtain and integrate views of the object from all sides, which not only complicates the system configuration but makes the whole process time-consuming. In this Letter, we present a high-resolution real-time 360° three-dimensional (3D) model reconstruction method that allows one to rotate an object manually and see a continuously updated 3D model during the scanning process. A multi-view fringe projection profilometry system acquires high-precision depth information about a handheld object from different perspectives and, meanwhile, the multiple views are aligned and merged together in real time. Our system employs stereo phase unwrapping and an adaptive depth constraint that can unwrap the phase of dense fringe images robustly without increasing the number of captured patterns. We then develop an efficient coarse-to-fine registration strategy to match the 3D surface segments rapidly. Our experiments demonstrate that our method can reconstruct the high-precision complete 3D model of complex objects under arbitrary rotation without any instrument assist and expensive pre/post-processing.

Three-dimensional (3D) modeling of non-linear objects from stylized sketches is a challenge even for computer graphics experts. The extrapolation of object parameters from a stylized sketch is a very complex and cumbersome task. In the present study, we propose a broker system that can transform a stylized sketch of a tree into a complete 3D model by mediating between a modeler and a 3D modeling software. The input sketches do not need to be accurate or detailed: They must only contain a rudimentary outline of the tree that the modeler wishes to 3D model. Our approach is based on a well-defined Deep Neural Network architecture, called TreeSketchNet (TSN), based on convolutions and capable of generating Weber and Penn [1995] parameters from a simple sketch of a tree. These parameters are then interpreted by the modeling software, which generates the 3D model of the tree pictured in the sketch. The training dataset consists of synthetically generated sketches that are associated with Weber–Penn parameters, generated by a dedicated Blender modeling software add-on. The accuracy of the proposed method is demonstrated by testing the TSN with synthetic and hand-made sketches. Finally, we provide a qualitative analysis of our results, by evaluating the coherence of the predicted parameters with several distinguishing features.

Multi-sensor point cloud data fusion for precise 3D mapping

Three-dimensional (3D) registration or matching is a crucial step in 3D model reconstruction. In this work, we develop a real-time 3D point cloud registration technology. Firstly, in order to achieve real-time 3D data acquisition, the stereo phase unwrapping method is utilized to eliminate the ambiguity of the wrapped phase, assisted with the depth constraint strategy without projecting any additional patterns or embedding any auxiliary signals. Then we implement SLAM-based coarse registration and ICP-based fine registration to match the point cloud data after the rapid identification of two-dimensional (2D) feature points. In order to improve the efficiency of 3D registration, the relative motion of the measured object at each coarse registration is quantified, through which only one fine registration is performed after several coarse registrations. The experiment shows that, the complex model can be registered in real time to reconstruct its whole 3D model with our method.

Image-based 3D shape reconstruction is essential for 3D modeling in recognition, virtual reality, generation of video games, 3D animation, and in the creation of 3D virtual fitting rooms - the sought-after application in this dissertation. In this work, we propose a novel reconstruction method that uses a generic 3D model that roughly controls the overall reconstruction shape which in turn, reduces the number of 2D images needed to just two (frontal and profile) for obtaining a 3D point model. The generic model consists of a massive set of 3D points that lack saliency, topological meaning, and relational interconnections. This is also true about points in the two canonical images. In this work, we propose finding a small set of interconnected ordered intrinsic salient points (control points) residing on the silhouette of the projections of the generic model onto the two image spaces, as well as finding those for the canonical images. Points on the generic model and the images are automatically given saliency with order and interconnections through loop-subdivision operating on those control points and their equivalent points on the canonical images. The transformation of the generic model into the specific object is done through morphing the points on the generic model to follow and be consistent with their equivalent points on the canonical images. The control points on the canonical images are automatically obtained using an active shape model (ASM) that modifies an average trained control point set attained using an offline training, to account for class shape variability and boundary regularization guaranteeing the control points to reside on the silhouette (boundary) of the individual. The reconstructed 3D model is furtherly refined through smoothing the reconstructed surface, while fixing local holes using a principle component analysis (PCA) decomposition. This reconstruction method is convenient, simple, efficient, and results in an average geometric error of less than 0.5% over 700 diverse body models from the CAESAR dataset. We then implement the contour-based articulated model to the personalized 3D model for pose recovery. The evolution of the model through the articulation process is captured in a video clip. 3D transformations are found and applied to the 3D model on its parts so that the projections of the 3D model 'match' those observed in the video sequence at corresponding frames. This is done by minimizing the error between the frontal projection body region points and the target points from the image for each independent moving part, with sub-resolution recovery errors. Once the personalized 3D model is obtained and the articulated model is implemented, the algorithm simulates how garments virtually appear on the reconstructed person under different poses. Different types of clothes are classified as tight or loose, where for the first type of garments, the garment is mapped onto the person's body, whereas for the second type, the mapping adjusts to accommodate the loose nature of the garment, and its fitting on the body of the reconstructed person model. This creates a virtual in-home fitting room for garment fitting, which is timely and is a needed application that would go hand in hand with online clothes shopping, allowing the trying of garments at home with changing poses before ordering them, and can be updated as often as necessary to accommodate changes in physique, weight, aging, etc. This is also done with minimum requirements from the user aside from taking two images and a video of pose articulations using their cam or smart phone, and stands as a useful, needed, and attractive App.

Objective To discuss the digital technique used in visualization of arteria saphena flaps and its clinical applications.Methods Two adult fresh cadaveric specimens,one male and one female,were subjected to radiographic CT scanning before and after perfusion with lead oxide-gelatine mixture,with collimation of 0.625 mm (120 kV,110 mA,512 ×512 matrix).Through Amira 5.4.1 software,the 2D CT images in.dicom format were transformed into 3D models of the entire region.Structures of the arteria saphena were observed and visible digital models of the arteria saphena flap were established through 3D computerized reconstruction by Amira 5.4.1 software.Next 6 patients with soft-tissue defects and bone exposure at the tibial region underwent contrast-enhanced CT angiography of the lower limb by a 64-row multi-slice spiral CT machine after median cubital vein injection with Ultravist (3.5 mL/s).The 2D CT images in.dicom format of the patients were transformed into individualized 3D reconstruction models of the arteria saphena flap using Mimics 14.0 software.The digital design of the individualized arteria saphena flap was used in clinic.Results All the 6 patients were followed up for an average of 18 months (from 8 to 24 months).The 3D reconstruction visible models established from the CT datasets perfectly displayed anatomy of the arteria saphena flap.In the 6 cases the main trunk and branches of the blood vessels in the designed flap were consistent with the surgical findings.The average distance from the starting point of the arteria saphena to the knee clearance was 119.2 ± 9.6 mm and the average diameter of the arteria saphena at the starting point was 1.5 ± 0.3 mm.The areas of flap ranged from 8.0 cm × 5.0 cm to 20.0 cm × 8.0 cm.All flaps survived well.The knee flexion was 120° to 140° and the knee extension 0° to 10°.Conclusion The 3D reconstruction models can provide perfect dynamic visualization of the anatomy of the arteria saphena which can be successfully used in clinic in individualized design of the arteria saphena flap to be transplanted for a specific patient. Key words: Imaging, three-dimensional; Models, anatomic; Surgical flaps

One might wonder what can be gained from the image-laser fusion and in which measuresuch a hybrid system can generate automatically complete and photorealist 3D models ofdifficult to access and unstructured underground environments.Our research work is focused on developing a vision-based system aimed at automaticallygenerating in-situ photorealist 3D models in previously unknown and unstructuredunderground environments from image and laser data. In particular we are interested inmodeling underground prehistoric caves. In such environments, special attention must begiven to the main issue standing behind the automation of the 3D modeling pipeline whichis represented by the capacity to match reliably image and laser data in GPS-denied andfeature-less areas. In addition, time and in-situ access constraints require fast and automaticprocedures for in-situ data acquisition, processing and interpretation in order to allow forin-situ verification of the 3D scene model completeness. Finally, the currently generated 3Dmodel represents the only available information providing situational awareness based onwhich autonomous behavior must be built in order to enable the system to act intelligentlyon-the-fly and explore the environment to ensure the 3D scene model completeness.This chapter evaluates the potential of a hybrid image-laser system for generating in-situcomplete and photorealist 3D models of challenging environments, while minimizing humanoperator intervention. The presented research focuses on two main aspects: (i) the automationof the 3D modeling pipeline, targeting the automatic data matching in feature-less andGPS-denied areas for in-situ world modeling and (ii) the exploitation of the generated 3Dmodels along with visual servoing procedures to ensure automatically the 3D scene modelcompleteness.We start this chapter by motivating the jointly use of laser and image data and by listing themain key issues which need to be addressed when aiming to supply automatic photorealist3D modeling tasks while coping with time and in-situ access constraints. The next foursections are dedicated to a gradual description of the 3D modeling system in which we projectthe proposed image-laser solutions designed to be embedded onboard mobile plateforms,providing them with world modeling capabilities and thus visual perception. This is an

UAVs have recently become a very popular tool for acquiring geospatial data. Photographs, films, images, and results of measurements of various sensors from them constitute source material for generating, among other things, photographic documentation, visualisation of places and objects, cartographic materials and 3D models. These models are not only material for the visualisation of objects but are also source material for spatial analysis, including the assessment and analyses of the solar potential of buildings. This research aims to benchmark the feasibility of using UAV-derived data acquired from three sensors, namely the DJI Zenmuse P1 camera, the Share PSDK102S v2 multi-lens camera and the DJI Zenmuse L1 laser scanner. The data from these were acquired for the construction of comprehensive and reliable 3D models, which will form the basis for generating solar potential maps. Various sensors, data storage formats, and geospatial data processing capabilities are analysed to determine the most optimal and efficient solution for providing accurate, complete and reliable 3D models of places and objects for the construction of solar potential maps. In this paper, the authors prepare a compilation of the results of the studies from different measurement combinations and analyse the strengths and weaknesses of the different solutions, as well as the integration of the results for an optimal 3D model, which was used to perform solar potential analyses for the selected built-up area. The results of the study show that the parameters for assessing the quality of a 3D model can be statistical parameters that determine the coplanarity of roof slope points (i.e., standard deviation, distances from the plane, and RMS value). The completeness of the model is defined as the percentage of the recorded area by sensors to the total area of the model.

To investigate the feasibility of MRI three-dimensional (3D) reconstruction model in quantifying glenoid bone defect by comparing with CT 3D reconstruction model measurement. Forty patients with shoulder anterior dislocation who met the selection criteria between December 2021 and December 2022 were admitted as study participants. There were 34 males and 6 females with an average age of 24.8 years (range, 19-32 years). The injury caused by sports injury in 29 cases and collision injury in 6 cases, and 5 cases had no obvious inducement. The time from injury to admission ranged from 4 to 72 months (mean, 28.5 months). CT and MRI were performed on the patients' shoulder joints, and a semi-automatic segmentation of the images was done with 3D slicer software to construct a glenoid model. The length of the glenoid bone defect was measured on the models by 2 physicians. The intra-group correlation coefficient ( ICC) was used to evaluate the consistency between the 2 physicians, and Bland-Altman plots were constructed to evaluate the consistency between the 2 methods. The length of the glenoid bone defects measured on MRI 3D reconstruction model was (3.83±1.36) mm/4.00 (0.58, 6.13) mm for physician 1 and (3.91±1.20) mm/3.86 (1.39, 5.96) mm for physician 2. The length of the glenoid bone defects measured on CT 3D reconstruction model was (3.81±1.38) mm/3.80 (0.60, 6.02) mm for physician 1 and (3.99±1.19) mm/4.00 (1.68, 6.38) mm for physician 2. ICC and Bland-Altman plot analysis showed good consistency. The ICC between the 2 physicians based on MRI and CT 3D reconstruction model measurements were 0.73 [95% CI (0.54, 0.85)] and 0.80 [95% CI (0.65, 0.89)], respectively. The 95% CI of the difference between the two measurements of physicians 1 and 2 were (-0.46, 0.49) and (-0.68, 0.53), respectively. The measurement of glenoid bone defect based on MRI 3D reconstruction model is consistent with that based on CT 3D reconstruction model. MRI can be used instead of CT to measure glenoid bone defects in clinic, and the soft tissue of shoulder joint can be observed comprehensively while reducing radiation.

The newly developed computer vision pose estimation technique in artificial intelligence (AI) is an emerging technology with potential advantages, such as high efficiency and contactless detection, for improving competitive advantage in the sports industry. The related literature is currently lacking an integrated and comprehensive discussion about the applications and limitations of using the pose estimation technique. The purpose of this study was to apply AI pose estimation techniques, and to discuss the concepts, possible applications, and limitations of these techniques in table tennis. This study implemented the OpenPose pose algorithm in a real-world video of a table tennis game. The research results show that the pose estimation algorithm performs well in estimating table tennis players’ poses from the video in a graphics processing unit (GPU)-accelerated environment. This study proposes an innovative two-stage AI pose estimation method for effectively addressing the current difficulties in applying AI to table tennis players’ pose estimation. Finally, this study provides several recommendations, benefits, and various perspectives (training vs. tactics) of table tennis and pose estimation limitations for the sports industry.

In order to better display and protect ancient Chinese architecture, a three-dimensional model reconstruction method based on multi-station laser scanning is proposed. This method mainly includes several steps, such as point cloud data collection, preconditioning, multi-site cloud data fusion, point cloud data compression, 3D model reconstruction and texture mapping, environment rendering, video processing, and 3D roaming. According to the requirement of video rendering and virtual roaming, we focus on cloud processing, 3D modeling and 3D model display in this paper. Experimental results show that an famous building named Enshi Dong Drum Tower is well digitally reconstructed, so as to result in ethnic characteristics and cultural heritage protection in practical application.

Acquiring realistic texture details for 3D models is important in 3D reconstruction. However, the existence of geometric errors, caused by noisy RGB-D sensor data, always makes the color images cannot be accurately aligned onto reconstructed 3D models. In this paper, we propose a global-to-local correction strategy to obtain more desired texture mapping results. Our algorithm first adaptively selects an optimal image for each face of the 3D model, which can effectively remove blurring and ghost artifacts produced by multiple image blending. We then adopt a non-rigid global-to-local correction step to reduce the seaming effect between textures. This can effectively compensate for the texture and the geometric misalignment caused by camera pose drift and geometric errors. We evaluate the proposed algorithm in a range of complex scenes and demonstrate its effective performance in generating seamless high fidelity textures for 3D models.