Low cost three-dimensional virtual model construction for remanufacturing industry

In recent years, due to the significant advancements in hardware sensors and software technologies, 3D environmental point cloud modeling has gradually been applied in the automation industry, autonomous vehicles, and construction engineering. With the high-precision measurements of 3D LiDAR, its point clouds can clearly reflect the geometric structure and features of the environment, thus enabling the creation of high-density 3D environmental point cloud models. However, due to the enormous quantity of high-density 3D point clouds, storing and processing these 3D data requires a considerable amount of memory and computing time. In light of this, this paper proposes a real-time 3D point cloud environmental contour modeling technique. The study uses the point cloud distribution from the 3D LiDAR body frame point cloud to establish structured edge features, thereby creating a 3D environmental contour point cloud map. Additionally, unstable objects such as vehicles will appear during the mapping process; these specific objects will be regarded as not part of the stable environmental model in this study. To address this issue, the study will further remove these objects from the 3D point cloud through image recognition and LiDAR heterogeneous matching, resulting in a higher quality 3D environmental contour point cloud map. This 3D environmental contour point cloud not only retains the recognizability of the environmental structure but also solves the problems of massive data storage and processing. Moreover, the method proposed in this study can achieve real-time realization without requiring the 3D point cloud to be organized in a structured order, making it applicable to unorganized 3D point cloud LiDAR sensors. Finally, the feasibility of the proposed method in practical applications is also verified through actual experimental data.

Abstract. A series of panoramic images are usually used to generate a 720° panorama image. Although panoramic images are typically used for establishing tour guiding systems, in this research, we demonstrate the potential of using panoramic images acquired from multiple sites to create not only 720° panorama, but also three-dimensional (3D) point clouds and 3D indoor models. Since 3D modeling is one of the goals of this research, the location of the panoramic sites needed to be carefully planned in order to maintain a robust result for close-range photogrammetry. After the images are acquired, panoramic images are processed into 720° panoramas, and these panoramas which can be used directly as panorama guiding systems or other applications. In addition to these straightforward applications, interior orientation parameters can also be estimated while generating 720° panorama. These parameters are focal length, principle point, and lens radial distortion. The panoramic images can then be processed with closerange photogrammetry procedures to extract the exterior orientation parameters and generate 3D point clouds. In this research, VisaulSFM, a structure from motion software is used to estimate the exterior orientation, and CMVS toolkit is used to generate 3D point clouds. Next, the 3D point clouds are used as references to create building interior models. In this research, Trimble Sketchup was used to build the model, and the 3D point cloud was added to the determining of locations of building objects using plane finding procedure. In the texturing process, the panorama images are used as the data source for creating model textures. This 3D indoor model was used as an Augmented Reality model replacing a guide map or a floor plan commonly used in an on-line touring guide system. The 3D indoor model generating procedure has been utilized in two research projects: a cultural heritage site at Kinmen, and Taipei Main Station pedestrian zone guidance and navigation system. The results presented in this paper demonstrate the potential of using panoramic images to generate 3D point clouds and 3D models. However, it is currently a manual and labor-intensive process. A research is being carried out to Increase the degree of automation of these procedures.

It is an important task to reconstruct surfaces from 3D point clouds. Current methods are able to reconstruct surfaces by learning Signed Distance Functions (SDFs) from single point clouds without ground truth signed distances or point normals. However, they require the point clouds to be dense, which dramatically limits their performance in real applications. To resolve this issue, we propose to reconstruct highly accurate surfaces from sparse point clouds with an on-surface prior. We train a neural network to learn SDFs via projecting queries onto the surface represented by the sparse point cloud. Our key idea is to infer signed distances by pushing both the query projections to be on the surface and the projection distance to be the minimum. To achieve this, we train a neural network to capture the on-surface prior to determine whether a point is on a sparse point cloud or not, and then leverage it as a differentiable function to learn SDFs from unseen sparse point cloud. Our method can learn SDFs from a single s parse point cloud without ground truth signed distances or point normals. Our numerical evaluation under widely used benchmarks demonstrates that our method achieves state-of-the-art reconstruction accuracy, especially for sparse point clouds. Code and data are available at https://github.com/mabaorui/OnSurfacePrior.

Smart mobile device and consumer level 3D visual sensor are becoming very cheap and widely accessible in people's daily lives. Visual information based indoor localization is getting more and more attention in the computer vision research community to achieve real time and precise localization result. In this paper, we present a 6-degree-of-freedom (6-DoF) pose estimation using a portable 3D visual sensor mounted on a mobile device. A detailed 3D model and a WiFi received signal strength model of the indoor environment are constructed in the offline training phase. During the online localization, we first use WiFi signals to locate the device in a 3D sub model. The initial pose is calculated through feature matching between the online captured 2D image and the key frame images used to build the 3D model. Then we employ iterative closest point (ICP) algorithm to estimate the rigid transform between online captured 3D point cloud and local 3D point cloud. The 6-DOF pose estimation is further refined by random sample consensus (RANSAC) algorithm. We implement the system on iOS platform and the experiments carried out in an indoor environment shows promising result of our approach. To the best of our knowledge, this is the first portable 3D visual sensor based indoor localization system on mobile device.

Photogrammetric surveying using unmanned aerial systems (UAS) and structure from motion (SfM) is a rapidly emerging technique that has been widely used as a valuable research tool in geodesy and related disciplines. We studied surveying and three-dimensional (3D) modeling based on UAS and SfM. In particular, we focus on the accuracy and quality of UAS-SfM in 3D modeling and volumetric calculation of a building with an overhanging feature. For this purpose, two aerial surveys were performed using a fixed-wing and a multirotor aircraft with nadir and oblique camera axes. For validating the results, field surveys were carried out using terrestrial laser scanner (TLS), differential global navigation satellite system, and total station. To study the performance of SfM software on the accuracy and quality of point clouds, UAS images were processed using three software packages: Agisoft PhotoScan, Pix4Dmapper, and 3Dsurvey. As a result of photogrammetric image processing, 3D dense point clouds, mesh models, and digital surface models were generated. For validating the UAS-SfM results, the models were compared with the TLS model and field measurements; absolute accuracy assessments and model versus model comparisons were performed. The most accurate and high-quality 3D models comparable to TLS models were obtained from oblique images. PhotoScan generated relatively more accurate 3D models with the horizontal and vertical accuracy of root mean square error 0.025 and 0.02 m, respectively. The maximum discrepancies in dimensions and volume measurements from the UAS-SfM-derived point cloud were at the level of 0.01 m and 0.66 m3 (0.03%). The results highlighted that using a low-cost UAS and SfM photogrammetry, high-accuracy and high-quality 3D models can be generated. The results confirm that with respect to its rapid data acquisition and centimeter-level accuracy, UAS-SfM can be considered as a suitable alternative approach to conventional measurement techniques, especially in engineering and emergency response.

Understanding normal structural and functional anatomy is critical for health professionals across various fields such as medicine, veterinary, and dental courses. The landscape of anatomical education has evolved tremendously due to several challenges and advancements in blended learning approaches, which have led to the adoption of the use of high‐fidelity 3D digital models in anatomical education. Cost‐effective methods such as photogrammetry, which creates digital 3D models from aligning 2D photographs, provide a viable alternative to expensive imaging techniques (i.e. computed tomography and magnetic resonance imaging) whilst maintaining photorealism and serving multiple purposes, including surgical planning and research. This study outlines a comprehensive workflow for producing realistic 3D digital models from embalmed veterinary specimens. The process begins with the preservation of specimens using the modified‐WhitWell (WhitWell‐Liverpool) embalming protocol, which ensures optimal tissue rigidity and improved colour enhancement, facilitating easier manipulation and better photogrammetry outcomes. Once embalmed, specimens are photographed to create digital 3D models using photogrammetry. Briefly, all images are processed to generate a sparse point cloud, which is then rendered into a 3D mesh. The mesh undergoes decimation and smoothing to reduce computational load, and a texture is applied to create a lifelike model. Additional colour enhancements and adjustments are made using digital tools to restore the natural appearance of the specimens. The 3D models are stored on a cloud repository and integrated into the University of Liverpool's Virtual Learning Environment, providing continuous, remote access to high‐quality anatomical resources. The switch to embalmed specimens during the COVID‐19 pandemic allowed for longer‐term use and detailed dissections, enhancing the quality of digital models. Fresh specimens, though naturally coloured, are less stable for photogrammetry, making embalmed specimens preferable for accurate 3D modelling. Our method ensures embalmed specimens are rigid enough for precise modelling while allowing texture adjustments to enhance digital representation. This approach has improved logistical efficiency, educational delivery, and specimen quality. Innovative embalming techniques and advanced photogrammetry have the power to revolutionise anatomical education with the creation of a vast digital library accessible online to students at any time. This approach paves the way for integrating digital 3D models into immersive environments and assessing their impact on learning outcomes.

Three-dimensional (3D) point clouds have a wide range of applications in the field of 3D vision. The quality of the acquired point cloud data considerably impacts the subsequent work of point cloud processing. Due to the sparsity and irregularity of point cloud data, processing point cloud data has always been challenging. However, existing deep learning-based point cloud dense reconstruction methods suffer from excessive smoothing of reconstruction results and too many outliers. The reason for this is that it is not possible to extract features for local and global features at different scales and provide different levels of attention to different regions in order to obtain long-distance dependence for dense reconstruction. In this paper, we use a parallel multi-scale feature extraction module based on graph convolution and an upsampling method with an added multi-head attention mechanism to process sparse and irregular point cloud data to obtain extended point clouds. Specifically, a point cloud training patch with 256 points is inputted. The PMS module uses three residual connections in the multi-scale feature extraction stage. Each PMS module consists of three parallel DenseGCN modules with different size convolution kernels and different averaging pooling sizes. The local and global feature information of the augmented receptive field is extracted efficiently. The scale information is obtained by averaging the different pooled augmented receptive fields. The scale information was obtained using the different average pooled augmented receptive fields. The upsampling stage uses an upsampling rate of r=4, The self-attentive features with a different focus on different point cloud data regions obtained by fusing different weights make the feature representation more diverse. This operation avoids the bias of one attention, and each focuses on extracting valuable fine-grained feature information. Finally, the coordinate reconstruction module obtains 1024 dense point cloud data. Experiments show that the proposed method demonstrates good evaluation metrics and performance and is able to obtain better visual quality. The problems of over-smoothing and excessive outliers are effectively mitigated, and the obtained sparse point cloud is more dense.

A state-of-the-art review of automated extraction of rock mass discontinuity characteristics using three-dimensional surface models

The 3D modeling of urban buildings has become a common research area in various disciplines such as photogrammetry and computer vision, with different applications such as intelligent city management, navigation of self-driving cars and architecture, just to name a few. The objective of this study is to produce a 3D model of the external facade of the buildings with the required precision, accuracy and level of detail according to the user’s requirements, while minimizing time and cost. This research focuses on the production of 3D models for blocks of residential buildings in Tehran, Iran. The Insta 360 One X2 spherical camera is selected to capture the data due to its low cost and 360 × 180° field of view. The camera specifications have facilitated more efficient data collection in terms of both time and cost. The proposed modeling method is based on extracting lines of external walls through the utilization of the point cloud density concept. Initially, photogrammetric point clouds are generated in with a reconstruction precision of 0.24 m from spherical camera images. In the next step, the 3D point cloud is projected into a 2D point cloud by setting the height component to zero. The 2D point cloud is then rotated based on the direction angle determined by the Hough transform so that the perpendicular walls are parallel to the axes of the coordinate system. Next, a 2D point cloud density analysis is performed by voxelizing the point cloud and counting the number of points in each voxel in both the horizontal and vertical directions. By determining the peaks in the density plot, the lines of the external vertical and horizontal walls are extracted. To extract the diagonal external walls, the density analysis is performed in the direction of the first principal component. Finally, by determining the height of each wall in the point cloud, a 3D model is created at the level of detail one. The resulting model has a precision of 0.32 m compared to real sizes, and the 2D plan has a precision of 0.31 m compared to the ground truth map. The use of the spherical camera and point cloud density analysis makes this method efficient and cost-effective, making it a promising approach for future urban modeling projects.

Digital twin-based industrial cloud robotics: Framework, control approach and implementation

Rockfall event is common on rocky terrain, and it threatens people. Recent advances in remote sensing technologies allow us to model the terrain with high accuracy and resolution. In this study, we show the application of modeling a rockfall site in three-dimensional (3D) using an unmanned aerial vehicle. The acquisition of information about discontinuities in the rock masses using 3D point clouds is vital to characterize rockfall dynamics. We obtained 3D spatial information and point clouds of a rocky slope from acquired images. We extracted 10 discontinuity sets from the 3D model as they have significant effects on the failure mechanism. Details in mapping discontinuities have improved significantly with the help of point clouds. The results are beneficial for modeling the terrain of inaccessible areas with high resolution and obtaining information from those areas. This paper shows how can point cloud analysis be applied to characterize discontinuity sets.

Surface reconstruction from 3D point clouds has a wide range of applications. In this paper, we focus on the reconstruction from raw, sparse point clouds. Although some existing methods work on this topic, the results often suffer from geometric defects. To solve this problem, we propose a novel method that optimizes a neural network (referred to as signed distance function) to fit the Signed Distance Field (SDF) from sparse point clouds. The signed distance function is optimized by projecting query points to its iso-surface accordingly. Our key idea is to encourage both the direction and distance of projection to be correct through the supervision provided by a nearest neighbor prior. In addition, we mitigate the error propagated from the prior function by augmenting the low-frequency components in the input. In our implementation, the nearest neighbor prior is trained with a large-scale local geometry dataset, and the positional encoding with a specified spectrum is used as a regularization for the optimization process. Experiments on the ShapeNetCore dataset demonstrate that our method achieves better accuracy than SDF-based methods while preserving smoothness.

Accurate human activity recognition (HAR) is the key to enable emerging context-aware applications that require an understanding and identification of human behavior, e.g., monitoring disabled or elderly people who live alone. Traditionally, HAR has been implemented either through ambient sensors, e.g., cameras, or through wearable devices, e.g., a smartwatch, with an inertial measurement unit (IMU). The ambient sensing approach is typically more generalizable for different environments as this does not require every user to have a wearable device. However, utilizing a camera in privacy-sensitive areas such as a home may capture superfluous ambient information that a user may not feel comfortable sharing. Radars have been proposed as an alternative modality for coarse-grained activity recognition that captures a minimal subset of the ambient information using micro-Doppler spectrograms. However, training fine-grained, accurate activity classifiers is a challenge as low-cost millimeter-wave (mmWave) radar systems produce sparse and non-uniform point clouds. In this paper, we propose RadHAR, a framework that performs accurate HAR using sparse and non-uniform point clouds. RadHAR utilizes a sliding time window to accumulate point clouds from a mmWave radar and generate a voxelized representation that acts as input to our classifiers. We evaluate RadHAR using a low-cost, commercial, off-the-shelf radar to get sparse point clouds which are less visually compromising. We evaluate and demonstrate our system on a collected human activity dataset with 5 different activities. We compare the accuracy of various classifiers on the dataset and find that the best performing deep learning classifier achieves an accuracy of 90.47%. Our evaluation shows the efficacy of using mmWave radar for accurate HAR detection and we enumerate future research directions in this space.

The fusion of laser point cloud and visual image depends on the point cloud density and the target framing effect, the traditional laser point cloud processing for sparse point cloud clustering effect is poor, it is difficult to frame small objects as well as medium and long distance objects. Then the subsequent sensor fusion is easy to miss the recognition of obstacles. In this paper, we improve the frame selection method for sparse point clouds, firstly build a deep learning framework pointpillar, use pointpillar to frame the sparse laser point clouds, then spatially calibrate the lidar coordinate system and camera coordinate system, project the lidar point clouds to the camera image, improve the late fusion method, effectively use the detection results of single sensor, and finally The late-fusion is performed with the target detection results of the camera image to output the exact distance as well as the category of the target. Experiments show that compared with the recognition effect of the traditional fusion algorithm, the number of frames is increased by 6 and the missed recognition rate is reduced from 31.41% to 12.31%.

Light Detection And Ranging (LiDAR) is an active laser measuring technology that combines laser scanning and Position and Orientation System (POS) in imaging for generation of accurate and dense 3D point clouds, Digital Elevation Models (DEMs) and Digital Surface Models (DSMs). Other value addition products such as contours, slope maps, tree and building height models and cut-and-fill models can also be produced from the primary LiDAR point cloud data.