Projection model, 3D reconstruction and rigid motion estimation from non-central catadioptric images

This work addresses the problem of rigid motion estimation and 3D reconstruction in vision systems where it is possible to recover the incident direction from image points. Such systems include pinhole cameras and catadioptric cameras. Given two images of the same scene acquired from two positions, the transformation is estimated by means of an iterative process. The estimation process aims at having corresponding incident rays intersecting at the same 3D point. Geometrical relationships are derived to support the estimation method. Experiments with real images are presented.

We present a technique for modeling non-central catadioptric cameras consisting of perspective cameras and curved mirrors. The real catadioptric cameras have to be treated as non-central cameras, since they do not possess a single viewpoint. We present a method for solving the correspondence problem, auto-calibrating cameras, and computing a 3D metric reconstruction automatically from two uncalibrated non-central catadioptric images. The method is demonstrated on spherical, parabolic, and hyperbolic mirrors. We observed that the reconstruction & auto-calibration with non-central catadioptric cameras is as easy (or as difficult) as with central catadioptric cameras, provided that the correspondence problem can be solved with a suitable approximate central model. It turns out that it is the number of parameters of the camera model that matters rather than the exact centrality of the projection. Our technique allows to autocalibrate catadioptric cameras even with genuinely non-central mirrors such as spheres (simple model, low blur, easy to manufacture) or uniform resolution mirrors (optimized projection).

We present a technique for modeling non-central catadioptric cameras consisting of a perspective camera and a rotationally symmetric conic reflector. While previous approaches use a central approximation and/or iterative methods for forward projection, we present an analytical solution. This allows computation of the optical path from a given 3D point to the given viewpoint by solving a 6th degree forward projection equation for general conic mirrors. For a spherical mirror, the forward projection reduces to a 4th degree equation, resulting in a closed form solution. We also derive the forward projection equation for imaging through a refractive sphere (non-central dioptric camera) and show that it is a 10th degree equation. While central catadioptric cameras lead to conic epipolar curves, we show the existence of a quartic epipolar curve for catadioptric systems using a spherical mirror. The analytical forward projection leads to accurate and fast 3D reconstruction via bundle adjustment. Simulations and real results on single image sparse 3D reconstruction are presented. We demonstrate ∼ 100 times speed up using the analytical solution over iterative forward projection for 3D reconstruction using spherical mirrors.

This study develops a 3D facial reconstruction system, which consists of five modules, using the orthographic projection model to approximate the perspective projection model. The first module identifies a number of feature points on the face and tracks these feature points over a sequence of facial images by the optical flow technique. The second module applies the factorization method to the orthographic model to reconstruct a 3D human face. The facial images are acquired using a pinhole camera, which are based on a perspective projection model. However, the face is reconstructed using an orthographic projection model. To compensate for the difference between these two models, the third module implements a simple and efficient method for approximating the perspective projection model. The fourth module overcomes the missing point problem, commonly arising in 3D reconstruction applications. Finally, the fifth module implements a smoothing process for the 3D surface by interpolating additional vertices.

Objective To study the image segmentation, three-dimensional (3D) reconstruction and simulation operation of choledocholithotomy and T-tube drainage based on the computed tomagraphy (CT) data of patient with choledocholithiasis. Methods Patient with choledocholithiasis underwent 64-slice spiral CT imaging. The images segmentation and 3D reconstruction were performed by Medical Image Process System (MIPS) to construct the 3D model of the hepatobiliary system. The model was modified by FreeForm Modeling System. The virtual surgical instruments were developed by GHOST SDK software, and were imported to the virtual surgery. Results The data of plain, arterial phase, venous phase and portal venous phase scanning were collected, and the data stored in DICOM format were transformed to BMP format. A model of the hepatobiliary system was constructed after the data was segmented by MIPS, and then the model was exported in the STL format. The data in STL format were imported to FreeForm Modeling System for smoothing the model. Different structures were assigned different colors to make the model more vivid. The self-developed virtual surgical instruments were imported to the system, and the virtual surgery for choledocholithiasis was performed with PHANToM. Conclusions With the help of MIPS, the image segmentation and 3D reconstruction of the model are finished rapidly and effectively. After the virtual surgical instruments are developed in FreeForm Modeling System, the virtual surgery can be achieved in the 3D model with the assistance of PHANToM. Key words: Choledocholithiasis; Three-dimensional reconstruction; Virtual surgery

Three-dimensional (3D) reconstruction technology has found widespread applications across various domains, including intelligent driving and underwater exploration. But the existing imaging systems and methods still have deficiencies in terms of reconstruction accuracy, detection distance and system volume. Herein, this paper presents a three-dimensional detection and reconstruction method based on a compact Risley-prism 3D imaging system that achieves multi-viewpoint imaging by rotating the Risley prism to adjust the camera's optical axis. A refractive camera model that integrates the pinhole camera model with the vector form of Snell's law is established to precisely describe beam trajectory. A forward projection method suitable for refractive interfaces is developed based on Fermat's principle, and the influence of systematic errors on the reconstruction is analyzed in detail through simulation. Furthermore, a new 3D reconstruction method combining error calibration based on the optimization iteration is introduced to avoid the influence of error and improve reconstruction quality. Experimental results demonstrate that the proposed approach markedly enhances 3D reconstruction accuracy, reducing the Normalized Root Mean Square Error (NRMSE) from 0.9076 to 0.0207.

Segmentation is a low-level processing aimed at the partition of an image in disjoint regions, each one homogeneous with respect to some properties like intensity, texture, shape, etc. Such a task is needed in many high-level processing and applications in such diverse fields as remote-sensing, medical imaging, video coding, and industrial automation, just to name a few. In this work of thesis, we first deal with the problem of color based image segmentation, that is based on some form of homogeneity in the color/spectral properties of the image, and then move to the more complex task of texture based segmentation, where the aim is to recognize complex structures in the image which are typically non homogeneous in terms of spectral properties. Concerning color based segmentation, we focus on a specific class of Markov Random Field (MRF) probability models, namely the Tree-Structured Markov Random Fields (TS-MRF), recently proposed to provide a hierarchical multiscale description of images. Based on such a model, the unsupervised image segmentation is carried out by means of a sequence of nested class splits, where each class is modeled as a local binary MRF. As a first contribution, we first extend the model to generic tree structures removing the unnecessary binary constraint, and then devise a new TS-MRF unsupervised segmentation technique which improves upon the original algorithm by selecting a better tree structure and eliminating spurious classes. Such results are obtained by using the Mean-Shift procedure to estimate the number of pdf modes at each node, and to obtain a more reliable initial clustering for subsequent MRF optimization. To this end, we devise a new reliable and fast clustering algorithm based on the Mean-Shift technique, which makes use of a variable-bandwidth strategy based on the k-Nearest Neighbors (k-NN) technique, and is implemented with a speed-up strategy that cuts significantly the computational complexity. Experimental results on synthetic data and real life images from the remote sensing domain prove the potential of the proposed method. Turning to texture based segmentation, we present a novel multiscale texture model, and a related algorithm for the unsupervised segmentation of color images. Elementary textures are characterized by their spatial interactions with neighboring regions along selected directions. Such interactions are modeled in turn by means of a set of Markov chains, one for each direction, whose parameters are collected in a feature vector that synthetically describes the texture. Based on the feature vectors, the texture are then recursively merged, giving rise to larger and more complex textures, which appear at different scales of observation: accordingly, the model is named Hierarchical Multiple Markov Chain (H-MMC). The Texture Fragmentation and Reconstruction (TFR) algorithm, addresses the unsupervised segmentation problem based on the H-MMC model. The ``fragmentation'' step allows one to find the elementary textures of the model, while the ``reconstruction'' step defines the hierarchical image segmentation based on a probabilistic measure (texture score) which takes into account both region scale and inter-region interactions. The performance of the proposed method was assessed through the Prague segmentation benchmark, based on mosaics of real natural textures, and also tested on real-world natural images from the Berkeley dataset. Finally, an application of the TFR algorithm for the unsupervised segmentation of multiresolution remotely sensed images is proposed, based on the significant presence of textural content in last generation high-resolution satellite images. The obtained results further assess the potential of the proposed method for such real life applications.

3D map reconstruction plays a critical role in fields such as autonomous driving, urban planning, and virtual reality. Using 2D images from real-world settings, multi-view image-based 3D map reconstruction offers a practical solution across these sectors. Specifically, generating 3D maps from realistic drone-view images has broad applicability. However, direct reconstruction from drone-view images is challenging due to extreme viewing angles and unstable image conditions like motion blur or varying light. Additionally, simultaneously generating and utilizing 3D maps from drone-captured images presents significant challenges, including fragmented sub-tasks such as 3D point cloud reconstruction, surface reconstruction, and shape fitting. To address these challenges, we introduce the 3D Scene Abstraction, which seamlessly integrates image matching, 3D reconstruction, 3D semantic segmentation, clustering, and shape fitting to produce 3D maps from RGB images. Our approach enhances the robustness of 3D map generation by merging 2D semantic information with 3D point clouds, ensuring reliable results under diverse drone-captured image conditions. For efficient data representation, we employ multiple primitives based on prior knowledge of various building shapes, improving both map formation performance and computational efficiency. Our approach not only enhances the effectiveness of 3D map reconstruction but also reduces the amount of data required to represent the scene, thereby improving overall efficiency and practical usability. These contributions address practical challenges in the field and lay the foundation for future advancements in 3D map reconstruction technologies.

In order to obtain the phenotypic parameters of apple quickly and accurately, which were commonly used as the basis of fruit sorting, a fast estimation method of apple phenotypic parameters based on three-dimensional (3D) reconstruction was proposed in this study. In this study, a three-dimensional model was constructed to estimate the phenotypic parameters of apple, such as volume, height, diameter, and fruit shape index. Firstly, an image acquisition system was built to capture sequence images of fruit with a binocular stereo vision system, and the images were extracted and matched using the Accelerated-KAZE algorithm to create the point cloud data. Secondly, the point cloud data were matched with the algorithm of Iterative Closest Point to establish a whole model of apple, and the surface reconstruction model of fruit was obtained by constructing irregular triangulation network. Finally, the apple phenotypic parameters were calculated by means of segmentation, surface complement and integral of the fruit model. Total of 200 apples were used as samples in the experiment. By this method, the phenotypic parameters of the apples were estimated based on their 3D reconstruction model, and the linear regression analysis was carried out between the estimated values and the real values. The results showed that R2 of the linear regression fitting of each parameter was higher than 0.90. Among them, the fitting of volume was the best with R2 of 0.97. In addition, the average errors of apple volume, height, fruit shape index, maximum diameter D and minimum diameter d were 8.73 cm3, 1.43 mm, 1.28%, 0.90 mm, and 1.23 mm, respectively. According to the Chinese national standard of “fresh Apple”, the average error of the estimated result is within the range of allowable error. It indicates that the method of apple phenotypic parameter estimation based on 3D reconstruction has a high accuracy and practicability, and it can be used as the support for fruit sorting. Keywords: apple, 3D reconstruction, phenotypic parameter, stereo vision system, sequence image, sorting DOI: 10.25165/j.ijabe.20211405.6258 Citation: Ma H, Zhu X, Ji J T, Wang H, Jin X, Zhao K X. Rapid estimation of apple phenotypic parameters based on 3D reconstruction. Int J Agric & Biol Eng, 2021; 14(5): 180–188.

ObjectivesThe primary objective was to determine which of the following angular insertion depth (AID) estimation methods showed the strongest correlation with the postoperative AID (AIDpostop) of a lateral wall (LW) electrode: the Escudé formula, based on the largest distance from the round window (RW) to the LW (distance A); the elliptic-circular approximation (ECA) method, based on both distance A and the perpendicular distance (distance B); and a three-dimensional (3D) reconstruction method. The secondary objective was to assess the impact of using the actual electrode insertion length on the evaluation of AID estimation methods.MethodsThis study included 45 cochleae implanted with the Advanced Bionics SlimJ electrode, with all 16 active electrode contacts positioned intracochlear. Distance A, distance B, and a 3D reconstruction along the LW were evaluated on preoperative magnetic resonance imaging using 3D Slicer. The AIDpostop and insertion length were assessed using cone-beam computed tomography. Two sets of estimated AID values were obtained: assuming a perfect 23-mm insertion length with the reference electrode at the RW (AID23mm); and using the actual insertion length (AIDinsertionlength). Spearman’s rank correlation coefficient was used to assess the association between the estimated AID values and AIDpostop.ResultsA moderately positive correlation was observed between AID23mm and AIDpostop for all three approaches, with no statistically significant differences among them (rs=0.410–0.579). When the analysis was repeated using AIDinsertionlength, the strength of the correlation with AIDpostop increased significantly for all three methods, with the 3D reconstruction method showing a stronger correlation (rs=0.952) than the Escudé formula (rs=0.734) or the ECA method (rs=0.787) (P<0.001).ConclusionThe 3D reconstruction approach has the potential to improve prediction of the AID for LW electrode arrays compared to spiral formulae based on 2D cochlear dimensions. Accurate assessment of the actual insertion length is important for the proper evaluation of AID estimation methods.

Omnidirectional and 360° images are becoming widespread in industry and in consumer society, causing omnidirectional computer vision to gain attention. Their wide field of view allows the gathering of a great amount of information about the environment from only an image. However, the distortion of these images requires the development of specific algorithms for their treatment and interpretation. Moreover, a high number of images is essential for the correct training of computer vision algorithms based on learning. In this paper, we present a tool for generating datasets of omnidirectional images with semantic and depth information. These images are synthesized from a set of captures that are acquired in a realistic virtual environment for Unreal Engine 4 through an interface plugin. We gather a variety of well-known projection models such as equirectangular and cylindrical panoramas, different fish-eye lenses, catadioptric systems, and empiric models. Furthermore, we include in our tool photorealistic non-central-projection systems as non-central panoramas and non-central catadioptric systems. As far as we know, this is the first reported tool for generating photorealistic non-central images in the literature. Moreover, since the omnidirectional images are made virtually, we provide pixel-wise information about semantics and depth as well as perfect knowledge of the calibration parameters of the cameras. This allows the creation of ground-truth information with pixel precision for training learning algorithms and testing 3D vision approaches. To validate the proposed tool, different computer vision algorithms are tested as line extractions from dioptric and catadioptric central images, 3D Layout recovery and SLAM using equirectangular panoramas, and 3D reconstruction from non-central panoramas.

Objective To apply imaging examinations for surveying the morphological characteristics of neurocentral synchondrosis (NCS) in fetal spine so as to deduce the function of NCS in the growth of vertebra and vertebral pedicle in normal spines. Methods A total of 15 fetal samples (23 w-33 w) without congenital spinal deformity were collected from October 2012 to December 2013. There were 8 males and 7 females with an average gestational age of 28+ 6 weeks. And informed consent form was signed. Based upon fetal age, the fetuses were divided into two groups of 7 samples (24 w) and 8 samples with an average gestational age of 32 w. All fetuses were visualized by magnetic resonance imaging (MRI) (T1-L5 vertebra) and full-spine spiral computed tomography (CT) + three-dimensional (3D) reconstruction. The above methods could observe the 3D morphology of NCS. Then the relevant parameters were measured by dicom image processing software. And the mean value of each group was calculated and line plotted by Origin software. All parameters of both sides were analyzed by paired t-test. And the test standard was set at P=0.05. Results As a pair of triangular structure with a wide exterior and a narrow interior in sagittal plane on T2 MRI image, NCS surrounds both vertebral body and posterior arch and inter-connected in anterior aspects. And it is reconstructed by two parallel lines perpendicular to vertebrae in sagittal plane on full-spine spiral CT + 3D reconstruction images. The mean parameters of two groups were calculated and the corresponding lines plotted. It showed that α angle increased faster than that of the second one. The others parameters of the second fetal group were bigger than those of the first one. The mean α angles of left and right sides were 60.827°±2.794° and 60.882°±2.840° for the first group and 58.878°±4.194 and 58.749±4.334 for the second group respectively. And the bilateral length-width ratios were 1.452±0.096 and 1.435±0.124 for the first group and 1.680±0.147 and 1.678±0.142 for the second group. The parameters of two groups were compared by paired t-test and the differences of two sides of two group were significant (P 0.05). Conclusions The position of NCS shifts constantly. It accounts for a bigger volume of vertebral body in fetal spines. And its role for the growth of spine is larger than that in infants and young children. Key words: Vertebral column; Fetus; Cartilage

Visual surveillance using a camera network has imposed new challenges to camera calibration. An essential problem is that a large number of cameras may not have a common field of view or even be synchronized well. We propose to use a hybrid camera network that consists of catadioptric and perspective cameras for a visual surveillance task. The relations between multiple views of a scene captured from different cameras can be then calibrated under the catadioptric camera's coordinate system. We address the important issue of how to calibrate the hybrid camera network. We calibrate the hybrid camera network in three steps. First, we calibrate the catadioptric camera using only the vanishing points. In order to reduce computational complexity, we calibrate the camera without the mirror first and then calibrate the catadioptric camera system. Second, we determine 3D positions of some points using as few as two spatial parallel lines and some equidistance points. Finally, we calibrate other perspective cameras based on these known spatial points.

The aim of the PhD thesis was to look for a relationship between the landslide-induced damage recorded on structures and facilities based on the results of several field campaigns and kinematic parameters quantitatively estimated by remote sensing techniques. Investigations were developed on two test sites: a deep-seated landslide in Colle Lapponi-Piano Ovetta in municipality of Agnone (Molise region, southern Italy) and the landslides affecting the southwestern sector of Volterra (Tuscany region, central Italy). First of all, a re-enactment of the evolution of both landslides were conducted, by means of 3D reconstructions based on historical aerial series of images and the analysis of Persistent Scatterers of ERS1/2, ENVISAT and COSMO-SkyMed satellites. The 3D Points Clouds and models were developed on several sets of aerial historical images dating from different years starting from 1945 and 1954 for Agnone and Volterra, respectively. To better understand the morpho-evolutionary stages, a qualitative assessment of changes of volume were made combining the oldest and the latest 3D reconstructed Points Clouds. This interpretation, even if qualitative and not quantitative, can be helpful for understanding possible effects of future reactivations and as a support to realize mitigation plans, susceptibility maps and other useful for the local administrators. The Persistent Scatterers were used to monitor the evolution in recent years, up to 2015. Then, for both case studies, the damage was revealed on structures and facilities by several field surveys and classified by means of five literature damage categorizations. During their application, some drawbacks and benefits of the methodologies were carried out and a new approach to improve the categorization of the damage on structures, facilities and ground surfaces was developed. This was conceived in two subsequent phases: i) a classification to use during the field campaign to quantify the severity of cracks and fractures on structures, facilities and ground surfaces; ii) an a posteriori ranking to apply on the entire structure, involving the extension of damage classes, performed by a cell-grid matrix. Furthermore, a damage recording scheme, useful for the recognition of cracks and fractures during the field surveys, was proposed. A critical comparison between the results obtained applying the different classification approaches, then followed. Buildings and facilities, for both sites, were categorized using also kinematic parameters such as velocity and maximum displacement measured along the Line Of Sight, derived by A-DInSAR, and their absolute values re-projected along the steepest local slope. Once characterized and categorized all structures and facilities of both sites of interest, a correlation between the surveyed damage classes and the deriving parameters by satellite were looked for. The investigation was carried to understand the behaviour of entire structures, subject to displacements. The first analysis was conducted on the Agnone test site where for several constructions an upper regression line between damage categories and velocity reprojected along the slope was recognized. Some outlayers were identified, mainly for low damage levels, then singularly investigated. To assess the reliability of all the structures, a matrix involving damage and velocity along the slope parameter acquired by ENVISAT and COSMO-SkyMed sensors was developed in order to obtain a classification. To validate the correlation and the reliability matrix the same procedure was applied to the Volterra site. Once asserted the validity of the relation between the velocity reprojected along the steepest slope and the classes of damage also for this area, the reliability matrix was applied on the constructions of the Volterra site. In this way, the relation between the displacement occurred during the period covered by ENVISAT and COSMO-SkyMed shows how the surveyed damage construction are related to the displacement. Some areas where damage occurred in the 2000 shows high reliability with ENVISAT recorded velocity, while others structures exhibits high reliability with COSMO-SkyMed data. The results were interesting because they highlight the fact that for some construction there is correlation between velocity of displacement of the entire structure and affecting damage; for others, instead, the high damage is related to the differential settlement and not necessarily to a high rate of displacement velocity.

Driving mutual advancement of 3D reconstruction and inpainting for masked faces