Ambiguity in Structure from Motion: Sphere versus Plane

If 3D rigid motion is estimated with some error a distorted version of the scene structure will in turn be computed. Of computational interest are these regions in space where the distortions are such that the depths become negative, because in order to be visible the scene has to lie in front of the image. The stability analysis for the structure-from-motion problem presented in this paper investigates the optimal relationship between the errors in the estimated translational and rotational parameters of a rigid motion, that results in the estimation of a minimum number of negative depth values. The input used is the value of the flow along some direction, which is more general than optic flow or correspondence. For a planar retina it is shown that the optimal configuration is achieved when the projections of the translational and rotational errors on the image plane are perpendicular. Furthermore, the projection of the actual and the estimated translation lie on a line passing through the image center. For a spherical retina given a rotational error, the optimal translation is the correct one, while given a translational error. The optimal rotational error is normal to the translational one at an equal distance from the real and estimated translations. The proofs, besides illuminating the confounding of translation and rotation in structure from motion, have an important application to ecological optics, explaining differences of planar and spherical eye or camera designs in motion and shape estimation.

This paper develops a novel solution to the problem of recovering the structure of a scene given an uncalibrated video sequence depicting the scene. The essence of the technique lies in a method for recovering the rigid transformation between the different views in the image sequence. Knowledge of this 3D motion allows for self-calibration and for subsequent recovery of 3D structure. The introduced method breaks away from applying only the traditionally used epipolar constraint and introduces a new constraint based on the interaction between 3D motion and shape. Up to now, structure from motion algorithms proceeded in two well defined steps, where the first and most important step is recovering the rigid transformation between two views, and the subsequent step is using this transformation to compute the structure of the scene in view. Here both aforementioned steps are accomplished in a synergistic manner. Existing approaches to 3D motion estimation are mostly based on the use of optic flow which however poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) estimate accurately flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. In the past this dilemma has been addressed by improving the estimation of flow through sophisticated optimization techniques, whose performance often depends on the scene in view. In this paper the main idea is based on the interaction between 3D motion and shape which allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, then we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that are locally unsmooth, i.e., they vary more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the \smoothest estimated depth as well as the image locations of scene discontinuities. Although no optic flow values are computed, we show that our algorithm is very much related to minimizing the epipolar constraint when the scene in view is smooth. When however the imaged scene is not smooth, the introduced constraint has in general different properties from the epipolar constraint and we present experimental results with real sequences where it performs better.

Objective To evaluate the effect of setup errors on the 2D image projection and image registration, and then propose an improved registration method based on mutual information. Methods An anthropomorphic head phantom was used to simulate the rotational and translational setup errors. The geometric disparities were reflected by the changes of mutual information. Known setup errors were intentionally introduced to twenty cases divided into two groups demarcated by 3 mm translation error and 3° rotation error: ten cases with larger errors and ten with smaller errors. Then the anterior-posterior and lateral portal images were captured by the electronic portal imaging device (EPID), based on which the setup errors were calculated using two mutual information registration method respectively: the vender provided one, and the improved method as proposed. The calculated errors were compared with the actual setup errors to evaluate robustness of the method. Results For the ten cases with smaller setup errors, the average translational registration disparities using the conventional method were 0.3, 0.4, and 0.3 mm in x, y and z directions respectively. The rotational disagreements were 0.4° in both x and z directions. The average time consumption was 28.7 s. The corresponding discrepancies analyzed using the improved method were 0.3, 0.4, 0.3 mm, 0.5° and 0.4°, respectively. On average, 31.1 s was needed for registration. For the ten cases with larger setup errors, the mean disparities of the conventional method were 0.9, 0.7, 0.8 mm, 0.9° and 0.8°, 29.9 s taken on average. The corresponding result of the improved method was 0.5, 0.4, 0.5 mm, 0.6° and 0.5°, 33.2 s taken on average. Conclusions Regarding smaller setup errors, the two methods showed little difference and both had good performance in image registration accuracy. For larger setup errors, however, the improved mutual information registration method exhibited significantly higher accuracy than the conventional method, at cost of clinically acceptable registration time. Key words: Setup errors; Electronic portal imaging device; Portal image; Image registration

Real-time 3D optical tracking of free-hand imaging devices or surgical tools has been studied and employed for object localization in many minimally invasive interventions. However, the surgical workspace for many interventional procedures is often sub-dermal with tool access through ports from surgical incisions or anatomical orifices. To maintain the optical line-of-sight criterion, external extensions of inserted imaging devices and rigid surgical tools must be tracked to localize the internal tool tips. Unfortunately, tracking by this form of correspondence is very susceptible to noise as orientation errors on the external tracked end compound into both rotational and translational errors on the internal, workspace position. These translational errors are proportional to the length of the probe and the sine of the angulation error, so small angulation errors can quickly compromise the accuracy of the tool tip localization. We propose a real-time tracking correction technique that uses the rotational fulcrum created by the device entry port to minimize the effect of translational and rotational noise errors for tool tip localization. Our technique could apply to many types of interventions, but we focus on the application to the prostate biopsy procedure for tracking a transrectal ultrasound (TRUS) probe commonly used for prostate biopsies. In vitro studies were performed using the Claron Technology MicronTracker 2 to track a TRUS probe in a fixed rotational device. Our experimental results showed an order of magnitude improvement in RMS localization of the internal TRUS probe tip using fulcrum correction over the raw tracking information.

Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

BackgroundDry eye syndrome has been recently reported in patients who underwent whole brain radiotherapy (WBRT). WBRT based on a couch with three‐degrees of freedom (3D) can occasionally be performed in which the rotational head motion is not corrected. This study assessed the dependency of the rotational errors on the mask and the dose variation of the lens and lacrimal gland in WBRT patients.MethodsTranslational and rotational setup errors were obtained at the first treatment with cone‐beam CT (CBCT) for patients under WBRT and frameless stereotactic radiosurgery (SRS) (n = 20 each) immobilized using a conventional WB mask and an SRS mask with a bite block, respectively. For the CT sets of SRS cases, WBRT plans were generated for the study. To simulate the rotational error, rotated CT images were created with each rotational error, on which initial WBRT plans were copied and doses were recalculated. The lens and lacrimal gland doses with and without rotation errors were compared.ResultsDespite similar translational setup errors for the two masks, the SRS mask showed a dramatic reduction in rotational errors compared to those of the WB mask. The errors varied within −2.9° to 2.9° and −1.2° to 0.7° for the WB and SRS masks, respectively. Accordingly, the SRS mask confined the change in the maximum lens dose, mean dose of the lacrimal gland, and lacrimal volume receiving 15 Gy to one‐third of those using the WB mask.ConclusionWhen the six‐degrees of freedom (6D) couch is not available, the frameless SRS mask is beneficial to WBRT for the faithful treatment as it was planned.

Evaluation of the dosimetric influence of interfractional 6D setup error in hypofractionated prostate cancer treated with IMRT and VMAT using daily kV-CBCT

We present a proof of principle, experimental validation of the potential of proton ‘Range Probes’ (RP) for patient positioning verification in proton therapy. In this work, we have evaluated experimentally the accuracy of RP by using tissue-like samples and an in-house developed multilayer ionization chamber (MLIC). In addition we build on our previous, simulation based work to present first experimental measurements of RP through anthropomorphic phantoms to detect either rotational or translational positioning errors. For this, a technique has been proposed to characterize the residual integral depth dose curve (RIDDC) after range mixing. This parametrization has been used to evaluate the similarity between Monte Carlo calculated error scenarios of the database and the measured RIDDC, while considering the intrinsic uncertainties of both modalities in order to deduce the positioning error. Finally, the additional dose applied to the patient when using clinical RP with known fluence has been estimated by measuring the local dose of a single RP. In tissue phantoms, the prediction accuracy of the water equivalent path length was 0.70%, with the highest deviations being found in low density samples (up to 5.67%). In addition, the results of the patient positioning verification measurements demonstrated that using carefully selected RPs, 1D translational or rotational errors could be detected with an accuracy of 1 mm and 2°, respectively, and that these would be associated with a low additional dose burden to the patient. In summary, these promising results suggest that the RP method could be a simple, fast and low-dose tool for verifying patient set-up during proton therapy treatment.

The aim of this study is to calculate setup margin based on six-dimensional (6D) corrected residual positional errors from kV cone beam computed tomography (CBCT) and from intrafraction projection kV imaging in coplanar and in noncoplanar couch positions in stereotactic radiotherapy. Six dimensional positional corrections were carried out before patient treatments, using a robotic couch and CBCT matching. A CBCT and stereoscopic ExacTrac image were acquired post-table position correction. Further, a series of intrafraction ExacTrac images were obtained for the variable couch position. Translational and rotational errors were identified as lateral (X), longitudinal (Y), vertical (Z); roll (Ɵ°), pitch (Φ°) and yaw (Ψ°). A total of 699 intrafraction image sets (361 coplanar and 338 noncoplanar) for 51 SRS/SRT patients were analysed. Rotational errors were corrected in terms of translational coordinates. Residual set-up margins were calculated from CBCT shifts. ExacTrac shifts give residual+intrafraction setup margins as a function of coplanar and noncoplanar couchpositions. The average residual positional error obtained from CBCT in X, Y, Z, Ɵ, Φ, Ψ were 0.1±0.4mm, 0.0±0.6mm,0.0±0.5mm, 0.2±0.8°, 0.1±0.6° and -0.1±0.7° respectively. For ExacTrac, the shits were -0.5±0.9mm, -0.0±1mm, -0.6±1.0mm, 0.4±0.9°, -0.2±0.6°, and -0.0±0.8°. CBCT calculated linear setup margins in X, Y, Z direction were 0.5, 1.2, and 1mm respectively. ExacTrac yielded coplanar and noncoplanar linear setup margins were 1.2, 1.3, 1.5, 1.4, 1.5, and 2.1mm respectively. CBCT-based gross residual set-up margin is equal to 1mm. ExacTrac calculated residual plus intrafraction setup margin falls within a 2mm range; attributed to intrafraction patient movement, table position inaccuracies, and poor image fusion in noncoplanar geometry. There could be variations in the required additional margin between centers and between machines, which require further studies.

Quality assurance of a system for improved target localization and patient set-up that combines real-time infrared tracking and stereoscopic X-ray imaging.

Head pose estimation is important for driver attention monitoring as well as for various human computer interaction tasks. In this paper, an adaptive head pose estimation method is proposed to overcome difficulties of existing approaches. The proposed method is based on the analysis of two approaches for head pose estimation from an image sequence, that is, principal component analysis (PCA) and 3D motion estimation. The algorithm performs accurate pose estimation by learning the subject appearance on-line. Depth information is used effectively in the algorithm to segment the head region even in a cluttered scene and to perform 3D head motion estimation based on optical flow constraints.

We introduce a novel approach for bronchoscopic navigation that leverages neural radiance fields (NeRF) to passively locate the endoscope solely from bronchoscopic images. This approach aims to overcome the limitations and challenges of current bronchoscopic navigation tools that rely on external infrastructures or require active adjustment of the bronchoscope. To address the challenges, we leverage NeRF for bronchoscopic navigation, enabling passive endoscope localization from bronchoscopic images. We develop a two-stage pipeline: offline training using preoperative data and online passive pose estimation during surgery. To enhance performance, we employ Anderson acceleration and incorporate semantic appearance transfer to deal with the sim-to-real gap between training and inference stages. We assessed the viability of our approach by conducting tests on virtual bronchscopic images and a physical phantom against the SLAM-based methods. The average rotation error in our virtual dataset is about 3.18 and the translation error is around 4.95 mm. On the physical phantom test, the average rotation and translation error are approximately 5.14 and 13.12 mm. Our NeRF-based bronchoscopic navigation method eliminates reliance on external infrastructures and active adjustments, offering promising advancements in bronchoscopic navigation. Experimental validation on simulation and real-world phantom models demonstrates its efficacy in addressing challenges like low texture and challenging lighting conditions.

A multi-segments stitching method is proposed to obtain two-dimensional profile of large scale aspheric components during grinding process. We firstly analyze the corresponding relation between surface features of aspherics and measurement range of profilometer. According to multi-body system theory and slope difference, a mathematical model of multi-segments stitching method is constructed. Further, the multi-segments stitching errors based on 400 mm aspheric profile are simulated with different translation amounts, rotation angles, translation errors as well as rotation errors. The simulation results indicate that the translation error is the key factor for measuring accuracy of the multi-segments stitching method. In addition, the standard deviations of multi-segments stitching errors are between 0.2 μm and 0.4 μm as the four factors are set to the proper values. To verify the simulation analysis results, an experimental setup, including Talysurf PGI 1240 and fixture, is applied to detect a 150 mm profile line on the aspheric surface. The experiment results show that the standard deviations of multi-segments stitching errors are almost within 1 μm when the rotation angle is under 7° and the translation amount is under 11 mm. The multi-segments stitching method can employ a small-range profilometer to achieve measurement of large scale aspheric surfaces with submicrometer accuracy.

This paper describes a 3D model-based unsupervised procedure for the segmentation of multiview image sequences using multiple sources of information. Using multiview information a 3D model representation of the scene is constructed. The articulation procedure is based on the homogeneity of parameters, such as rigid 3D motion, color and depth, estimated for each sub-object, which consists of a number of interconnected triangles of the 3D model. The rigid 3D motion of each sub-object for subsequent frames is estimated using a Kalman filtering algorithm taking into account the temporal correlation between consecutive frames. Information from all cameras is combined during the formation of the equations for the rigid 3D motion parameters. The parameter estimation for each sub-object and the 3D model segmentation procedures are interleaved and repeated iteratively until a satisfactory object segmentation emerges. The performance of the resulting segmentation method is evaluated experimentally.

In recent years, SLAM, visual odometry, and structure-from-motion approaches have widely addressed the problems of 3D reconstruction and ego-motion estimation. Of the many input modalities that can be used to solve these ill-posed problems, the pure visual alternative using a single monocular RGB camera has attracted the attention of multiple researchers due to its low cost and widespread availability in handheld devices. One of the best proposals currently available is the Direct Sparse Odometry (DSO) system, which has demonstrated the ability to accurately recover trajectories and depth maps using monocular sequences as the only source of information. Given the impressive advances in single-image depth estimation using neural networks, this work proposes an extension of the DSO system, named DeepDSO. DeepDSO effectively integrates the state-of-the-art NeW CRF neural network as a depth estimation module, providing depth prior information for each candidate point. This reduces the point search interval over the epipolar line. This integration improves the DSO algorithm’s depth point initialization and allows each proposed point to converge faster to its true depth. Experimentation carried out in the TUM-Mono dataset demonstrated that adding the neural network depth estimation module to the DSO pipeline significantly reduced rotation, translation, scale, start-segment alignment, end-segment alignment, and RMSE errors.