Beyond the Epipolar Constraint: Integrating 3D Motion and Structure Estimation

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patchess but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images. The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the “least varying” estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.

Effects of Errors in the Viewing Geometry on Shape Estimation

Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

If 3D rigid motion can be correctly estimated from image sequences, the structure of the scene can be correctly derived using the equations for image formation. However, an error in the estimation of 3D motion will result in the computation of a distorted version of the scene structure. Of computational interest are these regions in space where the distortions are such that the depths become negative, because in order for the scene to be visible it has to lie in front of the image, and thus the corresponding depth estimates have to be positive. The stability analysis for the structure from motion problem presented in this paper investigates the optimal relationship between the errors in the estimated translational and rotational parameters of a rigid motion that results in the estimation of a minimum number of negative depth values. The input used is the value of the flow along some direction, which is more general than optic flow or correspondence. For a planar retina it is shown that the optimal configuration is achieved when the projections of the translational and rotational errors on the image plane are perpendicular. Furthermore, the projection of the actual and the estimated translation lie on a line through the center. For a spherical retina, given a rotational error, the optimal translation is the correct one; given a translational error, the optimal rotational negative deptherror depends both in direction and value on the actual and estimated translation as well as the scene in view. The proofs, besides illuminating the confounding of translation and rotation in structure from motion, have an important application to ecological optics. The same analysis provides a computational explanation of why it is easier to estimate self-motion in the case of a spherical retina and why shape can be estimated easily in the case of a planar retina, thus suggesting that nature‘s design of compound eyes (or panoramic vision) for flying systems and camera-type eyes for primates (and other systems that perform manipulation) is optimal.

Since estimation of camera motion requires knowledge of independent motion, and moving object detection and localization requires knowledge about the camera motion, the two problems of motion estimation and segmentation need to be solved together in a synergistic manner. This paper provides an approach to treating both these problems simultaneously. The technique introduced here is based on a novel concept, ruggedness which parameterizes the variation in estimated scene depth with the error in the underlying three-dimensional (3D) motion. The idea is that incorrect 3D motion estimates cause distortions in the estimated depth map, and as a result smooth scene patches are computed as rugged surfaces. The correct 3D motion can be distinguished, as it does not cause any distortion and thus gives rise to the background patches with the least depth variation between depth discontinuities, with the locations corresponding to independent motion being rugged. The algorithm presented employs a binocular observer whose nature is exploited in the extraction of depth discontinuities, a step that facilitates the overall procedure, but the technique can be extended to a monocular observer in a variety of ways.

Head pose estimation is important for driver attention monitoring as well as for various human computer interaction tasks. In this paper, an adaptive head pose estimation method is proposed to overcome difficulties of existing approaches. The proposed method is based on the analysis of two approaches for head pose estimation from an image sequence, that is, principal component analysis (PCA) and 3D motion estimation. The algorithm performs accurate pose estimation by learning the subject appearance on-line. Depth information is used effectively in the algorithm to segment the head region even in a cluttered scene and to perform 3D head motion estimation based on optical flow constraints.

In automated feature-based motion analysis of multiple frames, correspondence data are usually noisy and fragmented. A technique that gradually refines the initial noisy correspondence data and links fragments of a single trajectory using feedback from 3D motion estimation is presented. First, 3D motion parameters are estimated using the initial correspondence data. Then, each noisy trajectory is partitioned into subsets of points, each of which conforms to the estimated motion. The best set is used as the input to the next motion estimation. This process is repeated, and the gaps in the refined correspondence data are filled by guidance from the predicted motion. Test results for a standard real image sequence are presented. >

In elastography we have previously developed a tracking and correctionmethod that estimates the axial and lateral strain components along andperpendicular to the compressor/scanning axis following an externallyapplied compression. However, the resulting motion is a three-dimensionalproblem. Therefore, in order to fully describe this motion we need toconsider a 3D model and estimate all three principal strain components,i.e. axial, lateral and elevational (out-of-plane), for a full 3D tensordescription.Since motion is coupled in all three dimensions, the three motioncomponents have to be decoupled prior to their estimation. In this paper,we describe a method that estimates and corrects motion in threedimensions, which is an extension of the 2D motion tracking and correctionmethod discussed before. In a similar way as in the 2D motion estimation,and by assuming that ultrasonic frames are available in more than oneparallel elevational plane, we used methods of interpolation andcross-correlation between elevationally displaced RF echo segments toestimate the elevational displacement and strain. In addition, the axial,lateral and elevational displacements were used to estimate all three shearstrain components that, together with the normal strain estimates, fullydescribe the full 3D normal strain tensor resulting from the uniformcompression. Results of this method from three-dimensional finite-elementsimulations are shown.

The optic flow generated when a person moves through the environment can be locally decomposed into several basic components, including radial, circular, translational and spiral motion. Since their analysis plays an important part in the visual perception and control of locomotion and posture it is likely that some brain regions in the primate dorsal visual pathway are specialized to distinguish among them. The aim of this study is to explore the sensitivity to different types of egomotion-compatible visual stimulations in the human motion-sensitive regions of the brain. Event-related fMRI experiments, 3D motion and wide-field stimulation, functional localizers and brain mapping methods were used to study the sensitivity of six distinct motion areas (V6, MT, MST+, V3A, CSv and an Intra-Parietal Sulcus motion [IPSmot] region) to different types of optic flow stimuli. Results show that only areas V6, MST+ and IPSmot are specialized in distinguishing among the various types of flow patterns, with a high response for the translational flow which was maximum in V6 and IPSmot and less marked in MST+. Given that during egomotion the translational optic flow conveys differential information about the near and far external objects, areas V6 and IPSmot likely process visual egomotion signals to extract information about the relative distance of objects with respect to the observer. Since area V6 is also involved in distinguishing object-motion from self-motion, it could provide information about location in space of moving and static objects during self-motion, particularly in a dynamically unstable environment.

The Radixact Synchrony system allows for target motion correction when tracking either fiducials in/around the target or a dense lesion in the lung. As such evaluation testing and quality assurance (QA) tests are required. To allow for QA procedures to be performed with a range of available phantoms evaluation of the dosimetric delivery accuracy was performed for a range of motions, phantoms, and motion platforms. A Computerized Imaging Reference Systems, Incorporated (CIRS) 1D motion platform and Accuray Tomotherapy "Cheese" phantom was utilized to perform absolute dose and GafChromic EBT3 film measurements. A HexaMotion platform and Delta4 phantom were utilized to quantify the effects of 1D and 3D motions. Inter-device comparison was performed with the ArcCHECK and Delta4 phantoms and GafChromic EBT3 film, five patient plans were delivered to each phantom when static and with two different motion types both with and without Synchrony motion correction. A range of QA tests are described. A phantom was designed to allow for daily verification of system functionality. This test allows for the detection of either fiducials or a dense silicone target with a stationary phantom. Monthly testing procedures are described that allow the user to verify the dosimetric improvement when utilizing synchrony delivery motion compensation versus uncorrected motions. These can be performed utilizing a 1D motion stage with an ion-chamber and GafChromic EBT3 film to allow for a 2D dosimetric validation. Alternatively, a 3D motion platform can be utilized where available. Monthly and annual imaging tests are described. Finally, annual test procedures designed to verify the coincidence of the imaging system and treatment isocenter are described. The evaluation of the Synchrony system using a range of QA devices shows consistently high dosimetric accuracy with similar trends in passing criteria found with GafChromic EBT3 film, ArcCHECK, and Delta4 phantoms for density-based respiratory model compensation. These results highlight the large improvements in the dose distribution when motion is accounted for with the Synchrony system as measured with a range of phantoms and motion platforms that the majority of users will have available. The testing methods and QA procedures described provide guidance for new users of the Radixact Synchrony system as they implement their own QA programs for this system, until such time as an AAPM task group report is made available. QA procedures including Kilovolts (kV) imaging quality metrics and imaging dose parameters, dose deposition accuracy, target detection coincidence, and target position detection accuracy are described.

Event Abstract Back to Event Area MT pattern motion selectivity by integrating 1D and 2D motion features from V1 - a neural model Problem The neural mechanisms of detecting, integrating, and disambiguating visual motion still remain largely a puzzle. While many neurons in area V1 show coarse direction and speed selectivity signaling component motion direction, evidence suggests that end-stop cells in V1 compute the true motion direction for 2D image features (Pack et al., Neuron 39, 2003). In area MT, neurons have been found that are selective to speed and respond to pattern motion direction in the case of plaids from combined differently oriented gratings (Movshon et al., Pattern Recognition Mechanisms, 1985). For elongated bars, MT neurons resolve the so-called aperture problem to signal the correct object motion after a temporal course of disambiguation (Pack&Born, Nature 409, 2001). The construction of selectivity to pattern motion from component selective input is a topic of ongoing research. Different experiments suggest that the visual system might use computational strategies of integrating different motion directions or selecting localized features (Pack&Born, The Senses: A Comprehensive Reference, 2008). Methods We propose a neurodynamical model of motion integration in areas V1, MT, and MSTv which unifies the integrationist and selectionist concepts. Normal flow responses are initially detected by model V1 simple/complex cells while inhibitory interactions between cells at different V1 laminae generate local end-stop responses after a temporal delay. These responses are integrated for a short temporal episode and enhanced to generate strong direction signals. MT neurons then receive input from V1 complex and end-stop neurons (size ratio 1:5, direction tuning +/-40°). The integration of component selective responses leads to weak pattern direction selectivity in MT, further sharpened by the input from localized end-stop neurons. Feedback from MT to V1 enhances neurons with feature selectivity and slightly reduces normal flow responses near localized features. Feedforward and feedback connections between MT and MSTv (size ratio 1:1.25) disambiguate MT responses along extended outline boundaries. Results and Conclusion To investigate the aperture problem, we tested the model with a vertically aligned bar moving in diagonal direction. Consistent with neurophysiological findings (Pack&Born, Nature 409, 2001), after few iterations MT neurons showed sharp speed and direction tuning for the correct velocity, while the tuning of V1 neurons only slightly changed. Originating from 2D signals at corners, the correct flow was further propagated in MT along the stimulus by each iteration while propagation time increased with bar length (Born et al., Prog. Brain Res. 140, 2002). For a plaid stimulus, model MT neurons were tuned to pattern motion after some iterations, while V1 neurons indicated normal flow responses along the plaid components, except for end-stop neurons located at crossings. Hence, V1 neurons already provided some of the correct 2D cues, their overall tuning nevertheless remained coarse in speed and direction. To conclude, the integration of 1D and 2D V1 motion features in MT as proposed by our model suggests the generation of MT cell pattern selectivity based on component integration and feature selection. The model explains findings of various neurophysiological experiments and unifies different modeling approaches into one framework. Supported by EU-project 027198 (Decisions-in-Motion) Conference: Computational and systems neuroscience 2009, Salt Lake City, UT, United States, 26 Feb - 3 Mar, 2009. Presentation Type: Poster Presentation Topic: Poster Presentations Citation: (2009). Area MT pattern motion selectivity by integrating 1D and 2D motion features from V1 - a neural model. Front. Syst. Neurosci. Conference Abstract: Computational and systems neuroscience 2009. doi: 10.3389/conf.neuro.06.2009.03.163 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 04 Feb 2009; Published Online: 04 Feb 2009. Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Google Google Scholar PubMed Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Dense stereo image matching in remotely sensed images is a challenging problem, though it has been studied for more than two decades, due to occlusions, discontinuities, geometric, and radiometric distortions. A novel multiobjective fitness function-based dense stereo matching approach using genetic algorithms (GAs) is proposed in this paper. The proposed method is useful for estimating dense disparity map with an improved number of inliers for a stereo image pair, despite the constraint of finding correct disparity at depth discontinuities. In this paper, the steps of GA, such as initialization of the population, fitness function, and crossover and mutation operation, are designed and implemented to effectively deal with the problem of dense stereo image matching. To initialize the population, a Scale Invariant Feature Transform (SIFT) descriptor is computed for each pixel and multiple-size window-based matching is performed, using the similarity measures: 1) Euclidean distance and 2) spectral angle mapper. The generated disparity maps are pruned to choose a suitable subset using the designed fitness functions, considering the constraints related to stereo image pair, such as epipolar constraint, which encodes the epipolar geometry and the similarity measure that is useful to decide accuracy of the correspondences. The two objective functions are the number of inliers computed using the fundamental matrix and an energy minimization function, considering discontinuities and occlusions. The usefulness of this approach for remotely sensed stereo image pairs is demonstrated by improving the number of inliers and favorably comparing with state-of-the-art dense stereo image matching methods.

This paper introduces a feature based method for the fast generation of sparse 3D point clouds from multiple images with known pose. We extract sub-pixel edge elements (2D position plus associated orientation) and use a space sweeping scheme to compute the accurate 3D location of these edge features. Our approach relies mainly on the geometric properties of the extracted primitives and incorporates a robust uncertainty estimation to detect outliers. Epipolar constraints between views are used to narrow down the search space of potential candidates in the images. In order to improve the efficiency of spatial queries, for detecting edgels lying close to an epipolar line, we utilise a pairwise stereo rectification scheme. The detection and verification of tentative hypotheses is carried out in 3D-space, thus allowing to perform an event driven search mode. An uncertainty measure that models the location inaccuracies in the feature extraction process and errors introduced in the camera pose estimation stage allows to assign a likelihood value to each hypothesis. Optionally an image-based similarity measure can be used to verify the 3D hypotheses and to identify false positives. We perform experiments on a synthetic data set and on several real datasets. The results indicate that the proposed method yields accurate measurements on depth discontinuities and thus represents a complementary technique to standard dense matching approaches.

Motion estimation algorithm plays a crucial role in video coding/processing and accounts for almost one quarter of the gate count or area in many digital video system-on-a-chip (SoC) due to its computational intensive nature. In this paper, we present a complexity-reduced three dimensional (3D) motion estimator (ME) which reduces the number of search candidates and hence gate count significantly. Moreover, by virtue of its recursive architecture, hardware estimated sum of absolute differences (SAD) and motion vectors (MV) can be passed backed to the embedded DSP processor for image segmentation, resulting in efficient software/hardware partition in SoC designs. In addition, the calculated SAD's can also be further analyzed by the embedded CPU for rate control in video coding. Alleviating traditional ME artifacts such as illumination, this motion estimator with segmentation capabilities is readily applicable to video object planes (VOP) processing in levels higher than the advance simple profile in MPEG4, H.264 and motion compensated format converters such as deinterlacers and scan-rate converters.

Dynamic Vision Sensors (DVS) provide low-latency, high-dynamic-range motion estimation, but their real-time applicability is often limited by the computational complexity and latency overheads introduced by iterative motion compensation techniques. In this work, we propose a novel probabilistic model that leverages the stochastic distribution of events along moving edges. Using our model, we introduce a lightweight patch-based algorithm that employs a linear combination of event spatial coordinates, making it highly suitable for implementation on specialized hardware. Our approach exhibits linear scalability with dimensionality, making it suitable for emerging event-based 3D sensors, such as Light-Field DVS (LF-DVS). Experimental results validate the efficiency and scalability of our method, establishing a solid foundation for real-time event-based ultra-efficient 2D and 3D motion estimation.