Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

This paper develops a novel solution to the problem of recovering the structure of a scene given an uncalibrated video sequence depicting the scene. The essence of the technique lies in a method for recovering the rigid transformation between the different views in the image sequence. Knowledge of this 3D motion allows for self-calibration and for subsequent recovery of 3D structure. The introduced method breaks away from applying only the traditionally used epipolar constraint and introduces a new constraint based on the interaction between 3D motion and shape. Up to now, structure from motion algorithms proceeded in two well defined steps, where the first and most important step is recovering the rigid transformation between two views, and the subsequent step is using this transformation to compute the structure of the scene in view. Here both aforementioned steps are accomplished in a synergistic manner. Existing approaches to 3D motion estimation are mostly based on the use of optic flow which however poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) estimate accurately flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. In the past this dilemma has been addressed by improving the estimation of flow through sophisticated optimization techniques, whose performance often depends on the scene in view. In this paper the main idea is based on the interaction between 3D motion and shape which allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, then we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that are locally unsmooth, i.e., they vary more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the \smoothest estimated depth as well as the image locations of scene discontinuities. Although no optic flow values are computed, we show that our algorithm is very much related to minimizing the epipolar constraint when the scene in view is smooth. When however the imaged scene is not smooth, the introduced constraint has in general different properties from the epipolar constraint and we present experimental results with real sequences where it performs better.

Distance estimation from visually simulated self-motion is imprecise. Depending on the evaluation method, travelled distance can be under- or over-estimated. One particular method consists of asking a stationary observer, exposed to an immersive optical flow, simulating forward self-motion, to indicate when s/he thinks s/he has reached the remembered position of a previously seen distant target. In this case, subjective evaluation of travelled distance is generally overestimated (i.e., the subject undershoots the target). Recent studies suggest that a translational optical flow with biological additional oscillations (simulating the optical effects of natural locomotion) would increase the sensation of walking and improve spatial perception, as compared to a purely translational optical flow. In the present study we tested this hypothesis, by measuring travelled distance estimation, according to two conditions of visual simulation of forward self-motion, at constant speed, in a CAVE setup: (1) an optical flow simulating pure forward translation (2) an optical flow with added "biological" oscillations, reproducing the optical effects of the natural motion of the head during walking. Our results show that an optical flow containing additional biological information enhances the accuracy of travelled distance estimation. The perceptual advantage provided by the biological oscillations in the optical flow is discussed.

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patchess but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images. The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the “least varying” estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.

If 3D rigid motion can be correctly estimated from image sequences, the structure of the scene can be correctly derived using the equations for image formation. However, an error in the estimation of 3D motion will result in the computation of a distorted version of the scene structure. Of computational interest are these regions in space where the distortions are such that the depths become negative, because in order for the scene to be visible it has to lie in front of the image, and thus the corresponding depth estimates have to be positive. The stability analysis for the structure from motion problem presented in this paper investigates the optimal relationship between the errors in the estimated translational and rotational parameters of a rigid motion that results in the estimation of a minimum number of negative depth values. The input used is the value of the flow along some direction, which is more general than optic flow or correspondence. For a planar retina it is shown that the optimal configuration is achieved when the projections of the translational and rotational errors on the image plane are perpendicular. Furthermore, the projection of the actual and the estimated translation lie on a line through the center. For a spherical retina, given a rotational error, the optimal translation is the correct one; given a translational error, the optimal rotational negative deptherror depends both in direction and value on the actual and estimated translation as well as the scene in view. The proofs, besides illuminating the confounding of translation and rotation in structure from motion, have an important application to ecological optics. The same analysis provides a computational explanation of why it is easier to estimate self-motion in the case of a spherical retina and why shape can be estimated easily in the case of a planar retina, thus suggesting that nature‘s design of compound eyes (or panoramic vision) for flying systems and camera-type eyes for primates (and other systems that perform manipulation) is optimal.

Head pose estimation is important for driver attention monitoring as well as for various human computer interaction tasks. In this paper, an adaptive head pose estimation method is proposed to overcome difficulties of existing approaches. The proposed method is based on the analysis of two approaches for head pose estimation from an image sequence, that is, principal component analysis (PCA) and 3D motion estimation. The algorithm performs accurate pose estimation by learning the subject appearance on-line. Depth information is used effectively in the algorithm to segment the head region even in a cluttered scene and to perform 3D head motion estimation based on optical flow constraints.

In this paper, investigations of brain activity response from experimental works, based on optic flow and wavelet using near infrared spectroscopy are presented. The outcomes of our research project showed that optical flow is a proper technology for the investigation of brain activity based on near-infrared spectroscopy (NIRS). Our research also showed that the results in the temporal domain, spatial domain, and wavelet domain are coherently supporting the analyses of our experimental results. Our wavelet analysis can obtain the majority of the brain activity for the spatiotemporal distribution. Based on the spatiotemporal analysis, we can clearly observe the existence of the dominant channel (DC), which plays a critical role in the brain activity. The spatial distribution of the cortical activity origin can be described by the hemodynamic response in the cerebral cortex after evoked stimulation using near infrared spectroscopy. Further application is expected in the next research outcomes.

Vision-based control is being aggressively pursued for autonomous systems. Such control is particularly valuable for path planning to achieve mission objectives like target tracking and obstacle avoidance. This paper presents a multi-rate strategy that utilises a fast-rate optic flow approach and a slow-rate scene reconstruction approach. The vehicle uses scene reconstruction to generate an accurate map for path planning; however, optic flow is used to avoid obstacles while the scene reconstruction is computed. A switch element is used in the feedback path to determine whether information relating to the reconstructed map or the optical flow should be used for navigation. The resulting controller is able to generate flight trajectories and perform obstacle avoidance within a computational cost which is reasonable given performance demands and computational resources on a wide range of aircraft. A simulation demonstrates the performance of an aircraft that uses the multi-rate controller to avoid an obstacle which is only observed after a turn. Essentially, the fast-rate optic flow indicates the presence of the obstacle during the time that slow-rate scene reconstruction is being performed. The resulting flight path is able to follow mission objectives and avoid a collision.

The fly visual system, although tiny when compared to the mammalian visual system, can still perform highly sophisticated spatial tasks like collision avoidance, landing on objects, pursuit of prey, etc. Flies outperform human-made autonomous flying systems in solving such spatial tasks by a long way. This is partly due to their ability to perceive and respond to optical flow generated by motion in the environment. They are also known to take actions that alter the image flow on their eyes. Higher level neurons in the fly visual system respond to different types of complex optical flows due to rotation and translation, by pooling information from local motion detectors called the elementary motion detectors (EMDs) in the lower level. In this sense, neuronal responses (spikes) from these optical flow detectors in the fly carry highly encoded signals: a single spike can represent a complex dynamical pattern of movement in the visual field. In this paper, we investigate how such highly encoded signals can be interpreted and utilized in the fly's brain, while solely operating on the encoded internal spike patterns within their brain and no direct external (unencoded) sensory information, i.e. a form of grounding. With a computational model of the optical flow detectors based on those in the fly, we show that specific pattern of action (or coordinated motor output) is the only way that a fly can decode its internal spikes and generate meaningful, relevant behavior based on that.

Effects of Errors in the Viewing Geometry on Shape Estimation

An autonomous system must have the capability of estimating or controlling its own motion parameters. There already exit tens of research work to fulfill the task. However, most of them are based on the motion correspondences establishment or full optical flows estimation. The above solutions put restrictions on the scene: either there must be presence of enough distinct features, or there must be dense texture. Different from the traditional works, utilizing no motion correspondences or epipolar geometry, we start from the normal flow data, ensure good use of every piece of them because they could only be sparsely available. We apply the spherical image model to avoid the ambiguity in describing the camera motion. Since each normal flow gives a locus for the location of the camera motion, the intersection of such loci offered by different data points will narrow the possibilities of the camera motion and even pinpoint it. A voting scheme in 3 - domain is applied to simplify the 3D voti ng space to a 2D votin g space. We tested the algorithms introduced above by using both synthetic imag e data and real image sequences. Experimental results are shown to illustrate the potential of the methods. Keywords: camera ego-motion, optical flow, normal flow.

Additive manufacturing has deeply simplified the production of prototypes by speeding up the realization of complex 3- dimensional objects. A well-designed mechanical part requires perfect knowledge of internal stresses and mechanical behavior of the used material. In this study, we propose a new application for the optical-flow method in the field of extensometry. This method has already proven its capabilities for measuring the displacements and speeds of moving objects in video sequences for many motion tracking applications. The orientation of the fused filament deposited strands influences strongly the behavior of the FDM-manufactured part. In order to study the behavior of a mechanical part as a function of the arrangement of the deposited strands, we produced, using FDM, tensile test specimens by superposing several homogeneous layers. Each of the layers is characterized by a fixed strands orientation. These specimens are then tested on a tensile test machine. Due to the different arrangements of the strands according to the layers, the deformations on the surface of the sample can be complex. A conventional extensometer is then no longer usable. Accordingly, we propose to apply the optical flow method to a video of the test sample surface recorded during a tensile test. We present the experimental setup of the measurement method, the implemented algorithm based on a Mathematica optic flow builtin function and some illustration results. The results show that this technique can be used on real parts subjected to stress in order to reveal weakened areas.

Optical flow based super-resolution: A probabilistic approach

Generation of a sequence of behaviors is necessary for the RoboCup Humanoid league to realize not simply an individual robot performance but also cooperative ones between robots. A typical example task is passing a ball between two humanoids, and the issues are: (1) basic skill decomposition, (2) skill learning, and (3) planning to connect the learned skills. This paper presents three methods for basic skill learning (trapping, approaching to, and kicking a ball) based on optic flow information by which a robot obtains sensorimotor mapping to realize the desired skill, assuming that skill decomposition and planning are given in advance. First, optic flow information of the ball is used to predict the trapping point. Next, the flow information caused by the self-motion is classified into the representative vectors, each of which is connected to motor modules and their parameters. Finally, optical flow for the environment caused by kicking motion is used to predict the ball trajectory after kicking. The experimental results are shown and discussion is given with future issues.

Sensitivity of 8-week-old infants to optical flow specifying the shape of a three-dimensional object was assessed. Infants viewed kinetic random-dot displays that specified three-dimensional cubes. The cubes were identical except for the presence or absence of an interior corner. Half of the infants viewed the full display. The other half viewed the central region of the displays, where the flow specifying the presence or absence of the corner differed. Infants in the full-view condition looked significantly longer to a novel cube than to the familiar cube following habituation. In contrast, infants in the partial-view condition looked equally to the novel and familiar cubes, ruling out the possibility that infants who viewed the full displays merely discriminated differences in motion in the central region of the two displays. These findings suggest that infants as young as 8 weeks perceive three-dimensional object shape from optic flow.

In this paper, we present the algorithm and operation of an aVLSI chip that can extract normal optical flow by using the gradient approach without interfering with the imaging process. This approach is feasible for scaling to larger arrays without affecting the processing or the processing area. Our system has a 92 x 52 photosensitive array of APS pixels at the core with processing circuits on the periphery. We discuss the approach and the different blocks in the design and then demonstrate the working of the individual blocks and of the system as a whole. The chip outputs the image, the spatial and temporal gradients and the normal flow at the read-out frame-rate with no penalty to the imaging process. The chip occupies an area of 4.5 mm² and consumes 2.6 mW (at V_dd = 5V). Once normal flow is obtained, the chip can be used to compute focus of expansion, time to contact and many other motion properties of images that can be used to control robots. Tracking systems can use the velocity and segmentation of moving objects can be realized using motion discontinuity.