Head pose estimation for driver monitoring

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patchess but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images. The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the “least varying” estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.

This paper develops a novel solution to the problem of recovering the structure of a scene given an uncalibrated video sequence depicting the scene. The essence of the technique lies in a method for recovering the rigid transformation between the different views in the image sequence. Knowledge of this 3D motion allows for self-calibration and for subsequent recovery of 3D structure. The introduced method breaks away from applying only the traditionally used epipolar constraint and introduces a new constraint based on the interaction between 3D motion and shape. Up to now, structure from motion algorithms proceeded in two well defined steps, where the first and most important step is recovering the rigid transformation between two views, and the subsequent step is using this transformation to compute the structure of the scene in view. Here both aforementioned steps are accomplished in a synergistic manner. Existing approaches to 3D motion estimation are mostly based on the use of optic flow which however poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) estimate accurately flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. In the past this dilemma has been addressed by improving the estimation of flow through sophisticated optimization techniques, whose performance often depends on the scene in view. In this paper the main idea is based on the interaction between 3D motion and shape which allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, then we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that are locally unsmooth, i.e., they vary more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the \smoothest estimated depth as well as the image locations of scene discontinuities. Although no optic flow values are computed, we show that our algorithm is very much related to minimizing the epipolar constraint when the scene in view is smooth. When however the imaged scene is not smooth, the introduced constraint has in general different properties from the epipolar constraint and we present experimental results with real sequences where it performs better.

If 3D rigid motion can be correctly estimated from image sequences, the structure of the scene can be correctly derived using the equations for image formation. However, an error in the estimation of 3D motion will result in the computation of a distorted version of the scene structure. Of computational interest are these regions in space where the distortions are such that the depths become negative, because in order for the scene to be visible it has to lie in front of the image, and thus the corresponding depth estimates have to be positive. The stability analysis for the structure from motion problem presented in this paper investigates the optimal relationship between the errors in the estimated translational and rotational parameters of a rigid motion that results in the estimation of a minimum number of negative depth values. The input used is the value of the flow along some direction, which is more general than optic flow or correspondence. For a planar retina it is shown that the optimal configuration is achieved when the projections of the translational and rotational errors on the image plane are perpendicular. Furthermore, the projection of the actual and the estimated translation lie on a line through the center. For a spherical retina, given a rotational error, the optimal translation is the correct one; given a translational error, the optimal rotational negative deptherror depends both in direction and value on the actual and estimated translation as well as the scene in view. The proofs, besides illuminating the confounding of translation and rotation in structure from motion, have an important application to ecological optics. The same analysis provides a computational explanation of why it is easier to estimate self-motion in the case of a spherical retina and why shape can be estimated easily in the case of a planar retina, thus suggesting that nature‘s design of compound eyes (or panoramic vision) for flying systems and camera-type eyes for primates (and other systems that perform manipulation) is optimal.

The main problem in stereo vision is the reconstruction of the 3D surface of objects in the scene from a pair of stereo images. This is performed by establishing correspondence between homologous points in each stereo image. Once correspondence is established it is straightforward to compute the coordinates of the corresponding 3D point. Although, important image analysis tasks like scene segmentation and 3D motion estimation may be performed using a monocular sequence, one may achieve better result by exploiting depth information from a stereo image sequence. Especially if depth estimation is combined with 3D motion estimation a considerable improvement in the accuracy of the estimates is expected. In this paper we present algorithms for the analysis of stereoscopic image sequences. Apart from the usefulness of stereoscopic imagin in compute revision, its application in advanced telecommunications is also of prime importance. Since the bandwidth required to transmit both stereoscopic image streams is large, efficient coding technique should be employed to reduce the data rate. In this paper we present an object-based approach that lacks the disadvantages of tradition block-based techniques. Also, the ability of this algorithm to describe a scene in a structural way, in contrast to traditional waveform-based coding techniques, opens new areas of applications.

In virtual environments, head pose and/or eye-gaze estimation can be employed to improve the visual experience of the user by enabling adaptive level of detail during rendering. In this study, we present a real-time system for rendering complex scenes in an immersive virtual environment based on head pose estimation and perceptual level of detail. In our system, the position and orientation of the head are estimated using stereo vision approach and markers placed on a pair of glasses used to view images projected on a stereo display device. The main innovation of our work is the incorporation of uncertainty estimates to improve the visual experience perceived by the user. The estimated pose and its uncertainty are used to determine the desired level of detail for different parts of the scene based on criteria originating from physiological and psychological aspects of human vision. Subject tests have been performed to evaluate our approach.

We present a method for head pose estimation for moving targets in multi-camera environments. Our approach utilizes an ensemble of exemplar classifiers for joint head detection and pose estimation and provides finer-grained predictions than previous approaches. We incorporate dynamic camera selection, which allows a variable number of cameras to be selected at each time step and provides a tunable trade-off between accuracy and speed. On a benchmark dataset for multi-camera head pose estimation, our method predicts head pan angle with a mean absolute error of ~ 8° for different moving targets.

The simultaneous recovery of three-dimensional (3D) structure from motion (SfM) for the sequences of images, is one of the more difficult problems in computer vision. Classical approaches to the problem rely on using algebraic techniques to solve for these unknowns given two or more image. Motion analysis and 3D shape estimation based on the estimated motion is an important problem in computer vision. The correspondence problem is an important tool in SfM where in this paper a general 3D motion based on a simple rotation, tilt, roll and translation is proposed, and then is used for 3D shape estimation. The current work expands the 2D motion estimation (shift, rotation) to accommodate general 3D motion (shift, rotation, tilt and roll). The proposed work in this paper of motion estimation of moving object is based on Fourier Descriptor Transformation (FDT) analysis before and after motion. The FDT is also used to resolve the correspondence problem in a sequence of images. We test our method on several large-scale photo collections, show the efficacy of the introduced approach to improve reconstruction accuracy.

To solve the problem of estimating velocities of gas bubbles in melted glass, a method based on optical flow constraint (OFC) has been extended to the 3D case. A single camera, whose distance to the fluid is variable in time, is used to capture a sequence of frames at different depths. Since objects are not static, we cannot obtain two frames of different height values at the same time, and to our knowledge, this prevents the use of common 3D motion estimation techniques. Since the information will be rather sparse, our estimation takes several measures around a given pixel and discards the erroneous ones, using a robust estimator. Along with the exposition of the practical application, the estimation proposed here is first contrasted in the 2D case to common benchmarks and then evaluated for a synthetic problem where velocities are known.

In automated feature-based motion analysis of multiple frames, correspondence data are usually noisy and fragmented. A technique that gradually refines the initial noisy correspondence data and links fragments of a single trajectory using feedback from 3D motion estimation is presented. First, 3D motion parameters are estimated using the initial correspondence data. Then, each noisy trajectory is partitioned into subsets of points, each of which conforms to the estimated motion. The best set is used as the input to the next motion estimation. This process is repeated, and the gaps in the refined correspondence data are filled by guidance from the predicted motion. Test results for a standard real image sequence are presented. >

Comunicació presentada a la 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), celebrada els dies 30 de maig a 3 de juny de 2017 a Washington DC, EUA.

Computer vision researchers have proved the feasibility of camera self-calibration —the estimation of a camera's internal parameters from an image sequence without any known scene structure. Various self-calibration algorithms have been published. Nevertheless, all of the recent sequential approaches to 3D structure and motion estimation from image sequences which have arisen in robotics and aim at real-time operation (often classed as visual SLAM or visual odometry) have relied on pre-calibrated cameras and have not attempted online calibration.

This paper proposes a new solution to improve the quality of depth map in 2D+depth map of 3D video coding. The proposed method based on H.264 variable block size motion estimation for 2D video coding in 3D video. As in the 2D video coding flow, motion estimation is also applied to depth map coding. However, the result of the predicted depth map is not precise enough only through 2D motion estimation. Therefore, a new idea grows, that is 3D motion estimation for 2D and depth video. The third dimensional search is necessary for depth information. The proposed 3D motion estimation raises the quality and the precision of the depth map after prediction, and reduces the coding bitrate. From the simulation results, the high accuracy depth map improves the effect of the 3D video viewing exported by the Depth image based rendering (DIBR) system.

A rate-distortion framework is used to define a very low-bit-rate coding scheme based on wireframe model adaptation and optimized selection of motion estimators. This technique achieves maximum reconstructed image quality under the constraint of a target bit rate for the coding of the vector field and the wireframe representation information. First, a complete scheme is proposed for hybrid two-dimensional (2D) and 3D motion estimation and compensation. The wireframe adaptation and updating is optimized for hybrid motion estimation in the rate distortion sense. A more sophisticated technique, adapted to the requirements of a very low-bit-rate coder is also proposed which considers also the transmission of the prediction error corresponding to the particular choice of the motion estimator for each object in the scene. Experimental results illustrating the performance of the proposed techniques in very low-bit-rate image sequence coding application areas are presented and evaluated. © 1998 John Wiley & Sons, Inc. Int J Imaging Syst Technol, 9, 238–247, 1998

Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

The paper presents an investigation into 3D structure and motion estimation from image sequences. The concept of a variable dimension 3D Kalman filter is outlined in which the structure and motion parameters of two or more images are reconstructed at each time instant. The developed procedure aims at applications in visual navigation. For a motion unit of two images the length of the state vector is restricted to N X 3 coordinates of N tracked natural landmarks and to 12 motion parameters. Even though new points appear in the sequence with each new processed image, a similar number of points leave the field of view. Therefore the length of the state vector is approximately constant (varies only by a small number of points from image to image) and does not depend on the number of images in the sequence. From a navigational point of view this feature of the proposed procedure is most important. A quality check is reported comparing the structure and motion parameters of the presented procedure to the results of a simultaneous bundle adjustment. The results refer to an experiment in which an observer moves through a stationary but unknown environment.