Camera motion estimation using normal flows

Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients

This work investigates the propagation of errors from the camera motion to the epipolar constraint. A relation between the perturbation of the motion parameters and the error in the epipolar constraint is derived. Based on this relation, the sensitivity of the motion parameters to the epipolar constraint is characterised, and a constraint on the allowed perturbation in a motion parameter in response to a threshold for the produced error in the epipolar constraint is determined. The presented error propagation model is useful to vision systems such as a mobile robot where the camera motion is provided by an odometry sensor. Experimental results on real images are presented.

Dynamic Vision Sensors (DVS) output asynchronous log intensity change events. They have potential applications in high-speed robotics, autonomous cars and drones. The precise event timing, sparse output, and wide dynamic range of the events are well suited for optical flow, but conventional optical flow (OF) algorithms are not well matched to the event stream data. This paper proposes an event-driven OF algorithm called adaptive block-matching optical flow (ABMOF). ABMOF uses time slices of accumulated DVS events. The time slices are adaptively rotated based on the input events and OF results. Compared with other methods such as gradient-based OF, ABMOF can efficiently be implemented in compact logic circuits. We developed both ABMOF and Lucas-Kanade (LK) algorithms using our adapted slices. Results shows that ABMOF accuracy is comparable with LK accuracy on natural scene data including sparse and dense texture, high dynamic range, and fast motion exceeding 30,000 pixels per second.

Unsupervised deep learning methods have demonstrated an impressive performance for understanding the structure of 3D scene from videos. These data-based learning methods are able to learn the tasks, such as depth, ego-motion, and optical flow estimation. In this paper, we propose a novel unsupervised deep learning method to jointly estimate scene depth, camera ego-motion, and optical flow from stereo images. Consecutive stereo images are used to train the system. After training stage, the system is able to estimate dense depth map, camera 6D pose, and optical flow by using a sequence of monocular images. No labelled data set is required for training. The supervision signals for training three deep neural networks of the system come from various forms of image warping. Due to the use of optical flow, the impact caused by occlusions and moving objects on the estimation results is alleviated. Experiments on the KITTI and Cityscapes datasets show that the proposed system demonstrates a better performance in terms of accuracy in depth, ego-motion, and optical flow estimation.

In this paper, we propose UnOS, an unified system for unsupervised optical flow and stereo depth estimation using convolutional neural network (CNN) by taking advantages of their inherent geometrical consistency based on the rigid-scene assumption. UnOS significantly outperforms other state-of-the-art (SOTA) unsupervised approaches that treated the two tasks independently. Specifically, given two consecutive stereo image pairs from a video, UnOS estimates per-pixel stereo depth images, camera ego-motion and optical flow with three parallel CNNs. Based on these quantities, UnOS computes rigid optical flow and compares it against the optical flow estimated from the FlowNet, yielding pixels satisfying the rigid-scene assumption. Then, we encourage geometrical consistency between the two estimated flows within rigid regions, from which we derive a rigid-aware direct visual odometry (RDVO) module. We also propose rigid and occlusion-aware flow-consistency losses for the learning of UnOS. We evaluated our results on the popular KITTI dataset over 4 related tasks, \ie stereo depth, optical flow, visual odometry and motion segmentation.

This paper presents a solution to the problem of estimating a camera's ego motion using its image data, namely, visual odometry. The implementation considered both synthetic and real-world scenarios. Analysis and performance comparisons of the algorithm against a defined benchmark are offered, identifying that the algorithm is resilient to gaussian image noise at SNR36dB, yet prone to scale ambiguity and gross errors in sparsely featured environments. The solution identifies that optimal RANSAC parameters are scene dependent and highlights the need for accurate and low noise secondary sensing to recover scale. The established importance of bundle adjustment in visual odometry optimisation is again confirmed by a reduction in gross trajectory errors. Additionally, 3D simulations enable accurate visualisation of the ground truth and estimated camera trajectories resulting from the algorithm. Providing simultaneous global and field-of-view perspectives of the camera's ego motion, they deliver an intuitive representation of the behaviour of the algorithm at all time intervals.

Precise estimation of optical flow, stereo depth and camera motion are important for the real-world 3D scene understanding and visual perception. Since the three tasks are tightly coupled with the inherent 3D geometric constraints, current studies have demonstrated that the three tasks can be improved through jointly optimizing geometric loss functions of several individual networks. In this paper, we show that effective feature-level collaboration of the networks for the three respective tasks could achieve much greater performance improvement for all three tasks than only loss-level joint optimization. Specifically, we propose a single network to combine and improve the three tasks. The network extracts the features of two consecutive stereo images, and simultaneously estimates optical flow, stereo depth and camera motion. The whole network mainly contains four parts: (I) a feature-sharing encoder to extract features of input images, which can enhance features' representation ability; (II) a pooled decoder to estimate both optical flow and stereo depth; (III) a camera pose estimation module which fuses optical flow and stereo depth information; (IV) a cost volume complement module to improve the performance of optical flow in static and occluded regions. Our method achieves state-of-the-art performance among the joint unsupervised methods, including optical flow and stereo depth estimation on KITTI 2012 and 2015 benchmarks, and camera motion estimation on KITTI VO dataset.

In this paper, we present the algorithm and operation of an aVLSI chip that can extract normal optical flow by using the gradient approach without interfering with the imaging process. This approach is feasible for scaling to larger arrays without affecting the processing or the processing area. Our system has a 92 x 52 photosensitive array of APS pixels at the core with processing circuits on the periphery. We discuss the approach and the different blocks in the design and then demonstrate the working of the individual blocks and of the system as a whole. The chip outputs the image, the spatial and temporal gradients and the normal flow at the read-out frame-rate with no penalty to the imaging process. The chip occupies an area of 4.5 mm² and consumes 2.6 mW (at V_dd = 5V). Once normal flow is obtained, the chip can be used to compute focus of expansion, time to contact and many other motion properties of images that can be used to control robots. Tracking systems can use the velocity and segmentation of moving objects can be realized using motion discontinuity.

A “theory” of optical flow

Proposes a new method of active vision which recognizes the 3-D shape of objects without knowing camera motion parameters. The motion parameters are calculated from the optical flows and the depth of object points whose 3-D shape is already known. Then, using these calculated motion parameters and the optical flows, the 3-D position of unknown points are reconstructed, which, in turn, will be used as the known points in the next frame of image. These processes are iterated for a sequence of images to recognize the 3-D scene. In this method, the effects of quantization errors are overcome by two approaches. The errors of camera motion parameters are compensated by using a large number of points to calculate them. Then, the Kalman filtering method is applied to the sequence of images to reduce the 3-D position errors of each unknown point. >

Movement estimation of the temporomandibular joint from sequential images of X-ray television comprises two procedures: (1) computation of the velocity field (optical flow) from two successive images; and (2) estimation of the velocity and the angular velocity (motion parameters) from data of the optical flow. The mandibular head of the X-ray image is observed merely as its blurred profile, superimposed by images of cranial bones. In order to achieve the movement tracking in such image data of inferior quality, the authors developed an accurate estimation of the optical flow using the Gabor wavelet transform, and an iterative estimation of the motion parameters. Through experiments employing an actual image-sequence of the TMJ, the authors' methodology was verified to be effective for improving the estimation accuracy of the movement, thereby the successful tracking of the mandibular head being achieved.

Joint optical flow estimation, segmentation, and 3D interpretation with level sets

Motion fields of real image sequences are typically piecewise smooth with discontinuities at object boundaries. Further, because of the aperture problem, only the normal component of the optical flow can be observed. Direct application of wavelet shrinkage to the normal component flow may yield undesirable results due to the correlated noise affecting the wavelet coefficients. In this paper, we present a new technique for estimating optical flow based on L/sub 1/ regularisation. The resulting flow estimate typically has many zero wavelet coefficients, but unlike wavelet shrinkage some of the remaining coefficients are allowed to grow. To highlight this difference, we have named our new technique wavelet zeroing. Additionally, we present a data-driven, sequence adaptive method for optimally choosing the penalty parameter.

We present a new approach to volumetric scene reconstruction which can produce accurate models from turntable image sequences. Instead of an epipolar plane image (EPI) volume, we consider a function on the 4D spatiotemporal volume valued with the intensity back projection of the camera (time) to the particular voxel. Using an optical flow technique we compute the local orientation of this spatiotemporal image and decide on occupancy based on the relative orientation between the viewing ray of the voxel at the particular time and the local image structure. Our method does not require a background compensation like the silhouette-based methods and is comparable in performance with space carving.

Three-stream fusion network for first-person interaction recognition