Self-Supervised Optical Flow Estimation by Projective Bootstrap

In the general structure-from-motion (SFM) problem involving several moving objects in a scene, the essential first step is to segment moving objects independently. We attempt to deal with the problem of optical flow estimation and motion segmentation over a pair of images. We apply a mean field technique to determine optical flow and motion boundaries and present a deterministic algorithm. Since motion discontinuities represented by line process are embedded in the estimation of the optical flow, our algorithm provides accurate estimates of optical flow especially along motion boundaries and handles occlusion and multiple motions. We show that the proposed algorithm outperforms other well-known algorithms in terms of estimation accuracy and timing.

We propose a new method for disparity assisted stereo optical flow estimation. This method is based on the linearization of the round-about compatibility constraint, which converts a stereo optical flow estimation problem to a single channel optical flow estimation problem. An over-determined system of optical flow equations can then be constructed for estimating the flow fields. The total least squares, instead of the traditional least squares is used to solve the estimation problem. We also investigate the extension of the locally constant flow model across the time domain. The experiments presented demonstrate that the proposed technique performs very well.

This paper introduces a 2D optical flow estimation method for cardiac ultrasound imaging based on a sparse representation. The optical flow problem is regularized using a classical gradient-based smoothness term combined with a sparsity inducing regularization that uses a learned cardiac flow dictionary. A particular emphasis is put on the influence of the spatial and sparse regularizations on the optical flow estimation problem. A comparison with state-of-the-art methods using realistic simulations shows the competitiveness of the proposed method for cardiac motion estimation in ultrasound images.

Self-supervised multi-body scene flow estimation

OVAD (Fraś et al. in Vision based systems for UAV applications, Springer International Publishing, pp. 123–136 [1]; Kwiatkowski et al. in Recent advances in electrical engineering and computer science, pp. 58–61 [2]) is a device, which is purposed for multi directional video acquisition. Further development of the device includes hardware implementation of time-absorbing calculations, connected with video stream processing, with utilization of FPGA (Field Programmable Gate Array). The paper presents the results of research on the problem of real time dense optical flow estimation. Furthermore, the described results are connected to the problem of parallel processing and hardware acceleration using FPGA. Proposed solution may be found as useful in many applications, both civilian and military and proves utility of FPGA based solutions in real time video processing.

Nowadays, convolutional neural networks achieve remarkable performance on optical flow estimation because of its strong non-linear fitting ability. Most of them adopt the U-Net architecture, which contains an encoder part and a decoder part. In the encoder part, the resolution of the feature map is reduced with the deepening of the network layer. In the decoder part, the feature map is enlarged by the deconvolution layer to recover the estimated flow as full resolution. However, the motion details are usually lost with the contracting and expanding operations. Moreover, learning methods, especially supervised networks, always ignore the advantages of many well-proven constraints used in the variational model. In this paper, we introduce a novel architecture named dilated residual networks for learning optical flow, which can avoid the loss of details of the U-Net architecture and can directly learn the residual functions rather than the unreferenced functions to enhance the learning ability of the network. Furthermore, inspired by variational methods, the traditional prior assumptions, such as brightness constancy, gradient constancy, and smoothness assumption, are used in the supervised network as extra auxiliary terms to guide the training of network. Our method is tested on several benchmarks, such as MPI-Sintel, KITTI2012, and KITTI2015. The experimental results show that the dilated residual network is suitable for dense optical flow estimation due to the capability of preserving motion details and can boost the accuracy of optical flow estimation.

Recursive Optical Flow Estimation—Adaptive Filtering Approach

A number of proposed optical flow estimation algorithms for image motion analysis use only a few subsequent frames from an image sequence to perform calculation. Others, that use many frames, as is case in frequency and energy based, as well as in stochastic optical flow algorithms, are better suited for off-line work. We present our approach in adopting a classic gradient optical flow algorithm for on-line work over extended image sequence. Our experiments on the usual optical flow testing image sequences, consisting of few dozens of image frames show significant improvements in accuracy of the estimated velocity flow fields and in speed of computation. This is achieved by modification and integration of three basic processing phases: preparation of a limited spatio-temporal volume, estimation of velocity flow fields and post filtering of the flow fields. In the preparation phase we use efficient spatio-temporal filters for image smoothing. In the estimation phase a modified iterative solution that propagates results from previous calculation steps is used. Linear and nonlinear post filtering additionally improve optical flow estimates. The whole process is less sensitive to the aperture problem and produces more accurate flow field estimates than the original algorithm. It better copes with motion discontinuities as well. We evidence our achievements by a number of experiments on extended image sequences.

In this paper, we propose UnOS, an unified system for unsupervised optical flow and stereo depth estimation using convolutional neural network (CNN) by taking advantages of their inherent geometrical consistency based on the rigid-scene assumption. UnOS significantly outperforms other state-of-the-art (SOTA) unsupervised approaches that treated the two tasks independently. Specifically, given two consecutive stereo image pairs from a video, UnOS estimates per-pixel stereo depth images, camera ego-motion and optical flow with three parallel CNNs. Based on these quantities, UnOS computes rigid optical flow and compares it against the optical flow estimated from the FlowNet, yielding pixels satisfying the rigid-scene assumption. Then, we encourage geometrical consistency between the two estimated flows within rigid regions, from which we derive a rigid-aware direct visual odometry (RDVO) module. We also propose rigid and occlusion-aware flow-consistency losses for the learning of UnOS. We evaluated our results on the popular KITTI dataset over 4 related tasks, \ie stereo depth, optical flow, visual odometry and motion segmentation.

This paper presents a method that estimates both the photometric mapping and the dense motion field in image sequences. The method extends the Horn-Schunck type dense variational optical flow estimation approach with the use of intensity mapping functions in an alternating optimization scheme. The intensity mapping functions are also updated through the iterations with the help of weighted histograms, where the weights reflect the visibility of pixels in one image from the other. We include experiments with both synthetic and real imagery to demonstrate the efficacy of the proposed method.

We address the problem of incremental optical flow estimation. Following Black et al., we design a cost function whose data and prior terms both involve robust M-estimators. The non-convex minimization is lead with an efficient multigrid algorithm which converges fast toward estimates of good quality while providing, at very low cost, crude estimates revealing the large discontinuity structures of the flow field.

Optical flow estimation is a research core of computer vision. In recent years, optical flow estimation methods based on convolutional neural networks (CNNs) have achieved great success. However, most of the existing CNN-based methods are incapable of modeling long-distance dependencies successfully due to the limited receptive fields of convolutions. This may lead to poor performance of optical flow estimation in regions of large displacements and local ambiguities. Furthermore, the problem of edge-blurring remains an open challenge for CNN-based optical flow methods because the general interpolation operation tends to amplify the optical flow errors during the upsampling process. To deal with the abovementioned problems, this paper proposes a novel optical flow estimation method based on local-global modeling and visual similarity guidance. First, an efficient and simple self-attention module is adopted to improve the local-global modeling ability of the network and to extract the more representative features, which decreases the number of optical flow errors caused by large displacements. Second, based on the assumption that the visual features of the objects are more similar, the motion is more similar, and a visual similarity-guided optical flow upsampling network is constructed to guide the upsampling process. By transforming the visual similarity of features into motion similarity, the proposed upsampling scheme improves the accuracy of optical flow estimation in regions of motion boundaries. Finally, we run the proposed method on MPI-Sintel and KITTI test datasets to conduct a comprehensive comparison with some state-of-the-art methods. The experimental results indicate that the proposed method achieves top performance among the comparable methods and, in particular, gains significant developments in regions of large displacements and motion boundaries.

Spatio-temporal edge-aware (STEA) filtering methods have recently received increased attention due to their ability to efficiently solve or approximate important image-domain problems in a temporally consistent manner - which is a crucial property for video-processing applications. However, existing STEA methods are currently unsuited for real-time, embedded stream-processing settings due to their high processing latency, large memory, and bandwidth requirements, and the need for accurate optical flow to enable filtering along motion paths. To this end, we propose an efficient STEA filtering pipeline based on the recently proposed permeability filter (PF), which offers high quality and halo reduction capabilities. Using mathematical properties of the PF, we reformulate its temporal extension as a causal, non-linear infinite impulse response filter, which can be efficiently evaluated due to its incremental nature. We bootstrap our own accurate flow using the PF and its temporal extension by interpolating a quasi-dense nearest neighbour field obtained with an improved PatchMatch algorithm, which employs binarized octal orientation maps (BOOM) descriptors to find correspondences among subsequent frames. Our method is able to create temporally consistent results for a variety of applications such as optical flow estimation, sparse data upsampling, visual saliency computation and disparity estimation. We benchmark our optical flow estimation on the MPI Sintel dataset, where we currently achieve a Pareto optimal quality-efficiency tradeoff with an average endpoint error of 7.68 at 0.59 s single-core execution time on a recent desktop machine.

Cryo-EM movie in single particle analysis has extremely low SNR and requires aligning multiple frames to achieve signal enhancement. Currently, signal processing technique is adopted to estimate the motion vector between a pair of cryo-EM movie frames at patch-level, and the estimated motion vector is used as the reference for frame alignment, whose accuracy will determine the resolution of the reconstructed 3D structure of the particle. The patch-level motion may not well represent the beam-induced motion of particles since particles in a patch move towards different directions due to beam striking. However, the low SNR of cryo-EM movie makes it difficult to estimate the motion of particles at pixel-level. Meanwhile, existing optical flow estimation models only consider the ideal case where high-quality videos are provided, which fail to obtain optical flow from cryo-EM movie. In this paper, we diminish this limitation by proposing a model called NoiseFlow, a deep learning network for optical flow estimation from low SNR cryo-EM movie. NoiseFlow makes use of the multi-frame stacking module and the denoising module to extract noise-invariant features, and then computes the correlation volume from noise-invariant features to learn optical flow. For evaluation, we train our model on two synthetic cryo-EM movie datasets and infer on real cryo-EM data. The experimental results illustrate that NoiseFlow achieves state-of-the-art performance on both synthetic and real cryo-EM datasets.

Multi-scale 3D scene flow from binocular stereo sequences