Robust optical flow estimation via edge preserving filtering

Optical flow estimation is a research core of computer vision. In recent years, optical flow estimation methods based on convolutional neural networks (CNNs) have achieved great success. However, most of the existing CNN-based methods are incapable of modeling long-distance dependencies successfully due to the limited receptive fields of convolutions. This may lead to poor performance of optical flow estimation in regions of large displacements and local ambiguities. Furthermore, the problem of edge-blurring remains an open challenge for CNN-based optical flow methods because the general interpolation operation tends to amplify the optical flow errors during the upsampling process. To deal with the abovementioned problems, this paper proposes a novel optical flow estimation method based on local-global modeling and visual similarity guidance. First, an efficient and simple self-attention module is adopted to improve the local-global modeling ability of the network and to extract the more representative features, which decreases the number of optical flow errors caused by large displacements. Second, based on the assumption that the visual features of the objects are more similar, the motion is more similar, and a visual similarity-guided optical flow upsampling network is constructed to guide the upsampling process. By transforming the visual similarity of features into motion similarity, the proposed upsampling scheme improves the accuracy of optical flow estimation in regions of motion boundaries. Finally, we run the proposed method on MPI-Sintel and KITTI test datasets to conduct a comprehensive comparison with some state-of-the-art methods. The experimental results indicate that the proposed method achieves top performance among the comparable methods and, in particular, gains significant developments in regions of large displacements and motion boundaries.

<p indent=0mm>Combining the deep learning model to compute end-to-end optical flow is a hot topic in current computer vision field. The optical flow estimation methods based on deep learning are summarized and reviewed. Firstly, the origin and concept of optical flow is introduced. Secondly, the optical flow datasets and evaluation metrics are summarized. Most importantly, classical methods are introduced and the deep optical flow estimation methods are reviewed in three aspects, including supervised deep optical flow estimation methods, unsupervised deep optical flow estimation methods and the performance of these methods. The analysis shows designing compact and generalized deep optical flow model is the future research direction. On this basis, the joint learning of optical flow estimation and specific video analysis tasks are introduced. It is pointed out that the design of task-driven deep optical flow network is a valuable research direction in practice application. Finally, the problems and challenges of deep optical flow estimation are summarized and the future work of this field is prospected.

Nowadays, convolutional neural networks achieve remarkable performance on optical flow estimation because of its strong non-linear fitting ability. Most of them adopt the U-Net architecture, which contains an encoder part and a decoder part. In the encoder part, the resolution of the feature map is reduced with the deepening of the network layer. In the decoder part, the feature map is enlarged by the deconvolution layer to recover the estimated flow as full resolution. However, the motion details are usually lost with the contracting and expanding operations. Moreover, learning methods, especially supervised networks, always ignore the advantages of many well-proven constraints used in the variational model. In this paper, we introduce a novel architecture named dilated residual networks for learning optical flow, which can avoid the loss of details of the U-Net architecture and can directly learn the residual functions rather than the unreferenced functions to enhance the learning ability of the network. Furthermore, inspired by variational methods, the traditional prior assumptions, such as brightness constancy, gradient constancy, and smoothness assumption, are used in the supervised network as extra auxiliary terms to guide the training of network. Our method is tested on several benchmarks, such as MPI-Sintel, KITTI2012, and KITTI2015. The experimental results show that the dilated residual network is suitable for dense optical flow estimation due to the capability of preserving motion details and can boost the accuracy of optical flow estimation.

Event cameras are novel bio-inspired sensors that offer advantages over traditional cameras (low latency, high dynamic range, low power, etc.). Optical flow estimation methods that work on packets of events trade off speed for accuracy, while event-by-event (incremental) methods have strong assumptions and have not been tested on common benchmarks that quantify progress in the field. Towards applications on resource-constrained devices, it is important to develop optical flow algorithms that are fast, light-weight and accurate. This work leverages insights from neuroscience, and proposes a novel optical flow estimation scheme based on triplet matching. The experiments on publicly available benchmarks demonstrate its capability to handle complex scenes with comparable results as prior packet-based algorithms. In addition, the proposed method achieves the fastest execution time ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\gt$</tex-math></inline-formula> 10 kHz) on standard CPUs as it requires only three events in estimation. We hope that our research opens the door to real-time, incremental motion estimation methods and applications in real-world scenarios.

In cardiac gated SPECT, image reconstruction suffers from several degrading factors including limited data counts, depth-dependent blur, attenuation, scatter, and motion blur. In our recent work it was demonstrated that use of motion-compensated temporal processing could lead to improved accuracy in 4D reconstruction. In this work we investigate how different optical flow estimation methods can affect the reconstruction in 4D, and in particular, we examine how much room is still left in optical flow estimation toward improving the reconstructed image quality. In our experiments we conducted a thorough, quantitative evaluation using simulated imaging study with multiple noise realizations from the NCAT phantom. As an upper bound, the known motion in NCAT was used for reconstruction. The results show that 4D reconstruction with a periodic optical flow model can achieve results almost matching that from the known motion.

Recently, learning to estimate optical flow via deep convolutional networks is attracting significant attention. In this paper, we introduce a spatial-channel attention module into optical flow estimation, which infers attention maps along two separated dimensions, channel and spatial, and then integrates these separated attention maps into a fusion attention map for feature refinement. We embed this module into spatial pyramid network, which can adaptively learn the channel and spatial attention maps at each level for modifying the different scaled features and can further improve the accuracy of optical flow estimation. Our network is trained on FlyingChairs and FlyingThings3D datasets with a supervised manner, and is further tested on MPI-Sintel benchmark. The experimental results show that using the spatial-channel attention unit is beneficial for dense flow estimation and our approach is comparable with the state-of-the-art methods.

Gradient-based optical flow estimation methods such as LucasKanade method work well for scenes with small displacements but fail when objects move with large displacements. Hierarchical matching-based methods do not suffer from large displacements but are less accurate. By utilizing the high speed imaging capability of CMOS image sensors, the frame rate can be increased to obtain more accurate optical flow with wide range of scene velocities in real time. Further, by integrating the memory and processing with the sensor on the same chip, optical flow estimation using high frame rate sequences can be performed without unduly increasing the off-chip data rate. The paper describes a method for obtaining high accuracy optical flow at a standard frame rate using high frame rate sequences. The Lucas-Kanade method is used to obtain optical flow estimates at high frame rate, which are then accumulated and refined to obtain optical flow estimates at a standard frame rate. The method is tested on video sequences synthetically generated by perspective warping. Results demonstrate significant improvements in optical flow estimation accuracy with moderate memory and computational power requirements.

Physics-based optical flow estimation under varying illumination conditions

The seminal work of Horn and Schunck is the first variational method for optical flow estimation. It introduced a novel framework where the optical flow is computed as the solution of a minimization problem. From the assumption that pixel intensities do not change over time, the optical flow constraint equation is derived. This equation relates the optical flow with the derivatives of the image. There are infinitely many vector fields that satisfy the optical flow constraint, thus the problem is ill-posed. To overcome this problem, Horn and Schunck introduced an additional regularity condition that restricts the possible solutions. Their method minimizes both the optical flow constraint and the magnitude of the variations of the flow field, producing smooth vector fields. One of the limitations of this method is that, typically, it can only estimate small motions. In the presence of large displacements, this method fails when the gradient of the image is not smooth enough. In this work, we describe an implementation of the original Horn and Schunck method and also introduce a multi-scale strategy in order to deal with larger displacements. For this multi-scale strategy, we create a pyramidal structure of downsampled images and change the optical flow constraint equation with a nonlinear formulation. In order to tackle this nonlinear formula, we linearize it and solve the method iteratively in each scale. In this sense, there are two common approaches: one approach that computes the motion increment in the iterations; or the one we follow, that computes the full flow during the iterations. The solutions are incrementally refined over the scales. This pyramidal structure is a standard tool in many optical flow methods.

Confidence measures are crucial to the interpretation of any optical flow measurement. Even though numerous methods for estimating optical flow have been proposed over the last three decades, a sound, universal, and statistically motivated confidence measure for optical flow measurements is still missing. We aim at filling this gap with this contribution, where such a confidence measure is derived, using statistical test theory and measurable statistics of flow fields from the regarded domain. The new confidence measure is computed from merely the results of the optical flow estimator and hence can be applied to any optical flow estimation method, covering the range from local parametric to global variational approaches. Experimental results using state-of-the-art optical flow estimators and various test sequences demonstrate the superiority of the proposed technique compared to existing ’confidence’ measures.KeywordsOptical FlowConfidence MeasureEndpoint ErrorIntrinsic DimensionalityOptical Flow EstimatorThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Time-lapse (4-D) seismic reservoir monitoring is a relative new technology that is gaining recognition in oil and gas producing areas around the world. In this paper, we propose an application of Horn-Schunck’s optical flow estimation method in order to obtain the movement field between pairs of seismic images. Optical flow estimation can provide important information about velocity of each image pixel. This way, we show that is possible to catch important displacements between pairs of seismic images separated by a time interval. In this work, each seismic image was deformed through the application of a synthetic field. The results obtained show that this method is able to recover with reasonable precision tiny displacements commonly found in time-lapse seismic data. Introduction The use of time-lapse technologies in order to perform analysis and monitoring of the fluid and pressure in oil and gas producing areas has advanced rapidly in two last decades (Lumley, 2004). These technologies involve the process of taking several seismic surveys separated by a calendar interval, at same site, in order to image and detect important changes in a producing reservoir. If each seismic survey is formed by 3-D data, the extra dimension is calendar time. So, the set of these techniques are often termed “4-D seismic”, and they are quickly becoming a very important engineering reservoir management tool, that can save hundred of million of dollars, when correctly applied (Lumley, 2001). Although 4-D seismic techniques can provide an improved seismic monitoring, residual differences, that are independent of changes in the subsurface geology, may exist in the repeated time-lapse data. These residual differences add noise to the model and consequently impact the effectiveness of the method. So, it is important to ensure seismic repeatability in time-lapse seismic monitoring. A variety of factors are involved in the seismic repeatability study, such as the depth of buried detectors small variations in water table, tides, currents and temperature, ambient noise, transition zone, subsidence, source and geophone positions, source signatures, geometry design, and CMP stack fold distribution. Some of the non-repeatability problems can be solved by the careful deployment of source and receiver positions. However, problems such as those caused by annual near surface variations are more complicated to solve at the acquisition stage. Commonly, these problems can only be solved in the post-processing steps (Zhang and Schmitt, 2006). There are several researches and study cases in seismic literature involving time-lapse techniques application in producing reservoirs. Van Gestel et al. (2008) present their experience in five years of continuous seismic monitoring of Vahal Field, located in the North Sea. The authors show that a combination of permanently installed seismic sensors and highly repeatable acquisition can provide high-quality 4-D images. These images can be used to improve reservoir model and help to plan and reduce risk when drilling new wells. Foster et al. (2008) present an overview of the status of BP’s three oceanbottom cables (OBC) monitoring systems and installation and operation of similar systems at Clair and ACG. OBC systems, in the way that they have been implemented in the three BP deployments, promise highly effective seismic monitoring data, providing both imaging quality and repeatability. Davies et al. (2008) shows that the use of permanent sensors deployed in the wellbore and along the tubing, in the surface production network and in the facilities provide a rich data flow to support advanced well and reservoir management techniques. Arts et al. (2004) present seismic interpretation of time-lapse seismic data provided by monitoring of CO2 injection into a saline aquifer. The authors conclude that the effect of CO2 on the seismic data is large in terms of seismic amplitude and in observed velocity pushdown effects. A new alternative to detect time-lapse seismic effects is proposed by Matos et al. (2004). The authors use self organized maps (SOM) combined with wavelet transform to detect time-lapse changes. Wavelet transform is used in order to detect seismic traces singularities of each time-lapse 3-D cube. Then, these detected objects are classified using the clustering of SOM. The authors also show a successful application of this technique to the Troll West gas province, offshore Norway. Claudino et al. (2008) examine the main effects of permeability barriers on seismic response using fluid flow simulations to generate pressure and saturation fields. The authors have performed several simulations in a simple reservoir model which has vertical and horizontal variations of porosity as well as permeability barriers. The authors use fluid substitution theory, Gassmann and patchy models, and Batzle and Wang’s empirical relationship to model the main seismic parameters, such as acoustic impedance and compressional velocity. Synthetics seismograms and some contrast sections were generated to compare the seismic images prior and

This paper presents a new optical flow estimation method using Markov Random Fields (MRFs) modelling. In the MRF framework, optical flow estimation leads to minimizing an energy function. To this end, we propose an Evolutionary Algorithm (EA) based on a Divide-and-Conquer strategy which adequately exploits the Markovian property. The image is split into small parts on which EAs are performed separately and independently. The elementary results obtained from all image parts are then combined into a final global solution. Experimental results using synthetic and real-world image sequences demonstrate the effectiveness of the method. Compared with traditional ICM and simulated annealing, results are improved. Moreover, computation cost is significantly reduced in comparison with a basic EA.

In this paper, we propose UnOS, an unified system for unsupervised optical flow and stereo depth estimation using convolutional neural network (CNN) by taking advantages of their inherent geometrical consistency based on the rigid-scene assumption. UnOS significantly outperforms other state-of-the-art (SOTA) unsupervised approaches that treated the two tasks independently. Specifically, given two consecutive stereo image pairs from a video, UnOS estimates per-pixel stereo depth images, camera ego-motion and optical flow with three parallel CNNs. Based on these quantities, UnOS computes rigid optical flow and compares it against the optical flow estimated from the FlowNet, yielding pixels satisfying the rigid-scene assumption. Then, we encourage geometrical consistency between the two estimated flows within rigid regions, from which we derive a rigid-aware direct visual odometry (RDVO) module. We also propose rigid and occlusion-aware flow-consistency losses for the learning of UnOS. We evaluated our results on the popular KITTI dataset over 4 related tasks, \ie stereo depth, optical flow, visual odometry and motion segmentation.

Robust depth estimation on real-world light field images using Gaussian belief propagation

This paper proposes a new method for robust and accurate optical flow estimation. The robustness of the new method is the ability of weakening the disturbance of varying illumination and preserving motion boundaries. The significance of this work is twofold: In the first part the generalized dynamic image model (GDIM) is adopted based on gradient field to encourage robustness to non-uniform brightness variations; otherwise, a Bayesian selecting scheme is adopted to encourage the ability of preserving motion boundaries and reduce the error, bias generated by smoothness constraint. We validated our results on a set of real and synthetic sequences. The qualitative and quantitative results indicate that the new method produces more accurate and robust results, which better sustain our conclusions.