Structure from Motion.

<p indent=0mm>Combining the deep learning model to compute end-to-end optical flow is a hot topic in current computer vision field. The optical flow estimation methods based on deep learning are summarized and reviewed. Firstly, the origin and concept of optical flow is introduced. Secondly, the optical flow datasets and evaluation metrics are summarized. Most importantly, classical methods are introduced and the deep optical flow estimation methods are reviewed in three aspects, including supervised deep optical flow estimation methods, unsupervised deep optical flow estimation methods and the performance of these methods. The analysis shows designing compact and generalized deep optical flow model is the future research direction. On this basis, the joint learning of optical flow estimation and specific video analysis tasks are introduced. It is pointed out that the design of task-driven deep optical flow network is a valuable research direction in practice application. Finally, the problems and challenges of deep optical flow estimation are summarized and the future work of this field is prospected.

The seminal work of Horn and Schunck is the first variational method for optical flow estimation. It introduced a novel framework where the optical flow is computed as the solution of a minimization problem. From the assumption that pixel intensities do not change over time, the optical flow constraint equation is derived. This equation relates the optical flow with the derivatives of the image. There are infinitely many vector fields that satisfy the optical flow constraint, thus the problem is ill-posed. To overcome this problem, Horn and Schunck introduced an additional regularity condition that restricts the possible solutions. Their method minimizes both the optical flow constraint and the magnitude of the variations of the flow field, producing smooth vector fields. One of the limitations of this method is that, typically, it can only estimate small motions. In the presence of large displacements, this method fails when the gradient of the image is not smooth enough. In this work, we describe an implementation of the original Horn and Schunck method and also introduce a multi-scale strategy in order to deal with larger displacements. For this multi-scale strategy, we create a pyramidal structure of downsampled images and change the optical flow constraint equation with a nonlinear formulation. In order to tackle this nonlinear formula, we linearize it and solve the method iteratively in each scale. In this sense, there are two common approaches: one approach that computes the motion increment in the iterations; or the one we follow, that computes the full flow during the iterations. The solutions are incrementally refined over the scales. This pyramidal structure is a standard tool in many optical flow methods.

Robust optical flow estimation via edge preserving filtering

Event cameras are novel bio-inspired sensors that offer advantages over traditional cameras (low latency, high dynamic range, low power, etc.). Optical flow estimation methods that work on packets of events trade off speed for accuracy, while event-by-event (incremental) methods have strong assumptions and have not been tested on common benchmarks that quantify progress in the field. Towards applications on resource-constrained devices, it is important to develop optical flow algorithms that are fast, light-weight and accurate. This work leverages insights from neuroscience, and proposes a novel optical flow estimation scheme based on triplet matching. The experiments on publicly available benchmarks demonstrate its capability to handle complex scenes with comparable results as prior packet-based algorithms. In addition, the proposed method achieves the fastest execution time ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\gt$</tex-math></inline-formula> 10 kHz) on standard CPUs as it requires only three events in estimation. We hope that our research opens the door to real-time, incremental motion estimation methods and applications in real-world scenarios.

The gradient-based methods for optical flow estimation are based on a constraint equation which is defined for each image pixel. The structure of the constraint equation makes the problem ill-posed so in the past some solutions have been proposed based on regularization. Under the assumption that in the immediate neighbourhood of a pixel the optical flow field is smooth, the constraint equations in that neighbourhood should have a common solution, in this case the problem is not ill-posed. Following this reasoning, an algorithm for evaluating the optical flow, which is suitable for parallel implementation is proposed in this paper. Moreover, parallel implementations of selected algorithms from the literature, for optical flow estimation, are presented in this paper with the intention of comparing their complexity and performance with respect to the proposed approach on a Connection Machine-2.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

In cardiac gated SPECT, image reconstruction suffers from several degrading factors including limited data counts, depth-dependent blur, attenuation, scatter, and motion blur. In our recent work it was demonstrated that use of motion-compensated temporal processing could lead to improved accuracy in 4D reconstruction. In this work we investigate how different optical flow estimation methods can affect the reconstruction in 4D, and in particular, we examine how much room is still left in optical flow estimation toward improving the reconstructed image quality. In our experiments we conducted a thorough, quantitative evaluation using simulated imaging study with multiple noise realizations from the NCAT phantom. As an upper bound, the known motion in NCAT was used for reconstruction. The results show that 4D reconstruction with a periodic optical flow model can achieve results almost matching that from the known motion.

Current deep learning methods for optical flow estimation often use spatial feature pyramids to extract image features. To get the correlation between images, they directly compute the cost volume of the obtained image features. In this process, fine object details tend to be ignored. To solve this fundamental problem, an object-scale adaptive optical flow estimation network is proposed, in which multi-scale features are selectively extracted and exploited using our developed feature selectable block (FSB). As a result, we can obtain the multi-scale receptive fields of objects at different scales in the image. To consolidate all image features generated from all scales, a new cost volume generation scheme called multi-scale cost volume generation block (MCVGB) is further proposed to aggregate information from different scales. Extensive experiments conducted on the Sintel and KITTI2015 datasets show that our proposed method can capture fine details of different scale objects with high accuracy and thus deliver superior performance over a number of state-of-the-art methods.KeywordsOptical flow estimationDeep learningFeature pyramidsReceptive fieldsMulti-scale

This paper presents a novel fast optical flow estimation algorithm and its application to real-lime obstacle avoidance of a guide-dog robot. The function of the laboratory-developed robot is to help blind or visually impaired pedestrians to move safely among obstacles. The proposed algorithm features a combination of the conventional correlation-based principle and the differential-based method for optical flow estimation. Employing image intensity gradients as features for pattern matching, we set up a brightness constraint to configure the search area. The merit of this scheme is that the computation load can be greatly reduced and in the mean time the possibility of estimation error is decreased. The vision system is installed on-board the robot to provide depth information of the immediate environment. The depth data are transformed to a safety distribution histogram and used for real-time obstacle avoidance. Experimental results demonstrate that the proposed method is effective for a guidance robot in a dynamic environment.

Abstract. Real-time change detection and analysis of natural disasters is of great importance to emergency response and disaster rescue. Recently, a number of video satellites that can record the whole process of natural disasters have been launched. These satellites capture high resolution video image sequences and provide researchers with a large number of image frames, which allows for the implementation of a rapid disaster procedure change detection approach based on deep learning. In this paper, pixel change in image sequences is estimated by optical flow based on FlowNet 2.0 for quick change detection in natural disasters. Experiments are carried out by using image frames from Digital Globe WorldView in Indonesia Earthquake took place on Sept. 28, 2018. In order to test the efficiency of FlowNet 2.0 on natural disaster dataset, 7 state-of-the-art optical flow estimation methods are compared. The experimental results show that FlowNet 2.0 is not only robust to large displacements but small displacements in natural disaster dataset. Two evaluation indicators: Root Mean Square Error (RMSE) and Mean Value are used to record the accuracy. For estimation error of RMSE, FlowNet 2.0 achieves 0.30 and 0.11 pixels in horizontal and vertical direction, respectively. The error in horizontal error is similar to other algorithms but the value in vertical direction is significantly lower than them. And the Mean Value are 1.50 and 0.09 pixels in horizontal and vertical direction, which are most close to the ground truth comparing to other algorithms. Combining the superiority of computing time, the paper proves that only the approach based on FlowNet 2.0 is able to achieve real-time change detection with higher accuracy in the case of natural disasters.

This paper proposes a method for detecting ships on real-time using navigational image sequence. As pointed out in the previous reports, our proposed method has problem in processing speed performance, although it is precise precision. First, this paper analyzes this problem, and proposes a new method for solving them. Our method distinguishes ships from the character of a moving vector of region. A moving vector is calculated on matching process of region. Matching process takes enormous-processing time. On the other hand, the Lucas-Kanade method (LKM) is popular version two-frame differential methods for optical flow estimation. The LKM can get a pixel flow first. We examined that applying the LKM in substitution for matching process, and we proposes a new method for detecting ships on real-time. We inspected an effect of prposed method by an experiment.

Many recent energy-based methods for optical flow estimation rely on a good initialization that is typically provided by some kind of feature matching. So far, however, these initial matching approaches are rather general: They do not incorporate any additional information that could help to improve the accuracy or the robustness of the estimation. In particular, they do not exploit potential cues on the camera poses and the thereby induced rigid motion of the scene. In the present paper, we tackle this problem. To this end, we propose a novel structure-from-motion-aware PatchMatch approach that, in contrast to existing matching techniques, combines two hierarchical feature matching methods: a recent two-frame PatchMatch approach for optical flow estimation (general motion) and a specifically tailored three-frame PatchMatch approach for rigid scene reconstruction (SfM). While the motion PatchMatch serves as baseline with good accuracy, the SfM counterpart takes over at occlusions and other regions with insufficient information. Experiments with our novel SfM-aware PatchMatch approach demonstrate its usefulness. They not only show excellent results for all major benchmarks (KITTI 2012/2015, MPI Sintel), but also improvements up to 50% compared to a PatchMatch approach without structure information.

Optical flow estimation is a research core of computer vision. In recent years, optical flow estimation methods based on convolutional neural networks (CNNs) have achieved great success. However, most of the existing CNN-based methods are incapable of modeling long-distance dependencies successfully due to the limited receptive fields of convolutions. This may lead to poor performance of optical flow estimation in regions of large displacements and local ambiguities. Furthermore, the problem of edge-blurring remains an open challenge for CNN-based optical flow methods because the general interpolation operation tends to amplify the optical flow errors during the upsampling process. To deal with the abovementioned problems, this paper proposes a novel optical flow estimation method based on local-global modeling and visual similarity guidance. First, an efficient and simple self-attention module is adopted to improve the local-global modeling ability of the network and to extract the more representative features, which decreases the number of optical flow errors caused by large displacements. Second, based on the assumption that the visual features of the objects are more similar, the motion is more similar, and a visual similarity-guided optical flow upsampling network is constructed to guide the upsampling process. By transforming the visual similarity of features into motion similarity, the proposed upsampling scheme improves the accuracy of optical flow estimation in regions of motion boundaries. Finally, we run the proposed method on MPI-Sintel and KITTI test datasets to conduct a comprehensive comparison with some state-of-the-art methods. The experimental results indicate that the proposed method achieves top performance among the comparable methods and, in particular, gains significant developments in regions of large displacements and motion boundaries.

Time-lapse (4-D) seismic reservoir monitoring is a relative new technology that is gaining recognition in oil and gas producing areas around the world. In this paper, we propose an application of Horn-Schunck’s optical flow estimation method in order to obtain the movement field between pairs of seismic images. Optical flow estimation can provide important information about velocity of each image pixel. This way, we show that is possible to catch important displacements between pairs of seismic images separated by a time interval. In this work, each seismic image was deformed through the application of a synthetic field. The results obtained show that this method is able to recover with reasonable precision tiny displacements commonly found in time-lapse seismic data. Introduction The use of time-lapse technologies in order to perform analysis and monitoring of the fluid and pressure in oil and gas producing areas has advanced rapidly in two last decades (Lumley, 2004). These technologies involve the process of taking several seismic surveys separated by a calendar interval, at same site, in order to image and detect important changes in a producing reservoir. If each seismic survey is formed by 3-D data, the extra dimension is calendar time. So, the set of these techniques are often termed “4-D seismic”, and they are quickly becoming a very important engineering reservoir management tool, that can save hundred of million of dollars, when correctly applied (Lumley, 2001). Although 4-D seismic techniques can provide an improved seismic monitoring, residual differences, that are independent of changes in the subsurface geology, may exist in the repeated time-lapse data. These residual differences add noise to the model and consequently impact the effectiveness of the method. So, it is important to ensure seismic repeatability in time-lapse seismic monitoring. A variety of factors are involved in the seismic repeatability study, such as the depth of buried detectors small variations in water table, tides, currents and temperature, ambient noise, transition zone, subsidence, source and geophone positions, source signatures, geometry design, and CMP stack fold distribution. Some of the non-repeatability problems can be solved by the careful deployment of source and receiver positions. However, problems such as those caused by annual near surface variations are more complicated to solve at the acquisition stage. Commonly, these problems can only be solved in the post-processing steps (Zhang and Schmitt, 2006). There are several researches and study cases in seismic literature involving time-lapse techniques application in producing reservoirs. Van Gestel et al. (2008) present their experience in five years of continuous seismic monitoring of Vahal Field, located in the North Sea. The authors show that a combination of permanently installed seismic sensors and highly repeatable acquisition can provide high-quality 4-D images. These images can be used to improve reservoir model and help to plan and reduce risk when drilling new wells. Foster et al. (2008) present an overview of the status of BP’s three oceanbottom cables (OBC) monitoring systems and installation and operation of similar systems at Clair and ACG. OBC systems, in the way that they have been implemented in the three BP deployments, promise highly effective seismic monitoring data, providing both imaging quality and repeatability. Davies et al. (2008) shows that the use of permanent sensors deployed in the wellbore and along the tubing, in the surface production network and in the facilities provide a rich data flow to support advanced well and reservoir management techniques. Arts et al. (2004) present seismic interpretation of time-lapse seismic data provided by monitoring of CO2 injection into a saline aquifer. The authors conclude that the effect of CO2 on the seismic data is large in terms of seismic amplitude and in observed velocity pushdown effects. A new alternative to detect time-lapse seismic effects is proposed by Matos et al. (2004). The authors use self organized maps (SOM) combined with wavelet transform to detect time-lapse changes. Wavelet transform is used in order to detect seismic traces singularities of each time-lapse 3-D cube. Then, these detected objects are classified using the clustering of SOM. The authors also show a successful application of this technique to the Troll West gas province, offshore Norway. Claudino et al. (2008) examine the main effects of permeability barriers on seismic response using fluid flow simulations to generate pressure and saturation fields. The authors have performed several simulations in a simple reservoir model which has vertical and horizontal variations of porosity as well as permeability barriers. The authors use fluid substitution theory, Gassmann and patchy models, and Batzle and Wang’s empirical relationship to model the main seismic parameters, such as acoustic impedance and compressional velocity. Synthetics seismograms and some contrast sections were generated to compare the seismic images prior and

This paper presents a new optical flow estimation method using Markov Random Fields (MRFs) modelling. In the MRF framework, optical flow estimation leads to minimizing an energy function. To this end, we propose an Evolutionary Algorithm (EA) based on a Divide-and-Conquer strategy which adequately exploits the Markovian property. The image is split into small parts on which EAs are performed separately and independently. The elementary results obtained from all image parts are then combined into a final global solution. Experimental results using synthetic and real-world image sequences demonstrate the effectiveness of the method. Compared with traditional ICM and simulated annealing, results are improved. Moreover, computation cost is significantly reduced in comparison with a basic EA.

We design a generic contrast and affine invariant planar shape recognition algorithm. By generic, we mean an algorithm which delivers a list of all shapes two digital images have in common, up to any affine transform or contrast change. Affine invariance must not be considered as a weak requirement in shape recognition since it is well known that if 3D objects with smooth boundaries are being observed, then the deformation of the objects’ images, far enough from the occlusion boundaries, is locally an affine transform. We define as ”shape elements” all pieces of level lines of the image. We then discuss an efficient local encoding of the shape elements. We finally show experiments. Applications aimed at include image registration, image indexing, optical flow.