Horn-Schunck Optical Flow with a Multi-Scale Strategy

Motion estimation based on the direction of intensity gradient

<p indent=0mm>Combining the deep learning model to compute end-to-end optical flow is a hot topic in current computer vision field. The optical flow estimation methods based on deep learning are summarized and reviewed. Firstly, the origin and concept of optical flow is introduced. Secondly, the optical flow datasets and evaluation metrics are summarized. Most importantly, classical methods are introduced and the deep optical flow estimation methods are reviewed in three aspects, including supervised deep optical flow estimation methods, unsupervised deep optical flow estimation methods and the performance of these methods. The analysis shows designing compact and generalized deep optical flow model is the future research direction. On this basis, the joint learning of optical flow estimation and specific video analysis tasks are introduced. It is pointed out that the design of task-driven deep optical flow network is a valuable research direction in practice application. Finally, the problems and challenges of deep optical flow estimation are summarized and the future work of this field is prospected.

Unmanned aerial vehicles (UAVs) have become an important part of today’s world. Micro Aerial Vehicles (MAVs) are small-sized UAVs that are suited for closed spaces, indoor or covert applications. Due to their small size and limited energy and load they can bear, there are several challenges in intelligent control of these vehicles. Also, due to their limitations, using energy-consuming sensors and complex control strategies which need high computational power, is not an option. To overcome MAV’s limitations, researchers are working on visual navigation systems. In this paper, we present an efficient strategy for the guidance and navigation of a 30gr MAV in indoor environments using a mixture of visual and inertial sensors. By taking inspiration from nature and flying insects, we combined optical flow and stereopsis methods to detect stationary obstacles and avoid collisions. Our control system uses two lateral cameras to avoid collision with lateral walls. To detect frontal walls and obstacles, instead of using optical flow methods, we use two frontal cameras with the same viewing direction and calculate the discrepancy between their images to detect walls and obstacles. For guidance, we designed a guidance system that functions well with our obstacle detection system and only needs destination and MAV’s current coordinates. By implementing our control system on a simulated coaxial helicopter, we were able to record more than 150 hours of collision-free flight and successful guidance to appointed destinations. Our control system is best suited for small sized UAVs in indoor environments and only needs inexpensive sensors and microcontrollers to implement

Despite progress made in the accuracy and robustness of optical flow in past years, the problem of over-segmentation and the blurring of image edge and motion boundary caused by the illumination change, complex texture, large displacement, and motion occlusion still remain. Recently, we developed a guided filtering scheme for flow field estimation, which is implemented as an add-on optimal operation during the coarse-to-fine optical flow computation. In this paper, we first review the research progress in optical flow computation and discuss limitations of the currently popular median filtering heuristic for a flow field optimization. We then introduce a general formulation of the guided filtering and provide the detailed illustration. Furthermore, we explore the potential of the guided filtering optimization for the flow field estimation under the coarse-to-fine computing scheme. Finally, we modify some typical and state-of-the-art optical flow methods by applying the proposed guided filtering operation to the baseline models, and test the performances of the basic and developed models through the Middlebury, MPI-Sintel, and KITTI data. The experimental results demonstrate that the guided filtering scheme is able to preserve the image edges and motion boundaries, and to improve the accuracy and robustness of optical flow estimation.

The authors developed an algorithm for computing optical flow in a differential framework. The image sequence is first convolved with a set of linear, separable spatiotemporal filters similar to those that have been used in other early vision. The brightness constancy constraint can then be applied to each of the resulting images, giving in general, an overdetermined system of equations for the optical flow at each pixel. There are three principal sources of error: (a) stochastic due to sensor noise, (b) systematic errors in the presence of large displacements, and (c) errors due to failure of the brightness constancy model. The analysis of these errors leads to development of an algorithm based on a robust version of total least squares. Each optical flow vector computed has an associated reliability measure which can be used in subsequent processing. The performance of the algorithm on the data set used by J. Barron et al. (1992) compares favorably with other techniques. In addition to being separable, the filters used are also causal, incorporating only past time frames. The algorithm is fully parallel and has been implemented on a multipleprocessor machine. >

Optical flow estimation is a research core of computer vision. In recent years, optical flow estimation methods based on convolutional neural networks (CNNs) have achieved great success. However, most of the existing CNN-based methods are incapable of modeling long-distance dependencies successfully due to the limited receptive fields of convolutions. This may lead to poor performance of optical flow estimation in regions of large displacements and local ambiguities. Furthermore, the problem of edge-blurring remains an open challenge for CNN-based optical flow methods because the general interpolation operation tends to amplify the optical flow errors during the upsampling process. To deal with the abovementioned problems, this paper proposes a novel optical flow estimation method based on local-global modeling and visual similarity guidance. First, an efficient and simple self-attention module is adopted to improve the local-global modeling ability of the network and to extract the more representative features, which decreases the number of optical flow errors caused by large displacements. Second, based on the assumption that the visual features of the objects are more similar, the motion is more similar, and a visual similarity-guided optical flow upsampling network is constructed to guide the upsampling process. By transforming the visual similarity of features into motion similarity, the proposed upsampling scheme improves the accuracy of optical flow estimation in regions of motion boundaries. Finally, we run the proposed method on MPI-Sintel and KITTI test datasets to conduct a comprehensive comparison with some state-of-the-art methods. The experimental results indicate that the proposed method achieves top performance among the comparable methods and, in particular, gains significant developments in regions of large displacements and motion boundaries.

Robust optical flow estimation via edge preserving filtering

The raindrops decrease the quality of outdoor surveillance system, and it brings difficulty for computer vision algorithm, especially in using feature information. This paper proposed a rain removal algorithm based on optical flow and hybrid properties constraint. To identify the candidate rain pixels, the optical flow is adopted. And then, the hybrid properties constraint of raindrops is used to refine the rain streaks. When the rain streaks are accurately detected, the scene can be restored by a weighted composing method. Experiments on rain-affected videos show the feasibility and validity of our algorithm.

This paper presents a new automatic image segmentation method for segmenting moving object in complex environment by combining the motion information with edge information. We propose an adaptive optical flow method based on the Horn-Schunck algorithm to estimate the optical flow field. Our method puts different smoothness constraints on different directions and optical flow constraint is used according to the gradient magnitude. Canny edge detector can obtain the most edge information but miss some pixels. In order to restore these missing pixels the edge has a growing based on the continuity of optical flow field. Next, by remaining the block that has the longest edge could delete the noise in the background, and then the last segmentation result is obtained. The experimental result demonstrates that this method can segment the moving object in complex environment precisely.

Digital subtraction angiography is the gold standard vascular imaging and it is used for all endovascular treatment of intracranial anerysms. Optical flow imaging has been described as a potential method to evaluate cerebral hemodynamics through DSA. In this study, we aimed to compare the flow patterns measured during angiography, by using an optical flow method, with those measured by using computational fluid dynamics in intracranial aneurysms. A consecutive series of 21 patients harboring unruptured saccular intracranial aneurysms who underwent diagnostic angiography before treatment was considered. High-frame-rate digital subtraction angiography was performed to obtain an intra-aneurysmal velocity field by following the cardiac-modulated contrast wave through the vascular structures by using optical flow principles. Additionally, computational fluid dynamics modeling was performed for every case by using patient-specific inlet-boundary conditions measured with the optical flow method from both DSA and 3D rotational angiography datasets. Three independent observers compared qualitatively both the inflow direction and the apparent recirculation in regular DSA, optical flow images, and computational fluid dynamics flow patterns for each patient; κ statistics were estimated. We included 21 patients. In 14 of these 21, the flow patterns were conclusive and matching between the optical flow images and computational fluid dynamics within the same projection view (κ = .91). However, in only 8 of these 14 patients the optical flow images were conclusive and matching regular DSA images (observer κ = 0.87). In 7 of the 21 patients, the flow patterns in the optical flow images were inconclusive, possibly due to improper projection angles. The DSA-based optical flow technique was considered qualitatively consistent with computational fluid dynamics outcomes in evaluating intra-aneurysmal inflow direction and apparent recirculation. Moreover, the optical flow technique may provide the premises for new solutions for improving the visibility of flow patterns when contrast motion in DSA is not apparent. This technique is a diagnostic method to evaluate intra-aneurysmal flow patterns and could be used in the future for validation and patient evaluation.

A shipborne infrared surveillance system, which is tasked with detecting real time target trajectories, needs a kinematic evaluation of the cloudy background. In effect, false alarms appear at the cloud edges; these false alarms reducing the effectiveness of the surveillance system. The elimination of clutter by means of static criteria on each frame is not possible because of the spatial coherence of these edges. The interest of this study is to bring together complementary information that concerns the temporal evolution of the 2D infrared signal. Thus the aim is to evaluate the kinematics of cloudy backgrounds. The originality of this paper resides in the use of the Generalized Hough Transform associated with an affine transform method using optical flow computation. The first transform gives an average displacement for a set of pels, while the second extracts more accurately the motion information by means of differential analysis based on the optical flow constraint equation. The complementary aspect of these two methods of motion estimation is shown.

The gradient-based methods for optical flow estimation are based on a constraint equation which is defined for each image pixel. The structure of the constraint equation makes the problem ill-posed so in the past some solutions have been proposed based on regularization. Under the assumption that in the immediate neighbourhood of a pixel the optical flow field is smooth, the constraint equations in that neighbourhood should have a common solution, in this case the problem is not ill-posed. Following this reasoning, an algorithm for evaluating the optical flow, which is suitable for parallel implementation is proposed in this paper. Moreover, parallel implementations of selected algorithms from the literature, for optical flow estimation, are presented in this paper with the intention of comparing their complexity and performance with respect to the proposed approach on a Connection Machine-2.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Gas–Liquid two-phase flows are a common flow in industrial production processes. Since these flows inherently consist of discrete phases, it is challenging to accurately measure the flow parameters. In this context, a novel approach is proposed that combines the pyramidal Lucas-Kanade (L–K) optical flow method with the Split Comparison (SC) model measurement method. In the proposed approach, videos of gas–liquid two-phase flows are captured using a camera, and optical flow data are acquired from the flow videos using the pyramid L–K optical flow detection method. To address the issue of data clutter in optical flow extraction, a dynamic median value screening method is introduced to optimize the corner point for optical flow calculations. Machine learning algorithms are employed for the prediction model, yielding high flow prediction accuracy in experimental tests. Results demonstrate that the gradient boosted regression (GBR) model is the most effective among the five preset models, and the optimized SC model significantly improves measurement accuracy compared to the GBR model, achieving an R2 value of 0.97, RMSE of 0.74 m3/h, MAE of 0.52 m3/h, and MAPE of 8.0%. This method offers a new approach for monitoring flows in industrial production processes such as oil and gas.

Abstract. Real-time change detection and analysis of natural disasters is of great importance to emergency response and disaster rescue. Recently, a number of video satellites that can record the whole process of natural disasters have been launched. These satellites capture high resolution video image sequences and provide researchers with a large number of image frames, which allows for the implementation of a rapid disaster procedure change detection approach based on deep learning. In this paper, pixel change in image sequences is estimated by optical flow based on FlowNet 2.0 for quick change detection in natural disasters. Experiments are carried out by using image frames from Digital Globe WorldView in Indonesia Earthquake took place on Sept. 28, 2018. In order to test the efficiency of FlowNet 2.0 on natural disaster dataset, 7 state-of-the-art optical flow estimation methods are compared. The experimental results show that FlowNet 2.0 is not only robust to large displacements but small displacements in natural disaster dataset. Two evaluation indicators: Root Mean Square Error (RMSE) and Mean Value are used to record the accuracy. For estimation error of RMSE, FlowNet 2.0 achieves 0.30 and 0.11 pixels in horizontal and vertical direction, respectively. The error in horizontal error is similar to other algorithms but the value in vertical direction is significantly lower than them. And the Mean Value are 1.50 and 0.09 pixels in horizontal and vertical direction, which are most close to the ground truth comparing to other algorithms. Combining the superiority of computing time, the paper proves that only the approach based on FlowNet 2.0 is able to achieve real-time change detection with higher accuracy in the case of natural disasters.

A wavelet-based optical flow method for high-resolution velocimetry based on tracer particle images is presented. The current optical flow estimation method (WOF-X) is designed for improvements in processing experimental images by implementing wavelet transforms with the lifting method and symmetric boundary conditions. This approach leads to speed and accuracy improvements over the existing wavelet-based methods. The current method also exploits the properties of fluid flows and uses the known behavior of turbulent energy spectra to semi-automatically tune a regularization parameter that has been primarily determined empirically in the previous optical flow algorithms. As an initial step in evaluating the WOF-X method, synthetic particle images from a 2D DNS of isotropic turbulence are processed and the results are compared to a typical correlation-based PIV algorithm and previous optical flow methods. The WOF-X method produces a dense velocity estimation, resulting in an order-of-magnitude increase in velocity vector resolution compared to the traditional correlation-based PIV processing. Results also show an improvement in velocity estimation by more than a factor of two. The increases in resolution and accuracy of the velocity field lead to significant improvements in the calculation of velocity gradient-dependent properties such as vorticity. In addition to the DNS results, the WOF-X method is evaluated in a series of two-dimensional vortex flow simulations to determine optimal experimental design parameters. Recommendations for optimal conditions for tracer particle seed density and inter-frame particle displacement are presented. The WOF-X method produces minimal error at larger particle displacements and lower relative error over a larger velocity dynamic range as compared to correlation-based processing.