Parameter optimisation for vision guided terrestrial locomotion: Multi-frame

Two-step phase extraction and random phase shift estimation in phase-shifting profilometry based on least-squared optical flow method

Abstract. Eastern China is one of the most economically developed and densely populated areas in the world. Due to its special geographical location and climate, eastern China is affected by different weather systems, such as monsoons, shear lines, typhoons, and extratropical cyclones. In the near future, the rainfall rate becomes difficult to predict precisely due to these systems. Traditional physics-based methods such as numerical weather prediction (NWP) tend to perform poorly on nowcasting problems due to the spin-up issue. Moreover, various meteorological stations are distributed in this region, generating a large amount of observation data every day, which have great potential for application to data-driven methods. Thus, it is important to train a data-driven model from scratch that is suitable for the specific weather situation of eastern China. However, due to the high degrees of freedom and nonlinearity of machine learning algorithms, it is difficult to add physical constraints. Therefore, with the intention of using various kinds of data as a proxy for physical constraints, we collected three kinds of data (radar, satellite, and precipitation data) in the flood season from 2017 to 2018 in this area and preprocessed them into tensors (256×256) that cover eastern China with a domain of 12.8×12.8∘. The developed multisource data model (MSDM) combines the optical flow, random forest, and convolutional neural network (CNN) algorithms. It treats the precipitation nowcasting task as an image-to-image problem, which takes radar and satellite data with an interval of 30 min as inputs and predicts radar echo intensity with a lead time of 30 min. To reduce the smoothing caused by convolutions, we use the optical flow algorithm to predict satellite data in the following 120 min. The predicted radar echoes from the MSDM together with satellite data from the optical flow algorithm are recursively implemented in the MSDM to achieve a 120 min lead time. The MSDM predictions are comparable to those of other baseline models with a high temporal resolution of 6 min. To solve blurry image problems, we applied a modified structural similarity (SSIM) index as a loss function. Furthermore, we use the random forest algorithm with predicted radar and satellite data to estimate the rainfall rate, and the results outperform those of the traditional, nonlinear radar reflectivity factor and rainfall rate (Z–R) relationships that use logarithmic functions. The experiments confirm that machine learning with multisource data provides more reasonable predictions and reveals a better nonlinear relationship between radar echo and precipitation rate. Apart from developing complicated machine learning algorithms, exploiting the potential of multisource data will yield more improvements.

We present a system that estimates the motion of a stereo head, or a single moving camera, based on video input. The system operates in real time with low delay, and the motion estimates are used for navigational purposes. The front end of the system is a feature tracker. Point features are matched between pairs of frames and linked into image trajectories at video rate. Robust estimates of the camera motion are then produced from the feature tracks using a geometric hypothesize‐and‐test architecture. This generates motion estimates from visual input alone. No prior knowledge of the scene or the motion is necessary. The visual estimates can also be used in conjunction with information from other sources, such as a global positioning system, inertia sensors, wheel encoders, etc. The pose estimation method has been applied successfully to video from aerial, automotive, and handheld platforms. We focus on results obtained with a stereo head mounted on an autonomous ground vehicle. We give examples of camera trajectories estimated in real time purely from images over previously unseen distances (600 m) and periods of time. © 2006 Wiley Periodicals, Inc.

Optical flow can greatly improve the robustness of visual tracking algorithms. While dense optical flow algorithms have various applications, they can not be used for real-time solutions without resorting to GPU calculations. Furthermore, most optical flow algorithms fail in challenging lighting environments due to the violation of the brightness constraint. We propose a simple but effective iterative regularisation scheme for real-time, sparse optical flow algorithms, that is shown to be robust to sudden illumination changes and can handle large displacements. The algorithm proves to outperform well known techniques in real life video sequences, while being much faster to calculate. Our solution increases the robustness of a real-time particle filter based tracking application, consuming only a fraction of the available CPU power. Furthermore, a new and realistic optical flow dataset with annotated ground truth is created and made freely available for research purposes.

Particle image velocimetry (PIV) has matured as a flow measurement technique. It enables the description of the instantaneous velocity field of the flow by analyzing the particle motion obtained from digitally recorded images. Correlation based PIV evaluation technique is widely used because of its good accuracy and robustness. Although very successful, correlation PIV technique has some weakness which can be avoided by optical flow based PIV algorithms. At present, most of the optical flow methods applied to PIV are based on brightness constancy assumption. However, some factors of flow imaging technology and the nature property of the fluids make the brightness constancy assumption less appropriate in real PIV cases. In this paper, an implementation of a 2D optical flow algorithm (GCOF) based on gradient constancy assumption is introduced. The proposed GCOF assumes the edges of the illuminated PIV particles are constant during motion. It comprises two terms: a combined local-global gradient data term and a first-order divergence and vorticity smooth term. The approach can provide accurate dense motion fields. The approach are tested on synthetic images and on two experimental flows. The comparison of GCOF with other optical flow algorithms indicates the proposed method is more accurate especially in conditions of illumination variation. The comparison of GCOF with correlation PIV technique shows that the proposed GCOF has advantages on preserving small divergence and vorticity structures of the motion field and getting less outliers. As a consequence, the GCOF acquire a more accurate and better topological description of the turbulent flow.

The existing optical flow algorithms don’t meet the requirements of human visual habits. In this chapter, a novel optical flow algorithm is proposed, in order to enhance the bionic features of optical flow. It is sensitive for new algorithms to extract the visual interesting feature such as color and motion. The robust tracing framework based on new optical flow algorithm can track more corners in the opponent color space which has the optimal bionic effect. Experiments show new framework can find new bionic features and trace more bionic corners.

Dense optical flow estimation is complex and time consuming, with state-of-the-art methods relying either on large synthetic data sets or on pipelines requiring up to a few minutes per frame pair. In this paper, we address the problem of optical flow estimation in the automotive scenario in a self-supervised manner. We argue that optical flow can be cast as a geometrical warping between two successive video frames and devise a deep architecture to estimate such transformation in two stages. First, a dense pixel-level flow is computed with a projective bootstrap on rigid surfaces. We show how such global transformation can be approximated with a homography and extend spatial transformer layers so that they can be employed to compute the flow field implied by such transformation. Subsequently, we refine the prediction by feeding a second, deeper network that accounts for moving objects. A final reconstruction loss compares the warping of frame X <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> with the subsequent frame X <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t+1</sub> and guides both estimates. The model has the speed advantages of end-to-end deep architectures while achieving competitive performances, both outperforming recent unsupervised methods and showing good generalization capabilities on new automotive data sets.

For the visual navigation and positioning of drones, the optical flow algorithm is used to calculate the flight speed of the drone, the calculation errors of its rotation state, and the vertical take-off and landing state. Based on the characteristics of its optical flow data, this paper compares various classification calculation methods. It uses the median KNN curve of the optical flow vector to simplify its feature quantity. Finally, it realizes the motion of the UAV with the optical flow characteristics based on the SVM algorithm. State recognition experiments have proved that this method can effectively identify the motion state of the UAV, and the calculation efficiency is high.

Optical flow can be used to segment a moving object from its backgrounds and track it. In this paper, an Enhanced Lucas-Kanade optical flow technique was used to improve human detection in terms of speed and accuracy. We combined object segmentation output with a human detector using an optical flow algorithm. The proposed technique used the optical flow to find the area of interest to complete object segmentation and use those results as an input for the human detector. This technique has been developed to be used in surveillance systems. Our experiments indicated that the proposed method was 37% faster and 118% more accurate than the standard Felzenszwalb (PFF) detector.

Towards computer-assisted neurosurgery, scene understanding algorithms for microscope video data are required. Previous work utilizes optical flow to extract spatiotemporal context from neurosurgical video sequences. However, to select an appropriate optical flow method, we need to analyze which algorithm yields the highest accuracy for the neurosurgical domain. Currently, there are no benchmark datasets available for neurosurgery. In our work, we present an approach to generate synthetic data for optical flow evaluation on the neurosurgical domain. We simulate image sequences and thereby take into account domainspecific visual conditions such as surgical instrument motion. Then, we evaluate two optical flow algorithms, Farneback and PWC-Net, on our synthetic data. Qualitative and quantitative assessments confirm that our data can be used to evaluate optical flow for the neurosurgical domain. Future work will concentrate on extending the method by modeling additional effects in neurosurgery such as elastic background motion.

Applications such as autonomous navigation, robot vision, and autonomous flying require depth map information of a scene. Depth can be estimated by using a single moving camera (depth from motion). However, the traditional depth from motion algorithms have low processing speeds and high hardware requirements that limit the embedded capabilities. In this work, we propose a hardware architecture for depth from motion that consists of a flow/depth transformation and a new optical flow algorithm. Our optical flow formulation consists in an extension of the stereo matching problem. A pixel-parallel/window-parallel approach where a correlation function based on the sum of absolute difference (SAD) computes the optical flow is proposed. Further, in order to improve the SAD, the curl of the intensity gradient as a preprocessing step is proposed. Experimental results demonstrated that it is possible to reach higher accuracy (90% of accuracy) compared with previous Field Programmable Gate Array (FPGA)-based optical flow algorithms. For the depth estimation, our algorithm delivers dense maps with motion and depth information on all image pixels, with a processing speed up to 128 times faster than that of previous work, making it possible to achieve high performance in the context of embedded applications.

Unsupervised deep learning methods have demonstrated an impressive performance for understanding the structure of 3D scene from videos. These data-based learning methods are able to learn the tasks, such as depth, ego-motion, and optical flow estimation. In this paper, we propose a novel unsupervised deep learning method to jointly estimate scene depth, camera ego-motion, and optical flow from stereo images. Consecutive stereo images are used to train the system. After training stage, the system is able to estimate dense depth map, camera 6D pose, and optical flow by using a sequence of monocular images. No labelled data set is required for training. The supervision signals for training three deep neural networks of the system come from various forms of image warping. Due to the use of optical flow, the impact caused by occlusions and moving objects on the estimation results is alleviated. Experiments on the KITTI and Cityscapes datasets show that the proposed system demonstrates a better performance in terms of accuracy in depth, ego-motion, and optical flow estimation.

It has been a challenging task to estimate optical flow for videos in which either foreground or background exhibits remarkable motion information (i.e., large displacement), or those with insufficient resolution due to artifacts like motion blur or noise. We present a novel optical flow algorithm, which approaches the above problem as solving the task of energy minimization, which exploits image data and smoothness terms at the superpixel level. Our proposed method can be considered as an extended mean-shift algorithm, which advances color and gradient information of superpixels across consecutive frames with smoothness guarantees. Since we do not require assumptions of linearlization during optimization (as standard optical flow approaches do), we are able to alleviate local minimum problems and thus produce improved estimation results. Empirical results on the MPI-Sintel video dataset verify the effectiveness of our proposed method.

In this paper, we consider a motion sense problem motivated by the International Aerial Robotics Competition Mission-7, where an aerial robot is required to provide detection and estimation about mobile vehicles. Dense optical flow computing is employed first to provide a velocity field from image sequences. Then, region growing based on the optical flow field is used to extract moving objects on the background, and motion estimation is eventually achieved while both camera and objects are moving. In addition, classical optical flow techniques do not work in the competition since there may be illumination changes, such as flashlights and reflections in the arena. To deal with this problem, the procedures of the brightness constancy relaxation and intensity normalization are combined in the optical flow algorithm. Experimental results have demonstrated the robustness against varying illumination. The proposed approach can provide motion estimation results of acceptable accuracy for several benchmark data sets and image sequences generated with micro aerial vehicles.

A number of proposed optical flow estimation algorithms for image motion analysis use only a few subsequent frames from an image sequence to perform calculation. Others, that use many frames, as is case in frequency and energy based, as well as in stochastic optical flow algorithms, are better suited for off-line work. We present our approach in adopting a classic gradient optical flow algorithm for on-line work over extended image sequence. Our experiments on the usual optical flow testing image sequences, consisting of few dozens of image frames show significant improvements in accuracy of the estimated velocity flow fields and in speed of computation. This is achieved by modification and integration of three basic processing phases: preparation of a limited spatio-temporal volume, estimation of velocity flow fields and post filtering of the flow fields. In the preparation phase we use efficient spatio-temporal filters for image smoothing. In the estimation phase a modified iterative solution that propagates results from previous calculation steps is used. Linear and nonlinear post filtering additionally improve optical flow estimates. The whole process is less sensitive to the aperture problem and produces more accurate flow field estimates than the original algorithm. It better copes with motion discontinuities as well. We evidence our achievements by a number of experiments on extended image sequences.