Neuromorphic Configurable Architecture for Robust Motion Estimation

Optical flow can greatly improve the robustness of visual tracking algorithms. While dense optical flow algorithms have various applications, they can not be used for real-time solutions without resorting to GPU calculations. Furthermore, most optical flow algorithms fail in challenging lighting environments due to the violation of the brightness constraint. We propose a simple but effective iterative regularisation scheme for real-time, sparse optical flow algorithms, that is shown to be robust to sudden illumination changes and can handle large displacements. The algorithm proves to outperform well known techniques in real life video sequences, while being much faster to calculate. Our solution increases the robustness of a real-time particle filter based tracking application, consuming only a fraction of the available CPU power. Furthermore, a new and realistic optical flow dataset with annotated ground truth is created and made freely available for research purposes.

Inherent limitations of the biological mechanism of vision have resulted in elegant algorithms to accomplish the required information processing. Biological and psychophysical evidence have led to hypothesizing just such an algorithm for the processing of motion information. The Temporal Frequency Spectrum (TFS) model for motion perception uses a transformation process analogous with those Fourier processes sometimes used in the study of spatial vision. The TFS can serve as a metric for describing complex moving scenes and as a model for human motion perception. This model can accomplish motion perception even when the viewer is in motion. Human psychophysical evidence is provided consistent with a TFS mechanism and the use of a TFS mechanism in machine vision is considered.

Two-step phase extraction and random phase shift estimation in phase-shifting profilometry based on least-squared optical flow method

In this paper, we consider a motion sense problem motivated by the International Aerial Robotics Competition Mission-7, where an aerial robot is required to provide detection and estimation about mobile vehicles. Dense optical flow computing is employed first to provide a velocity field from image sequences. Then, region growing based on the optical flow field is used to extract moving objects on the background, and motion estimation is eventually achieved while both camera and objects are moving. In addition, classical optical flow techniques do not work in the competition since there may be illumination changes, such as flashlights and reflections in the arena. To deal with this problem, the procedures of the brightness constancy relaxation and intensity normalization are combined in the optical flow algorithm. Experimental results have demonstrated the robustness against varying illumination. The proposed approach can provide motion estimation results of acceptable accuracy for several benchmark data sets and image sequences generated with micro aerial vehicles.

Temporal segmentation of a video sequence into different shots is fundamental to a number of video retrieval and analysis applications. Motion estimation has been wildly used in many applications of video processing, since it provides the most essential information for an image sequence. In this paper, we explore the possibility to exploit motion and illumination estimation in a video sequence to detect various types of shot changes. Optical flow is the motion vector computed at each pixel in an image sequence from intensity variation. Traditionally, optical flow computational algorithms were derived from the brightness constancy assumption. In this paper, we employ a generalized optical flow constraint that includes an illumination parameter to model local illumination changes. An iterative optical flow and illumination estimation algorithm is developed in this paper to refine the generalized optical flow constraints step by step, thus leading to a very accurate estimation of the optical flow and illumination parameters. Two robust measures are defined from the mean and standard deviation of the estimated intensity compensation values for all the blocks in the same image. Either of these two measures corresponds significantly to various types of shot changes. We show the usefulness of these two measures through experiments.

Abstract. Eastern China is one of the most economically developed and densely populated areas in the world. Due to its special geographical location and climate, eastern China is affected by different weather systems, such as monsoons, shear lines, typhoons, and extratropical cyclones. In the near future, the rainfall rate becomes difficult to predict precisely due to these systems. Traditional physics-based methods such as numerical weather prediction (NWP) tend to perform poorly on nowcasting problems due to the spin-up issue. Moreover, various meteorological stations are distributed in this region, generating a large amount of observation data every day, which have great potential for application to data-driven methods. Thus, it is important to train a data-driven model from scratch that is suitable for the specific weather situation of eastern China. However, due to the high degrees of freedom and nonlinearity of machine learning algorithms, it is difficult to add physical constraints. Therefore, with the intention of using various kinds of data as a proxy for physical constraints, we collected three kinds of data (radar, satellite, and precipitation data) in the flood season from 2017 to 2018 in this area and preprocessed them into tensors (256×256) that cover eastern China with a domain of 12.8×12.8∘. The developed multisource data model (MSDM) combines the optical flow, random forest, and convolutional neural network (CNN) algorithms. It treats the precipitation nowcasting task as an image-to-image problem, which takes radar and satellite data with an interval of 30 min as inputs and predicts radar echo intensity with a lead time of 30 min. To reduce the smoothing caused by convolutions, we use the optical flow algorithm to predict satellite data in the following 120 min. The predicted radar echoes from the MSDM together with satellite data from the optical flow algorithm are recursively implemented in the MSDM to achieve a 120 min lead time. The MSDM predictions are comparable to those of other baseline models with a high temporal resolution of 6 min. To solve blurry image problems, we applied a modified structural similarity (SSIM) index as a loss function. Furthermore, we use the random forest algorithm with predicted radar and satellite data to estimate the rainfall rate, and the results outperform those of the traditional, nonlinear radar reflectivity factor and rainfall rate (Z–R) relationships that use logarithmic functions. The experiments confirm that machine learning with multisource data provides more reasonable predictions and reveals a better nonlinear relationship between radar echo and precipitation rate. Apart from developing complicated machine learning algorithms, exploiting the potential of multisource data will yield more improvements.

Particle image velocimetry (PIV) has matured as a flow measurement technique. It enables the description of the instantaneous velocity field of the flow by analyzing the particle motion obtained from digitally recorded images. Correlation based PIV evaluation technique is widely used because of its good accuracy and robustness. Although very successful, correlation PIV technique has some weakness which can be avoided by optical flow based PIV algorithms. At present, most of the optical flow methods applied to PIV are based on brightness constancy assumption. However, some factors of flow imaging technology and the nature property of the fluids make the brightness constancy assumption less appropriate in real PIV cases. In this paper, an implementation of a 2D optical flow algorithm (GCOF) based on gradient constancy assumption is introduced. The proposed GCOF assumes the edges of the illuminated PIV particles are constant during motion. It comprises two terms: a combined local-global gradient data term and a first-order divergence and vorticity smooth term. The approach can provide accurate dense motion fields. The approach are tested on synthetic images and on two experimental flows. The comparison of GCOF with other optical flow algorithms indicates the proposed method is more accurate especially in conditions of illumination variation. The comparison of GCOF with correlation PIV technique shows that the proposed GCOF has advantages on preserving small divergence and vorticity structures of the motion field and getting less outliers. As a consequence, the GCOF acquire a more accurate and better topological description of the turbulent flow.

The existing optical flow algorithms don’t meet the requirements of human visual habits. In this chapter, a novel optical flow algorithm is proposed, in order to enhance the bionic features of optical flow. It is sensitive for new algorithms to extract the visual interesting feature such as color and motion. The robust tracing framework based on new optical flow algorithm can track more corners in the opponent color space which has the optimal bionic effect. Experiments show new framework can find new bionic features and trace more bionic corners.

For the visual navigation and positioning of drones, the optical flow algorithm is used to calculate the flight speed of the drone, the calculation errors of its rotation state, and the vertical take-off and landing state. Based on the characteristics of its optical flow data, this paper compares various classification calculation methods. It uses the median KNN curve of the optical flow vector to simplify its feature quantity. Finally, it realizes the motion of the UAV with the optical flow characteristics based on the SVM algorithm. State recognition experiments have proved that this method can effectively identify the motion state of the UAV, and the calculation efficiency is high.

Optical flow can be used to segment a moving object from its backgrounds and track it. In this paper, an Enhanced Lucas-Kanade optical flow technique was used to improve human detection in terms of speed and accuracy. We combined object segmentation output with a human detector using an optical flow algorithm. The proposed technique used the optical flow to find the area of interest to complete object segmentation and use those results as an input for the human detector. This technique has been developed to be used in surveillance systems. Our experiments indicated that the proposed method was 37% faster and 118% more accurate than the standard Felzenszwalb (PFF) detector.

Towards computer-assisted neurosurgery, scene understanding algorithms for microscope video data are required. Previous work utilizes optical flow to extract spatiotemporal context from neurosurgical video sequences. However, to select an appropriate optical flow method, we need to analyze which algorithm yields the highest accuracy for the neurosurgical domain. Currently, there are no benchmark datasets available for neurosurgery. In our work, we present an approach to generate synthetic data for optical flow evaluation on the neurosurgical domain. We simulate image sequences and thereby take into account domainspecific visual conditions such as surgical instrument motion. Then, we evaluate two optical flow algorithms, Farneback and PWC-Net, on our synthetic data. Qualitative and quantitative assessments confirm that our data can be used to evaluate optical flow for the neurosurgical domain. Future work will concentrate on extending the method by modeling additional effects in neurosurgery such as elastic background motion.

Quantization analysis and enhancement of a VLSI gradient-based motion estimation architecture

It has been a challenging task to estimate optical flow for videos in which either foreground or background exhibits remarkable motion information (i.e., large displacement), or those with insufficient resolution due to artifacts like motion blur or noise. We present a novel optical flow algorithm, which approaches the above problem as solving the task of energy minimization, which exploits image data and smoothness terms at the superpixel level. Our proposed method can be considered as an extended mean-shift algorithm, which advances color and gradient information of superpixels across consecutive frames with smoothness guarantees. Since we do not require assumptions of linearlization during optimization (as standard optical flow approaches do), we are able to alleviate local minimum problems and thus produce improved estimation results. Empirical results on the MPI-Sintel video dataset verify the effectiveness of our proposed method.

A number of proposed optical flow estimation algorithms for image motion analysis use only a few subsequent frames from an image sequence to perform calculation. Others, that use many frames, as is case in frequency and energy based, as well as in stochastic optical flow algorithms, are better suited for off-line work. We present our approach in adopting a classic gradient optical flow algorithm for on-line work over extended image sequence. Our experiments on the usual optical flow testing image sequences, consisting of few dozens of image frames show significant improvements in accuracy of the estimated velocity flow fields and in speed of computation. This is achieved by modification and integration of three basic processing phases: preparation of a limited spatio-temporal volume, estimation of velocity flow fields and post filtering of the flow fields. In the preparation phase we use efficient spatio-temporal filters for image smoothing. In the estimation phase a modified iterative solution that propagates results from previous calculation steps is used. Linear and nonlinear post filtering additionally improve optical flow estimates. The whole process is less sensitive to the aperture problem and produces more accurate flow field estimates than the original algorithm. It better copes with motion discontinuities as well. We evidence our achievements by a number of experiments on extended image sequences.

This paper proposes a novel particle image velocimetry (PIV) technique to generate an instantaneous two-dimensional velocity field for sediment-laden fluid based on the optical flow algorithm of ultrasound imaging. In this paper, an ultrasonic PIV (UIV) system is constructed by integrating a medical ultrasound instrument and an ultrasonic particle image velocimetry algorithm. The medical ultrasound instrument with a phased sensor array is used to acquire acoustic echo signals of particles in water and generate two-dimensional underwater ultrasound images. Based on the optical flow field, the instantaneous velocity of the particles in water corresponding to the pixels in the ultrasonic particle images is derived from the grayscale change between adjacent images under the L-K local constraint based on the optical flow field, and finally, the two-dimensional flow field is obtained. Through multiple sets of experiments, the proposed algorithm is verified. The experimental results are compared with those of the conventional cross-correlation algorithms. The results show that the L-K optical flow method can not only obtain the underwater velocity field accurately, but also has the advantages of good smoothness and extensive suitability, especially for the flow field measurement in sediment-laden fluid than conventional algorithms.