Physics-based optical flow estimation under varying illumination conditions

<p indent=0mm>Combining the deep learning model to compute end-to-end optical flow is a hot topic in current computer vision field. The optical flow estimation methods based on deep learning are summarized and reviewed. Firstly, the origin and concept of optical flow is introduced. Secondly, the optical flow datasets and evaluation metrics are summarized. Most importantly, classical methods are introduced and the deep optical flow estimation methods are reviewed in three aspects, including supervised deep optical flow estimation methods, unsupervised deep optical flow estimation methods and the performance of these methods. The analysis shows designing compact and generalized deep optical flow model is the future research direction. On this basis, the joint learning of optical flow estimation and specific video analysis tasks are introduced. It is pointed out that the design of task-driven deep optical flow network is a valuable research direction in practice application. Finally, the problems and challenges of deep optical flow estimation are summarized and the future work of this field is prospected.

This paper proposed a moving vehicle detection algorithm based on optical flow estimation on an edge image. Using the Canny operator, the image edge is obtained and refined. Then a set of feature points is extracted from the edge image. The pyramid model of Lucas-Kanade optical flow is used to calculate the optical flow information of the feature point set. A new algorithm, which is called the weighted Kmeans optical flow clustering algorithm, is proposed to cluster feature points and used to identify vehicle pattern of the feature point set on the optical flow so that the moving vehicle can be efficiently extracted from the complicated dynamic background. In this paper the vehicle videos, which are captured by a camera on a moving car, are used as the test data set. The experimental results show that this algorithm can effectively detect the moving vehicles in the videos from the camera on the moving car.

This paper proposes a new semitransparency-based optical-flow model with a point trajectory (PT) model for particle-like video. Previous optical-flow models have used ranging from image brightness constancy to image brightness change models as constraints. However, two important issues remain unsolved. The first is how to track/match a semitransparent object with a very large displacement between frames. Such moving objects with different shapes and sizes in an outdoor scene move against a complicated background. Second, due to semitransparency, the image intensity between frames can also violate a previous image brightness-based optical-flow model. Thus, we propose a two-step optimization for the optical-flow estimation model for a moving semitransparent object, i.e., particle. In the first step, a rough optical flow between particles is estimated by a new alpha constancy constraint that is based on an image generation model of semitransparency. In the second step, the optical flow of a particle with a continuous trajectory in a definite temporal interval based on a PT model can be refined. Many experiments using various falling-snow and foggy scenes with multiple moving vehicles show the significant improvement of the optical flow compared with a previous optical-flow model.

In general, discretization in the motion vector domain yields an intractable number of labels. In this paper, we propose an approach that can reduce general optical flow to the constrained matching problem by pre-estimating a 2-D disparity labeling map of the desired discrete motion vector function. One of the goals of the proposed paper is estimating coarse distribution of motion vectors and then utilizing this distribution as global constraints for discrete optical flow estimation. This pre-estimation is done with a simple frame-to-frame correlation technique also known as the digital symmetric-phase-only-filter (SPOF). We discover a strong correlation between the output of the SPOF and the motion vector distribution of the related optical flow. A two step matching paradigm for optical flow estimation is applied: pixel accuracy (integer flow) and subpixel accuracy estimation. The matching problem is solved by global optimization. Experiments on the Middlebury optical flow datasets confirm our intuitive assumptions about strong correlation between motion vector distribution of optical flow and maximal peaks of SPOF outputs. The overall performance of the proposed method is promising and achieves state-of-the-art results on the Middlebury benchmark.

In this paper, we propose UnOS, an unified system for unsupervised optical flow and stereo depth estimation using convolutional neural network (CNN) by taking advantages of their inherent geometrical consistency based on the rigid-scene assumption. UnOS significantly outperforms other state-of-the-art (SOTA) unsupervised approaches that treated the two tasks independently. Specifically, given two consecutive stereo image pairs from a video, UnOS estimates per-pixel stereo depth images, camera ego-motion and optical flow with three parallel CNNs. Based on these quantities, UnOS computes rigid optical flow and compares it against the optical flow estimated from the FlowNet, yielding pixels satisfying the rigid-scene assumption. Then, we encourage geometrical consistency between the two estimated flows within rigid regions, from which we derive a rigid-aware direct visual odometry (RDVO) module. We also propose rigid and occlusion-aware flow-consistency losses for the learning of UnOS. We evaluated our results on the popular KITTI dataset over 4 related tasks, \ie stereo depth, optical flow, visual odometry and motion segmentation.

We present an unsupervised learning framework for simultaneously training single-view depth prediction and optical flow estimation models using unlabeled video sequences. Existing unsupervised methods often exploit brightness constancy and spatial smoothness priors to train depth or flow models. In this paper, we propose to leverage geometric consistency as additional supervisory signals. Our core idea is that for rigid regions we can use the predicted scene depth and camera motion to synthesize 2D optical flow by backprojecting the induced 3D scene flow. The discrepancy between the rigid flow (from depth prediction and camera motion) and the estimated flow (from optical flow model) allows us to impose a cross-task consistency loss. While all the networks are jointly optimized during training, they can be applied independently at test time. Extensive experiments demonstrate that our depth and flow models compare favorably with state-of-the-art unsupervised methods.

Optical flow estimation is a research core of computer vision. In recent years, optical flow estimation methods based on convolutional neural networks (CNNs) have achieved great success. However, most of the existing CNN-based methods are incapable of modeling long-distance dependencies successfully due to the limited receptive fields of convolutions. This may lead to poor performance of optical flow estimation in regions of large displacements and local ambiguities. Furthermore, the problem of edge-blurring remains an open challenge for CNN-based optical flow methods because the general interpolation operation tends to amplify the optical flow errors during the upsampling process. To deal with the abovementioned problems, this paper proposes a novel optical flow estimation method based on local-global modeling and visual similarity guidance. First, an efficient and simple self-attention module is adopted to improve the local-global modeling ability of the network and to extract the more representative features, which decreases the number of optical flow errors caused by large displacements. Second, based on the assumption that the visual features of the objects are more similar, the motion is more similar, and a visual similarity-guided optical flow upsampling network is constructed to guide the upsampling process. By transforming the visual similarity of features into motion similarity, the proposed upsampling scheme improves the accuracy of optical flow estimation in regions of motion boundaries. Finally, we run the proposed method on MPI-Sintel and KITTI test datasets to conduct a comprehensive comparison with some state-of-the-art methods. The experimental results indicate that the proposed method achieves top performance among the comparable methods and, in particular, gains significant developments in regions of large displacements and motion boundaries.

Robust depth estimation on real-world light field images using Gaussian belief propagation

The estimation of optical flow is an important problem in the dynamic image analysis. There are various methods of estimating the optical flow. The gradient method and matching method are based on the operation in the space-time domain. While spectral energy method is based on the frequency analysis and has robustness against noise. In this paper, we propose a hierarchical estimation of the optical flow using spectral energy method. In the proposed method, we make a set of hierarchical images for each frame. A global motion which is estimated at the coarsest resolution image sequences is corrected by using finer resolution image sequences. Consequently we can estimate both global and local motion accurately with robustness to noise.

This paper focuses on crowd motion analysis and consists two parts. Firstly, we propose a new foreground detection approach called optical flow and background model (OFBM) based on Lucas-Kanade optical flow and Gaussian background model methods. This approach overcomes the shortages of optical flow and background subtract, such as sensitiveness of light changing and producing accumulate errors. Secondly, according to moment analysis, we propose a new feature based on the zeroth-order Tehebichef discrete orthogonal moment (TOM), which is employed for crowd density estimation. Some experimental results show that this approach is useful and efficient in crowd density estimation.

This paper presents the analytical of non-Gaussian noise resistance with the aid of the use of bilateral in reverse confidential with the optical flow. In particular, optical flow is the sample of the image’s motion from the consecutive images caused by the object’s movement. It is a 2-D vector where every vector is a displacement vector displaying the motion from the first image to the second. When the noise interferes with the image flow, the approximated performance on the vector in optical flow is poor. We ensure greater appropriate noise resistance by applying bilateral in reverse confidential in optical flow in the experiment by concerning the error vector magnitude (EVM). Many noise resistance models of the global method optical flow are using for comparison in our experiment. And many sequenced image data sets where they are interfered with by several types of non-Gaussian noise are used for experimental analysis.

Optical flow estimation has made great progress, but usually suffers from degradation under adverse weather. Although semi/full-supervised methods have made good attempts, the domain shift between the synthetic and real adverse weather images would deteriorate their performance. To alleviate this issue, our start point is to unsupervisedly transfer the knowledge from source clean domain to target degraded domain. Our key insight is that adverse weather does not change the intrinsic optical flow of the scene, but causes a significant difference for the warp error between clean and degraded images. In this work, we propose the first unsupervised framework for adverse weather optical flow via hierarchical motion-boundary adaptation. Specifically, we first employ image translation to construct the transformation relationship between clean and degraded domains. In motion adaptation, we utilize the flow consistency knowledge to align the cross-domain optical flows into a motion-invariance common space, where the optical flow from clean weather is used as the guidance-knowledge to obtain a preliminary optical flow for adverse weather. Furthermore, we leverage the warp error inconsistency which measures the motion misalignment of the boundary between the clean and degraded domains, and propose a joint intra- and inter-scene boundary contrastive adaptation to refine the motion boundary. The hierarchical motion and boundary adaptation jointly promotes optical flow in a unified framework. Extensive quantitative and qualitative experiments have been performed to verify the superiority of the proposed method.

A motion estimation algorithm using wavelet approximation as an optical flow model has been developed to estimate accurate dense optical flow from an image sequence. This wavelet motion model is particularly useful in estimating optical flows with large displacement. Traditional pyramid methods which use the coarse-to-fine image pyramid by image burring in estimating optical flow often produce incorrect results when the coarse-level estimates contain large errors that cannot be corrected at the subsequent finer levels. This happens when regions of low texture become flat or certain patterns result in spatial aliasing due to image blurring. Our method, in contrast, uses large-to-small full-resolution regions without blurring images, and simultaneously optimizes the coarser and finer parts of optical flow so that the large and small motion can be estimated correctly. We compare results obtained by using our method with those obtained by using one of the leading optical flow methods, the Szeliski pyramid spline-based method. The experiments include cases of small displacement (less than 4 pixels under 128/spl times/128 image size or equivalent displacement under other image sizes), and those of large displacement (10 pixels). While both methods produce comparable results when the displacements are small, our method outperforms pyramid spline-based method when the displacements are large.

Optical Flow Estimation aims to find the 2D dense motion field between two frames. Due to the limitation of model structures and training datasets, existing methods often rely too much on local clues and ignore the integrity of objects, resulting in fragmented motion estimation. Through theoretical analysis, we find the pre-trained large vision models are helpful in optical flow estimation, and we notice that the recently famous Segment Anything Model (SAM) demonstrates a strong ability to segment complete objects, which is suitable for solving the fragmentation problem. We thus propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. To address the challenge of in-depth utilizing SAM in non-segmentation tasks like optical flow estimation, we propose an Optical Flow Task-Specific Adaption scheme, including a Context Fusion Module to fuse the SAM encoder with the optical flow context encoder, and a Context Adaption Module to adapt the SAM features for optical flow task with Learned Task-Specific Embedding. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%. Furthermore, our model achieves state-of-the-art performance on the Sintel and KITTI-15 benchmarks, ranking #1 among all two-frame methods on Sintel clean pass.

Unsupervised deep learning methods have demonstrated an impressive performance for understanding the structure of 3D scene from videos. These data-based learning methods are able to learn the tasks, such as depth, ego-motion, and optical flow estimation. In this paper, we propose a novel unsupervised deep learning method to jointly estimate scene depth, camera ego-motion, and optical flow from stereo images. Consecutive stereo images are used to train the system. After training stage, the system is able to estimate dense depth map, camera 6D pose, and optical flow by using a sequence of monocular images. No labelled data set is required for training. The supervision signals for training three deep neural networks of the system come from various forms of image warping. Due to the use of optical flow, the impact caused by occlusions and moving objects on the estimation results is alleviated. Experiments on the KITTI and Cityscapes datasets show that the proposed system demonstrates a better performance in terms of accuracy in depth, ego-motion, and optical flow estimation.