Constrained Optical Flow Estimation as a Matching Problem

This paper reports an optical based method for quantification of cardiac motion in MRI images. The cardiac motion is quantified in a Short-Axis (SAX) slice located at the mid-cavity of the left ventricle. In this slice, the left ventricle wall is segmented for extracting the endocardium, the epicardium and the mid‑wall. The velocity field for these contours is available after performing the optical flow estimation for the given slice. Post-processing of this motion field enables estimation of the radial displacement. The core of this cardiac motion quantification method is a sparse based algorithm for optical flow estimation. This algorithm is presented and compared with several classical optical flow estimation algorithms. The comparison is performed using several video sequences available from the Middlebury benchmark where the ground truth optical flow is known. The algorithms with better performance are also tested using a 4‑ D cardiac Magnetic Resonance Image (MRI) sequence. Results show that he sparse algorithm estimates an optical flow field that represents the motion of contraction during the systole interval. Moreover, results about radial displacement estimation on real MRI sequences for a normal subject and a patient with hypertrophy show that the proposed quantification method could be useful for quantification of left ventricle motion.

In this paper, we propose UnOS, an unified system for unsupervised optical flow and stereo depth estimation using convolutional neural network (CNN) by taking advantages of their inherent geometrical consistency based on the rigid-scene assumption. UnOS significantly outperforms other state-of-the-art (SOTA) unsupervised approaches that treated the two tasks independently. Specifically, given two consecutive stereo image pairs from a video, UnOS estimates per-pixel stereo depth images, camera ego-motion and optical flow with three parallel CNNs. Based on these quantities, UnOS computes rigid optical flow and compares it against the optical flow estimated from the FlowNet, yielding pixels satisfying the rigid-scene assumption. Then, we encourage geometrical consistency between the two estimated flows within rigid regions, from which we derive a rigid-aware direct visual odometry (RDVO) module. We also propose rigid and occlusion-aware flow-consistency losses for the learning of UnOS. We evaluated our results on the popular KITTI dataset over 4 related tasks, \ie stereo depth, optical flow, visual odometry and motion segmentation.

Optical flow estimation is a research core of computer vision. In recent years, optical flow estimation methods based on convolutional neural networks (CNNs) have achieved great success. However, most of the existing CNN-based methods are incapable of modeling long-distance dependencies successfully due to the limited receptive fields of convolutions. This may lead to poor performance of optical flow estimation in regions of large displacements and local ambiguities. Furthermore, the problem of edge-blurring remains an open challenge for CNN-based optical flow methods because the general interpolation operation tends to amplify the optical flow errors during the upsampling process. To deal with the abovementioned problems, this paper proposes a novel optical flow estimation method based on local-global modeling and visual similarity guidance. First, an efficient and simple self-attention module is adopted to improve the local-global modeling ability of the network and to extract the more representative features, which decreases the number of optical flow errors caused by large displacements. Second, based on the assumption that the visual features of the objects are more similar, the motion is more similar, and a visual similarity-guided optical flow upsampling network is constructed to guide the upsampling process. By transforming the visual similarity of features into motion similarity, the proposed upsampling scheme improves the accuracy of optical flow estimation in regions of motion boundaries. Finally, we run the proposed method on MPI-Sintel and KITTI test datasets to conduct a comprehensive comparison with some state-of-the-art methods. The experimental results indicate that the proposed method achieves top performance among the comparable methods and, in particular, gains significant developments in regions of large displacements and motion boundaries.

Motion and structure estimation are elementary problems of computer vision. These are active areas of research, even though the first methods were proposed several decades ago. We develop new approaches for motion and structure estimation for autonomous driving. An autonomous vehicle requires an accurate model of its environment, wrong decisions made by an autonomous car can have severe consequences. We assume the monocular setup, where only a single camera is mounted on the car. Outdoor traffic sequences are challenging for optical flow estimation. The high speed of the car causes large displacements in the optical flow field, the lighting conditions are unstable and there can be strong distortions due to reflections and difficult weather conditions. We propose new discrete methods, which determine optical flow as optimal configuration of probabilistic graphical models. The first approach selects sparse locations in the reference frame, and matches them across the second image. The best correspondences, which match constraints from a multiple view configuration, are considered motion vectors in a graphical model. In a second approach, we solve for dense optical flow by approximating the original infeasible graphical model with a sequence of reduced models. The monocular configuration poses challenges to the estimation of scene structure, camera positions and scene parameters need to be estimated jointly. The geometry of multiple views creates blind spots in the images, and adds a scale ambiguity, which both to not exist in a setup with multiple cameras. We propose two methods for structure estimation. The first approach determines the energy optimal camera track, given optical flow and depth observations. A further approach estimates camera motion and a piecewise planar scene description jointly from a single optical flow field. The scene description contains depth and plane normal information. We evaluate our approaches for motion and structure estimation on different real world and rendered datasets. In addition to evaluation on publicly available evaluation data, we evaluate on a new rendered dataset with ground truth plane normals.

<p indent=0mm>Combining the deep learning model to compute end-to-end optical flow is a hot topic in current computer vision field. The optical flow estimation methods based on deep learning are summarized and reviewed. Firstly, the origin and concept of optical flow is introduced. Secondly, the optical flow datasets and evaluation metrics are summarized. Most importantly, classical methods are introduced and the deep optical flow estimation methods are reviewed in three aspects, including supervised deep optical flow estimation methods, unsupervised deep optical flow estimation methods and the performance of these methods. The analysis shows designing compact and generalized deep optical flow model is the future research direction. On this basis, the joint learning of optical flow estimation and specific video analysis tasks are introduced. It is pointed out that the design of task-driven deep optical flow network is a valuable research direction in practice application. Finally, the problems and challenges of deep optical flow estimation are summarized and the future work of this field is prospected.

Recent CNN-based optical flow approaches have a separated structure of feature extraction and flow estimation. The core task of optical flow is finding the corresponding points while rich representation is just the key part of such matching problems. However, the prior work usually pays more attention to the design of flow decoder than the feature extraction. In this paper, we present a novel optical flow estimation network to enrich the feature representation of each pyramid level, with a hierarchical dilated architecture and a bottom-up aggregation scheme. In addition, inspired by edge guided classical methods, we bring the edge-aware idea into our approach and propose an edge-aware refinement (EAR) subnetwork to handle motion boundaries. Using the same decoding structure as PWC-Net, our network outperforms it by a large margin and leads all its derivatives both on KITTI-2012 and KITTI-2015. Further performance analysis proves the effectiveness of proposed ideas.

Optical Flow Estimation aims to find the 2D dense motion field between two frames. Due to the limitation of model structures and training datasets, existing methods often rely too much on local clues and ignore the integrity of objects, resulting in fragmented motion estimation. Through theoretical analysis, we find the pre-trained large vision models are helpful in optical flow estimation, and we notice that the recently famous Segment Anything Model (SAM) demonstrates a strong ability to segment complete objects, which is suitable for solving the fragmentation problem. We thus propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. To address the challenge of in-depth utilizing SAM in non-segmentation tasks like optical flow estimation, we propose an Optical Flow Task-Specific Adaption scheme, including a Context Fusion Module to fuse the SAM encoder with the optical flow context encoder, and a Context Adaption Module to adapt the SAM features for optical flow task with Learned Task-Specific Embedding. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%. Furthermore, our model achieves state-of-the-art performance on the Sintel and KITTI-15 benchmarks, ranking #1 among all two-frame methods on Sintel clean pass.

Unsupervised deep learning methods have demonstrated an impressive performance for understanding the structure of 3D scene from videos. These data-based learning methods are able to learn the tasks, such as depth, ego-motion, and optical flow estimation. In this paper, we propose a novel unsupervised deep learning method to jointly estimate scene depth, camera ego-motion, and optical flow from stereo images. Consecutive stereo images are used to train the system. After training stage, the system is able to estimate dense depth map, camera 6D pose, and optical flow by using a sequence of monocular images. No labelled data set is required for training. The supervision signals for training three deep neural networks of the system come from various forms of image warping. Due to the use of optical flow, the impact caused by occlusions and moving objects on the estimation results is alleviated. Experiments on the KITTI and Cityscapes datasets show that the proposed system demonstrates a better performance in terms of accuracy in depth, ego-motion, and optical flow estimation.

Multi-scale 3D scene flow from binocular stereo sequences

Self-supervised multi-body scene flow estimation

Gradient-based optical flow estimation methods such as LucasKanade method work well for scenes with small displacements but fail when objects move with large displacements. Hierarchical matching-based methods do not suffer from large displacements but are less accurate. By utilizing the high speed imaging capability of CMOS image sensors, the frame rate can be increased to obtain more accurate optical flow with wide range of scene velocities in real time. Further, by integrating the memory and processing with the sensor on the same chip, optical flow estimation using high frame rate sequences can be performed without unduly increasing the off-chip data rate. The paper describes a method for obtaining high accuracy optical flow at a standard frame rate using high frame rate sequences. The Lucas-Kanade method is used to obtain optical flow estimates at high frame rate, which are then accumulated and refined to obtain optical flow estimates at a standard frame rate. The method is tested on video sequences synthetically generated by perspective warping. Results demonstrate significant improvements in optical flow estimation accuracy with moderate memory and computational power requirements.

Optical flow estimation constitutes a fundamental computational challenge in computer vision, with critical applications object trajectory prediction, depth reconstruction, and autonomous navigation systems. The emergence of neuromorphic vision systems, integrating event-driven cameras with spiking neural networks (SNNs), has recently gained attention as a promising paradigm for edge deployment of optical flow estimation due to their advantages in ultralow power and resource efficiency. However, current neuromorphic computing platforms lack specialized architectures optimized for this problem domain. Existing implementations either prioritize configurable architectures at the expense of energy efficiency or employ intricate hardware control mechanisms to manage the asynchronous and sparse computing patterns inherent in SNNs. To address these limitations, we present ASNA-Flow, an event-driven asynchronous neuromorphic accelerator featuring a pioneering algorithm–hardware co-design framework specifically tailored for event-based optical flow estimation. Our methodology encompasses three key innovations: 1) a hardware-aware algorithm optimization that maintains computational fidelity while enhancing implementation efficiency; 2) systematic data pattern analysis to inform architectural decisions; and 3) novel exploitation of optical flow’s spatial locality characteristics to enable efficient sparse computing. Implemented in TSMC 28-nm CMOS technology, ASNA-Flow achieves real-time performance of 104 frames per second (FPS) with ultralow power consumption of 7.9 mW, demonstrating superior energy efficiency of 0.3 pJ per synaptic operation (SOP). This work establishes the first dedicated neuromorphic computing solution that simultaneously addresses the temporal sparsity, event-driven processing, and energy constraints inherent in optical flow estimation tasks.

A dense estimation of optical flow field within the MPEG-2 compressed domain is proposed, which utilizes only the compressed-domain information, i.e. motion vectors and DCT coefficients. First, motion vectors are pre-processed to estimate the DCT coefficients for P and B frames. Second, initial optical flow is estimated with Black's optical flow estimation framework, in which DC image is substituted by DC+2AC image to provide more intensity information. Third, high confidence test is exploited to generate the dense and accurate motion vector field by removing false and noisy motion vectors. It preserves the advantages of compressed domain processing and improves the existent MPEG velocity field in terms of accuracy and density. Experimental results demonstrate that the proposed approach can provide a satisfactory motion analysis for compressed-domain video object extraction.

Confidence measures are crucial to the interpretation of any optical flow measurement. Even though numerous methods for estimating optical flow have been proposed over the last three decades, a sound, universal, and statistically motivated confidence measure for optical flow measurements is still missing. We aim at filling this gap with this contribution, where such a confidence measure is derived, using statistical test theory and measurable statistics of flow fields from the regarded domain. The new confidence measure is computed from merely the results of the optical flow estimator and hence can be applied to any optical flow estimation method, covering the range from local parametric to global variational approaches. Experimental results using state-of-the-art optical flow estimators and various test sequences demonstrate the superiority of the proposed technique compared to existing ’confidence’ measures.KeywordsOptical FlowConfidence MeasureEndpoint ErrorIntrinsic DimensionalityOptical Flow EstimatorThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

A number of proposed optical flow estimation algorithms for image motion analysis use only a few subsequent frames from an image sequence to perform calculation. Others, that use many frames, as is case in frequency and energy based, as well as in stochastic optical flow algorithms, are better suited for off-line work. We present our approach in adopting a classic gradient optical flow algorithm for on-line work over extended image sequence. Our experiments on the usual optical flow testing image sequences, consisting of few dozens of image frames show significant improvements in accuracy of the estimated velocity flow fields and in speed of computation. This is achieved by modification and integration of three basic processing phases: preparation of a limited spatio-temporal volume, estimation of velocity flow fields and post filtering of the flow fields. In the preparation phase we use efficient spatio-temporal filters for image smoothing. In the estimation phase a modified iterative solution that propagates results from previous calculation steps is used. Linear and nonlinear post filtering additionally improve optical flow estimates. The whole process is less sensitive to the aperture problem and produces more accurate flow field estimates than the original algorithm. It better copes with motion discontinuities as well. We evidence our achievements by a number of experiments on extended image sequences.