A superresolution imaging method based on dense subpixel-accurate motion fields

An embedded coding scheme for dense motion (optical flow) fields is proposed. Such a scheme is particularly useful in scalable video compression where one must compensate for inter-frame motion at various visual qualities and resolutions. However, the high cost of coding such fields has often made this option prohibitive. Using our previously developed ‘breakpoint’-adaptive wavelet transform, we show that it is possible to code dense motion fields efficiently while simultaneously endowing the coded motion representation with embedded resolution and quality scalability attributes. Performance comparisons with the traditional non-scalable block-based model are also made and presented with the aid of a modified H.264/AVC JM reference encoder.

In this paper, a dense motion field estimation technique based on the Bayesian framework is proposed to estimate the true dense motion fields of video sequences. Previous stochastic techniques of dense motion field estimation adopts piecewise smooth motion model and use MAP estimation to find the motion field with joint minimization of motion compensation errors and maximization of motion smoothness. However, such random process does not guarantee to converge to the true motion field. In this paper, the motion of landmark points in the video sequence is introduced into the MAP estimation process to regularize the estimated motion field. Experimental results show that the proposed algorithm produces estimated motion fields which preserve piecewise smooth nature and are visually close to the true motion of the video sequences.

In this paper, we present a new algorithm of video coding for very low bit-rate applications. The algorithm is based on: (1) dense motion field, which can achieve better motion compensation than sparse (say, block-wise) motion field; (2) DCT applied to the dense motion field to drastically save overhead information; (3) region-based segmentation with morphological techniques which can segment video frames into different regions according to their content significance; (4) discrete wavelet transform (DWT) applied to residual data with adaptive bit allocation. Consequently, this algorithm avoids annoying block artifacts, thus making reconstructed video frames much more visually pleasant, while maintains similar bit-rate to H.263.

In two-view correspondence learning, prevalent multi-layer perceptron (MLP)-based methods struggle with context capturing. To remedy this issue, recent advances innovatively stack convolutional neural network (CNN)-based Resblocks sequentially, showing an inherent proficiency in local context extraction. Yet, such non-issue-specific designs inherit the drawback of CNN’s difficulty in aggregating global context, leading to performance bottlenecks. To address this problem, this prospective study further explores the potential of the CNN-based framework and proposes MC-Net, a top-performing network that integrates both local and global context elegantly and seamlessly. Specifically, considering that sparse motion vectors and a dense motion field can be converted into each other through interpolation and sampling, we first transform unordered matches into image-structured data by estimating the dense motion field implicitly. Then, we design a hierarchical rectifying module to rectify the error of each ordered motion vector with CNN at multiple levels, enabling MC-Net to perceive global context from coarse-level features and local context from fine-level features simultaneously, which facilitates to tackle the discontinuities of the motion field in case of large scene disparity. Finally, we reconstruct comprehensive context-embedded features from rectified motion fields at all levels. Also, instead of using the residuals between rectified and pre-rectified motion vectors at the same layer to reject outliers as in previous studies, which seriously affects the inlier prediction accuracy, we rethink this operation meticulously and modify it to the difference between motion vectors obtained from each layer’s reconstruction and ones from the first layer before transformation, ensuring purer residuals and enhancing the matching performance without extra computational burden. Extensive experiments show that MC-Net outperforms state-of-the-arts on multiple domains and datasets.

In this paper a new iterative motion-based segmentation algorithm is presented for object-based video coding. At first, dense motion fields between image pairs are obtained by a multiresolution motion estimation algorithm. Based on these motion fields a set of global motion vectors are determined by an iterative vector quantisation procedure. Using the global motion vectors, the images are segmented into homogenous regions. Covered and uncovered areas of the background and objects are properly recognized by examining two consecutive motion fields. Experimental results for artificial as well for real-life video have shown that the proposed algorithm results in good quality segmentation label fields while it requires only moderate computations.

A method to obtain motion artifact-free reconstructed images of the coronary arteries is proposed and evaluated. The method relies on the integration of coronary motion estimation in an iterative computed tomography reconstruction technique. Coronary motion fields are derived from a set of coronary centerlines extracted at multiple cardiac phases within the R-R interval. Start and end points are provided by the user in one time-frame only. Corresponding centerline positions are used to determine the motion fields from phase to phase. Finally, dense motion fields are achieved by thin-plate-spline interpolation and are used to perform a motion-corrected iterative reconstruction of a selected region of interest, which results in an effective improvement of the reconstructed image quality.

Numerical phantoms are a common tool for the evaluation of registration and reconstruction algorithms. For applications concerning motion, dense deformation fields are of particular interest. Phantoms, however, are often described as surfaces and thus motion vectors can only be generated at these surfaces. In order to create dense motion fields, interpolation is required. A frequently used method for this purpose is the Parzen interpolator. However, with a high number of surface motion vectors and a high voxel count, its run time increases dramatically. In this paper, we investigate different methods to accelerate the creation of these motion fields using hierarchical sampling and the random ball cover. In the results, we show that a 64 3 volume can be sampled in less than one second with an error below 0.1 mm. Furthermore, we accelerate the interpolation of a 256 3 dense deformation field to only ˜. 5m inutes using the proposed methods from days with previous methods.

We present a novel hierarchical MRFs optimization method for dense and deformable motion extraction in dynamic scenes. In particular, this hierarchical MRFs structure consists of two layers, the segmentation and the correspondence layer. Firstly, dynamic RGB-D foreground data is segmented through a pixel-level MRF in the segmentation layer. Subsequently, the extracted foreground data is transformed into a 3D point-level MRF in the correspondence layer. A new surface descriptor named deformable color and shape histogram is proposed. It is combined with photometric and geometric features to represent a deformable surface. Finally, the dynamic scene motion is retrieved from correspondences established in the image sequence. Discrete optimization schemes are used for the binary classification and multi-labeling problems. We provide an RGB-D dataset of dynamic scenes, which involves different motion patterns and surface properties of foreground objects. The effectiveness and efficiency of our proposed approach for high accurate foreground segmentation and motion extraction is validated in experiments.

Designing a 3D representation of a dynamic scene for fast optimization and rendering is a challenging task. While recent explicit representations enable fast learning and rendering of dynamic radiance fields, they require a dense set of input viewpoints. In this work, we focus on learning a fast representation for dynamic radiance fields with sparse input viewpoints. However, the optimization with sparse input is under-constrained and necessitates the use of motion priors to constrain the learning. Existing fast dynamic scene models do not explicitly model the motion, making them difficult to be constrained with motion priors. We design an explicit motion model as a factorized 4D representation that is fast and can exploit the spatio-temporal correlation of the motion field. We then introduce reliable flow priors including a combination of sparse flow priors across cameras and dense flow priors within cameras to regularize our motion model. Our model is fast, compact and achieves very good performance on popular multi-view dynamic scene datasets with sparse input viewpoints. The source code for our model can be found on our project page: https://nagabhushansn95.github.io/publications/2024/RF-DeRF.html.

Analyzing fluid motion is essential in number of domains and can rarely be handled using generic computer vision techniques. In this particular application context, we address two distinct problems. First we describe a dedicated dense motion estimator. The approach relies on constraints issuing from fluid motion properties and allows us to recover dense motion fields of good quality. Secondly, we address the problem of analyzing such velocity fields. We present a kind of motion-based segmentation relying on an analytic representation of the motion field that permits to extract important quantities such as singularities, stream-functions or velocity potentials. The proposed method has the advantage to be robust, simple, and fast.KeywordsSingular PointMotion EstimationVelocity PotentialBhattacharyya DistanceDense MotionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The quantitative estimation of regional cardiac deformation from 3D image sequences has important clinical implications for the assessment of viability in the heart wall. Such estimates have so far been obtained almost exclusively from Magnetic Resonance (MR) images, specifically MR tagging. In this paper we describe a methodology for estimating cardiac deformations from 3D ultrasound images. The images are segmented interactively and then initial correspondence is established using a shape-tracking approach. A dense motion field is then estimated using an anisotropic linear elastic model, which accounts for the fiber directions in the left-ventricle. The dense motion field is in turn used to calculate the deformation of the heart wall in terms of strain in cardiac specific directions. The strains obtained using this approach in open-chest dogs before and after coronary occlusion related to changes in blood flow, show good agreement with previously published results in the literature. This proposed method provides quantitative regional 3D estimates of heart deformation from ultrasound images.KeywordsUltrasound ImageRisk AreaCircumferential StrainStrain Energy FunctionHeart WallThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The quantitative estimation of regional cardiac deformation from three-dimensional (3-D) image sequences has important clinical implications for the assessment of viability in the heart wall. We present here a generic methodology for estimating soft tissue deformation which integrates image-derived information with biomechanical models, and apply it to the problem of cardiac deformation estimation. The method is image modality independent. The images are segmented interactively and then initial correspondence is established using a shape-tracking approach. A dense motion field is then estimated using a transversely isotropic, linear-elastic model, which accounts for the muscle fiber directions in the left ventricle. The dense motion field is in turn used to calculate the deformation of the heart wall in terms of strain in cardiac specific directions. The strains obtained using this approach in open-chest dogs before and after coronary occlusion, exhibit a high correlation with strains produced in the same animals using implanted markers. Further, they show good agreement with previously published results in the literature. This proposed method provides quantitative regional 3-D estimates of heart deformation.

Unsupervised methods have showed promising results on monocular depth estimation. However, the training data must be captured in scenes without moving objects. To push the envelope of accuracy, recent methods tend to increase their model parameters. In this paper, an unsupervised learning framework is proposed to jointly predict monocular depth and complete 3D motion including the motions of moving objects and camera. (1) Recurrent modulation units are used to adaptively and iteratively fuse encoder and decoder features. This improves the single-image depth inference without overspending model parameters. (2) Instead of using a single set of filters for upsampling, multiple sets of filters are devised for the residual upsampling. This facilitates the learning of edge-preserving filters and leads to the improved performance. (3) A warping-based network is used to estimate a motion field of moving objects without using semantic priors. This breaks down the requirement of scene rigidity and allows to use general videos for the unsupervised learning. The motion field is further regularized by an outlier-aware training loss. Despite the depth model just uses a single image in test time and 2.97M parameters, it achieves state-of-the-art results on the KITTI and Cityscapes benchmarks.

Non-invasive estimation of regional cardiac function is important for assessment of myocardial contractility. The use of MR tagging technique enables acquisition of intra-myocardial tissue motion by placing a spatially modulated pattern of magnetization whose deformation with the myocardium over the cardiac cycle can be imaged. Quantitative computation of parameters such as wall thickening, shearing, rotation, torsion and strain within the myocardium is traditionally achieved by processing the tag-marked MR image frames to 1) segment the tag lines and 2) detect the correspondence between points across the time-indexed frames. In this paper, we describe our approach to solving this problem using the Large Deformation Diffeomorphic Metric Mapping (LDDMM) algorithm in which tag-line segmentation and motion reconstruction occur simultaneously. Our method differs from earlier proposed non rigid registration based cardiac motion estimation methods in that our matching cost incorporates image intensity overlap via the L 2 norm and the estimated tranformations are diffeomorphic. We also present a novel method of generating synthetic tag line images with known ground truth and motion characteristics that closely follow those in the original data; these can be used for validation of motion estimation algorithms. Initial validation shows that our method is able to accurately segment tag-lines and estimate a dense 3D motion field describing the motion of the myocardium in both the left and the right ventricle.

Facial expression recognition (FER) for emotion inference has become one of the most important research fields in human-computer interaction. Existing study on FER mainly focuses on visible images, whereas varying lighting conditions may influence their performances. Recent studies have demonstrated the advantages of infrared thermal images reflecting the temperature distributions, which are robust to lighting changes. In this paper, a novel infrared image sequence based FER method is proposed using spatiotemporal feature analysis and deep Boltzmann machines (DBM). Firstly, a dense motion field among infrared image sequences is generated using optical flow algorithm. Then, PCA is applied for dimension reduction and a three-layer DBM structure is designed for final expression classification. Finally, the effectiveness of the proposed method is well demonstrated based on several experiments conducted on NVIE database.