Hybrid Object-Based Video Compression Scheme Using a Novel Content-Based Automatic Segmentation Algorithm

In two-view correspondence learning, prevalent multi-layer perceptron (MLP)-based methods struggle with context capturing. To remedy this issue, recent advances innovatively stack convolutional neural network (CNN)-based Resblocks sequentially, showing an inherent proficiency in local context extraction. Yet, such non-issue-specific designs inherit the drawback of CNN’s difficulty in aggregating global context, leading to performance bottlenecks. To address this problem, this prospective study further explores the potential of the CNN-based framework and proposes MC-Net, a top-performing network that integrates both local and global context elegantly and seamlessly. Specifically, considering that sparse motion vectors and a dense motion field can be converted into each other through interpolation and sampling, we first transform unordered matches into image-structured data by estimating the dense motion field implicitly. Then, we design a hierarchical rectifying module to rectify the error of each ordered motion vector with CNN at multiple levels, enabling MC-Net to perceive global context from coarse-level features and local context from fine-level features simultaneously, which facilitates to tackle the discontinuities of the motion field in case of large scene disparity. Finally, we reconstruct comprehensive context-embedded features from rectified motion fields at all levels. Also, instead of using the residuals between rectified and pre-rectified motion vectors at the same layer to reject outliers as in previous studies, which seriously affects the inlier prediction accuracy, we rethink this operation meticulously and modify it to the difference between motion vectors obtained from each layer’s reconstruction and ones from the first layer before transformation, ensuring purer residuals and enhancing the matching performance without extra computational burden. Extensive experiments show that MC-Net outperforms state-of-the-arts on multiple domains and datasets.

Micro-expression is an expression when a person tries to held or hidden, but the leak of this emotion still occurs in one or two areas of the face or maybe a short expression that across in the whole-face. Not more than 500ms, micro-expressions can be difficult to recognize and detect where the leakage area is located. This study presents a new method to recognize and detect the subtle motion on the facial components area using Phase Only Correlation algorithm with All Block Search (POC-ABS) to estimate the motion of all block areas. This block matching method is proposed by comparing each block in the two frames to determine whether there is movement or not. If the two blocks are identical, then the motion vector value is not displayed, whereas if the blocks are non-identical, the motion vector value of the POC is displayed. The motion vector, which is as a motion feature, estimates whether or not there are movements in the same block. In order to further confirm the reliability of the proposed method, two different classifiers were used for the micro-expression recognition of the CASME II dataset. The highest performance results are for SVM at 94.3 percent and for KNN at 95.6 percent. Finally, this algorithm detects leaks of motion based on the ratio of the motion vectors. The left and right eyebrows are dominant when expressing disgust, sadness, and surprise. Meanwhile, the movements of the right eye and left eye were the most dominant when the happiness expression.

Numerical phantoms are a common tool for the evaluation of registration and reconstruction algorithms. For applications concerning motion, dense deformation fields are of particular interest. Phantoms, however, are often described as surfaces and thus motion vectors can only be generated at these surfaces. In order to create dense motion fields, interpolation is required. A frequently used method for this purpose is the Parzen interpolator. However, with a high number of surface motion vectors and a high voxel count, its run time increases dramatically. In this paper, we investigate different methods to accelerate the creation of these motion fields using hierarchical sampling and the random ball cover. In the results, we show that a 64 3 volume can be sampled in less than one second with an error below 0.1 mm. Furthermore, we accelerate the interpolation of a 256 3 dense deformation field to only ˜. 5m inutes using the proposed methods from days with previous methods.

The performance of superresolution video enhancement relies heavily on the robustness and accuracy of motion estimation techniques. In this paper, we propose a novel and efficient block matching motion estimation algorithm suitable for estimating general motion existing in low resolution video frames. We exploit the spatial correlations between motion vectors and apply a coarse-to-fine multi-stage scheme to get the dense motion fields. We incorporate our motion estimation technique into the Projection Onto Convex Sets (POCS) superresolution framework. Experimental results show that the resulting high resolution images yield visual sharper enhanced images with significant PSNR improvement.

In this paper, a new low-complexity true-motion estimation (TME) algorithm is proposed for video processing applications, such as motion-compensated temporal frame interpolation (MCTFI) or motion-compensated frame rate up-conversion (MCFRUC). Regular motion estimation, which is often used in video coding, aims to find the motion vectors (MVs) to reduce the temporal redundancy, whereas TME aims to track the projected object motion as closely as possible. TME is obtained by imposing implicit and/or explicit smoothness constraints on the block-matching algorithm. To produce better quality-interpolated frames, the dense motion field at interpolation time is obtained for both forward and backward MVs; then, bidirectional motion compensation using forward and backward MVs is applied by mixing both elegantly. Finally, the performance of the proposed algorithm for MCTFI is demonstrated against recently proposed methods and smoothness constraint optical flow employed by a professional video production suite. Experimental results show that the quality of the interpolated frames using the proposed method is better when compared with the MCFRUC techniques.

Video segmentation for object based video coding according to MPEG-4 should be able to segment interested objects in video sequence clearly. This paper presents the object segmentation algorithm which image features are combined to use in segmentation process following to characteristic of video signal. Because the combination of many features in video sequence is a method that can achieve high quality object segmentation. In addition, this algorithm is an adaptive method that many parameters can be adjusted in order to give clearly segmentation. The significant features are used in segmentation process including color, motion vector and change information. A fast shortest spanning tree (SST) algorithm is adapted to use for fast segmenting image boundary. Motion vectors are estimated and searched by thresholding hierarchical block matching (HBM) which want quite low computation and give a few groups of motion vectors. The change information is used to detect moving objects that can separate between moving object and static background. After that, each feature will be considered in segmentation decision process to segment interested objects. Then the post processing refines the final segmentation. The results from many test sequences have good quality and show object boundary clearly.

In this paper we present a novel blind video forgery detection method by applying Markov models to motion in videos. Motion is an important aspect of video forgery detection as it effects forgery detection in videos. Most of the current video forgery detection algorithms do not consider motion in their approach. Motion is usually captured from motion vectors and prediction error frame. However capturing motion for I-frame is computationally expensive, so in this paper we extract the motion information by applying collusion on successive frames. First a base frame is obtained by applying collusion on successive frames and the difference between actual and estimate gives information about motion. Then we apply Markov models on this motion residue and apply pattern recognition on this. We used Support Vector Machines (SVMs) in our experiment. We obtained an accuracy of 87% even for reduced feature set.

A motion vector estimation (ME) method for VLSI implementation, according to MPEG (ISO/DIS-11172) is presented. The algorithm is based on a block-matching method, and a two-step tree search is applied. Moreover, a simple 2-D finite impulse response (FIR) filter is used just before subsampling, in order to avoid nonapplicable vectors. The filter is realized by additions only, so implementation is very easy. The computation amount decreases to one-fortieth compared with full-search ME through using this algorithm. The efficiency of 2-D FIR filter is evaluated by simulation. This VLSI can estimate motion vectors for MPEG, CCITT H.261 and CCIR Rec.723 in real time. >

Measuring river surface velocity is crucial for hydrological monitoring. However, traditional methods exhibit significant limitations under extreme weather conditions and emergency scenarios. In this study, we propose OF-PhyNet, an innovative hybrid neural network architecture designed as an image-based velocimetry method to improve the accuracy of river surface velocity estimation. By integrating an optical flow algorithm with a physics-informed neural network, OF-PhyNet achieves deep coupling of motion feature extraction with fluid dynamics constraints. To address the challenges posed by uneven particle distribution and illumination variations in natural rivers, we introduce a feature extraction strategy based on K-means clustering and morphological operations, which enables effective flow field segmentation and reliable identification of particle regions. Experimental validation conducted in the Heihe River Basin, using Stalker Pro II surface velocity radar (SVR) as a reference, demonstrates that OF-PhyNet achieves a mean absolute error of 0.054 m/s and a relative error of 7.22% across 12 cross-sectional measurement points, substantially outperforming PIVLab with 11.65% and the Horn-Schunck algorithm with 25.84% relative error. As a UAV-based image velocimetry approach, OF-PhyNet offers exceptional mobility for rapid flood monitoring and enables robust reconstruction of flow fields in particle-sparse regions, delivering precise non-contact river velocity measurements beyond the reach of traditional methods.

The majority of hyperspectral target detection algorithms are developed from statistical data models employing stationary background statistics or white Gaussian noise models. Stationary background models are inaccurate as a result of two separate physical processes. First, varying background classes often exist in the imagery that possess different clutter statistics. Many algorithms can account for this variability through the use of subspaces or clustering techniques. The second physical process, which is often ignored, is a signal-dependent sensor noise term. For photon counting sensors that are often used in hyperspectral imaging systems, sensor noise increases as the measured signal level increases as a result of Poisson random processes. This work investigates the impact of this sensor noise on target detection performance. A linear noise model is developed describing sensor noise variance as a linear function of signal level. The linear noise model is then incorporated for detection of targets using data collected at Wright Patterson Air Force Base.

In this paper, we propose a fast block motion estimation scheme by exploiting the correlations of motion vectors (MV) existing in spatially- and temporally-adjacent as well as hierarchically-related blocks. The basic idea is to use the information obtained from the corresponding block at a coarser resolution level and spatio-temporal neighboring blocks at the same level in order to select a good set of initial MV candidates and then perform further local search to find the best matching MV. In order to further reduce computational complexity, the sign pattern vector (SPV) is defined and used for block matching, as opposed to the pixel intensity values used in the conventional block matching methods. By using the SPV definition, a data block can be presented by a mean and a set of binary sign patterns with a much reduced data set. The block matching motion estimation is then divided into mean matching and binary phase matching. The proposed technique enables a significant reduction in computational complexity compared with the conventional block matching motion estimation (ME) because binary phase matching only involves Boolean logic operations. This scheme also significantly reduces the data transfer time between the frame buffer and motion estimator. Our test results indicate that the performance of the proposed scheme is comparable with the full-search block matching under the same search ranges, however, the proposed scheme has a speed-up factor ranging from 250 to 370 in comparison with full search.

In this paper a new iterative motion-based segmentation algorithm is presented for object-based video coding. At first, dense motion fields between image pairs are obtained by a multiresolution motion estimation algorithm. Based on these motion fields a set of global motion vectors are determined by an iterative vector quantisation procedure. Using the global motion vectors, the images are segmented into homogenous regions. Covered and uncovered areas of the background and objects are properly recognized by examining two consecutive motion fields. Experimental results for artificial as well for real-life video have shown that the proposed algorithm results in good quality segmentation label fields while it requires only moderate computations.

Expression transfer for facial sketch animation

In this paper, we propose an algorithm that can efficiently detect and monitor multiple ships in real-time. The proposed algorithm uses morphological operations and edge information for detecting and tracking ships. We used smoothing filter with a <TEX>$3{\times}3$</TEX> Gaussian window and luminance component instead of RGB components in the captured image. Additionally, we applied Sobel operator for edge detection and a threshold for binary images. Finally, object labeling with connectivity and morphological operation with open and erosion were used for ship detection. Compared with conventional methods, the proposed method is meant to be used mainly in coastal surveillance systems and monitoring systems of harbors. A system based on this method was tested for both stationary and non-stationary backgrounds, and the results of the detection and tracking rates were more than 97% on average. Thousands of image frames and 20 different video sequences in both online and offline modes were tested, and an overall detection rate of 97.6% was achieved.

In this paper, we present a novel approach for pose estimation and tracking of license plates from monocular camera view. Given an initial estimate, we try to track the location, motion vector and pose of the object in 3D in the successive video frames. We utilize Condensation algorithm for estimating the state of the object and filtering the measurements, according to the extracted image features. We utilize directional gradients as the image features. Each sample of the Condensation algorithm is projected to the image plane by perspective camera model. The overlapping of the image gradients and the sample boundaries, gives a likelihood for each sample of the Condensation algorithm. Our contribution is utilizing condensation algorithm for rigid object tracking, where the object is tracked in 3D. We demonstrate the performance of the approach by tracking license plates in outdoor environment with different motion trajectories.KeywordsMotion VectorImage GradientLicense PlateExtract Image FeatureCondensation AlgorithmThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.