Compressed domain motion vector resampling for downscaling of MPEG video

Digital video databases in compressed form are becoming widely available. In applications such as video browsing, and picture in picture, for a lower bitrate there is a need to down-sample the video before transmission. The conventional approach to downscale a compressed video sequence is to decompress it at the video server, perform the down-sampling in the pixel domain and recompress it again for efficient delivery. This process is computationally expensive due to the motion estimation process needed during the recompression phase. In the alternative compressed domain based approach the downscaled motion vectors are computed directly from the original ones. We propose a method to improve the process of downscaling in the compressed domain by better selecting the motion prediction type of P and B frame macroblocks. Simulations suggest that the proposed scheme improves the compressed domain down-sampling quality by up to 2.2 dB without any substantial increase in computation.

Motion vector outlier removal using dissimilarity measure

A critical evaluation of image and video indexing techniques in the compressed domain

This paper presents a novel approach for moving object extraction in the H.264/AVC compressed domain, which based on Ant Colony clustering Algorithm (ACA) and threshold method in macro block layer. Firstly, the Motion Vector (MV) field and the macro block types are extracted from the H.264/AVC compressed video, and then merge MVs with the same characteristic. Secondly, an improved ACA is used to classify the MV field into different motion homogenous regions. At the same time, use macro block types to determine the location of objects. Finally, using the complementarities of macro block template and MVs clustering template to obtain final objects. Experimental results for several video sequences demonstrate that in the case of ensuring accuracy, the proposed approach can extract moving object faster.

Accurate and efficient video quality assessment (VQA) methods are important guidelines for optimizing network video quality, improving video compression performance, and recommending compression coding parameters. No-reference VQA methods are mainly divided into bitstream-based methods, which are less time-consuming but less accurate, and pixel-based methods, which are more accurate but more time-consuming and require high hardware requirements. Both methods are not applicable to the quality assessment of a large number of network videos. To address the above problem, we propose an efficient and accurate cross-domain no-reference network video quality assessment method (CDNRVQA) based on bitstream. CDNRVQA takes I-frames as representative frames and uses deep neural network to obtain video high semantic features and distortion features from the representative frames, then obtains temporal features and some spatial features of the video based on the macroblock and motion vector in the compressed videos, and uses feature fusion strategy to fuse features from pixel domain and compressed domain, and finally obtains comprehensive cross-domain features which can represent the video distortion brought by content, motion and compression. CDNRVQA performs well on large VQA datasets. CDNRVQA achieves similar performance to the state-of-the-art VQA models, but greatly improves prediction time, and greatly reduces hardware consumption on the device. CDNRVQA makes it possible to apply VQA methods to video platforms.

The rapid growth of digital libraries, e-governance, and internet based applications has caused an exponential escalation in the volume of ‘Big-data’ particularly due to texts, images, audios and videos that are being both archived and transmitted on a daily basis. In order to make their storage and transfer efficient, different data compression techniques are used in the literature. The ultimate motive behind data compression is to transform a big size data into small size data, which eventually implies less space while archiving, and less time in transferring. However, in order to operate/analyze compressed data, it is usually necessary to decompress it, so as to bring back the data to its original form, which unfortunately warrants an additional computing cost. In this backdrop, if operating upon the compressed data itself can be made possible without going through the stage of decompression, then the advantage that could be accomplished due to compression would escalate. Further due to compression, from the data structure and storage perspectives, the original visibility structure of the data also being lost, it turns into a potential challenge to trace the original information in the compressed representation. This challenge is the motivation behind exploring the idea of direct processing on the compressed data itself in the literature. The proposed survey paper specifically focuses on compressed document images and brings out two original contributions. The first contribution is that it presents a critical study on different image analysis and image compression techniques, and highlights the motivational reasons for pursuing document image analysis in the compressed domain. The second contribution is that it summarizes the different compressed domain techniques in the literature so far based on the type of compression and operations performed by them. Overall, the paper aims to provide a perspective for pursuing further research in the area of document image analysis and pattern recognition directly based on the compressed data.

Visual database systems require efficient indexing to enable fast access to the images and video sequences in the database. Since image and video data are voluminous, compression is essential for storage and transmission. The International Standards Organization has proposed the JPEG and MPEG standards for image and video compression, respectively. In order to avoid the unnecessary decompression and recompression operations in the searching process, it is efficient to index the image/video in compressed form. In this paper, we present image and video indexing techniques in the compressed domain.

This paper investigates a novel motion vector outlier rejection method based on using mean shift clustering on block motion vectors. The accuracy of compressed domain global motion estimation techniques is largely influenced by its ability to counter the outlier motion vectors. These outliers occur in the block motion vector field due to moving objects, noise or due to large matching errors as a result of the encoders priority on rate distortion optimization. In the present work it is shown that by using mean shift clustering on block motion vectors, those clusters which correspond to outlier motion vectors can be identified. Once detected these clusters are kept out of the global motion estimation process thereby increasing the robustness of estimated camera parameters. The proposed method is compared with existing state-of-the-art outlier removal methods using synthetic and real video sequences to establish and validate its superiority.

Huge number of images is available on the internet. Efficient and effective retrieval system is needed to retrieve these images by the contents or features of the images like color, texture and shape. This system is called content based image retrieval (CBIR). Conventionally features are extracted from images in pixel domain. But at present almost all images are represented in compressed form using DCT (Discrete Cosine Transformation) blocks transformation. Some critical information is removed in compression and only perceptual information is left which has significant attraction for information retrieval in compressed domain. In this paper we study the problem that how to retrieve perceptual information in compressed domain JPEG such that to improve image retrieval. Our approach is based on quantized histogram statistical texture features in DCT blocks. We show that to get best image retrieval performance by extracting the statistical texture features of quantized histogram in DCT blocks using JPEG compressed format images. Experiments on the Corel animal database using the proposed approach, give results which show that the statistical texture features of histogram are robust in retrieval of images. This shows that texture features in local compression is a significant step for effective image retrieval.

With low computation cost, motion vectors can be readily extracted from MPEG video streams and processed to estimate vehicle motion speed. A statistical model is proposed to model vehicle speed and noise. In order to achieve high estimation accuracy and also study the limitations of the proposed algorithm, we quantitatively evaluated four parameters used in our algorithm: temporal filter window size T, video resolution R v (CIF/QCIF), motion vector frame distance m, and video bit-rates. Our experiments showed that the mean vehicle speed can be estimated with high accuracy, up to 85 to 92% by proper spatial and temporal processing. The proposed algorithm is especially suitable for Skycam-based application, where the traditional tracking-based or virtual-loop-based approaches perform poorly because of their requirements of high-resolution images. Although extensive work has been done in extracting motion information directly from MPEG video data in compressed domain, to our best knowledge, this paper is the very first work in which stationary motion (speed) of moving objects can be estimated with high accuracy directly from MPEG motion vectors. Furthermore the proposed method is not limited to vehicle speed estimation by nature and it can be applied to other applications where the stationary motion assumption is satisfied.

The problem of generating textual descriptions for the visual data has gained research attention in the recent years. In contrast to that the problem of generating visual data from textual descriptions is still very challenging, because it requires the combination of both Natural Language Processing (NLP) and Computer Vision techniques. The existing methods utilize the Generative Adversarial Networks (GANs) and generate the uncompressed images from textual description. However, in practice, most of the visual data are processed and transmitted in the compressed representation. Hence, the proposed work attempts to generate the visual data directly in the compressed representation form using Deep Convolutional GANs (DCGANs) to achieve the storage and computational efficiency. We propose GAN models for compressed image generation from text. The first model is directly trained with JPEG compressed DCT images (compressed domain) to generate the compressed images from text descriptions. The second model is trained with RGB images (pixel domain) to generate JPEG compressed DCT representation from text descriptions. The proposed models are tested on an open source benchmark dataset Oxford-102 Flower images using both RGB and JPEG compressed versions, and accomplished the state-of-the-art performance in the JPEG compressed domain. The code will be publicly released at GitHub after acceptance of paper.

Multimedia products today broadcast over networks and are typically compressed and transmitted from host to client. Adding watermarks to the compressed domain ensures content integrity, protects copyright, and can be detected without quality degradation. Hence, watermarking video data in the compressed domain is important. This work develops a novel video watermarking system with the aid of computational intelligence, in which motion vectors define watermark locations. The number of watermark bits varies dynamically among frames. The current study employs several intelligent computing methods including K-means clustering, Fuzzy C-means clustering, Swarm intelligent clustering and Swarm intelligence based Fuzzy C-means (SI-FCM) clustering to determine the motion vectors and watermark positions. This study also discusses and compares the advantages and disadvantages among various approaches. The proposed scheme has three merits. First, the proposed watermarking strategy does not involve manually setting watermark bit locations. Second, the number of embedded motion vector clusters differs according to the motion characteristics of each frame. Third, the proposed special exclusive-OR operation closely relates the watermark bit to the video context, preventing attackers from discovering the real watermark length of each frame. Therefore, the proposed approach is highly secure. The proposed watermark-extracting scheme immediately detects forgery through changes in motion vectors. Experimental results reveal that the watermarked video retains satisfactory quality with very low degradation.

A dense estimation of optical flow field within the MPEG-2 compressed domain is proposed, which utilizes only the compressed-domain information, i.e. motion vectors and DCT coefficients. First, motion vectors are pre-processed to estimate the DCT coefficients for P and B frames. Second, initial optical flow is estimated with Black's optical flow estimation framework, in which DC image is substituted by DC+2AC image to provide more intensity information. Third, high confidence test is exploited to generate the dense and accurate motion vector field by removing false and noisy motion vectors. It preserves the advantages of compressed domain processing and improves the existent MPEG velocity field in terms of accuracy and density. Experimental results demonstrate that the proposed approach can provide a satisfactory motion analysis for compressed-domain video object extraction.

Block-matching algorithm is the most common technique applied in block-based motion estimation technique. There are several block-matching algorithm based on block-based motion estimation techniques have been developed. Full search (FS), three step search (TSS), new three step search (NTSS), diamond search (DS) and hexagon based search (HS) are the most well known block-matching algorithm. These techniques are applied to video sequences to remove the temporal redundancy for compression purposes and to gauge the motion vector estimation. In addition, the mentioned block-matching algorithms are the baseline techniques that have been used to further develop all the enhanced or improved algorithms. In order to develop the proposed methods, the baseline techniques are studied to develop the proposed algorithms. This chapter proposes modelling of fast unrestricted hexagon search (FUHS16) and unrestricted hexagon-diamond search (UHDS16) algorithms for motion vector estimation, which is based on the theory and application of block-based motion estimation. Both of these algorithms are designed using 16 × 16 block size. In particular, the motion vector estimation, quality performance, computational complexity, and elapsed processing time are emphasised. These parameters have been used to measure the experimental results. It is the aim of this study that this work provides a common framework with which to evaluate and understand block-based matching motion estimation performance. On the theoretical side, four fundamental issues are explored: (1) division of frame, (2) basic block-based matching, (3) motion vector estimation, and (4) block-matching algorithm development. Various existing block-matching motion estimation algorithms have been analysed to develop the fundamental research. Based on the theoretical and fundamental research analysis the FUHS16 and UHDS16 algorithms using 16 × 16 block-based motion estimation formulations were developed. To improve the UHDS16 algorithm, 8 × 8 block-matching technique has been tested. The 8 × 8 block-matching technique is known as UHDS8. The results show positive improvements. From an application perspective, the UHDS8 algorithm efficiently captured the motion vectors in many video sequences. For example, in video compression, the use of motion vectors on individual macro-blocks optimized the motion vector information. The UHDS8 algorithm also offers improvement in terms of image quality performance, computational complexity and elapsed processing time. Thus, this chapter offers contributions in certain areas such as reducing the mechanism of computational complexity in estimating the motion from the video sequences. In particular, the FUHS16, UHDS16 and UHDS8 algorithms were developed to estimate the motion vectors field in the video sequences. Theoretical analysis block-based matching criteria are adapted to FUHS16, UHDS16 and UHDS8 algorithms, which are based on search points technique. Basically, the proposed of FUHS16, UHDS16 and UHDS8 algorithm produces the best motion vector estimation finding based on the block-based matching criteria. Besides that, the UHDS8 algorithm also improves the image quality performances and the search points in terms of the computational complexity. Overall, the study shows that the UHDS8 algorithm produces better results compared to the FUHS16 and UHDS16 algorithm.

Confident Recovery for Lost Motion Vectors in Video Transmission