No-reference perceptual quality assessment of streamed videos using optical flow features

A new block-based multi-metric fusion (BMMF) approach is proposed for perceptual image quality assessment. The proposed BMMF scheme automatically detects image content and distortion types in a block via machine learning, which is motivated by the observation that the performance of an image quality metric is highly influenced by these factors. Locally, image block content is classified into three types; namely, smooth, edge and texture. Image distortion is detected and grouped into five types. An appropriate image quality metric is adopted for each block by considering its content and distortion types, and then all block-based quality metrics are fused to result in one final score. Furthermore, a corrected version of BMMF is derived for a specific group of distortions based on image complexity analysis. The proposed BMMF scheme is tested on TID database with its Spearman Correlation equal to 0.9471, which outperforms today's state-of-the-art image quality metrics.

In this paper, we investigated the impact of spatial resolution and quantization on the perceptual quality of a compressed video coded from the SVC reference software - JSVM. Two subjective quality tests have been carried out on the TI Zoom2 mobile development platform (MDP). 5 different spatial resolutions (i.e., from QCIF to 4CIF) and 4 different quantization levels (i.e., QP 22, 28, 36, 44) are chosen to create the processed video sequences (PVS) from 7 different video sources. First test focuses on the effect of spatial resolution at fixed QP 22, while second test explores the impact of both spatial resolution and quantization artifacts. Videos coded at different spatial resolutions are displayed at the full screen size of the mobile platform. We report the test results in terms of how MOS reduces as the spatial resolution decreases at constant QP, and how MOS reduces as QP increases under the same spatial resolution. We found that the dropping rate of the MOS vs. spatial resolution increases as QP increases and the dropping rate of the MOS vs. QP increases as spatial resolution reduces. In addition, the dropping rate against spatial resolution is faster when the video is changing in a lower motion speed; while the dropping rate against QP is faster when the video has less texture details. Also, the dropping rate of MOS against spatial resolution can be captured by a one-parameter falling exponential function in most of cases.

This paper reviews the development of perceptually-motivated models for quality assessment of speech transmission/storage systems. The aim is to predict a subjective mean opinion score (MOS) for non-linear, time-variant distortions such as lossy coders, channel errors or noise reduction, particularly for telecommunications applications. Because linear methods have proven unsuitable for this purpose, many researchers have studied perceptual quality assessment using a comparison of auditory transforms to estimate quality. This work has led to several ITU standards. Non-intrusive models, arguably more suited to network monitoring, are the focus of much current interest. Intrusive, signal-based non-intrusive, and parametric non-intrusive models are discussed.

This paper is about a theoretical framework on perceptual reproduction quality assessment. — Visual perception is the main theme for two approaches on image metrics, visual performance and image quality. Visual performance looks at the perception of standard elementary images, such as gratings, by different observers or by a single observer in different visual conditions; image quality looks at the perception of variations of the same complex scene by a standard observer. — Visual performance, for which Barten proposed a model in 1999, is studying threshold responses in visual perception. For image quality, on the other hand, indexes are built which must evaluate responses to a variety of suprathreshold visual characteristics, attempting to give results as close as possible to human visual perception. — In audiovisual field, all the configurations approved by the creative team must be reproduced in the final viewing. This cannot be measured by visual performance or image quality indexes. A third approach is needed. This may be called perceptual reproduction quality assessment. Visual perception assessment is not measuring defects, per se; it is built to evaluate the perceptual effectiveness of a given scene coding through a given reproduction technology. — It is only dependent on the number of configurations being reproduced by a given theater system relatively to the various configurations reproduced and approved in the review room. As such, it is a statistical measure. It is in fact similar to the definition of relative entropy. One important fact is that this evaluation must include a model of visual perception. — This paper shows how relative entropy, and entropy loss, can be estimated while taking into account the coding transform, the projector performances, maximum light and the characteristics of human visual system. — This scheme is carried easily for monochromatic content. Color, trichromacy, is quite challenging. The size of configurations goes from 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">12</sup> to 2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">36</sup> , around 6 billion configurations. Calculation and memory requirements are huge. Non-linearities are adding complexity leading to calculation of an upper limit only of entropy loss. — Presentation of results is also a challenge. One way to solve this is to process separately lightness and chromatic information. — In summary the main arguments in this paper are: — • Besides image quality and visual performance there is a need for perceptual reproduction quality assessment — • Entropy is the proper statistical estimator to evaluate the perceptual reproduction quality — • Color reproduction quality estimation requires processing lightness and chromatic attributes separately

This paper studies the problem of full reference visual quality assessment of denoised images with a special emphasis on images with low contrast and noise-like texture. Denoising of such images together with noise removal often results in image details loss or smoothing. A new test image database, FLT, containing 75 noise-free "reference" images and 300 filtered ("distorted") images is developed. Each reference image, corrupted by an additive white Gaussian noise, is denoised by the BM3D filter with four different values of threshold parameter (four levels of noise suppression). After carrying out a perceptual quality assessment of distorted images, the mean opinion scores (MOS) are obtained and compared with the values of known full reference quality metrics. As a result, the Spearman Rank Order Correlation Coefficient (SROCC) between PSNR values and MOS has a value close to zero, and SROCC between values of known full-reference image visual quality metrics and MOS does not exceed 0.82 (which is reached by a new visual quality metric proposed in this paper). The FLT dataset is more complex than earlier datasets used for assessment of visual quality for image denoising. Thus, it can be effectively used to design new image visual quality metrics for image denoising.

The frame rate (FR) of a video plays an important role in affecting the perceptual video quality. Most studies about the effect of FR on the video quality mainly focused on low frame rate, e.g. less than 30 frames per second (fps), at low resolutions like CIF or QCIF. As the video frame rate and resolution advance, we reconsider this issue and investigate the relationship between frame rate and the perceptual video quality under high frame rate and high resolution. In this paper, we discuss the impact of frame rate on the perceptual quality of High Definition (HD) video with high frame rates (up to 120 fps) considered. Firstly, we design and conduct subjective experiment to construct the video dataset, which includes video sequences at different frame rates and the corresponding mean opinion scores (MOS) which represent the perceptual video quality. Based on the MOS results, we analyze how perceptual video quality changes as frame rate varies among different video sequences and propose some meaningful findings. The video dataset will be made publicly available. We deem that this study will enrich video quality assessment and benefit the development of high frame rate and high definition video business.KeywordsVideo quality assessmentSubjective experienceHigh frame rateHigh definition

The authors previously proposed an image-based technique [Y. Lin et al. Med. Phys. 39, 7019-7031 (2012)] to assess the perceptual quality of clinical chest radiographs. In this study, an observer study was designed and conducted to validate the output of the program against rankings by expert radiologists and to establish the ranges of the output values that reflect the acceptable image appearance so the program output can be used for image quality optimization and tracking. Using an IRB-approved protocol, 2500 clinical chest radiographs (PA/AP) were collected from our clinical operation. The images were processed through our perceptual quality assessment program to measure their appearance in terms of ten metrics of perceptual image quality: lung gray level, lung detail, lung noise, rib-lung contrast, rib sharpness, mediastinum detail, mediastinum noise, mediastinum alignment, subdiaphragm-lung contrast, and subdiaphragm area. From the results, for each targeted appearance attribute/metric, 18 images were selected such that the images presented a relatively constant appearance with respect to all metrics except the targeted one. The images were then incorporated into a graphical user interface, which displayed them into three panels of six in a random order. Using a DICOM calibrated diagnostic display workstation and under low ambient lighting conditions, each of five participating attending chest radiologists was tasked to spatially order the images based only on the targeted appearance attribute regardless of the other qualities. Once ordered, the observer also indicated the range of image appearances that he/she considered clinically acceptable. The observer data were analyzed in terms of the correlations between the observer and algorithmic rankings and interobserver variability. An observer-averaged acceptable image appearance was also statistically derived for each quality attribute based on the collected individual acceptable ranges. The observer study indicated that, for each image quality attribute, the averaged observer ranking strongly correlated with the algorithmic ranking (linear correlation coefficient R > 0.92), with highest correlation (R = 1) for lung gray level and the lowest (R = 0.92) for mediastinum noise. There was a strong concordance between the observers in terms of their rankings (i.e., Kendall's tau agreement > 0.84). The observers also generally indicated similar tolerance and preference levels in terms of acceptable ranges, as 85% of the values were close to the overall tolerance or preference levels and the differences were smaller than 0.15. The observer study indicates that the previously proposed technique provides a robust reflection of the perceptual image quality in clinical images. The results established the range of algorithmic outputs for each metric that can be used to quantitatively assess and qualify the appearance quality of clinical chest radiographs.

Although a variety of successful no-reference (blind) picture quality analyzers have been proposed, progress on the blind video quality analysis problem has been slow. We break down the problem of perceptual blind video quality assessment (VQA) into components, which we address individually, before proposing a holistic solution. The idea is to tackle the challenges that comprise the blind VQA problem individually in order to gain a better understanding of it. Publisher’s Note: The first printing of this volume was completed prior to the SPIE Digital Library publication and this paper has since been replaced with a corrected/revised version.

Quality assessment is crucial to developments of video communication systems. While subjective test yields most accurate results, its expensive cost and human biasness make it an unsatisfactory figure of merit. The peak-signal to noise ratio (PSNR), the most acceptable objective video quality assessment metric, is based on error sensitivity; its failure to capture perceptual errors is well documented. Other perceptual quality metrics fail to detect the localized errors that are common in sequences produced by low bit-rate video communication systems. In this work, a novel perceptual quality assessment metric, the structural similarity (SSIM) metric, is modified to better detect spatial artifacts often seen in video communication systems

Current action recognition algorithms in ice hockey do not fully exploit the temporal cues available in video. To solve this challenge, we introduce a two-stream network utilizing player pose sequences and optical flow features for recognizing hockey actions. Player pose sequences are compact representations consisting of frame by frame human and stick joint locations and angles between joints. The optical flow features are obtained by a state-of-the-art optical flow algorithm. The player pose sequences are processed by a two-layered Long short-term memory (LSTM) network. The LSTM output is fused with optical flow features processed by a convolutional neural network (CNN). Experimental results demonstrate the efficacy of the method by achieving 90.48% test accuracy on the HARPET (Hockey Action Recognition Pose Estimation, Temporal) dataset thus surpassing current benchmark by 5%. The network performs better than the current benchmark in segregating similar classes like passing and shooting. It achieves a 90% reduction in parameters and 80% reduction in floating point operations per second (FLOPs) than the benchmark on the HARPET dataset, thus furthering the effectiveness of the network.

Human motion recognition is traditionally approached by either recognizing basic motions from features derived from video input or by interpreting complex motions by applying a high-level hierarchy of motion primitives. The former method is usually limited to rather simple motions while the latter requires human expert knowledge to build up a suitable hierarchy. In this paper we propose a new approach that uses the strength of both methods while overcoming their respective limitations. Our approach is able to recognize the motion units within complex motion sequences. The recognition process applies Hidden Markov Models (HMM) based on features consisting of optical flow gradient histograms. For each primitive motion unit we train one HMM and then concatenate these primitive motion units to form complex motion sequences. Modeling sequences with HMMs allows for a very flexible combination of motion units into motion sequences. They can either be combined in a restrictive rule-based formulation using predefined grammars or be more flexibly combined using a statistical model of sequence probabilities. In this paper we are mainly interested in the comparison of the optical flow features with marker-based features, therefore we do not use a motion grammar. We apply our approach to 24 motion units forming five complex motion sequences as they appear in a real-world kitchen tasks. The results show that the proposed approach allows for a very fast low-level recognition of human motion units without the need for any complex reconstruction, post processing or pose estimation. Straight-forward characteristic flow fields in combination with HMM sequence modeling are sufficient to reliably recognize complex motions even with an unrestricted search. Our results show that this search already achieves 13.1 % recognition error rate. We compare HMM models based on the optical flow features to those derived from a marker-based system. Our recognition results indicate that optical flow features achieve a competitive performance.

There is a need for video encoders to generate bit streams of quality that matches human evaluation. This is often achieved by adopting perceptual quality metrics in video encoders. A huge body of work has been devoted to the development of perceptual video quality metrics that take the properties of human visual system into account. It is our interest in this paper to study how the existing video coding systems can benefit from such perceptual quality metrics. Specifically, we examine the existing perceptual video coding systems and introduce the concept of i§codec-friendliness, i¨ meaning how the perceptual quality metrics can be nicely incorporated into the coding process of standard video coding algorithms. We conclude the study by suggesting guidelines for the development of future quality metrics from the video coding perspective.

The demand of an accurate objective image quality assessment tool is important in modern multimedia systems. Image coding algorithms introduce highly structured coding artifacts and distortions. In this paper, we present a novel approach to predict the perceived image quality. Properties of the Human Visual System (HVS) were exploited to select a set of suitable metrics. These metrics are extracted while comparing the reference and distorted image. Mutual Information (MI) and Principal Component Analysis (PCA) were used to obtain an optimal set of objective features that best describe the perceived image quality in respect to subjective scores from human observers. The impairment feature vector is forwarded to the Artificial Neural Network (ANN) where the features are combined and the predicted quality score is computed. Parameters of the ANN are adjusted using Mean Opinion Scores (MOS) obtained from the group of assessors. It is shown that the proposed image quality assessment model can achieve high correlation with the subjective image quality ratings. Possible incorporation of the model into a perceptual image-coding algorithm is proposed. Such a system is capable to ensure that only visually important information is encoded and consequently that the required communication bandwidth is minimized.

Bai, Y., 2019. Target detection method of underwater moving image based on optical flow characteristics. In: Guido-Aldana, P.A. and Mulahasan, S. (eds.), Advances in Water Resources and Exploration. Journal of Coastal Research, Special Issue No. 93, pp. 668–673. Coconut Creek (Florida), ISSN 0749-0208.The detection and recognition ability of underwater moving image target is improved, and an automatic target detection method based on optical flow feature is proposed. The image acquisition model of underwater moving image target detection is constructed, and the local information entropy feature of underwater moving target image is detected by optical flow field scale decomposition method. The contrast feature of the distribution area of the optical flow field of the underwater moving image is extracted, and the information fusion model of the underwater moving target image is constructed according to the statistical value of the inter-frame difference of the underwater moving image. The center position of the candidate target is obtained, and the adaptive feature extraction of the underwater moving image target is carried out by using the correlation fusion method between the pixel neighborhoods, and the automatic detection of the underwater moving image target is realized according to the optical flow feature distribution. The simulation results show that the proposed method has good adaptability, high accuracy and strong anti-optical flow interference ability in underwater moving image target recognition.

A Pan-Sharpening (PS) technique integrates spatial details of a high-resolution panchromatic (PAN) image with spectral information of a low-resolution multi-spectral (MS) image, creating a single high-resolution color image. Since different PS methods produce images of a different quality, they should be compared using Image Quality Assessment (IQA) approaches that mimics human visual perception. In this paper, a MATLAB-based Toolbox for perceptual Image Quality Assessment of Pan-Sharpened Images (TIQA-PSI) is presented. TIQA-PSI aims to stimulate the development and facilitate the evaluation of new PS or PS-IQA approaches as it contains a large number of state-of-the-art IQA methods and images generated by representative PS techniques with corresponding subjective scores. They can be used to assess the output of any new PS algorithm or to serve as baselines for the comparison with new PS-IQA methods. Furthermore, apart from popular PS-IQA methods, the toolbox offers an opportunity to employ perceptual IQA approaches that can be effectively trained on scores obtained in tests with human subjects for superior image evaluation. The code of TIQA-PSI is available on GitHub.