Learning from multi metrics for stereoscopic 3D image quality assessment

The quality assessment of stereoscopic images has attracted considerable attention and become an important issue in 3D multimedia applications. The 3D image quality assessment (IQA) encounters many challenges and simple extension of the 2D quality metrics to the 3D case is not satisfying. In this paper, we propose a new perceptual quality assessment scheme for stereoscopic 3D images by considering the local and global visual characteristics. The design of this scheme is motivated by studies on the perception of distorted stereoscopic images. To be more specific, after the log-Gabor filter processing, the local amplitude and phase from the left and right views of the reference and distorted 3D images are utilized as features in local quality evaluation. Meanwhile, the global structure changes of the left and right views are also incorporated into the final quality pooling. The overall 3D quality score is obtained by combining the local and global quality indexes together. The effectiveness of the designed metric is verified on publicly available 3D image quality assessment databases. Experimental results show that the proposed scheme exhibits better performance than other related algorithms in terms of consistency with subjective assessment of stereoscopic 3D images.

View synthesis brings geometric distortions which are not handled efficiently by existing image quality assessment metrics. Despite the widespread of 3-D technology and notably 3D television (3DTV) and free-viewpoints television (FTV), the field of view synthesis quality assessment has not yet been widely investigated and new quality metrics are required. In this study, we propose a new full-reference objective quality assessment metric: the View Synthesis Quality Assessment (VSQA) metric. Our method is dedicated to artifacts detection in synthesized view-points and aims to handle areas where disparity estimation may fail: thin objects, object borders, transparency, variations of illumination or color differences between left and right views, periodic objects... The key feature of the proposed method is the use of three visibility maps which characterize complexity in terms of textures, diversity of gradient orientations and presence of high contrast. Moreover, the VSQA metric can be defined as an extension of any existing 2D image quality assessment metric. Experimental tests have shown the effectiveness of the proposed method.

Image and video compression aims at finding an optimal trade-off between rate and distortion. This is done through Rate-Distortion Optimization (RDO) in traditional en-coders with the use of Image Quality Assessment (IQA) metrics. While it is known that most IQA metrics are designed to be correlated with human perception, there is no evidence that this observation can be generalized in a Video Coding for Machines (VCM) context, where the receiver is not a human anymore but a machine. In this paper, we propose an evaluation protocol to measure the correlation level between conventional Full-Reference (FR) IQA metrics and machine perception through the semantic segmentation vision task. Experiments showed a relatively low correlation between them when measured on the block-level. This observation implies the need of RDO algorithms that are better suited for Machine-to-Machine (M2M) communications. In order to facilitate the emergence of IQA metrics that better reflect machine perception, the code and dataset used to perform this study is made freely available at https://github.com/albmarie/iqa_m2m_segmentation.

This paper presents a full reference quality assessment metric for stereoscopic images based on perceptual binocular characteristics. To ensure that the predicted 3D quality of experience is as reliable and close as possible to 3D human perception, the proposed stereoscopic image quality assessment (SIQA) method is relying on the cyclopean image. Our approach is motivated by the fact that in case of asymmetric quality, 3D perception mechanisms place more emphasis on the view providing the most important and contrasted information. We integrated this psychophysical findings in the proposed 3D-IQA framework thanks to a weighting factor based on local information content. Add to that, to take into account the disparity/depth masking effect, we modulate the obtained quality score of each pixel of the cyclopean image according to its location in the scene. Experimental results show that the proposed metric correlates better with human judgement than the state-of-the-art metrics.

Today, thanks to the developing technology, changes are experienced in many areas and these changes sometimes bring some problems, issues that need to be solved or improved. In the field of digital image and video processing, the viewer experience has been greatly improved with the development of image and video coding methods and technologies in recent years. However, due to these developments, the importance of video and image quality evaluation techniques has increased. In the field of imaging technology, although the human visual system and human judgment experiments will give us the most accurate results, this rapidly growing field has made human judgment experiments unsuitable with the increasing content. At this point, effective and automatic quality assessment metrics are needed to evaluate and optimize advanced compression technologies in terms of visual perception quality. In this paper, our aim is to compare the performances of some of the new generation image and video quality assessment metrics.

No-reference image quality assessment (NR-IQA) uses only the test image for its quality assessment, and as video is essentially comprised of image frames with additional temporal dimension, video quality assessment (VQA) requires a thorough understanding of image quality assessment metrics and models. Therefore, in order to identify features that deteriorate video quality, a fundamental analysis of spatial and temporal artifacts with respect to individual video frames needs to be performed. Existing IQA and VQA metrics are primarily for capturing few distortions and hence may not be good for all types of images and videos. In this paper, we propose an NR-IQA model by combining existing three methods (namely NIQE, BRISQUE and BLIINDS-II) using multi-linear regression. We also present a holistic no-reference video quality assessment (NR-VQA) model by exploring quantification of certain distortions like ringing, frame difference, blocking, clipping and contrast in video frames. For the proposed NR-IQA model, the results represent improved performance as compared to the state-of-the-art methods and it requires very low fraction of samples for training to provide a consistent accuracy over different training-to-testing ratios. The performance of NR-VQA model is examined using a simple neural network model to attain high value of goodness of fit.

Understanding the characteristics of high-quality stereoscopic 3D (S3D) images has great significance for S3D image classification, quality assessment, and quality enhancement. Existing works assess the quality of an S3D image from a single perspective and use databases with subjective opinion scores obtained by conducting subjective experiments in labs. So the performance of these works in real applications is unclear and questionable. In this paper, we propose an S3D image quality assessment index based on two important factors, namely DisTortion and DisParity (DTDP). Distortion reflects the extent of distortion of each view of an S3D image, respectively. Disparity is a distinguishing factor for an S3D image as compared with a monocular image. We design some features to represent these two factors and use a random forest regression (RFR) to learn the mapping between the features and subjective opinion scores. The database we choose is the NVIDIA 3D VISION LIVE Highest Rated database which is from real application. The images and the subjective opinion scores are directly come from the website viewers all over the world. Experimental results demonstrate a superior performance of the proposed DTDP index as compared to the existing methods.

A reference-free underwater image quality assessment metric in frequency domain

In previous quality assessment studies on stereoscopic 3D (S3D) images, researchers have concentrated on deriving manually extracted features which represent the quality of images. These features are based on the human visual system or natural scene statistics, but they have not been revealed as a deterministic function, preventing to guarantee the robustness of features. To solve this problem, we introduce a deep learning method for predicting the quality of S3D images without a reference. A convolutional neural network (CNN) model is trained through two-step learning. First, to overcome the lack of training data, patch-based CNNs are introduced. And then, automatically extracted patch features are pooled into image features. Finally, the trained CNN model parameters are updated iteratively using holistic image labeling, i.e., mean opinion score (MOS). The proposed method represents a significant improvement compared to other no-reference (NR) S3D image quality assessment (IQA) algorithms.

New Stereoscopic Image Quality Assessment Metric Based on Three Dimensional-discrete Cosine Transform for 3d Media

Currently, many tone mapping operators (TMOs) have been provided to compress high dynamic range images to low dynamic range (LDR) images for visualizing them on the common displays. Since quality degradation is inevitably induced by compression, how to evaluate the obtained LDR images is indeed a headache problem. Until now, only a few full reference (FR) image quality assessment metrics have been proposed. However, they highly depend on reference image and neglect human visual system characteristics, hindering the practical applications. In this paper, we propose an effective blind quality assessment method of tone-mapped image without access to reference image. Inspired by that the performance of existing TMOs largely depend on the brightness and chromatic and structural properties of a scene, we evaluate the perceptual quality from the perspective of color information processing in the brain. Specifically, motivated by the physiological and psychological evidence, we simulate the responses of single-opponent (SO) and double-opponent (DO) cells, which play an important role in the processing of the color information. To represent the textural information, we extract three features from gray-level co-occurrence matrix (GLCM) calculated from SO responses. Meanwhile, both GLCM and local binary pattern descriptor are employed to extract texture and structure in the responses of DO cells. All these extracted features and associated subjective ratings are learned to reveal the connection between feature space and human opinion score. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art blind quality assessment methods and is comparable with the popular FR methods on two recently published tone-mapped image databases.

Three-dimensional (3D) imaging technology has been growingly prevalent in today's world. But objective quality assessment of 3D images is a challenging task. In this paper, we try to investigate and develop the 2D image quality metrics into stereoscopic image quality assessment using a dual-weight model within the concept of free energy principle and human visual system. On the basis of a psychological measure, the free energy is a principle telling where supervises more and attracts human attention. We believe that the "surprise" can account for the binocular rivalry and introduce the inter-weight. Also, the intra-weight is computed according to the physiology of human visual system. We first do the intra-weighted 2D image quality assessment. With inter-weights, we then calculate the combining score of stereopairs. Experiments are performed on the symmetric LIVE3D-I and asymmetric LIVE3D-II databases. Results confirm that the proposed 3D IQA technique, appropriately computing and combining the inter- and intra-weights, is able to faithfully predict the visual quality of stereopairs.

영상 처리 및 컴퓨터 비전 분야에 있어서, 평균 제곱 오차(Mean Squared Error: MSE)는 좋은 수학적 특성(예를 들어, 척도성(metricability), 미분가능성(differentiability) 및 볼록 성질(convexity))을 가짐으로 인해 많은 영상 화질 최적화 문제의 객관적 척도로 사용되어 왔다. 그러나 MSE가 영상의 왜곡 신호에 대한 시각적 인지 화질과 상관도가 높지 않다는 것이 알려지면서, 이를 해결하기 위해 위에서 언급한 좋은 수학적 특성과 높은 영상 화질 예측 성능을 동시에 가지는 객관적 영상 화질 측정(Image Quality Assessment: IQA)척도가 활발히 연구되어 왔다. 비록 최근 제안된 좋은 수학적 성질을 만족시키는 IQA 척도들은 MSE와 비교하여 매우 향상된 주관적 화질 예측 성능을 보이지만, 상대적으로 높은 계산 복잡도를 가진다. 본 논문은 이를 해결하기 위해, 단순 라플라스 연산자를 이용한 좋은 수학적 특성을 가지는 새로운 IQA 척도를 제안한다. 제안 IQA 방법에 도입한 단순 라플라스 연산자는 인간 시각 체계의 망막에서의 광도 자극에 대한 시신경 반응을 효과적으로 모사할 뿐만 아니라 계산이 매우 단순하기 때문에, 제안 IQA 척도는 단순 라플라스 연산자를 사용하여 매우 빠른 계산 속도와 높은 주관적 화질 점수 예측력을 확보하였다. 제안 IQA 척도의 효과를 검증하기 위해, 최신 IQA 척도들과 광범위한 성능비교 실험을 수행하였다. 실험 결과, 제안하는 IQA 척도는 모든 테스트 IQA 척도들 중 MSE를 제외하고 가장 빠른 처리 속도를 보였을 뿐만 아니라, 가장 높은 주관적 화질예측 성능을 보였다. In image processing and computer vision fields, mean squared error (MSE) has popularly been used as an objective metric in image quality optimization problems due to its desirable mathematical properties such as metricability, differentiability and convexity. However, as known that MSE is not highly correlated with perceived visual quality, much effort has been made to develop new image quality assessment (IQA) metrics having both the desirable mathematical properties aforementioned and high prediction performances for subjective visual quality scores. Although recent IQA metrics having the desirable mathematical properties have shown to give some promising results in prediction performance for visual quality scores, they also have high computation complexities. In order to alleviate this problem, we propose a new fast IQA metric using a simple Laplace operator. Since the Laplace operator used in our IQA metric can not only effectively mimic operations of receptive fields in retina for luminance stimulus but also be simply computed, our IQA metric can yield both very fast processing speed and high prediction performance. In order to verify the effectiveness of the proposed IQA metric, our method is compared to some state-of-the-art IQA metrics. The experimental results showed that the proposed IQA metric has the fastest running speed compared the IQA methods except MSE under comparison. Moreover, our IQA metric achieves the best prediction performance for subjective image quality scores among the state-of-the-art IQA metrics under test.

Blind quality assessment for screen content images via convolutional neural network

Phase map of the images captures the most fundamental cognitive features and thus is widely used in various digital image processing tasks. In this paper, we propose the Log Gabor Phase Similarity (LGPS), a novel full reference image quality assessment metrics based on measuring of similarities between phases in log Gabor transform domain. Phase can capture any changes in image details regardless of the fluctuation in contrast, and the similarity between phase maps provides a measure of the perceptual quality of images. An image is firstly decomposed by a filter bank consisting of a pair of log Gabor filters. The phase maps are then computed from the responses of each filter pair. We have developed a window-based similarity metric to evaluate the resemblance between phase maps so as to measure the quality of the image. Experimental results and comparative studies suggest that LGPS can be used to predict the perceived quality of images with different distortions.