Stereoscopic Image Quality Assessment Based on both Distortion and Disparity

In previous quality assessment studies on stereoscopic 3D (S3D) images, researchers have concentrated on deriving manually extracted features which represent the quality of images. These features are based on the human visual system or natural scene statistics, but they have not been revealed as a deterministic function, preventing to guarantee the robustness of features. To solve this problem, we introduce a deep learning method for predicting the quality of S3D images without a reference. A convolutional neural network (CNN) model is trained through two-step learning. First, to overcome the lack of training data, patch-based CNNs are introduced. And then, automatically extracted patch features are pooled into image features. Finally, the trained CNN model parameters are updated iteratively using holistic image labeling, i.e., mean opinion score (MOS). The proposed method represents a significant improvement compared to other no-reference (NR) S3D image quality assessment (IQA) algorithms.

This paper presents a full reference quality assessment metric for stereoscopic images based on perceptual binocular characteristics. To ensure that the predicted 3D quality of experience is as reliable and close as possible to 3D human perception, the proposed stereoscopic image quality assessment (SIQA) method is relying on the cyclopean image. Our approach is motivated by the fact that in case of asymmetric quality, 3D perception mechanisms place more emphasis on the view providing the most important and contrasted information. We integrated this psychophysical findings in the proposed 3D-IQA framework thanks to a weighting factor based on local information content. Add to that, to take into account the disparity/depth masking effect, we modulate the obtained quality score of each pixel of the cyclopean image according to its location in the scene. Experimental results show that the proposed metric correlates better with human judgement than the state-of-the-art metrics.

Quality assessment of stereoscopic 3D images is much more complex than that of 2D images. Quality assessment metrics for 3D images have achieved good performance on symmetric distorted 3D images, while given poor performance on asymmetric distorted 3D images. To further improve the perception consistency for symmetric and asymmetric distorted 3D images, this paper presents a machine learning-based full-reference 3D image quality assessment method that learns from multi IQA metrics. Considering that changes in left view, right view and depth information all impact human perception, our method considers the quality of two views and depth information. Since symmetric and asymmetric distortions impact left and right views of a 3D image differently, we propose different features for symmetric and asymmetric distorted 3D images. The experimental results show that our method has better performance than both 2D-based 3D quality assessment metrics and the state of the art specially designed 3D quality assessment metrics.

The quality assessment of stereoscopic images has attracted considerable attention and become an important issue in 3D multimedia applications. The 3D image quality assessment (IQA) encounters many challenges and simple extension of the 2D quality metrics to the 3D case is not satisfying. In this paper, we propose a new perceptual quality assessment scheme for stereoscopic 3D images by considering the local and global visual characteristics. The design of this scheme is motivated by studies on the perception of distorted stereoscopic images. To be more specific, after the log-Gabor filter processing, the local amplitude and phase from the left and right views of the reference and distorted 3D images are utilized as features in local quality evaluation. Meanwhile, the global structure changes of the left and right views are also incorporated into the final quality pooling. The overall 3D quality score is obtained by combining the local and global quality indexes together. The effectiveness of the designed metric is verified on publicly available 3D image quality assessment databases. Experimental results show that the proposed scheme exhibits better performance than other related algorithms in terms of consistency with subjective assessment of stereoscopic 3D images.

영상 처리 및 컴퓨터 비전 분야에 있어서, 평균 제곱 오차(Mean Squared Error: MSE)는 좋은 수학적 특성(예를 들어, 척도성(metricability), 미분가능성(differentiability) 및 볼록 성질(convexity))을 가짐으로 인해 많은 영상 화질 최적화 문제의 객관적 척도로 사용되어 왔다. 그러나 MSE가 영상의 왜곡 신호에 대한 시각적 인지 화질과 상관도가 높지 않다는 것이 알려지면서, 이를 해결하기 위해 위에서 언급한 좋은 수학적 특성과 높은 영상 화질 예측 성능을 동시에 가지는 객관적 영상 화질 측정(Image Quality Assessment: IQA)척도가 활발히 연구되어 왔다. 비록 최근 제안된 좋은 수학적 성질을 만족시키는 IQA 척도들은 MSE와 비교하여 매우 향상된 주관적 화질 예측 성능을 보이지만, 상대적으로 높은 계산 복잡도를 가진다. 본 논문은 이를 해결하기 위해, 단순 라플라스 연산자를 이용한 좋은 수학적 특성을 가지는 새로운 IQA 척도를 제안한다. 제안 IQA 방법에 도입한 단순 라플라스 연산자는 인간 시각 체계의 망막에서의 광도 자극에 대한 시신경 반응을 효과적으로 모사할 뿐만 아니라 계산이 매우 단순하기 때문에, 제안 IQA 척도는 단순 라플라스 연산자를 사용하여 매우 빠른 계산 속도와 높은 주관적 화질 점수 예측력을 확보하였다. 제안 IQA 척도의 효과를 검증하기 위해, 최신 IQA 척도들과 광범위한 성능비교 실험을 수행하였다. 실험 결과, 제안하는 IQA 척도는 모든 테스트 IQA 척도들 중 MSE를 제외하고 가장 빠른 처리 속도를 보였을 뿐만 아니라, 가장 높은 주관적 화질예측 성능을 보였다. In image processing and computer vision fields, mean squared error (MSE) has popularly been used as an objective metric in image quality optimization problems due to its desirable mathematical properties such as metricability, differentiability and convexity. However, as known that MSE is not highly correlated with perceived visual quality, much effort has been made to develop new image quality assessment (IQA) metrics having both the desirable mathematical properties aforementioned and high prediction performances for subjective visual quality scores. Although recent IQA metrics having the desirable mathematical properties have shown to give some promising results in prediction performance for visual quality scores, they also have high computation complexities. In order to alleviate this problem, we propose a new fast IQA metric using a simple Laplace operator. Since the Laplace operator used in our IQA metric can not only effectively mimic operations of receptive fields in retina for luminance stimulus but also be simply computed, our IQA metric can yield both very fast processing speed and high prediction performance. In order to verify the effectiveness of the proposed IQA metric, our method is compared to some state-of-the-art IQA metrics. The experimental results showed that the proposed IQA metric has the fastest running speed compared the IQA methods except MSE under comparison. Moreover, our IQA metric achieves the best prediction performance for subjective image quality scores among the state-of-the-art IQA metrics under test.

We propose visual comfort assessment for stereoscopic 3D (S3D) images based on salient discomfort regions. Color-based saliency successfully represents visual attention because that the human visual system (HVS) focuses on the most salient region in an image. Disparity-based saliency effectively expresses visual comfort in S3D images. Based on two saliencies, we extract salient discomfort regions which mostly determine the overall visual comfort degree in S3D images. Then, we predict the visual comfort score from salient discomfort regions by a disparity feature vector which combines saliency-weighted disparity and maximum disparity. Finally, we provide visual comfort index maps of S3D images based on visual comfort scores, thus showing perceptually salient discomfort regions and their degrees. Experimental results demonstrate that the proposed method achieves significantly high accuracy in visual comfort assessment as compared to existing methods.

Objective Image Quality Assessment (IQA) of a stereo pair entails considering the depth information in the 3D scene. Initially 2D Quality Assessment algorithms were used on stereopairs but failed to address IQA of all distortion types in stereo pairs as depth information was missing in it. Conventional QA measures like MSE and PSNR involved only error detection between reference and distorted images. As recent study shows Human Visual System (HVS) is more adapted to recognizing edges and contours from natural scenes than pixel level errors, Mean Edge Structure Similarity (MESSIM) algorithm has been developed and observation studied. It is compared with other IQA tools like MSSIM, MSE, PSNR, AD, SC, IF, Spectral Activity, Spatial Frequency and Entropy. MESSIM is observed to give better correlation with the subjective score than the rest of the IQA tools. A single, cyclopean image is synthesized from the left and right images of a stereo pair for quality assessment between reference and distorted image. IQA based on cyclopean image gives better correlation results than using 2D QA tool of averaging quality indices of left and right images.

An optimized CNN-based quality assessment model for screen content image

This research aims to determine 1) the possibility of integrating stereoscopic 3D images into textbooks as the medium giving material illustrations, and 2) the effect of using textbooks integrated with stereoscopic 3D images on the learning process. This research was conducted through a quasi-experiment in primary schools by using control class and experimental class. This research was joined by 52 participants (26 students in control class and 26 students in experimental class). The research conclusions indicate that, 1) stereoscopic 3D images can be integrated into textbooks as illustrations increasing the primary school students’ perception about classroom activities (i.e., motivation, interest, challenge and excitement). and 2) textbooks integrated with stereoscopic 3D images are proven effective in improving the students' concept mastery, as seen from the independent sample t-test by using SPSS resulting in a probability (p) value of 0.000 with a significance level of 0.05 so that p < 0, 05.

However, in order to create a stereoscopic 3D image, independently, it is required to spend expensive manufacturing costs and to have special techniques. As 3D display devices have been generalized, there is an increasing need for implementing a stereoscopic 3D image without a burden of expensive costs. This paper proposes some methods to implement stereoscopic 3D images easily by utilizing a marker tracking technology using a single camera. In addition, the study made it possible for the resolution of an image to be adjustable dynamically. This paper will be committed to the promotion of the field of UCC (User Created Contents) using stereoscopic 3D images by attracting the active participation of general users in the field of the implementation of stereoscopic 3D images.

Traditional 2D image-based saliency prediction models suffer from unsatisfactory performance when dealing with stereoscopic 3D (S3D) images because eye movements in the case of freely viewing S3D images are demonstrated to be guided by both RGB and depth features. This paper studies the problem of saliency prediction on S3D images, where the interactions between RGB and depth modalities are both taken into account. Specifically, we design a novel deep neural network named Cross-modality Fusion and Progressive Integration Network (CFPI-Net) to address this problem. It consists of a Multi-level Cross-modality Feature Fusion (MCFF) module and a Multi-stage Progressive Feature Integration (MPFI) module. The MCFF module first captures hierarchical contexture features from each modality and then effectively fuses the hierarchical contexture features from different modalities at each level. The MPFI module involves multiple cascaded deeply supervised feature integration (DSFI) blocks in which the low-level and high-level cross-modality features are progressively integrated using the integrated features in the previous stage as a guidance. Our proposed CFPI-Net benefits from the advantages of multi-level feature representation, cross-modality feature fusion, and multi-stage progressive feature integration, which hereby fully boost the performance. Experimental results on two benchmark datasets demonstrate that CFPI-Net outperforms state-of-the-art saliency prediction methods both quantitatively and qualitatively. All the results and relevant codes will be made available to the public.

This paper proposes a new methodology to address the image quality problem encountered as the use of an unmanned aerial vehicle (UAV) in the field of bridge inspection increased. When inspecting a bridge, the image obtained from the UAV was degraded by various interference factors such as vibration, wind, and motion of UAV. Image quality degradation such as blur, noise, and low-resolution is a major obstacle in utilizing bridge inspection technology based on UAV. In particular, in the field of bridge inspection where damages must be accurately and quickly detected based on data obtained from UAV, these quality issues weaken the advantage of using UAVs by requiring re-take of images through re-flighting. Therefore, in this study, image quality assessment (IQA) based on local blur map (LBM) and image quality enhancement (IQE) using the variational Dirichlet (VD) kernel estimation were proposed as a solution to address the quality issues. First, image data was collected by setting different camera parameters for each bridge member. Second, a blur map was generated through discrete wavelet transform (DWT) and a new quality metric to measure the degree of blurriness was proposed. Third, for low-quality images with a large degree of blurriness, the blind kernel estimation and blind image deconvolution were performed to enhance the quality of images. In the validation tests, the proposed quality metric was applied to material image sets of bridge pier and deck taken from UAV, and its results were compared with those of other quality metrics based on singular value decomposition (SVD), sum of gray-intensity variance (SGV) and high-frequency multiscale fusion and sort transform (HiFST) methods. It was validated that the proposed IQA metric showed better classification performance on UAV images for bridge inspection through comparison with the classification results by human perception. In addition, by performing IQE, on average, 26% of blur was reduced, and the images with enhanced quality showed better damage detection performance through the deep learning model (i.e., mask and region-based convolutional neural network).

No-reference stereoscopic image quality assessment using a multi-task CNN and registered distortion representation

We study non-reference image and video quality assessment methods, which are of great importance for computational video editing. The object of our work is image quality assessment (IQA) applicable for fast and robust frame-by-frame multipurpose video quality assessment (VQA) for short videos.

최근 들어 다양한 스테레오 3D 디스플레이 장치의 출시가 활발히 이루어지면서 임의의 스테레오 3D 영상에 대한 효과적인 크기 변환 방법, 즉 스케일링(scaling) 방법에 대한 요구가 증가되고 있다. 그런데 CG 등을 통하여 생성된 경우와 같이 장면의 기하학적 정보를 완벽히 파악할 수 있는 경우를 제외하고는, 정확한 스테레오 3D 영상의 스케일링은 현실적으로 불가능하다. 이러한 제한으로 가장 널리 이용되는 방법은 좌우 영상에 대하여 적절한 2D 스케일링을 수행하고, 그 결과로 적절한 스테레오 3D 영상스케일링이 발생하기를 기대하는 것이다. 본 논문에서는 2D 영상 스케일링을 통한 스테레오 3D 영상 스케일링에 있어서, 양안시(binocular vision) 모델에 근거한 기하학적 분석과 주파수 변화 분석을 제시한다. 기하학적 분석을 통해서는 스테레오 3D 영상에서 발생 가능한 왜곡들에 대해서 설명하고, 주파수 변화 분석을 통해서는 디지털 신호의 스케일링에 필수적으로 요구되는 aliasing 왜곡 방지 방법에 대해서도 제안한다. 제안된 방법은 주관적 화질 평가 결과 샘플링 이론 (sampling theory)에만 근거한 필터링 방식에 비해 화질 왜곡을 더욱 효과적으로 억제할 수 있다는 것을 파악할 수 있었다. 따라서, 제안된 방법은 다양한 해상도를 지니는 3D 디스플레이 장치에서의 스테레오 3D 영상 재생에 효과적으로 이용될 것으로 기대된다. Image resizing (or scaling) is one of the most essential issues for the success of visual service because image data has to be adapted to the variety of display features. For 2D imaging, the image scaling is generally accomplished by 2D image re-sampling (i.e., up-/down-sampling). However, when it comes to stereoscopic 3D images, 2D re-sampling methods are inadequate because additional consideration on the third dimension of depth is not incorporated. Practically, stereoscopic 3D image scaling is process with left/right images, not stereoscopic 3D image itself, because the left/right Images are only tangible data. In this paper, we analyze stereoscopic 3D image scaling from two aspects: geometrical deformation and frequency-domain aliasing. A number of 3D displays are available in the market and they have various screen dimensions. As we have more varieties of the displays, efficient stereoscopic 3D image scaling is becoming more emphasized. We present the recommendations for the 3D scaling from the geometric analysis and propose a disparity-adaptive filter for anti-aliasing which could occur during the image scaling process.