No-reference stereoscopic image quality assessment based on binocular collaboration

In contemporary society full of stereoscopic images, how to assess visual quality of 3D images has attracted an increasing attention in field of Stereoscopic Image Quality Assessment (SIQA). Compared with 2D-IQA, SIQA is more challenging because some complicated features of Human Visual System (HVS), such as binocular interaction and binocular fusion, must be considered. In this paper, considering both binocular interaction and fusion mechanisms of the HVS, a hierarchical no-reference stereoscopic image quality assessment network (StereoIF-Net) is proposed to simulate the whole quality perception of 3D visual signals in human cortex, including two key modules: BIM and BFM. In particular, Binocular Interaction Modules (BIMs) are constructed to simulate binocular interaction in V2-V5 visual cortex regions, in which a novel cross convolution is designed to explore the interaction details in each region. In the BIMs, different output channel numbers are designed to imitate various receptive fields in V2-V5. Furthermore, a Binocular Fusion Module (BFM) with automatic learned weights is proposed to model binocular fusion of the HVS in higher cortex layers. The verification experiments are conducted on the LIVE 3D, IVC and Waterloo-IVC SIQA databases and three indices including PLCC, SROCC and RMSE are employed to evaluate the assessment consistency between StereoIF-Net and the HVS. The proposed StereoIF-Net achieves almost the best results compared with advanced SIQA methods. Specifically, the metric values on LIVE 3D, IVC and WIVC-I are the best, and are the second-best on the WIVC-II.

No-reference stereoscopic image quality assessment using a multi-task CNN and registered distortion representation

A no-reference stereoscopic image quality assessment method is proposed that takes two primary binocular effects, visual discomfort, and binocular rivalry, into consideration. The adaptive segmentation, local energy, and visual saliency are employed for estimating binocular effects and constructing a quality baseline, which together composes a comprehensive image quality description. Moreover, a compact parallel framework is designed for efficiency. Experiments have validated the high accuracy and robustness of the proposed method.

With continuous progress of Internet of Things, multimedia analysis in it has attracted more and more attention. Specially, stereoscopic display technology plays an important role in the multimedia analysis processing. In the Internet of Things system, the quality of stereoscopic image will be reduced in the transmission process. In this mode, it will have a great impact on multimedia analysis to judge whether the quality of stereoscopic image meets the requirements. In this paper, a new no-reference stereoscopic image quality assessment model for multimedia analysis towards Internet of Things is built, which is based on a deep learning model to learn from the class labels and image representations. In our framework, images are represented by natural scene statistics features that are extracted from discrete cosine transform domain, and a regression model is employed to shine upon the quality from the feature vector. The training process of the proposed model contains an unsupervised pretraining phase and a supervised fine-tuning phase, enabling it to generalize over the whole distortion types and severity. The proposed model greatly shows the correlation with subjective assessment as demonstrated by experiments on the LIVE 3-D Image Quality Database and IVC 3-D Image Quality Database.

Quality assessment of stereoscopic 3D images is much more complex than that of 2D images. Quality assessment metrics for 3D images have achieved good performance on symmetric distorted 3D images, while given poor performance on asymmetric distorted 3D images. To further improve the perception consistency for symmetric and asymmetric distorted 3D images, this paper presents a machine learning-based full-reference 3D image quality assessment method that learns from multi IQA metrics. Considering that changes in left view, right view and depth information all impact human perception, our method considers the quality of two views and depth information. Since symmetric and asymmetric distortions impact left and right views of a 3D image differently, we propose different features for symmetric and asymmetric distorted 3D images. The experimental results show that our method has better performance than both 2D-based 3D quality assessment metrics and the state of the art specially designed 3D quality assessment metrics.

Stereoscopic image retargeting plays an important role in adaptive 3D stereoscopic displays. It aims to fit displays with various resolutions while preserving visually salient content and geometric consistency. We propose a stereoscopic image retargeting method based on stereoscopic visual attention guided seam carving. Firstly, stereoscopic saliency map is generated by combining 2D saliency and depth saliency maps, and significant energy map is generated by considering binocular disparity binocular and binocular just-noticeable-difference (BJND). Then, seam selection is applied to the left image based on stereoscopic saliency and energy maps, and seam replacement is performed for the occluded regions to prevent the geometry inconsistency. Finally, according to the matched left and right seams, the retargeted stereoscopic image is generated. In the experiments, subjective and objective analysis on three stereoscopic image databases shows that the proposed approach produces better seam carving results than the related existing methods.

Stereoscopic image quality assessment has been widely studied in last decades; however, the research on 3D quality of experience (QoE) is proposed recently. As a part of human stereo perception, 3D QoE plays an important role to stereoscopic image quality assessment. In this paper, an objective metric is proposed based on the hypothesis that binocular vision system is sensitive to the structure of low-level features and its discrepancy between the two view images of a stereoscopic image pair. Specifically, the correlation between the left and right views of a stereoscopic image pair could reflect the QoE. To represent the structure of low-level features, in each view of the stereoscopic image pair, the phase congruency (PC) and the saliency map are employed as the primary and secondary features to compose a feature map. To compute the correlation between the two views, a local matching function is suggested to weight the discrepancy between the two feature maps and generate a local quality. Then these local quality values are combined to derive a single quality score. The proposed metric is evaluated on one public subjective assessment database. The experimental results indicate that our metric exhibits good performance.

Objective quality assessment of distorted stereoscopic images is a challenging problem, especially when the distortions in the left and right views are asymmetric. Existing studies suggest that simply averaging the quality of the left and right views well predicts the quality of symmetrically distorted stereoscopic images, but generates substantial prediction bias when applied to asymmetrically distorted stereoscopic images. In this paper, we first build a database that contains both single-view and symmetrically and asymmetrically distorted stereoscopic images. We then carry out a subjective test, where we find that the quality prediction bias of the asymmetrically distorted images could lean toward opposite directions (overestimate or underestimate), depending on the distortion types and levels. Our subjective test also suggests that eye dominance effect does not have strong impact on the visual quality decisions of stereoscopic images. Furthermore, we develop an information content and divisive normalization-based pooling scheme that improves upon structural similarity in estimating the quality of single-view images. Finally, we propose a binocular rivalry-inspired multi-scale model to predict the quality of stereoscopic images from that of the single-view images. Our results show that the proposed model, without explicitly identifying image distortion types, successfully eliminates the prediction bias, leading to significantly improved quality prediction of the stereoscopic images.

The quality assessment of stereoscopic images has attracted considerable attention and become an important issue in 3D multimedia applications. The 3D image quality assessment (IQA) encounters many challenges and simple extension of the 2D quality metrics to the 3D case is not satisfying. In this paper, we propose a new perceptual quality assessment scheme for stereoscopic 3D images by considering the local and global visual characteristics. The design of this scheme is motivated by studies on the perception of distorted stereoscopic images. To be more specific, after the log-Gabor filter processing, the local amplitude and phase from the left and right views of the reference and distorted 3D images are utilized as features in local quality evaluation. Meanwhile, the global structure changes of the left and right views are also incorporated into the final quality pooling. The overall 3D quality score is obtained by combining the local and global quality indexes together. The effectiveness of the designed metric is verified on publicly available 3D image quality assessment databases. Experimental results show that the proposed scheme exhibits better performance than other related algorithms in terms of consistency with subjective assessment of stereoscopic 3D images.

We propose a new watermarking algorithm for stereoscopic image tamper detection and self-recovery in three-dimensional multimedia services. Initially, left and right views of stereoscopic image are divided into nonoverlapping 2×2 blocks in order to improve the accuracy of tamper localization in an image. As the left and right views of a stereoscopic image are not independent from each other but have an inter-view relationship, every block of a stereoscopic image is classified into matching block or nonmatching block and then block disparities are obtained. Both matching blocks in the left and right views have similar pixel values, so that fewer bits are allocated for recovery watermark generation, which can increase the quality of watermarked stereoscopic images. A hierarchical tamper-detection strategy with a four-level checkup is presented to improve the accuracy of tamper localization. Additionally, two copies of block (matching block and nonmatching block) information are embedded into the stereoscopic image, and it assures the quality of tampered recovery. For the nonmatching block recovery, two copies of the partner block are embedded into their chaotic mapping blocks, which supply the second chance for tamper recovery. For the matching block recovery, the inter-view relationship between tampers of left and right views supplies the third chance for tamper recovery. Experimental results show that the proposed algorithm can not only detect and locate tampers in stereoscopic image more accurately but also recover the tampered regions better, compared with other algorithms.

Objective To investigate the visual function after part-time occlusion therapy in amblyopia children whose corrected visual acuity recovered to normal. Methods Investigate the binocular vision of 70 cases amblyopia children whose corrected visual acuity had recovered to normal.According to whether patching dominant eye during treatment the children were divided into 2 groups (A and B).The statistically difference of simultaneous perception,fusion function and stereopsis in two groups was analyzed.Results Forty-two (91.3％) and 26 (95.8％) children got the simultaneous perception in group A and group B respectively,and the differences between two groups were not significant (P ＞0.05).Groups A had two children (4.3％) and groups B had only one child (4.2％)own the normal aggregative fusion range,two groups did not show any difference (P ＞0.05),respectively,36 children (78.3％) and 17 children (70.8％) had the normal divided fusion range in group A and group B,the recovery rate did not show any significant difference (P ＞0.05).Fifteen (32.6％)and 10 (41.7％) patients acquired the far stereoscopic vision in group A and group B respectively,the recovery rate also did not show any significant difference.Six (13.0％) and 3 (12.5％) children got the normal near stereoscopic vision in group A and group B respectively,the recovery rate did not show any significant difference (P ＞0.05).After treatment,among the 70 cases of amblyopia children 25 (35.7％) got the normal far stereoscopic vision and only 7 (10.0％) got the near stereoscopic vision,there were statistically significant difference between far and near stereoscopic vision (P ＜0.05). Conclusions After amblyopia treatment,although the children obtain the normal visual acuity,in the short term the binocular vision still can not completely become normal.Part-time occlusion therapy dose not produce significant effect to binocular vision development. Key words: Amblyopia; Part-time occlusion; Binocular vision

The effectiveness of depth information in saliency detection has been fully proved. However, it is still worth exploring how to utilize the depth information more efficiently. Erroneous depth information may cause detection failure, while non-salient objects may be closer to the camera which also leads to erroneously emphasis on non-salient regions. Moreover, most of the existing RGB-D saliency detection models have poor robustness when the salient object touches the image boundaries. To mitigate these problems, we propose a multi-stage saliency detection model with the bilateral absorbing Markov chain guided by depth information. The proposed model progressively extracts the saliency cues with three level (low-, mid-, and high-level) stages. First, we generate low-level saliency cues by explicitly combining color and depth information. Then, we design a bilateral absorbing Markov chain to calculate mid-level saliency maps. In mid-level, to suppress boundary touch problem, we present the background seed screening mechanism (BSSM) for improving the construction of the two-layer sparse graph and better selecting background-based absorbing nodes. Furthermore, the cross-modal multi-graph learning model (CMLM) is designed to fully explore the intrinsic complementary relationship between color and depth information. Finally, to obtain a more highlighted and homogeneous saliency map in high-level, we structure a depth-guided optimization module which combines cellular automata and suppression-enhancement function pair. This optimization module refines the saliency map in color space and depth space, respectively. Comprehensive experiments on three challenging benchmark datasets demonstrate the effectiveness of our proposed method both qualitatively and quantitatively.

Stereoscopic perception is an important part of human visual system that allows the brain to perceive depth. However, depth information has not been well explored in existing saliency detection models. In this letter, a novel saliency detection method for stereoscopic images is proposed. Firstly, we propose a measure to evaluate the reliability of depth map, and use it to reduce the influence of poor depth map on saliency detection. Then, the input image is represented as a graph, and the depth information is introduced into graph construction. After that, a new definition of compactness using color and depth cues is put forward to compute the compactness saliency map. In order to compensate the detection errors of compactness saliency when the salient regions have similar appearances with background, foreground saliency map is calculated based on depth-refined foreground seeds selection mechanism and multiple cues contrast. Finally, these two saliency maps are integrated into a final saliency map through weighted-sum method according to their importance. Experiments on two publicly available stereo datasets demonstrate that the proposed method performs better than other 10 state-of-the-art approaches.

© IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

In the domain of stereo vision, Stereo High Dynamic Range Imaging (SHDRI) is a revolutionary concept. Traditionally, monocular depth estimators had to be trained with ground truth and binocular stereo data. With the use of deep learning, this paper suggests a new SHDRI approach that uses a single image. The Left View (LV) which is the input, is given to a pretrained monocular depth estimator, which uses depth hints obtained from off-the-shelf stereo algorithms. The estimator provides us with the LV’s depth map as well as the disparity values which are then combined with the input to create the Right View (RV), resulting in the stereo pair. Using Power Law Transformation, we make multiple exposed images of both the LV and the RV. A unified and unsupervised end-to-end exposure fusion network fuses the LV and RV exposures and generates HDR images for both the views, forming the stereo HDR image pair. Additionally, we produce anaglyphs from the resulting HDR image pair, which can be employed to intuitively perceive three-dimensional structures.