Blind light field image quality assessment by analyzing angular-spatial characteristics

No-reference light field image quality assessment based on depth, structural and angular information

With the development of immersive media, Light Field Image (LFI) quality assessment is becoming more and more important, which helps to better guide light field acquisition, processing and application. However, almost all existing LFI quality assessment schemes utilize the 2D or 3D quality assessment methods while ignoring the intrinsic high dimensional characteristics of LFI. Therefore, we adopt the tensor theory to explore the LF 4D structure characteristics and propose the first Blind quality Evaluator of LIght Field image (BELIF). We generate cyclopean images tensor from the original LFI and then the features are extracted by the tucker decomposition. Specifically, Tensor Spatial Characteristic Features (TSCF) for spatial quality and Tensor Structure Variation Index (TSVI) for angular consistency are designed to fully assess the LFI quality. Extensive experimental results on the public LFI databases demonstrate that BELIF signifi-cantly outperforms the existing image quality assessment algorithms.

Light field images can record multiple information about the light rays in a scene and provide multiple views from a single image, offering a new data source for 3D reconstruction. However, ensuring the quality of light field images themselves is challenging, and distorted image inputs may lead to poor reconstruction results. Accurate light field image quality assessment can pre-judge the quality of light field images used as input for 3D reconstruction, providing a reference for the reconstruction results before the reconstruction work, significantly improving the efficiency of 3D reconstruction based on light field images. In this paper, we propose an Adaptive Vision Transformer-based light-field image-quality assessment model (AViT-LFIQA). The model adopts a multi-view sub-aperture image sequence input method, greatly reducing the number of input images while retaining as much information as possible from the original light field image, alleviating the training pressure on the neural network. Furthermore, we design an adaptive learnable attention layer based on ViT, which addresses the lack of inductive bias in ViT by using adaptive diagonal masking and a learnable temperature coefficient strategy, making the model more suitable for training on small datasets of light field images. Experimental results demonstrate that the proposed model is effective for various types of distortions and shows superior performance in light-field image-quality assessment.

Splitting the backbone: A novel hierarchical method for assessing light field image quality

Light Field Images (LFIs) capture both angular and spatial information, enabling immersive visual experiences for observers. However, during acquisition and processing, LFIs often suffer from various degradations, accompanied by intricate spatio-angular distortions. Consequently, objective LFI Quality Assessment (LFIQA) has garnered significant research interest. Existing blind LFIQA methods predominantly operate in a content-agnostic manner. They primarily model low-level distortions while neglecting the semantic context, a limitation that significantly diverges from the perception mechanisms of the Human Visual System (HVS). To bridge the gap between artifact analysis and scene understanding, we propose LV-BLiF, a novel dual-branch LFIQA network based on language-vision representation learning. Firstly, the textual prompts-assisted branch utilizes a pre-trained quality-aware Large Multimodal Model (LMM) to extract deep semantic representations, which are transformed into semantic context vectors sensitive to spatial-angular quality variations via pseudo-label guided learning strategy. Subsequently, semantic quality scores are generated based on this high-level information. Secondly, the visual cues-assisted branch employs disentangling convolutions to decompose light field features into spatial, angular, and epipolar plane image subspaces. These features are processed by Light Field Residual Blocks (LFRBs) and adaptively integrated via a Mixture-of-Experts-like (MoE-like) gating mechanism to predict visual quality scores. Finally, the semantic and visual scores are fused through a learnable adaptive gating mechanism to derive the overall LFI quality score. Extensive experiments on benchmark datasets, including NBU, SHU, and Win5-LID, demonstrate that LV-BLiF significantly outperforms state-of-the-art methods and exhibits high consistency with human perception. The proposed metric will be publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/shuaibilx/LV-BLiF</uri>

Faithful measurement of perceptual quality is of significant importance to various multimedia applications. By fully utilizing reference images, full-reference image quality assessment (FR-IQA) methods usually achieves better prediction performance. On the other hand, no-reference image quality assessment (NR-IQA), also known as blind image quality assessment (BIQA), which does not consider the reference image, makes it a challenging but important task. Previous NR-IQA methods have focused on spatial measures at the expense of information in the available frequency bands. In this paper, we present a multiscale deep blind image quality assessment method (BIQA, M.D.) with spatial optimal-scale filtering analysis. Motivated by the multi-channel behavior of the human visual system and contrast sensitivity function, we decompose an image into a number of spatial frequency bands by multiscale filtering and extract features for mapping an image to its subjective quality score by applying convolutional neural network. Experimental results show that BIQA, M.D. compares well with existing NR-IQA methods and generalizes well across datasets.

Light Field imaging provides a wide range of interactive features, such as view point changing and refocusing, and it has been developing as a solution for six degree of freedom applications. Many ongoing efforts, specifically related to pre/post processing, compression, and rendering, have been devoted to the development of this technology. To benchmark processing algorithms, the knowledge of degraded or improved quality level is important. However, the definition of validated subjective and objective quality assessment methods is still an open and challenging issue. In this scenario, this contribution deals with: the impact of visualization techniques on Light Filed image Quality of Experience, a visualization technique recommended for subjective quality assessment, perceptual visual impact of compression and noise artifacts, performance analysis of 2D image quality measures applied to Light Field images, and a Light Field image quality dataset.

An investigation of diseases using magnetic resonance (MR) imaging requires automatic image quality assessment methods able to exclude low-quality scans. Such methods can be also employed for an optimization of parameters of imaging systems or evaluation of image processing algorithms. Therefore, in this paper, a novel blind image quality assessment (BIQA) method for the evaluation of MR images is introduced. It is observed that the result of filtering using non-maximum suppression (NMS) strongly depends on the perceptual quality of an input image. Hence, in the method, the image is first processed by the NMS with various levels of acceptable local intensity difference. Then, the quality is efficiently expressed by the entropy of a sequence of extrema numbers obtained with the thresholded NMS. The proposed BIQA approach is compared with ten state-of-the-art techniques on a dataset containing MR images and subjective scores provided by 31 experienced radiologists. The Pearson, Spearman, Kendall correlation coefficients and root mean square error for the method assessing images in the dataset were 0.6741, 0.3540, 0.2428, and 0.5375, respectively. The extensive experimental evaluation of the BIQA methods reveals that the introduced measure outperforms related techniques by a large margin as it correlates better with human scores.

State of the art blind image quality assessment (IQA) methods generally extract perceptual features from the training images, and send them into support vector machine (SVM) to learn the regression model, which could be used to further predict the quality scores of the testing images. However, these methods need complicated training and learning, and the evaluation results are sensitive to image contents and learning strategies. In this paper, two novel blind IQA metrics without training and learning are firstly proposed. The new methods extract perceptual features, i.e., the shape consistency of conditional histograms, from the joint histograms of neighboring divisive normalization transform coefficients of distorted images, and then compare the length attribute of the extracted features with that of the reference images and degraded images in the LIVE database. For the first method, a cluster center is found in the feature attribute space of the natural reference images, and the distance between the feature attribute of the distorted image and the cluster center is adopted as the quality label. The second method utilizes the feature attributes and subjective scores of all the images in the LIVE database to construct a dictionary, and the final quality score is calculated by interpolating the subjective scores of nearby words in the dictionary. Unlike the traditional SVM based blind IQA methods, the proposed metrics have explicit expressions, which reflect the relationships of the perceptual features and the image quality well. Experiment results in the publicly available databases such as LIVE, CSIQ and TID2008 had shown the effectiveness of the proposed methods, and the performances are fairly acceptable.

The evaluation of the quality of light field images is a demanding task given the peculiarities of this media. In the literature, attempts to assess quality have been done by considering specific coding approaches or visualization techniques. In this paper we intend to i) investigate whether the distortions of the light field are reflected in the distortion of the saliency map and ii) propose a metric for image quality assessment of light fields based on a convolutional neural network that exploits the measure of the distortion of the saliency map. In our tests, the annotated SMART dataset has been used. The achieved results confirm the importance of saliency for improving the performance of quality metrics.

Light filed imaging is becoming popular for its diversity of post-processing and a wide range of applications. Various kinds of research about light field such as light field compression methods are coming out one after the other in recent years. For better evaluation of the quality of light field images and the performance of compression algorithm, the study on quality assessment of light field is in desperate need. In this paper, in order to establish a light field quality assessment database for the subsequent research, we propose a methodology of subjective evaluation for light field image and use a 2D objective evaluation method to verify the methodology. Results show that this methodology can be successfully used to assess the quality of light field content.

Light field image (LFI) now is becoming increasingly popular in immersive media applications. Unlike traditional 2D and 3D images, images taken by light field cameras can capture both angular and spatial information. However, the spatial and angular information of LFI is highly inter-twined with varying disparities, which poses a higher challenge to the quality assessment of LFI. To address this issue, this paper proposes a full-reference light field image quality assessment (LFIQA) index that attempts to disentangle the coupling information from macro-pixel image (MacPI) to accurately evaluate the entire LFI quality. The proposed framework can be divided into three steps. Firstly, the LFIs are converted into the MacPIs, and then the spatial and angular feature maps are disentangled by using the spatial, angular and epipolar plane image (EPI) convolutions in the MacPI mode. Secondly, the structural similarity (SSIM) maps are calculated between the disentangled feature maps of the original and distorted LFIs. Furthermore, the quality-aware features of LFIs are extracted on the SSIM maps by utilized local binary patterns (LBP) and natural scene statistics (NSS). Finally, support vector regression (SVR) is utilized to predict the qualities of LFIs. Extensive experiments show that the proposed model outperforms multiple classical and state-of-the-art methods.

Due to the rapid development of multimedia technologies, over the last decades image quality assessment (IQA) has become an important topic. As a consequence, a great research effort has been made to develop computational models that estimate image quality. Among the possible IQA approaches, blind IQA (BIQA) is of fundamental interest as it can be used in most multimedia applications. BIQA techniques measure the perceptual quality of an image without using the reference (or pristine) image. This paper proposes a new BIQA method that uses a combination of texture features and saliency maps of an image. Texture features are extracted from the images using the local binary pattern (LBP) operator at multiple scales. To extract the salient of an image, i.e. the areas of the image that are the main attractors of the viewers' attention, we use computational visual attention models that output saliency maps. These saliency maps can be used as weighting functions for the LBP maps at multiple scales. We propose an operator that produces a combination of multiscale LBP maps and saliency maps, which is called the multiscale salient local binary pattern (MSLBP) operator. To define which is the best model to be used in the proposed operator, we investigate the performance of several saliency models. Experimental results demonstrate that the proposed method is able to estimate the quality of impaired images with a wide variety of distortions. The proposed metric has a better prediction accuracy than state-of-the-art IQA methods.

In this paper, a universal blind image quality assessment (IQA) algorithm is proposed that works in presence of various distortions. The proposed algorithm is a Blind Image Quality metric based on Wakeby Statistics (BIQWS) which extracts local mean subtraction and contrast normalization (MSCN) coefficients in spatial domain from input image. The MSCN coefficients are used for generating a Wakeby distribution statistical model to extract quality-aware features. The statistical studies indicate that the MSCN coefficients histogram is altered in the presence of various distortions with different severities. These changes are regular and can be used to estimate the type of the distortion and its severity. We extended our previous studies to extract efficient Wakeby distribution model parameters which are more sensitive to changes in MSCN coefficients. These parameters are used to form a quality-aware feature vector. This feature vector is then fed to an SVM (support vector machine) regression model with a nonlinear Kernel to predict the quality score of the input image without any information about the distortion type or reference image. Experimental results show that the image quality index obtained by the proposed method has higher correlation with respect to human perceptual opinions and it is superior in some distortions when compared to some full-reference and other state-of-the-art blind image quality assessment methods.

Image quality is related to image content and distortion information. Most learning-based image quality assessment (IQA) methods extract quality-oriented features with auxiliary tasks like detecting the distortion type and level. However, the perceptual quality degradation caused by the same distortion type and level varies substantially for different content in an image. To deal with this problem, in this article, we propose a blind IQA method based on Deep Response fEAture decoMposition and aggregation (DREAM), which considers two factors affecting the image quality simultaneously. First, we use a convolutional neural network (CNN) to extract the basic features from the input image. Second, several parallel fully connected (FC) layers are employed to decompose these basic features into response features related to the image content, distortion type, and distortion level. Third, the graph attention network (GAT) is leveraged to aggregate these response features corresponding to the visual quality. Finally, a regression network is used to predict the quality score. The success of our method lies in the feature decomposition to obtain the response features of different content to a specific distortion in the given distorted image and the quality-oriented features obtained by feature aggregation using the internal relation of these response features. Experimental results indicate that our proposed DREAM achieves state-of-the-art (SOTA) performance on both synthetic and authentic distortion IQA datasets.