Blind Image Quality Assessment by Learning from Multiple Annotators

Image quality assessment (IQA) is very important for both end-users and service providers since a high-quality image can significantly improve the user's quality of experience (QoE) and also benefit lots of computer vision algorithms. Most existing blind image quality assessment (BIQA) models were developed for synthetically distorted images, however, they perform poorly on in-the-wild images, which are widely existed in various practical applications. In this paper, we propose a novel BIQA model for in-the-wild images by addressing two critical problems in this field: how to learn better quality-aware feature representation, and how to solve the problem of insufficient training samples in terms of their content and distortion diversity. Considering that perceptual visual quality is affected by both low-level visual features (e.g. distortions) and high-level semantic information (e.g. content), we first propose a staircase structure to hierarchically integrate the features from intermediate layers into the final feature representation, which enables the model to make full use of visual information from low-level to high-level. Then an iterative mixed database training (IMDT) strategy is proposed to train the BIQA model on multiple databases simultaneously, so the model can benefit from the increase in both training samples and image content and distortion diversity and can learn a more general feature representation. Experimental results show that the proposed model outperforms other state-of-the-art BIQA models on six in-the-wild IQA databases by a large margin. Moreover, the proposed model shows an excellent performance in the cross-database evaluation experiments, which further demonstrates that the learned feature representation is robust to images with diverse distortions and content. The code is available at https://github.com/sunwei925/StairIQA.

PATCH-IQ: A patch based learning framework for blind image quality assessment

Unsupervised blind image quality assessment based on joint structure and natural scene statistics features

Objective assessment of image quality is fundamentally important in many image processing tasks. In this paper, we focus on learning blind image quality assessment (BIQA) models, which predict the quality of a digital image with no access to its original pristine-quality counterpart as reference. One of the biggest challenges in learning BIQA models is the conflict between the gigantic image space (which is in the dimension of the number of image pixels) and the extremely limited reliable ground truth data for training. Such data are typically collected via subjective testing, which is cumbersome, slow, and expensive. Here, we first show that a vast amount of reliable training data in the form of quality-discriminable image pairs (DIPs) can be obtained automatically at low cost by exploiting large-scale databases with diverse image content. We then learn an opinion-unaware BIQA (OU-BIQA, meaning that no subjective opinions are used for training) model using RankNet, a pairwise learning-to-rank (L2R) algorithm, from millions of DIPs, each associated with a perceptual uncertainty level, leading to a DIP inferred quality (dipIQ) index. Extensive experiments on four benchmark IQA databases demonstrate that dipIQ outperforms the state-of-the-art OU-BIQA models. The robustness of dipIQ is also significantly improved as confirmed by the group MAximum Differentiation competition method. Furthermore, we extend the proposed framework by learning models with ListNet (a listwise L2R algorithm) on quality-discriminable image lists (DIL). The resulting DIL inferred quality index achieves an additional performance gain.

We propose a blind image quality assessment (BIQA) model using learning to rank (LTR) method in order to overcome the defects of the existing BIQA model. In this model, we first simplify the problem of predicting exact image quality score with the classification of image quality preference and we then use feature fusion and multiple kernel learning based on group lasso (MKLGL) methods to train a classifier. Finally, based on the voting strategy, a simple and effective image quality prediction model is constructed. Experimental results on the existing IQA datasets show that our proposed method is highly consistent with the subjective perception of human eyes and is superior to existing BIQA methods. In addition, our method is highly scalable.

Almost all present successful general purpose blind image quality assessment (BIQA) models are designed only towards singly and synthetically distorted images. However, real world images generally contain multiply types of distortions other than single type of distortion. Even some state of the art BIQA methods cannot work well on authentically distorted images. In this paper we propose a novel effective BIQA method for authentically distorted images to remedy the shortage of the popular BIQA models. First, except for traditional natural scene statistics (NSS) based features, we introduce a series of quality aware features based on human visual system (HVS), such as image dynamic range, color information, blurriness. Then support vector regression (SVR) is utilized to learn the mapping between the combined effective features and human opinion scores. Experimental results demonstrate the promising performance of the proposed method.

The high-quality pathological microscopic images are essential for physicians or pathologists to make a correct diagnosis. Image quality assessment (IQA) can quantify the visual distortion degree of images and guide the imaging system to improve image quality, thus raising the quality of pathological microscopic images. Current IQA methods are not ideal for pathological microscopy images due to their specificity. In this paper, we present deep learning-based blind image quality assessment model with saliency block and patch block for pathological microscopic images. The saliency block and patch block can handle the local and global distortions, respectively. To better capture the area of interest of pathologists when viewing pathological images, the saliency block is fine-tuned by eye movement data of pathologists. The patch block can capture lots of global information strongly related to image quality via the interaction between different image patches from different positions. The performance of the developed model is validated by the home-made Pathological Microscopic Image Quality Database under Screen and Immersion Scenarios (PMIQD-SIS) and cross-validated by the five public datasets. The results of ablation experiments demonstrate the contribution of the added blocks. The dataset and the corresponding code are publicly available at: https://github.com/mikugyf/PMIQD-SIS.

In this paper, we explore the suitability of multi-resolution and multi-scale band-pass image representation generated by difference of Gaussian (DoG) operator for blind image quality assessment model. The developed model is based on general regression neural network (GRNN). The proposed model is consistent with human perception when evaluated on DIBR-synthesized image dataset.

Learning-based Image Quality Assessment (IQA) models have obtained impressive performance with the help of reliable subjective quality labels, where Mean Opinion Score (MOS) is the most popular choice. However, in view of the subjective bias of individual annotators, the Labor-Abundant MOS (LA-MOS) typically requires large collections of opinion scores from multiple annotators for each image, which significantly increases the learning cost. In this paper, we aim to learn robust IQA models from Low-Cost MOS (LC-MOS), which only requires very few opinion scores or even a single opinion score for each image. More specifically, we consider the LC-MOS as the noisy observation of LA-MOS and enforce the IQA model learned from LC-MOS to approach the unbiased estimation of LA-MOS. Thus, we represent the subjective bias between LC-MOS and LA-MOS, and the model bias between IQA predictions learned from LC-MOS and LA-MOS (i.e., dual-bias) as two latent variables with unknown parameters. By means of the expectation-maximization-based alternating optimization, we can jointly estimate the parameters of the dual-bias, which suppresses the misleading of LC-MOS via a gated dual-bias calibration (GDBC) module. To the best of our knowledge, this is the first exploration of robust IQA model learning from noisy low-cost labels. Theoretical analysis and extensive experiments on four popular IQA datasets show that the proposed method is robust toward different bias rates and annotation numbers and significantly outperforms the other Learning-based IQA models when only LC-MOS is available. Furthermore, we also achieve comparable performance with respect to the other models learned with LA-MOS.

Objective image quality assessment (IQA) models have been developed by effective features to imitate the characteristics of human visual system (HVS). Actually, HVS is extremely sensitive to color degradation and complex texture changes. In this paper, we firstly reveal that many existing full reference image quality assessment (FR-IQA) methods can hardly measure the image quality with contrast and masking texture changes. To solve this problem, considering texture masking effect, we proposed a novel FR-IQA method, called Texture and Color Quality Index (TCQI). The proposed method considers both in the masking effect texture and color visual perceptual threshold, which adopts three kinds of features to reflect masking texture, color difference and structural information. Furthermore, random forest (RF) is used to address the drawbacks of existing pooling technologies. Compared with other traditional learning-based tools (support vector regression and neural network), RF can achieve the better prediction performance. Experiments conducted on five large-scale databases demonstrate that our approach is highly consistent with subjective perception, outperforms twelve the state-of-the-art IQA models in terms of prediction accuracy and keeps a moderate computational complexity. The cross database validation also validates our approach achieves the ability to maintain high robustness.

We propose a highly unsupervised, training free, no reference image quality assessment (IQA) model that is based on the hypothesis that distorted images have certain latent characteristics that differ from those of “natural” or “pristine” images. These latent characteristics are uncovered by applying a “topic model” to visual words extracted from an assortment of pristine and distorted images. For the latent characteristics to be discriminatory between pristine and distorted images, the choice of the visual words is important. We extract quality-aware visual words that are based on natural scene statistic features [1]. We show that the similarity between the probability of occurrence of the different topics in an unseen image and the distribution of latent topics averaged over a large number of pristine natural images yields a quality measure. This measure correlates well with human difference mean opinion scores on the LIVE IQA database [2].

The visual quality of images resulting from Super Resolution (SR) techniques is predicted with blind image quality assessment (BIQA) models trained on a database(s) of human rated distorted images and associated human subjective opinion scores. Such opinion-aware (OA) methods need a large amount of training samples with associated human subjective scores, which are scarce in the field of SR. By contrast, opinion distortion unaware (ODU) methods do not need human subjective scores for training. This paper presents an opinion-unaware BIQA measure of super resolved images based on optimally extracted perceptual features. This set of features was selected using a floating forward search whose objective function is the correlation with human judgment. The proposed BIQA method does not need any distorted images nor subjective quality scores for training, yet the experiments demonstrate its superior quality-prediction performance relative to state-of-the-art opinion-unaware BIQA methods, and that it is competitive to state-of-the-art opinion-aware BIQA methods.

This paper presents an approach to predict perceived quality of compressed images while incorporating real visual attention coordinates. Information about the visual attention is not usually taken into account in models for image quality assessment. The idea of implementing gaze information into the image quality assessment system lies in that the artefacts are more disturbing to human observer in the region with higher saliency than in other parts of an image. Impact of the region of interest on estimation accuracy of a simple image quality metric is investigated. The gaze coordinates were calculated using calibrated electro-oculogram records of human observers while watching a number of test images. The same images were used for subjective image quality assessment. Obtained mean opinion scores of perceived image quality and eye tracking data were used to verify potential improvement of assessment accuracy for a simple image quality metric.

To monitor objects of interest, such as wildlife and people, image-capturing devices are used to collect a large number of images with and without objects of interest. As we are recording valuable information about the behavior and activity of objects, the quality of images containing objects of interest should be better than that of images without objects of interest, even if the former exhibits more severe distortion than the latter. However, according to current methods, quality assessments produce the opposite results. In this study, we propose an end-to-end model, named DETR-IQA (detection transformer image quality assessment), which extends the capability to perform object detection and blind image quality assessment (IQA) simultaneously by adding IQA heads comprising simple multi-layer perceptrons at the top of the DETRs (detection transformers) decoder. Using IQA heads, DETR-IQA carried out blind IQAs based on the weighted fusion of the distortion degree of the region of objects of interest and the other regions of the image; the predicted quality score of images containing objects of interest was generally greater than that of images without objects of interest. Currently, the subjective quality score of all public datasets is in accordance with the distortion of images and does not consider objects of interest. We manually extracted the images in which the five predefined classes of objects were the main contents of the largest authentic distortion dataset, KonIQ-10k, which was used as the experimental dataset. The experimental results show that with slight degradation in object detection performance and simple IQA heads, the values of PLCC and SRCC were 0.785 and 0.727, respectively, and exceeded those of some deep learning-based IQA models that are specially designed for only performing IQA. With the negligible increase in the computation and complexity of object detection and without a decrease in inference speeds, DETR-IQA can perform object detection and IQA via multi-tasking and substantially reduce the workload.

Artificial Intelligence (AI) is the enhancement and method of computer system that handles tasks which requires human like intelligence such as recognition, language translation and visual interpretation. Subjective image quality assessment (IQA) is difficult to be implemented in real-time systems, methodology for enhancing the involvement in producing IQA model is to improve the quality of image by significant evaluation. Intuitively, human eyes are not sensitive to the distortion and damage from the area with lesser visual saliency (VS), VS is closely related to IQA. With this consideration, an effective IQA was proposed, which involved two processes. The local quality map of a distorted image was computed using the structural similarity function of its feature attributes, such as brightness, chrominance and gradient. Second, the local quality map was weighted with visual saliency (VS) to get the objective evaluation of image quality. The VS was modeled by extracting the saliency of low-level features of the image, wiping off the molestation information from these saliency based on an apriori threshold, and combining the effective information to construct the saliency map. Image processing using fuzzy is gathering features and segments as fuzzy set while processing images. The experiments on the two largest database for six classical IQA metrics demonstrate that performance of weighted-VS IQA metrics is superior to the performance of no weighted-VS IQA metrics, and the proposed IQA method has higher computational accuracy than the other IQA metrics under a moderate computational complexity, especially for two types of distortion images, such as local block-wise (Block) and fast-fading (FTF).