A distortion-agnostic video quality metric based on multi-scale spatio-temporal structural information

In video communication, the quality of video is mainly determined by bitrate in general. Moreover, the effect of video contents and their visual perception on video quality assessment (VQA) is often overlooked. However, in fact, for different videos, although the bitrates are the same, their VQA scores are still significantly different. Hence, it is assumed that the bitrate, video contents, and human visual characteristics mainly affect the VQA. Based on the above three aspects, in this paper, we designed a bitrate-based no-reference (NR) VQA metric combining the visual perception of video contents, namely, BRVPVC. In this metric, first an initial VQA model was proposed by only considering the bitrate alone. Then, the visual perception model for video contents was designed based on the texture complexity and local contrast of image, temporal information of video, and their visual perception features. Finally, two models were synthesized by adding certain weight coefficients into an overall VQA metric, namely, BRVPVC. Furthermore, ten reference videos and 150 distorted videos in the LIVE video database were used to test the metric. Moreover, based on the results of evaluating the videos in LIVE, VQEG, IRCCyN, EPFL-PoliMI, IVP, CSIQ, and Lisbon databases, the performance of BRVPVC is respectively compared with that of six full-reference (FR) metrics and ten NR VQA metrics. The results show that our VQA metric has a higher accuracy than six common FR VQA metrics and eight NR VQA metrics, and it is close to other two NR VQA metrics in accuracy. The corresponding values of Pearson linear correlation coefficient and Spearman rank order correlation coefficient reached 0.8547 and 0.8260, respectively. In addition, the computational complexity of proposed VQA metric is lower than video signal-to-noise ratio, video quality model, motion-based video integrity evaluation, spatiotemporal most apparent distortion, V-BLINDS, and V-CORNIA metrics. Moreover, the proposed metric has a better generalization property than these metrics.

Modelling events in densely crowded environments remains challenging, due to the diversity of events and the noise in the scene. We propose a novel approach for anomalous event detection in crowded scenes using dynamic textures described by the Local Binary Patterns from Three Orthogonal Planes (LBP-TOP) descriptor. The scene is divided into spatio-temporal patches where LBP-TOP based dynamic textures are extracted. We apply hierarchical Bayesian models to detect the patches containing unusual events. Our method is an unsupervised approach, and it does not rely on object tracking or background subtraction. We show that our approach outperforms existing state of the art algorithms for anomalous event detection in UCSD dataset.

In this paper, we propose a general-purpose no-reference (NR) video quality assessment (VQA) metric based on the cascade combination of 2D convolutional neural network (CNN), multi-layer perceptron (MLP), and support vector regression (SVR) model. The features are extracted from both spatial and spatiotemporal domains by using a 2D CNN. These features can capture different aspects of video frames for predicting quality scores, and we take these features as inputs of MLP to obtain a few estimated quality scores on different perspectives. Finally, these estimated scores are combined as a final quality score by an SVR model. The proposed method is evaluated on the well-known LIVE Video database with other state-of-the-art and well-performing VQA metrics. And the experimental result demonstrates that our method is competitive with other full-reference and NR VQA metrics.

Objective video quality assessment plays an important role in a variety of video processing applications such as video compression, transmission, visualization, and display. This paper proposes a no-reference video quality assessment model based on visual memory understanding. Inspired by the findings of neuroscience researchers, who argue there is a large overlap between the active human brain area when performing video quality assessment and saliency detection tasks, saliency maps are employed here to assist the quality assessment. To this end, we first generate the saliency maps using CLBP (Complete Local Binary Patterns) features of the residual frames. Then, a model of visual memory is created from the statistics of saliency maps. This is followed by learning the video quality from the visual memory, saliency, and frame features through a support vector regression pipeline. The experimental results on the state-of-the-art LIVE and SJTU video datasets indicate that the proposed no-reference video quality assessment algorithm is effective and performs statistically better than several other state-of-the-art approaches.

A general-purpose no-reference video quality assessment algorithm based on a long short-term memory (LSTM) network and a pretrained convolutional neural network (CNN) is introduced. Considering video sequences as a time series of deep features extracted with the help of a CNN, an LSTM network is trained to predict subjective quality scores. In contrast to previous methods, the resulting algorithm was trained on the recently published Konstanz Natural Video Quality Database (KoNViD-1k), which is the only publicly available database that contains sequences with authentic distortions. The results of experiments on KoNViD-1k demonstrate that the proposed method outperforms other state-of-the-art algorithms. Furthermore, these results are also confirmed using tests on the LIVE Video Quality Assessment Database, which consists of artificially distorted videos.

This paper proposes a novel approach in image compression based on Local Binary Pattern (LBP). LBP has already been used as a simple texture descriptor, labeling the image pixels by looking at the points surrounding a central point (usually on a 3×3 neighborhood) and examining whether these neighbors' color values are greater or less than the central point and accordingly assigning a binary value to the corresponding bit. The description of image's local pattern results in an eight-bit binary description, but in order to restore the image from such a LBP description, the value of each central pixel is also needed. These two pieces of information, i.e. the LBP description and the actual original value for each local neighborhood central pixel, are stored in a newly proposed Local Binary Compressed format, denoted .LBC, from which the image can be reconstructed by employing statistical methods, i.e. generating smaller or larger sets of random numbers to fill in the missing information within each local neighborhood, based on the LBP descriptor. Two statistical distributions were tested and, apart from the compression performance, a Structural Similarity Index Metric was used to evaluate the results.

No-reference image quality assessment (NR-IQA) uses only the test image for its quality assessment, and as video is essentially comprised of image frames with additional temporal dimension, video quality assessment (VQA) requires a thorough understanding of image quality assessment metrics and models. Therefore, in order to identify features that deteriorate video quality, a fundamental analysis of spatial and temporal artifacts with respect to individual video frames needs to be performed. Existing IQA and VQA metrics are primarily for capturing few distortions and hence may not be good for all types of images and videos. In this paper, we propose an NR-IQA model by combining existing three methods (namely NIQE, BRISQUE and BLIINDS-II) using multi-linear regression. We also present a holistic no-reference video quality assessment (NR-VQA) model by exploring quantification of certain distortions like ringing, frame difference, blocking, clipping and contrast in video frames. For the proposed NR-IQA model, the results represent improved performance as compared to the state-of-the-art methods and it requires very low fraction of samples for training to provide a consistent accuracy over different training-to-testing ratios. The performance of NR-VQA model is examined using a simple neural network model to attain high value of goodness of fit.

Methods for No-Reference Video Quality Assessment (NR-VQA) of consumer-produced video content are largely investigated due to the spread of databases containing videos affected by natural distortions. In this work, we design an effective and efficient method for NR-VQA. The proposed method exploits a novel sampling module capable of selecting a predetermined number of frames from the whole video sequence on which to base the quality assessment. It encodes both the quality attributes and semantic content of video frames using two lightweight Convolutional Neural Networks (CNNs). Then, it estimates the quality score of the entire video using a Support Vector Regressor (SVR). We compare the proposed method against several relevant state-of-the-art methods using four benchmark databases containing user generated videos (CVD2014, KoNViD-1k, LIVE-Qualcomm, and LIVE-VQC). The results show that the proposed method at a substantially lower computational cost predicts subjective video quality in line with the state of the art methods on individual databases and generalizes better than existing methods in cross-database setup.

We present a framework to detect and retrieve CPR activity scenes from medical simulation videos. Medical simulation is a modern training method for medical practitioners, where an emergency patient condition is simulated on humanlike mannequins and the students act upon. These simulation sessions are recorded for later debriefing. With the increasing number of simulation videos, automatic detection and retrieval of specific scenes became necessary. In this paper, we present an automated CPR scene retrieval system which can identify and retrieve CPR activity scenes, using Local Binary Patterns Over Three Orthogonal Planes (LBP-TOP) features. We present a comparative study of LBP-TOP features and Histogram of Orientation of Gradients (HOG3D) features for CPR action detection. We also present the results of decision level fusion of different classifier outputs. Promising classification results have been achieved, suggesting the proposed technique to be effective in detecting and retrieving CPR activity scenes from simulation videos.

No-reference video quality assessment (NR-VQA) for user generated content (UGC) is crucial for understanding and improving visual experience. Unlike video recognition tasks, VQA tasks are sensitive to changes in input resolution. Since large amounts of UGC videos nowadays are 720p or above, the fixed and relatively small input used in conventional NR-VQA methods results in missing high-frequency details for many videos. In this paper, we propose a novel Transformer-based NR-VQA framework that preserves the high-resolution quality information. With the multi-resolution input representation and a novel multi-resolution patch sampling mechanism, our method enables a comprehensive view of both the global video composition and local high-resolution details. The proposed approach can effectively aggregate quality information across different granularities in spatial and temporal dimensions, making the model robust to input resolution variations. Our method achieves state-of-the-art performance on large-scale UGC VQA datasets LSVQ and LSVQ-1080p, and on KoNViD-1k and LIVE-VQC without fine-tuning.

In this paper, the performance of two new promising operators for the analysis of 3D textures based on feature extraction is validated. The first operator is the decimal descriptor patterns from three orthogonal planes (DDP-TOP), a new feature descriptor that considers the co-occurrences on three orthogonal planes. The second operator is the grey level co-occurrence matrix from three orthogonal planes (GLCM-TOP) which is an extension of the 2D grey level co-occurrence matrix method. In order to classify the MR images of brain into healthy and diseased, several tests are performed in the same conditions of work using the classifier multiclass support vector machines (SVMs). The local binary pattern (LBP), a best known method of texture analysis is used for comparison. Using the DDP-TOP operator, excellent experimental results are obtained that prove the robustness of our approach with respect to the noise level and to different image contrasts.

Iris recognition is a means of biometric identification. A key part of the recognition system using iris is the extraction of prominent texture information or features in the iris. The identification delay in iris recognition can be reduced by reducing the feature vector generated from the feature extraction of iris images. A new form of local binary pattern (LBP) termed LBPX is proposed in this paper as an iris feature extraction method. For this, input eye images are processed and converted to normalized iris images employing circular Hough transformation and Daugman’s rubber sheet model. Next, LBPX is applied to the normalized images. In this LBPX stage, rotation-invariant LBP operation takes place. The performance of LBPX based recognition system adopting iris image is evaluated in terms of accuracy and feature vector length. This is done for three datasets CASIA-IRIS-V4, UBIRIS and IITD. Results indicate that LBPX can achieve acceptable accuracy values of 97.20%, 96% and 96.40% for CASIA-IRIS-V4, UBIRIS and IITD datasets, respectively. Furthermore, results show LBPX outperforms existing feature extraction methods in terms of reduced feature-length, ensuring faster iris recognition.

Texture feature of the froth image is widely used in the working condition recognition of froth flotation. However, due to the complexity of the froth image, the current texture features vary greatly and are difficult to identify the work condition accurately. Therefore, we propose a dynamic texture feature named LBP on the TOP and GLCM Histograms (LTGH) which integrates the local binary patterns (LBPs) and gray-level co-occurrence matrix (GLCM) histograms on the three orthogonal planes (TOP). First, we use the rotation invariant LBPs to enhance rotation invariance and illumination robustness. Then, we implement the TOP on the enhanced texture feature map to generate the multiple dimensional enhanced feature maps. After that, we calculate the GLCM and supplementary features (SFs) on the multiple dimensional enhanced feature map. Finally, we integrate the histogram of the GLCM and SFs to discriminate the texture feature. The LTGH feature considers the froth structures both in the macrolevel and microlevel and captures the temporal information between the froth images. Experiments have demonstrated the effectiveness and stability of the proposed texture feature for work condition recognition in froth flotation. Compared with other traditional texture features, the accuracy of the LTGH feature has been increased by at least 7.76%.

Local binary pattern (LBP) is widely used to extract image features in various visual recognition tasks. LBP is formulated in quite a simple form and thus enables us to extract effective image features with a low computational cost. There, however, are some limitations mainly regarding sensitivity to noise and loss of image contrast information. In this paper, we propose a novel LBP-based image feature to remedy those drawbacks without degrading the simplicity of the original LBP formulation. Encoding local pixel intensities into binary patterns can be regarded as separating them into two modes (clusters). We introduce Fisher discriminant criterion to optimize the LBP coding for exploiting binary patterns stably and discriminatively with robustness to noise. Besides, image contrast information is incorporated in a unified way by leveraging the discriminant score as a weight on the corresponding binary pattern; thereby, the prominent patterns are emphasized. In the experiments on pedestrian detection, the proposed method exhibits superior performance compared to the ordinary LBP and the other methods, especially in the case of lower-dimensional features.

In recent years, significant efforts have been dedicated to advancing 3D content generation. However, existing quality assessment research predominantly focuses on evaluating Text-to-3D Content (T23DC) while ignoring Single Image-to-3D Content (SI23DC). In this paper, we establish the first Single Image-to-3D Content Quality Assessment (SI23DCQA) database to comprehensively study the perceptual quality of SI23DCs. The database contains 1500 SI23DCs, which are generated by 5 common SI23DC algorithms from 300 images including realistic images, AI generated images, and model rendered images. Afterward, we carry out a well-designed subjective experiment to collect subjective quality ratings for SI23DCs from three perspectives including overall, color, and shape. Additionally, a benchmark experiment is conducted with the state-of-the-art no reference image quality assessment (NR-IQA), no reference video quality assessment (NR-VQA), and no reference 3D quality assessment (NR-3DQA) and the experimental results show that current quality assessment methods are limited in evaluating the perceptual loss of SI23DCs. The database is released on https://github.com/ZedFu/SI23DCQA.