Point cloud subjective evaluation methodology based on reconstructed surfaces

An increased interest in immersive applications has drawn attention to\nemerging 3D imaging representation formats, notably light fields and point\nclouds (PCs). Nowadays, PCs are one of the most popular 3D media formats, due\nto recent developments in PC acquisition, namely with new depth sensors and\nsignal processing algorithms. To obtain high fidelity 3D representations of\nvisual scenes a huge amount of PC data is typically acquired, which demands\nefficient compression solutions. As in 2D media formats, the final perceived PC\nquality plays an important role in the overall user experience and, thus,\nobjective metrics capable to measure the PC quality in a reliable way are\nessential. In this context, this paper proposes and evaluates a set of\nobjective quality metrics for the geometry component of PC data, which plays a\nvery important role in the final perceived quality. Based on the popular PSNR\nPC geometry quality metric, the novel improved PSNR-based metrics are proposed\nby exploiting the intrinsic PC characteristics and the rendering process that\nmust occur before visualization. The experimental results show the superiority\nof the best-proposed metrics over the state-of-the-art, obtaining an\nimprovement of up to 32% in the Pearson correlation coefficient.\n

In this paper, we propose a new statistical approach to investigate the performance of some objective quality metrics used in the literature in order to determine the most suitable quality metric for watermark transparency evaluation. To this end, we have defined a new procedure based on the ANOVA (ANalysis Of VAriance) tests and the subjective performance evaluation. Firstly, a set of selected quality metrics is statistically analyzed by means of ANOVA technique to identify the specific metric that provides the best discrimination to watermarking artifacts. So, the obtained results will answer the question: which metrics are sensitive to watermarking artifacts? Secondly, subjective tests were performed and some correlation measures between MOS (Mean Opinion Score) and each quality metric are computed. It is clear that the best quality metric is the one that provides the best consistency with subjective experiments. Results from both objective and subjective investigations were discussed to give some concluding remarks. All conclusions drawn in the paper are supported by extensive experiments in terms of used quality metrics, watermarking algorithms and image database.

Recent advances in high dynamic range (HDR) capture and display technologies have attracted a lot of interest from scientific, professional, and artistic communities. As in any technology, the evaluation of HDR systems in terms of quality of experience is essential. Subjective evaluations are time consuming and expensive, and thus objective quality assessment tools are needed as well. In this paper, we report and analyze the results of an extensive benchmarking of objective quality metrics for HDR image quality assessment. In total, 35 objective metrics were benchmarked on a database of 20 HDR contents encoded with 3 compression algorithms at 4 bit rates, leading to a total of 240 compressed HDR images, using subjective quality scores as ground truth. Performance indexes were computed to assess the accuracy, monotonicity, and consistency of the metric estimation of subjective scores. Statistical analysis was performed on the performance indexes to discriminate small differences between metrics. Results demonstrated that metrics designed for HDR content, i.e., HDR-VDP-2 and HDR-VQM, are the most reliable predictors of perceived quality. Finally, our findings suggested that the performance of most full-reference metrics can be improved by considering non-linearities of the human visual system, while further efforts are necessary to improve performance of no-reference quality metrics for HDR content.

The use of point clouds to digitally represent three-dimensional objects with both geometry and color attributes is rapidly increasing in several applications. Since storage and transmission of uncompressed point cloud data are often impractical, several lossy compression algorithms have been proposed, each exhibiting specific types of visual distortion. This creates a challenging environment for objective quality metrics, which might be effective only on a restricted number of distortion types and contexts. To evaluate the performance of the most recent objective quality metrics in predicting distortions as perceived by humans, a benchmarking study is conducted using subjective scores from observers examining models distorted with a conventional as well as a learning-based compression method, while rendering them on both a traditional flat monitor and an eye-sensing light field display. The results are then analyzed and conclusions are drawn on the correlation between recent state-of-the-art objective quality metrics and human perception on two different types of visualization devices.

The goal of speech super-resolution (SSR) or speech bandwidth expansion is to generate the missing high-frequency components for a given low-resolution speech signal. It has the potential to improve the quality of telecommunications. We propose a new method for SSR that leverages the generative adversarial networks (GANs) and a regularization method for stabilizing the GAN training. The generator network is a convolutional autoencoder with 1D convolution kernels, operating along time-axis and generating the high-frequency log-power spectra from the low-frequency log-power spectra input. We employ two recent deep neural network (DNN) based approaches to compare them with our proposed method, including both objective speech quality metrics and subjective perceptual tests. We show that our proposed method outperforms the baseline methods in terms of both objective and subjective evaluations.

This contribution presents the subjective evaluation of the compressed light field datasets obtained with four state-of- the-art codecs: two from the JPEG Pleno Light Field Verification Model and two recent methods for which codecs are publicly available. To the best of our knowledge, currently no subjective testing has been carried out to compare the performances of the four considered codecs. The evaluation methodology is based on Bradley-Terry scores, obtained from pairwise comparisons of the four codecs at four target bit-rates, for four light field datasets. The subset of pairs for which the comparisons are performed is selected according to the square design method, under two design variants, resulting in two datasets of subjective results. The analysis of the collected data, obtained by ranking the subjective scores of the codecs at various bitrates, shows high correlation with the available objective quality metrics.

Color-to-gray (C2G) image conversion is the process of transforming a color image into a grayscale one. Despite its wide usage in real-world applications, little work has been dedicated to compare the performance of C2G conversion algorithms. Subjective evaluation is reliable but is also inconvenient and time consuming. Here, we make one of the first attempts to develop an objective quality model that automatically predicts the perceived quality of C2G converted images. Inspired by the philosophy of the structural similarity index, we propose a C2G structural similarity (C2G-SSIM) index, which evaluates the luminance, contrast, and structure similarities between the reference color image and the C2G converted image. The three components are then combined depending on image type to yield an overall quality measure. Experimental results show that the proposed C2G-SSIM index has close agreement with subjective rankings and significantly outperforms existing objective quality metrics for C2G conversion. To explore the potentials of C2G-SSIM, we further demonstrate its use in two applications: 1) automatic parameter tuning for C2G conversion algorithms and 2) adaptive fusion of C2G converted images.

Omnidirectional image and video have gained popularity thanks to availability of capture and display devices for this type of content. Recent studies have assessed performance of objective metrics in predicting visual quality of omnidirectional content. These metrics, however, have not been rigorously validated by comparing their prediction results with ground-truth subjective scores. In this paper, we present a set of 360-degree images along with their subjective quality ratings. The set is composed of four contents represented in two geometric projections and compressed with three different codecs at four different bitrates. A range of objective quality metrics for each stimulus is then computed and compared to subjective scores. Statistical analysis is performed in order to assess performance of each objective quality metric in predicting subjective visual quality as perceived by human observers. Results show the estimated performance of the state-of-the-art objective metrics for omnidirectional visual content. Objective metrics specifically designed for 360-degree content do not outperform conventional methods designed for 2D images.

With the increasing complexity of system-on-chip (SoC) design, intellectual property (IP)-based design flow is becoming an inevitable trend to achieve high performance and short time-to-market. However the transfer of IP core remains a complex and time-consuming process. Different IP vendors usually use different design rules and tool flows, which make it difficult for IP integrators to select suitable IP cores and integrate them rapidly. In this paper, a common platform for IP qualification (CPIPQ) is presented to provide and establish a set of common standards to qualify Soft/Hard IP. There are two kinds of qualification methods: subjective quality metric and objective quality metric. We use IEEE QIP Metric to evaluate the subjective quality metrics of IP cores. Objective metrics are measured automatically by using commercial EDA tools, as well as tools that developed by ASTRI IC Design Group.

The main goal of hiding data is to conceal the very existence of the hidden information, therefore there is a significant demand for steganographic approaches that can ensure imperceptibility of such infromation. However, there is a limited corresponding evaluation parameters available. Most of the studies use the Peak Signal to Noise Ratio (PSNR) as a metric for imperceptibility evaluation, although it could provide less accurate results than the Human Visual System (HVS) evaluation. This paper provides a review of the existent evaluation metrics that are used to assess the quality of steganography. The examination of the correlation between the existing objective and subjective metrics is also conducted. Pixel differences metrics have a poor correlation with the subjective metrics, hence the HSV based metrics have better correlation than pixel metrics.

Point cloud geometry (PCG) is used to precisely represent arbitrary-shaped 3D objects and scenes, is of great interest to vast applications which puts forward the pressing desire of high-efficiency PCG compression for transmission and storage. Existing PCG coding mostly relies on the octree model by which point-wise processing is applied without exploring nonlocal regional geometry similarity across the entire 3D surface. This work, instead, suggests the region-wise processing to leverage the region similarity to exploit inter-region redundancy for efficient lossy point cloud geometry compression. Towards this goal, a given PCG is first segmented into numerous local regions each of which comprises a portion of point cloud surface, and can be represented by a surface vector that describes the geometry shape numerically in a projected principal space. Subsequently, these regions are grouped into several discriminative clusters, assuring that inter-cluster similarity is minimized and intra-cluster similarity is maximized simultaneously, where the similarity is calculated using the regional surface vectors. In each cluster, we set a reference region having the largest similarity score to the others, which enables the non-reference region prediction from the reference one using alignment transform. In the end, we encode the reference regions directly using the lossless mode of the Geometry-based Point Cloud Compression (G-PCC), while corresponding non-reference regions are signaled using associated transform parameters. Compared with the state-of-the-art G-PCC using octree model, our region-wise approach can offer remarkable coding efficiency improvement, e.g., 32.4% and 22.0% Bjontegaard-delta rate (BD-Rate) gains for respective point-to-point ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$D1$ </tex-math></inline-formula> ) and point-to-plane ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$D2$ </tex-math></inline-formula> ) distortion evaluations, across a variety of common test sequences used in standard committee.

Abstract. Training LiDAR point clouds object detectors requires a significant amount of annotated data, which is time-consuming and effort-demanding. Although weakly supervised 3D LiDAR-based methods have been proposed to reduce the annotation cost, their performance could be further improved. In this work, we propose a weakly supervised LiDAR-based point clouds vehicle detector that does not require any labels for the proposal generation stage and needs only a few labels for the refinement stage. It comprises two primary modules. The first is an unsupervised proposal generation module based on the geometry of point clouds. The second is the pseudo-label refinement module. We validate our method on two point clouds based object detection datasets, namely KITTI and ONCE, and compare it with various existing weakly supervised point clouds object detection methods. The experimental results demonstrate the method’s effectiveness with a small amount of labeled LiDAR point clouds.

The Video Quality Experts Group (VQEG) is preparing a programme of subjective multimedia quality tests. The results from these tests will be used to evaluate the performance of competing objective multimedia quality metrics. The reliability of the subjective test data is of great importance for VQEG's task. This paper provides an overview of VQEG's multimedia ad-hoc group. The work of this group will require subjective tests to be performed by laboratories located in Europe, Asia and North America. For VQEG's multimedia work to be successful, the subjective assessment methodology must be precisely defined and produce reliable and repeatable subjective quality data. Although international standards covering multimedia quality assessment methods are in force, there remains some uncertainty regarding the most effective approach to assessing the subjective quality of multimedia. A review of existing methods is provided. Two experiments are presented investigating the suitability of alternative subjective assessment methods (single-stimulus ACR and SAMVIQ). The results of these experiments are discussed within the context of the VQEG multimedia testing programme.

This paper presents the details of the SRIB-LEAP submission to the ConferencingSpeech challenge 2021. The challenge involved the task of multi-channel speech enhancement to improve the quality of far field speech from microphone arrays in a video conferencing room. We propose a two stage method involving a beamformer followed by single channel enhancement. For the beamformer, we incorporated self-attention mechanism as inter-channel processing layer in the filter-and-sum network (FaSNet), an end-to-end time-domain beamforming system. The single channel speech enhancement is done in log spectral domain using convolution neural network (CNN)-long short term memory (LSTM) based architecture. We achieved improvements in objective quality metrics - perceptual evaluation of speech quality (PESQ) of 0.5 on the noisy data. On subjective quality evaluation, the proposed approach improved the mean opinion score (MOS) by an absolute measure of 0.9 over the noisy audio.

Video quality can be measured using both subjective and objective assessment methods. Subjective experiments are crucially important as they constitute a benchmark for evaluating the performance of objective quality metrics. Subjective quality assessment of television pictures has received extensive attention from experts over the past decades. On the other hand, emerging applications such as PC-based video streaming and mobile video streaming require new subjective test methodologies. Although some recent studies have compared different test methodologies and procedures, most concerned television pictures. No studies for multimedia-type video really validated the repeatability and reliability of the assessment method and the experimental procedure. This paper outlines a methodology for conducting subjective evaluation of video quality for multimedia applications in a repeatable and reliable manner across different laboratories. Using video material at low-resolution, low-bit rate and low-frame rate, the same experiment was conducted by two different laboratories, i.e. test material was identical in both experiments. Laboratory set-up was different, i.e. different computers and display panels were used, and viewing distance was not fixed. Results show that quality ratings obtained in both experiments are statistically identical. This is an important validation step for the Video Quality Experts Group, which will conduct an ambitious campaign of subjective experiments using many different test laboratories.