An Efficient Method for No-Reference Video Quality Assessment

In this paper, we propose a general-purpose no-reference (NR) video quality assessment (VQA) metric based on the cascade combination of 2D convolutional neural network (CNN), multi-layer perceptron (MLP), and support vector regression (SVR) model. The features are extracted from both spatial and spatiotemporal domains by using a 2D CNN. These features can capture different aspects of video frames for predicting quality scores, and we take these features as inputs of MLP to obtain a few estimated quality scores on different perspectives. Finally, these estimated scores are combined as a final quality score by an SVR model. The proposed method is evaluated on the well-known LIVE Video database with other state-of-the-art and well-performing VQA metrics. And the experimental result demonstrates that our method is competitive with other full-reference and NR VQA metrics.

In recent years, significant efforts have been dedicated to advancing 3D content generation. However, existing quality assessment research predominantly focuses on evaluating Text-to-3D Content (T23DC) while ignoring Single Image-to-3D Content (SI23DC). In this paper, we establish the first Single Image-to-3D Content Quality Assessment (SI23DCQA) database to comprehensively study the perceptual quality of SI23DCs. The database contains 1500 SI23DCs, which are generated by 5 common SI23DC algorithms from 300 images including realistic images, AI generated images, and model rendered images. Afterward, we carry out a well-designed subjective experiment to collect subjective quality ratings for SI23DCs from three perspectives including overall, color, and shape. Additionally, a benchmark experiment is conducted with the state-of-the-art no reference image quality assessment (NR-IQA), no reference video quality assessment (NR-VQA), and no reference 3D quality assessment (NR-3DQA) and the experimental results show that current quality assessment methods are limited in evaluating the perceptual loss of SI23DCs. The database is released on https://github.com/ZedFu/SI23DCQA.

A general-purpose no-reference video quality assessment algorithm based on a long short-term memory (LSTM) network and a pretrained convolutional neural network (CNN) is introduced. Considering video sequences as a time series of deep features extracted with the help of a CNN, an LSTM network is trained to predict subjective quality scores. In contrast to previous methods, the resulting algorithm was trained on the recently published Konstanz Natural Video Quality Database (KoNViD-1k), which is the only publicly available database that contains sequences with authentic distortions. The results of experiments on KoNViD-1k demonstrate that the proposed method outperforms other state-of-the-art algorithms. Furthermore, these results are also confirmed using tests on the LIVE Video Quality Assessment Database, which consists of artificially distorted videos.

In video communication, the quality of video is mainly determined by bitrate in general. Moreover, the effect of video contents and their visual perception on video quality assessment (VQA) is often overlooked. However, in fact, for different videos, although the bitrates are the same, their VQA scores are still significantly different. Hence, it is assumed that the bitrate, video contents, and human visual characteristics mainly affect the VQA. Based on the above three aspects, in this paper, we designed a bitrate-based no-reference (NR) VQA metric combining the visual perception of video contents, namely, BRVPVC. In this metric, first an initial VQA model was proposed by only considering the bitrate alone. Then, the visual perception model for video contents was designed based on the texture complexity and local contrast of image, temporal information of video, and their visual perception features. Finally, two models were synthesized by adding certain weight coefficients into an overall VQA metric, namely, BRVPVC. Furthermore, ten reference videos and 150 distorted videos in the LIVE video database were used to test the metric. Moreover, based on the results of evaluating the videos in LIVE, VQEG, IRCCyN, EPFL-PoliMI, IVP, CSIQ, and Lisbon databases, the performance of BRVPVC is respectively compared with that of six full-reference (FR) metrics and ten NR VQA metrics. The results show that our VQA metric has a higher accuracy than six common FR VQA metrics and eight NR VQA metrics, and it is close to other two NR VQA metrics in accuracy. The corresponding values of Pearson linear correlation coefficient and Spearman rank order correlation coefficient reached 0.8547 and 0.8260, respectively. In addition, the computational complexity of proposed VQA metric is lower than video signal-to-noise ratio, video quality model, motion-based video integrity evaluation, spatiotemporal most apparent distortion, V-BLINDS, and V-CORNIA metrics. Moreover, the proposed metric has a better generalization property than these metrics.

With the development of Video on Demand applications due to the availability of high-speed internet access, adaptive streaming algorithms have been developing and improving. The focus is on improving user’s Quality of Experience (QoE) and taking it into account as one of the parameters for the adaptation algorithm. Users often experience changing network conditions, so the goal is to ensure stable video playback with satisfying QoE level. Although subjective Video Quality Assessment (VQA) methods provide more accurate results regarding user’s QoE, objective VQA methods cost less and are less time-consuming. In this article, nine different objective VQA methods are compared on a large set of video sequences with various spatial and temporal activities. VQA methods used in this analysis are: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), MultiScale Structural Similarity Index (MS-SSIM), Video Quality Metric (VQM), Mean Sum of Differences (DELTA), Mean Sum of Absolute Differences (MSAD), Mean Squared Error (MSE), Netflix Video Multimethod Assessment Fusion (Netflix VMAF) and Visual Signal-to-Noise Ratio (VSNR). The video sequences used for testing purposes were encoded according to H.264/AVC with twelve different target coding bitrates, at three different spatial resolutions (resulting in a total of 190 sequences). In addition to objective quality assessment, subjective quality assessment was performed for these sequences. All results acquired by objective VQA methods have been compared with subjective Mean Opinion Score (MOS) results using Pearson Linear Correlation Coefficient (PLCC). Measurement results obtained on a large set of video sequences with different spatial resolutions show that VQA methods like SSIM and VQM correlate better with MOS results compared to PSNR, SSIM, VSNR, DELTA, MSE, VMAF and MSAD. However, the PLCC results for SSIM and VQM are too low (0.7799 and 0.7734, respectively), for the usage of these methods in streaming services instead of subjective testing. These results suggest that more efficient VQA methods should be developed to be used in streaming testing procedures as well as to support the video segmentation process. Furthermore, when comparing results obtained for different spatial resolutions, it can be concluded that the quality of video sequences encoded at lower spatial resolutions in cases of lower target coding bitrate is higher compared to the quality of video sequences encoded at higher spatial resolutions at the same target coding bitrate, particularly when video sequences with higher spatial and temporal information are used.

Unmanned aerial vehicles (UAVs) are invaluable technologies concerning their remote control and monitoring capabilities. Convolutional neural networks (CNNs), known for their high pattern recognition capabilities, are appropriate for forest fire detection with UAVs. Deep convolutional neural networks show substantial performance on hardware with high processing capabilities. While these networks can be operated in unmanned aerial vehicles controlled from ground control stations equipped with GPU-supported hardware, the execution on a typical UAV’s limited computational resources necessitates the use of lightweight, small-sized networks. To overcome these impediments, this article presents a lightweight and attention-based approach for performing forest fire detection tasks using UAV vision data (images acquired by cameras mounted on UAVs). In this paper, we also present comprehensive research for different approaches such as transfer learning, deep CNNs, and lightweight CNNs. Among the experimented models, the attention-based EfficientNetB0 backboned model emerged as the most successful architecture for forest fire detection. With the test accuracy of , the F1-score of , the recall of , and the precision of have strongly reinforced the efficiency of the EfficientNetB0-based model in wildfire recognition. Moreover, the network has a less parameter size than the experimented networks. It proves the model’s suitability for wildfire detection with UAVs having limited hardware resources.

Several deep neural networks have been introduced for finger vein recognition over time, and these networks have demonstrated high levels of performance. However, most current state-of-the-art deep learning systems use networks with increasing layers and parameters, resulting in greater computational costs and complexity. This can make them impractical for real-time implementation, particularly on embedded hardware. To address these challenges, this article concentrates on developing a lightweight convolutional neural network (CNN) named FV-EffResNet for finger vein recognition, aiming to find a balance between network size, speed, and accuracy. The key improvement lies in the utilization of the proposed novel convolution block named the Efficient Residual (EffRes) block, crafted to facilitate efficient feature extraction while minimizing the parameter count. The block decomposes the convolution process, employing pointwise and depthwise convolutions with a specific rectangular dimension realized in two layers (n × 1) and (1 × m) for enhanced handling of finger vein data. The approach achieves computational efficiency through a combination of squeeze units, depthwise convolution, and a pooling strategy. The hidden layers of the network use the Swish activation function, which has been shown to enhance performance compared to conventional functions like ReLU or Leaky ReLU. Furthermore, the article adopts cyclical learning rate techniques to expedite the training process of the proposed network. The effectiveness of the proposed pipeline is demonstrated through comprehensive experiments conducted on four benchmark databases, namely FV-USM, SDUMLA, MMCBNU_600, and NUPT-FV. The experimental results reveal that the EffRes block has a remarkable impact on finger vein recognition. The proposed FV-EffResNet achieves state-of-the-art performance in both identification and verification settings, leveraging the benefits of being lightweight and incurring low computational costs.

In this study, a lightweight convolutional neural network (CNN) is employed to identify Lamb modes. The proposed approach consists of five convolutional and pooling layers, then a fully-connected layer and a sigmoid layer. In which, the first convolutional layer is a wide-scale kernel. Lamb wave responses based on froward modelling are obtained for different plate materials (aluminium, steel and titanium), different excitation frequencies (250 kHz, 500 kHz), and different excitation cycles (4-cycle, 5-cycle). 16800 Lamb wave samples labelled by ‘A0 mode’ and ‘S0 mode’ are beforehand and hosted in a database, then trained via the lightweight CNN. In validation process, the lightweight CNN reaches 100% accuracy. The performance of light-weight CNN is also compared with some popular networks. Now, the well-trained network can be used to identify Lamb mode. Some responses are stimulated by ABAQUS under different excitation signal, different propagating distance, different plate material, and the predicted results via the lightweight CNN are all right. In addition, the extensibility of the network is validated by identifying new-converted Lamb mode correctly.

As urbanization accelerates, the prevalence of fire incidents leads to significant hazards. Enhancing the accuracy of remote fire detection systems while reducing computation complexity and power consumption in edge hardware are crucial. Therefore, this paper investigates an innovative lightweight Convolutional Spiking Neural Network (CSNN) method for fire detection based on acoustics. In this model, Poisson encoder and convolution encoder strategies are considered and compared. Additionally, the study investigates the impact of observation time steps, surrogate gradient functions, and the threshold and decay rate of membrane potential on network performance. A comparison is made between the classification metrics of the traditional Convolutional Neural Network (CNN) approaches and the proposed lightweight CSNN method. To assess the generalization performance of the proposed lightweight method, publicly available datasets are merged with our experimental data for training, which results in a high accuracy of 99.02%, a precision of 99.37%, a recall of 98.75%, and an F1 score of 99.06% on the test datasets.

The quality assessment of user-generated content (UGC) videos is a challenging problem due to the absence of reference videos and their complex distortions. Traditional no-reference video quality assessment (NR-VQA) algorithms mainly target specific synthetic distortions. Less attention has been paid to authentic distortions in UGC videos, which are not distributed evenly in both the spatial and temporal domains. In this paper, we propose an end-to-end neural network model for UGC video quality assessment based on the attention mechanism. The key step in our approach is to embed the attention modules in the feature extraction network, which effectively extracts local distortion information. In addition, to exploit the temporal perception mechanism of the human visual system (HVS), the gated recurrent unit (GRU) and temporal pooling layer are integrated into the proposed model. We validate the proposed model on three public in-the-wild VQA databases: KoNViD-1k, CVD2014, and LIVE-Qualcomm. Experimental results demonstrate that the proposed method outperforms state-of-the-art NR-VQA models. The implementation of our method is released at https://github.com/qingshangithub/AB-VQA.

The key to preventing the COVID-19 is to diagnose patients quickly and accurately. Studies have shown that using Convolutional Neural Networks (CNN) to analyze chest Computed Tomography (CT) images is helpful for timely COVID-19 diagnosis. However, personal privacy issues, public chest CT data sets are relatively few, which has limited CNN's application to COVID-19 diagnosis. Also, many CNNs have complex structures and massive parameters. Even if equipped with the dedicated Graphics Processing Unit (GPU) for acceleration, it still takes a long time, which is not conductive to widespread application. To solve above problems, this paper proposes a lightweight CNN classification model based on transfer learning. Use the lightweight CNN MobileNetV2 as the backbone of the model to solve the shortage of hardware resources and computing power. In order to alleviate the problem of model overfitting caused by insufficient data set, transfer learning is used to train the model. The study first exploits the weight parameters trained on the ImageNet database to initialize the MobileNetV2 network, and then retrain the model based on the CT image data set provided by Kaggle. Experimental results on a computer equipped only with the Central Processing Unit (CPU) show that it consumes only 1.06 s on average to diagnose a chest CT image. Compared to other lightweight models, the proposed model has a higher classification accuracy and reliability while having a lightweight architecture and few parameters, which can be easily applied to computers without GPU acceleration. Code:github.com/ZhouJie-520/paper-codes.

Convolutional neural networks (CNNs) have achieved significant accuracy improvement in many intelligent applications at the cost of intensive convolution operations and massive data movements. To efficiently deploy CNNs on low power embedded platforms in real time, the depthwise separable convolution has been proposed to replace the standard convolution, especially in lightweight CNNs, which remarkably reduces the computation complexity and model size. However, it is difficult for a general convolution engine to obtain the theoretical performance improvement as the decreased data dependency of depthwise convolution significantly reduces the data reuse opportunity. To address this issue, a flexible and highperformance accelerator based on FPGA is proposed to efficiently process the inference of both large-scale and lightweight CNNs. Firstly, by sharing the activation dataflow between the depthwise convolution and pooling layers, the control logic and data bus of the two layers are reused to maximize the data utilization and minimize the logic overhead. Furthermore, these two layers can be processed either directly after standard convolutions to eliminate the external memory accesses or independently to gain better flexibility. Thirdly, a performance model is proposed to automatically explore the optimal design options of the accelerator. The proposed hardware accelerator is evaluated on Intel Arria 10 SoC FPGA and demonstrates state-of-the-art performance on both large-scale CNNs, e.g., VGG, and lightweight ones, e.g., MobileNet.

No-reference image quality assessment (NR-IQA) uses only the test image for its quality assessment, and as video is essentially comprised of image frames with additional temporal dimension, video quality assessment (VQA) requires a thorough understanding of image quality assessment metrics and models. Therefore, in order to identify features that deteriorate video quality, a fundamental analysis of spatial and temporal artifacts with respect to individual video frames needs to be performed. Existing IQA and VQA metrics are primarily for capturing few distortions and hence may not be good for all types of images and videos. In this paper, we propose an NR-IQA model by combining existing three methods (namely NIQE, BRISQUE and BLIINDS-II) using multi-linear regression. We also present a holistic no-reference video quality assessment (NR-VQA) model by exploring quantification of certain distortions like ringing, frame difference, blocking, clipping and contrast in video frames. For the proposed NR-IQA model, the results represent improved performance as compared to the state-of-the-art methods and it requires very low fraction of samples for training to provide a consistent accuracy over different training-to-testing ratios. The performance of NR-VQA model is examined using a simple neural network model to attain high value of goodness of fit.

Objective video quality assessment plays an important role in a variety of video processing applications such as video compression, transmission, visualization, and display. This paper proposes a no-reference video quality assessment model based on visual memory understanding. Inspired by the findings of neuroscience researchers, who argue there is a large overlap between the active human brain area when performing video quality assessment and saliency detection tasks, saliency maps are employed here to assist the quality assessment. To this end, we first generate the saliency maps using CLBP (Complete Local Binary Patterns) features of the residual frames. Then, a model of visual memory is created from the statistics of saliency maps. This is followed by learning the video quality from the visual memory, saliency, and frame features through a support vector regression pipeline. The experimental results on the state-of-the-art LIVE and SJTU video datasets indicate that the proposed no-reference video quality assessment algorithm is effective and performs statistically better than several other state-of-the-art approaches.

A distortion-agnostic video quality metric based on multi-scale spatio-temporal structural information