Perceptual Quality Improvement in Videoconferencing Using Keyframes-Based GAN

Deep learning methods have achieved impressive performance in compressed video quality enhancement tasks. However, these methods rely excessively on practical experience by manually designing the network structure and do not fully exploit the potential of the feature information contained in the video sequences, i.e., not taking full advantage of the multiscale similarity of the compressed artifact information and not seriously considering the impact of the partition boundaries in the compressed video on the overall video quality. In this article, we propose a novel Mixed Difference Equation inspired Transformer (MDEformer) for compressed video quality enhancement, which provides a relatively reliable principle to guide the network design and yields a new insight into the interpretable transformer. Specifically, drawing on the graphical concept of the mixed difference equation (MDE), we utilize multiple cross-layer cross-attention aggregation (CCA) modules to establish long-range dependencies between encoders and decoders of the transformer, where partition boundary smoothing (PBS) modules are inserted as feedforward networks. The CCA module can make full use of the multiscale similarity of compression artifacts to effectively remove compression artifacts, and recover the texture and detail information of the frame. The PBS module leverages the sensitivity of smoothing convolution to partition boundaries to eliminate the impact of partition boundaries on the quality of compressed video and improve its overall quality, while not having too much impacts on non-boundary pixels. Extensive experiments on the MFQE 2.0 dataset demonstrate that the proposed MDEformer can eliminate compression artifacts for improving the quality of the compressed video, and surpasses the state-of-the-arts (SOTAs) in terms of both objective metrics and visual quality.

The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.

Summary form only given. Block-based Discrete Cosine Transform (BDCT) has been widely used in image and video compression due to its energy compacting property and relative ease of implementation. However, BDCT has a major drawback, which is usually referred to as blocking artifacts. Blocking artifacts appear as grid noise along the block boundaries because each block is transformed and quantized independently. Image deblocking techniques can reduce these distortions and alleviate the conflict between bit rate reduction and visual quality preservation. Many state-of-the-art image deblocking algorithms treated the blocking artifacts reduction of the compressed image as an inverse restoration problem. Natural image prior models are well utilized into the blocking artifacts reduction processing, such as the local sparsity prior model and non-local similarity property of natural images. These two local and non-local models characterize the image prior information in two complementary perspectives. Therefore, it is necessary to combine these two models in a unified framework. In this paper, we propose a novel method to reduce the blocking artifacts of blockcoded images via patch clustering and low-rank minimization, which simultaneously exploits the local and non-local sparse representations in a unified framework. First, the whole compressed image are divided into small patches. For each patch, we perform patch clustering to collect similar patches into a group. Then the whole group are simultaneously reconstructed by a low-rank minimization approach. Singular value thresholding (SVT) algorithm is employed to solve the low-rank minimization problem. To further improve the performance of the proposed algorithm, we adopt an iterative procedure to utilize the newly output data in each iteration and update the noise and signal variance adaptively. Experimental results show that the proposed method achieves higher PSNR and SSIM than the state-of-the-art methods. Comparing to the state-of-theart algorithms and, the proposed algorithm achieves about 0.37dB and 0.11dB improvement on average. For visual quality assessment, the deblocking images produced by the proposed algorithm reveal much more sharp edge structures and richer textures.

We have seen a rise in video based user communication in the last year, unfortunately fueled by the spread of COVID-19 disease. Efficient low-latency delay of transmission of video is a challenging problem which must also deal with the segmented nature of network infrastructure not always allowing a high throughput. Lossy video compression is a basic requirement to enable such technology widely. While this may compromise the quality of the streamed video there are recent deep learning based solutions to restore quality of a lossy compressed video. Considering the very nature of video conferencing, bitrate allocation in video streaming could be driven semantically, differentiating quality between the talking subjects and the background. Currently there have not been any work studying the restoration of semantically coded video using deep learning. In this work we show how such videos can be efficiently generated by shifting bitrate with masks derived via computer vision and how a deep generative adversarial network can be trained to restore video quality. Our study shows that the combination of semantic coding and learning based video restoration can provide superior results.

The compressed domain video watermarking has been less expensive since complete decoding and then re-encoding is not required for watermark embedding. Handling synchronization error due to embedding is a challenging task, especially, due to motion compensation. In this paper, we propose an embedding scheme that can minimize desynchronization in the watermark extraction process at the decoder for embedding in compressed videos with better visual quality. Experimental results show the effectiveness of the proposed scheme.

This study aims to improve image super-resolution techniques by balancing distortion reduction with perceptual quality improvement. It introduces a new framework called Toward Improving Super-Resolution, which focuses on producing high-quality knee magnetic resonance images. The framework uses a lightweight, resolution-independent, feedforward convolutional network with 266,000 parameters, which includes smoothing and denoising preprocessing and Leaky Rectified Linear Unit activations for stable training. The model builds on a baseline deep learning architecture to improve training stability and visual quality while maintaining computational efficiency. The fast Magnetic Resonance Imaging knee dataset was compared to established super-resolution methods like Super-Resolution Convolutional Neural Network, Very Deep Super-Resolution, and Enhanced Deep Super-Resolution. Toward Improving Super-Resolution achieved a peak signal-to-noise ratio of 38.405 ± 0.129 decibels and a structural similarity index of 0.9815 ± 0.0021, surpassing Super-Resolution Convolutional Neural Network, Very Deep Super-Resolution, and Enhanced Deep Super-Resolution. It maintained high performance at scales three and four, demonstrating accuracy and statistical robustness. The study shows that the proposed framework can enhance diagnostic imaging, reduce the need for repeated scans, and speed up clinical decision-making.

DDFusion: An efficient multi-exposure fusion network with dense pyramidal convolution and de-correlation fusion

Three-dimensional (3D) point clouds have a wide range of applications in the field of 3D vision. The quality of the acquired point cloud data considerably impacts the subsequent work of point cloud processing. Due to the sparsity and irregularity of point cloud data, processing point cloud data has always been challenging. However, existing deep learning-based point cloud dense reconstruction methods suffer from excessive smoothing of reconstruction results and too many outliers. The reason for this is that it is not possible to extract features for local and global features at different scales and provide different levels of attention to different regions in order to obtain long-distance dependence for dense reconstruction. In this paper, we use a parallel multi-scale feature extraction module based on graph convolution and an upsampling method with an added multi-head attention mechanism to process sparse and irregular point cloud data to obtain extended point clouds. Specifically, a point cloud training patch with 256 points is inputted. The PMS module uses three residual connections in the multi-scale feature extraction stage. Each PMS module consists of three parallel DenseGCN modules with different size convolution kernels and different averaging pooling sizes. The local and global feature information of the augmented receptive field is extracted efficiently. The scale information is obtained by averaging the different pooled augmented receptive fields. The scale information was obtained using the different average pooled augmented receptive fields. The upsampling stage uses an upsampling rate of r=4, The self-attentive features with a different focus on different point cloud data regions obtained by fusing different weights make the feature representation more diverse. This operation avoids the bias of one attention, and each focuses on extracting valuable fine-grained feature information. Finally, the coordinate reconstruction module obtains 1024 dense point cloud data. Experiments show that the proposed method demonstrates good evaluation metrics and performance and is able to obtain better visual quality. The problems of over-smoothing and excessive outliers are effectively mitigated, and the obtained sparse point cloud is more dense.

This paper proposes a method of artifact reduction in compressed videos using dynamic texture map together with artifact maps and 3D - fuzzy filters. To preserve better details during filtering process, the authors introduce a novel method to construct a texture map for video sequences called dynamic texture map. Then, temporal arifacts such as flicker artifacts and mosquito artifacts are also estimated by advanced flicker maps and mosquito maps. These maps combined with fuzzy filters are applied to intraframe and interframe pixels to enhancecompressed videos. Simulation results verify the advanced performance of the proposed fuzzy filtering scheme in term of visual quality, SSIM, PSNR and flicker metrics in comparisionwith existing state of the art methods.

The H.264 video compression technique enjoys the merit of good video quality with high compression rate; therefore, it has been widely applied in delivering video content. In IEEE 802.11ac wireless local area networks, enhanced distributed channel access (EDCA) is the primarily adopted access control mechanism for multimedia traffic transmission. However, the performance is degraded due to the unpredictable delay caused by its inefficient backoff procedure. To achieve high quality of service/experience, this paper proposes a multipolling controlled access (MPCA) scheme to guarantee the latency in delivering important video frames while reducing transmission overhead. We also propose a cross-layer designed quality adjustment strategy to maximize the visual experience. Furthermore, the downlink multiuser multiple-input-multiple-output feature in IEEE 802.11ac is exploited to enhance the reliability of MPCA. Theoretical analysis is provided to show high efficiency of MPCA. Simulation results indicate that MPCA has higher throughput, lower packet delay, lower packet loss, and higher peak signal-to-noise ratio than the EDCA.

In modern world more image and video compression technologies improving gradually. But major barrier of the new technologies concepts are repeated from authors. Because authors can’t find in depth of the papers from various technologies. So Survey of Video Compression Techniques very helpful for these types problems in video compression areas. Video compression techniques such as DCT coding, Quantization, Entropy coding, Motion estimation are widely used in video compression techniques. The main focus of this paper is to analyse video compression techniques required for video processing especially to discover how much amount of data to compressed, which techniques is faster and visual quality better and so on. We evaluate the video compression techniques for finding compression ratio in terms of performance, speed and accuracy. We also compare video compression techniques with conventional methods.

Solving the video compression problem requires a multi-faceted approach, balancing quality, efficiency, and computational demands. By leveraging advancements in technology and adapting to the evolving needs of video applications, it is possible to develop compression methods that meet the challenges of the present and future digital landscape. To address these objectives, machine learning and AI approaches can be utilized to predict and remove redundancies more effectively, optimizing compression algorithms dynamically based on content. Still, state-of-the art neural network-based video compression models need large and diverse datasets to generalize well across different types of video content. Wavelets can provide both time (spatial) and frequency localization, making them highly effective for video compression. This dual localization allows wavelet transforms to handle both rapid changes in video content and slow-moving scenes efficiently, leading to better compression ratios. Yet, some wavelet coefficients may be more critical for maintaining visual quality than others. Inaccurate quantization can lead to noticeable degradation. For the first time, the suggested model combine Quantum Wavelet Transform (QWT) and Neural Networks (NN) for video compression. This fusion model aims to achieve higher compression ratios, maintain video quality, and reduce computational complexity by utilizing QWT’s efficient data representation and NN’s powerful pattern recognition and predictive capabilities. Quantum bits (qubits) can encode large amounts of information in their quantum states, enabling more efficient data representation. This is especially useful for encoding large video files. Furthermore, quantum entanglement allows for correlated data representation across qubits, which can be exploited to capture intricate details and redundancies in video data more effectively than classical methods. The experimental results reveal that QWT achieves a compression ratio of almost twice that of traditional WT for the same video, maintaining superior visual quality due to more efficient redundancy elimination.

In this paper we propose a novel saliency-based computational model for visual attention. This model processes both top-down (goal directed) and bottom-up information. Processing in the top-down channel creates the so called skin conspicuity map and emulates the visual search for human faces performed by humans. This is clearly a goal directed task but is generic enough to be context independent. Processing in the bottom-up information channel follows the principles set by Itti et al. but it deviates from them by computing the orientation, intensity and color conspicuity maps within a unified multi-resolution framework based on wavelet subband analysis. In particular, we apply a wavelet based approach for efficient computation of the topographic feature maps. Given that wavelets and multiresolution theory are naturally connected the usage of wavelet decomposition for mimicking the center surround process in humans is an obvious choice. However, our implementation goes further. We utilize the wavelet decomposition for inline computation of the features (such as orientation angles) that are used to create the topographic feature maps. The bottom-up topographic feature maps and the top-down skin conspicuity map are then combined through a sigmoid function to produce the final saliency map. A prototype of the proposed model was realized through the TMDSDMK642-0E DSP platform as an embedded system allowing real-time operation. For evaluation purposes, in terms of perceived visual quality and video compression improvement, a ROI-based video compression setup was followed. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

A variety of approaches have been proposed in the literature for region-of-interest (ROI) estimation. In most of them the ROI definition is highly subjective, i.e., lacks scientific evidence in supporting the claim that the areas defined as ROIs are indeed perceptually interesting. In this paper we attempt to model ROIs as the visually attended areas indicated by a saliency map in order to lower as much as possible the subjectivity of selection. For evaluation purposes we follow a ROI-based video compression setup and present comparisons with state-of-the-art algorithms in terms of perceived visual quality and video compression improvement. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

To realize the multi-focus image fusion task, an end-to-end deep convolutional neural network (DCNN) model that produces the final fused image directly from the source images is presented in this paper. In order to promote the fusion precision, the innovative multi-focus fusion DCNN introduces a multi-scale feature extraction (MFE) unit to collect more complementary features from different spatial scales and fuse them to excavate more spatial information. Moreover, a visual attention unit is designed to help the network locate the focused region more accurately and pick more useful features for perfectly splicing the details in the fusion process. Experimental results illustrate that the proposed method is superior to several existing multi-focus image fusion methods in both of the subjective visual effects and objective quality metrics.