AN EMBEDDED SALIENCY MAP ESTIMATOR SCHEME: APPLICATION TO VIDEO ENCODING

A variety of approaches have been proposed in the literature for region-of-interest (ROI) estimation. In most of them the ROI definition is highly subjective, i.e., lacks scientific evidence in supporting the claim that the areas defined as ROIs are indeed perceptually interesting. In this paper we attempt to model ROIs as the visually attended areas indicated by a saliency map in order to lower as much as possible the subjectivity of selection. For evaluation purposes we follow a ROI-based video compression setup and present comparisons with state-of-the-art algorithms in terms of perceived visual quality and video compression improvement. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

Various computational models of visual attention rely on the extraction of salient points or proto-objects, i.e., discrete units of attention, computed from bottom-up image features. In recent years, different solutions integrating top-down mechanisms were implemented, as research has shown that although eye movements initially are solely influenced by bottom-up information, after some time goal driven (high-level) processes dominate the guidance of visual attention towards regions of interest (Hwang, Higgins & Pomplun, 2009). However, even these improved modeling approaches are unlikely to generalize to a broader range of application contexts, because basic principles of visual attention, such as cognitive control, learning and expertise, have thus far not sufficiently been taken into account (Tatler, Hayhoe, Land & Ballard, 2011). In some recent work, the authors showed the functional role and representational nature of long-term memory structures for human perceptual skills and motor control. Based on these findings, the chapter extends a widely applied saliency-based model of visual attention (Walther & Koch, 2006) in two ways: first, it computes the saliency map using the cognitive visual attention approach (CVA) that shows a correspondence between regions of high saliency values and regions of visual interest indicated by participants’ eye movements (Oyekoya & Stentiford, 2004). Second, it adds an expertise-based component (Schack, 2012) to represent the influence of the quality of mental representation structures in long-term memory (LTM) and the roles of learning on the visual perception of objects, events, and motor actions.

The compressed domain video watermarking has been less expensive since complete decoding and then re-encoding is not required for watermark embedding. Handling synchronization error due to embedding is a challenging task, especially, due to motion compensation. In this paper, we propose an embedding scheme that can minimize desynchronization in the watermark extraction process at the decoder for embedding in compressed videos with better visual quality. Experimental results show the effectiveness of the proposed scheme.

This paper presents an evaluation of the latest MPEG-5 Part 2 Low Complexity Enhancement Video Coding (LCEVC) for video streaming applications using best effort protocols. LCEVC is a new video standard by MPEG, which enhances any base codec through an additional low bitrate stream, improving both video compression efficiency and and transmission. However, there is an interplay between packetization, packet loss visibility, choice of codec and video quality, which implies that prior studies with other codecs may be not as relevant. The contributions of this paper is, therefore in twofold: It evaluates the compression performance of LCEVC and then the impact of packet loss on its video quality when compared to H.264 and HEVC.The results from this evaluation suggest that, regarding compression, LCEVC outperformed its base codecs, overall in terms average encoding bitrate savings when using the constant rate factor (CRF) rate control. For example at a CRF of 19, the average encoding bitrate was reduced by 18.7% and 15.8% when compared with the base H.264 and HEVC codecs respectively. Furthermore, LCEVC produced better visual quality across the packet loss range compared to its base codecs and the quality only started to decrease once packet loss exceeded 0.8-1%, and decreases at a slower pace compared to its equivalent base codecs. This suggests that the LCEVC enhancement layer also provides error concealment. The results presented in this paper will be of interest to those considering the LCEVC standard and expected video quality in error-prone environments

Speakers adapt their speech based on both prior expectations and incoming cues about their addressees' informational needs (Kuhlen and Brennan 2010). Here, we investigate whether top-down information, such as speakers' expectations about addressees' attentiveness, and bottom-up cues, such as addressees' feedback during conversation, also influence speakers' gestures. In 39 dyads, addressees were either attentive when speakers told a joke or else distracted by a second task, while speakers expected addressees to be either attentive or distracted. Independently of adjustments in speech, both speakers' expectations and addressees' feedback shaped quantitative and qualitative aspects of gesturing. Speakers gestured more frequently when their prior expectations matched addressees' actual behavior. Moreover, speakers with attentive addressees gestured more in the periphery of gesture space when they expected addressees to be attentive. These systematic adjustments in gesturing suggest that speakers flexibly adapt to their addressees by integrating bottom-up cues available during the interaction in light of attributions made from top-down expectations. That these sources of information lead to adjustments patterning similarly in speech and gesture informs theoretical frameworks of how different modalities are deployed and coordinated in dialogue.

Top-down and bottom-up effects in pure alexia: Evidence from eye movements

Blind Image Quality Assessment (BIQA) aims to predict the image quality automatically by mimicking the perception mechanism of the human vision system (HVS). As a perception task, BIQA should consider the synergy of bottom-up and top-down information. The bottom-up and top-down information are sensory information and task-relevant prior information, respectively. Specifically, we consider that quality perception derives from the synergy of bottom-up information obtained from vision sensory and the quality-shared prior as top-down information, which is learned by HVS during daily life. Recently, with the successful application of large pre-trained vision-language models like CLIP in prompting downstream tasks, a series of prompt-based BIQA methods have been proposed to obtain top-down information and synergize it with bottom-up information for the final quality-oriented features. However, directly deploying such models for BIQA is challenging since additional annotations and well-designed prompt templates are required, and a slight change in template wording could have a huge impact on performance. To avoid these challenges, we proposed a novel end-to-end method, dubbed PPD-IQA, which learns the top-down information from a proxy classification task without a pre-trained vision-language model by prototype learning. The learned top-down information is referred to as a group of quality prototypes, representing centers of samples with similar quality perception. Prompted by the quality prototypes, we can disentangle the bottom-up tokens extracted by a pre-trained Vision Transformer (ViT) into quality-shared and content-specific features in the perception space, which generate final quality predictions synergistically. The experimental results demonstrate that our method can achieve competitive performance on both synthetic and authentic IQA datasets without any additional annotations and well-designed prompt templates, which is more practical in real-world applications.

Deep learning methods have achieved impressive performance in compressed video quality enhancement tasks. However, these methods rely excessively on practical experience by manually designing the network structure and do not fully exploit the potential of the feature information contained in the video sequences, i.e., not taking full advantage of the multiscale similarity of the compressed artifact information and not seriously considering the impact of the partition boundaries in the compressed video on the overall video quality. In this article, we propose a novel Mixed Difference Equation inspired Transformer (MDEformer) for compressed video quality enhancement, which provides a relatively reliable principle to guide the network design and yields a new insight into the interpretable transformer. Specifically, drawing on the graphical concept of the mixed difference equation (MDE), we utilize multiple cross-layer cross-attention aggregation (CCA) modules to establish long-range dependencies between encoders and decoders of the transformer, where partition boundary smoothing (PBS) modules are inserted as feedforward networks. The CCA module can make full use of the multiscale similarity of compression artifacts to effectively remove compression artifacts, and recover the texture and detail information of the frame. The PBS module leverages the sensitivity of smoothing convolution to partition boundaries to eliminate the impact of partition boundaries on the quality of compressed video and improve its overall quality, while not having too much impacts on non-boundary pixels. Extensive experiments on the MFQE 2.0 dataset demonstrate that the proposed MDEformer can eliminate compression artifacts for improving the quality of the compressed video, and surpasses the state-of-the-arts (SOTAs) in terms of both objective metrics and visual quality.

In the latest years, videoconferencing has taken a fundamental role in interpersonal relations, both for personal and business purposes. Lossy video compression algorithms are the enabling technology for videoconferencing, as they reduce the bandwidth required for real-time video streaming. However, lossy video compression decreases the perceived visual quality. Thus, many techniques for reducing compression artifacts and improving video visual quality have been proposed in recent years. In this work, we propose a novel GAN-based method for compression artifacts reduction in videoconferencing. Given that, in this context, the speaker is typically in front of the camera and remains the same for the entire duration of the transmission, we can maintain a set of reference keyframes of the person from the higher-quality I-frames that are transmitted within the video stream and exploit them to guide the visual quality improvement; a novel aspect of this approach is the update policy that maintains and updates a compact and effective set of reference keyframes. First, we extract multi-scale features from the compressed and reference frames. Then, our architecture combines these features in a progressive manner according to facial landmarks. This allows the restoration of the high-frequency details lost after the video compression. Experiments show that the proposed approach improves visual quality and generates photo-realistic results even with high compression rates. Code and pre-trained networks are publicly available at https://github.com/LorenzoAgnolucci/Keyframes-GAN.

In modern world more image and video compression technologies improving gradually. But major barrier of the new technologies concepts are repeated from authors. Because authors can’t find in depth of the papers from various technologies. So Survey of Video Compression Techniques very helpful for these types problems in video compression areas. Video compression techniques such as DCT coding, Quantization, Entropy coding, Motion estimation are widely used in video compression techniques. The main focus of this paper is to analyse video compression techniques required for video processing especially to discover how much amount of data to compressed, which techniques is faster and visual quality better and so on. We evaluate the video compression techniques for finding compression ratio in terms of performance, speed and accuracy. We also compare video compression techniques with conventional methods.

Solving the video compression problem requires a multi-faceted approach, balancing quality, efficiency, and computational demands. By leveraging advancements in technology and adapting to the evolving needs of video applications, it is possible to develop compression methods that meet the challenges of the present and future digital landscape. To address these objectives, machine learning and AI approaches can be utilized to predict and remove redundancies more effectively, optimizing compression algorithms dynamically based on content. Still, state-of-the art neural network-based video compression models need large and diverse datasets to generalize well across different types of video content. Wavelets can provide both time (spatial) and frequency localization, making them highly effective for video compression. This dual localization allows wavelet transforms to handle both rapid changes in video content and slow-moving scenes efficiently, leading to better compression ratios. Yet, some wavelet coefficients may be more critical for maintaining visual quality than others. Inaccurate quantization can lead to noticeable degradation. For the first time, the suggested model combine Quantum Wavelet Transform (QWT) and Neural Networks (NN) for video compression. This fusion model aims to achieve higher compression ratios, maintain video quality, and reduce computational complexity by utilizing QWT’s efficient data representation and NN’s powerful pattern recognition and predictive capabilities. Quantum bits (qubits) can encode large amounts of information in their quantum states, enabling more efficient data representation. This is especially useful for encoding large video files. Furthermore, quantum entanglement allows for correlated data representation across qubits, which can be exploited to capture intricate details and redundancies in video data more effectively than classical methods. The experimental results reveal that QWT achieves a compression ratio of almost twice that of traditional WT for the same video, maintaining superior visual quality due to more efficient redundancy elimination.

The scheduling algorithm is one of the most important portions of embedded operating systems especially for real-time embedded operating systems. The performance of scheduling algorithm influences the performance of the whole system. Real-time embedded operating system needs better response time for real-time process; it is more rigid on response time for hard real-time embedded operating system. A high-performance scheduling algorithm is required in order to make embedded operating system more efficient. Most of the mainstream scheduling algorithms are able to ensure response time for most processes, but for specific real-time applications for embedded system, a specialized real-time scheduling algorithm is required to ensure real-time response. In this paper, we propose a novel real-time scheduling algorithm for real-time embedded system. This algorithm divides all tasks into different sets and schedules the tasks according to the features of the set.

In human visual saliency, top-down and bottom-up information are combined as a basis of visual attention. Recently, deep Convolutional Neural Networks (CNN) have demonstrated strong performance on RGB salient object detection, providing an effective mechanism for combining top-down semantic information with low level features. Although depth information has been shown to be important for human perception of salient objects, the use of top-down information and the exploration of CNNs for RGB-D salient object detection remains limited. Here we propose a novel deep CNN architecture for RGB-D salient object detection that utilizes both top-down and bottom-up cues. In order to produce such an architecture, we present novel depth features that capture the ideas of background enclosure, depth contrast and histogram distance in a manner that is suitable for a learned approach. We show improved results compared to state-of-the-art RGB-D salient object detection methods. We also show that the low-level and mid-level depth features both contribute to improvements in results. In particular, the F-Score of our method is 0.848 on RGBD1000, which is 10.7% better than the current best.

Active and selective perception seeks regions of interest in an image in order to reduce the computational complexity associated with time-consuming processes such as object recognition. We describe in this paper a visual attention system that extracts regions of interest by integrating multiple image cues. Bottom-up cues are detected by decomposing the image into a number: of feature and conspicuity maps, while a-priori knowledge (i.e. models) about objects is used to generate top-down attention cues. Bottom-up and top-down information is combined through a non-linear relaxation process using energy minimization-like procedures. The functionality of the attention system is expanded by the introduction of an alerting (motion-based) system able to explore and avoid obstacles. Experimental results are reported, using cluttered and noisy scenes.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

It is possible to improve the features supported by devices with embedded systems by increasing the processor computing power, but this always results in higher costs, complexity, and power consumption. An interesting alternative is to use the growing networking infrastructures to do remote processing and visualization, with the embedded system mainly responsible for communications and user interaction. This enables devices to behave as if much more “intelligent” to users, at very low costs and power. In this article we explain how compression can make some of these solutions more bandwidth-efficient, enabling devices to simply decompress very rich graphical information and user interfaces that had been rendered elsewhere. The mixture of natural images and video with text, graphics, and animations simultaneously in the same frame is called compound video. We present a new method for compression of compound images and video, which is able to efficiently identify the different components during compression, and use an appropriate coding method. Our system uses lossless compression for graphics and text, and, on natural images and highly detailed parts, it uses lossy compression with dynamically varying quality. Since it was designed for embedded systems with very limited resources, and it has small executable size, and low complexity for classification, compression and decompression. Other compression methods (e.g., MPEG) can do the same, but are very inefficient for compound content. High-level graphics languages can be bandwidth-efficient, but are much less reliable (e.g., supporting Asian fonts), and are many orders of magnitude more complex. Numerical tests show the very significant gains in compression achieved by these systems.