Dolby (Netherlands)

Audio synthesis has broad applications in multimedia. Recent advancements have made it possible to generate relevant audios from inputs describing an audio scene, such as images or texts. However, the immersiveness and expressiveness of the generation are limited. One possible problem is that existing methods solely rely on the global scene and overlook details of local sounding objects (i.e., sound sources). To address this issue, we propose a Sound Source-Aware Audio (SS2A) generator. SS2A is able to locally perceive multimodal sound sources from a scene with visual detection and cross-modality translation. It then contrastively learns a Cross-Modal Sound Source (CMSS) Manifold to semantically disambiguate each source. Finally, we attentively mix their CMSS semantics into a rich audio representation, from which a pretrained audio generator outputs the sound. To model the CMSS manifold, we curate a novel single-sound-source visual-audio dataset VGGS3 from VGGSound. We also design a Sound Source Matching Score to clearly measure localized audio relevance. With the effectiveness of explicit sound source modeling, SS2A achieves state-of-the-art performance in extensive image-to-audio tasks. We also qualitatively demonstrate SS2A's ability to achieve intuitive synthesis control by compositing vision, text, and audio conditions. Furthermore, we show that our sound source modeling can achieve competitive video-to-audio performance with a straightforward temporal aggregation mechanism.

Animation retargetting applies sparse motion description (e.g., keypoint sequences) to a character mesh to produce a semantically plausible and temporally coherent full-body mesh sequence. Existing approaches come with restrictions - they require access to template-based shape priors or artist-designed deformation rigs, suffer from limited generalization to unseen motion and/or shapes, or exhibit motion jitter. We propose Self-supervised Motion Fields (SMF), a self-supervised framework that is trained with only sparse motion representations, without requiring dataset-specific annotations, templates, or rigs. At the heart of our method are Kinetic Codes, a novel autoencoder-based sparse motion encoding, that exposes a semantically rich latent space, simplifying large-scale training. Our architecture comprises dedicated spatial and temporal gradient predictors, which are jointly trained in an end-to-end fashion. The combined network, regularized by the Kinetic Codes' latent space, has good generalization across both unseen shapes and new motions. We evaluated our method on unseen motion sampled from AMASS, D4D, Mixamo, and raw monocular video for animation transfer on various characters with varying shapes and topology. We report a new SoTA on the AMASS dataset in the context of generalization to unseen motion.

Abstract Collaborating with AI as a teammate rather than a tool is reshaping how research teams approach qualitative analysis. This paper draws on six case studies from Stby, a design research agency, to share empirical evidence into this shift. The examples reveal how AI integration accelerates analysis, alters methods, reshapes team dynamics, and requires researchers to adopt new roles. Alongside opportunities, challenges have emerged around AI overload, trust gaps, participant privacy, and the development of early‐career researchers. By experimenting with AI in both R&D and client projects, Stby fosters transparency, shared learning, and evolving workflows. These experiences highlight the practical impacts of AI on qualitative research and the cultural shifts needed to integrate it meaningfully. The paper advocates for researchers to treat AI as a collaborator, offering Stby's journey as a starting point for navigating the complexities of human‐AI collaboration in research.

Our visual system has evolved to process a dynamic world. We can rapidly summarize the average speeds and trajectories of moving objects at high accuracies (Haberman & Whitney, 2012; Watamaniuk & Duchon, 1992; Williams & Sekuler, 1984). Dynamic data visualizations can convey large amounts of information across time, such as using movements to depict changes in data values for multiple entities. Such dynamic displays demand our visual processing capacities, yet our visual system is limited when perceiving motion. When tracking multiple objects across space and time, we can typically track up to four objects, and the capacity is even lower if we also need to remember the history of the objects’ features (Horowitz et al., 2007). Several techniques have been shown to improve the processing of dynamic displays. Staging the animation to sequentially show steps in a transition and tracing object movement by displaying trajectory histories can increase processing by reducing the cognitive load (Fisher, 2010; Robertson et al., 2008). We examine the effectiveness of staging and tracing in dynamic displays. We showed participants animated line charts depicting the movements of four lines and asked them to identify the line with the highest mean and variance. We manipulated the animation to display the lines sequentially or synchronously, either with or without tracing, and compared the results to a static chart as control. Preliminary results showed that, for the mean comparison task, displaying lines sequentially without tracing is associated with increased accuracy, despite participants expressing low preferences for this animation style. For the variance comparison task, tracing is associated with higher accuracy, with the static chart outperforming most dynamic displays. Overall, the benefit of dynamic displays can be task-dependent and misaligned with human preferences. These findings help inform real-world best practices for constructing dynamic displays to harness our visual processing.

Efficiently transmitting media while maintaining or improving the quality of experience (QoE) poses a constant challenge. On the media playback side, there exists a variety of factors affecting the resulting QoE perceived by end users, such as device characteristics, viewing conditions, and individual visual acuity. Understanding the characteristics of playback-side context can provide opportunities for optimizing media delivery tailored to each user. — We present a novel approach to estimating viewers’ visible frequency ranges by creating hybrid images in a quantitatively controlled manner. The proposed approach, which can be applied to adaptive bitrate (ABR) streaming optimization, combines low- and high-frequency components of two different images providing multiscale image perception. The proposed method is designed to adjust the hybrid image size and the cutoff frequencies of lowpass and highpass filters of hybrid image synthesis according to the given frequency range under test. This method can be used to estimate the minimum video resolution for maintaining maximum QoE, in any viewing conditions and devices. — Our experimental results reveal a strong correlation between the minimum resolution obtained from hybrid image perception tasks and the traditional just noticeable difference (JND) of real images of different resolutions when presented side-by-side. These JND tasks exhibit a wider spread both in the measured minimum resolution and elapsed time to complete the tasks, potentially due to the complexity of the cognitive process and the variability caused by contents and subjects. In contrast, the hybrid image technique achieves more stable results with much less effort from subjects.

Ambient light can significantly degrade picture quality by reducing visibility of details and contrast. In many cases, it is difficult to control or eliminate ambient light, making it essential to compensate for its effect. This is especially true in the context of media and entertainment, where viewers often consume content in different lighting conditions and environments. Traditional image processing techniques, such as adjusting the backlight, brightness/contrast, or more professionally using a PLUGE signal, do not sufficiently model the human visual system. The result is that detail is lost under changes in ambient light. This is especially problematic for very dark scenes, which has become apparent with cinematic high dynamic range content where numerous consumer complaints have graced headline news. — This paper proposes a method of ambient light compensation by adaptively modeling the contrast sensitivity functions of the human visual system. By estimating global eye adaptation to both the content and the surround environment, the contrast sensitivity models may be used to maintain perceptual detail and contrast under varying ambient illumination. We build upon existing models of human vision and show how we can use cone sensitivity to maintain global contrast. Since displays have limited dynamic range, we show how this can be adaptive to both the content and environment.

SMPTE was formed in 1916 as narrative performance was ushered onto the technological stage allowing for mass distribution and transmission. While there is no need to list those advantages, there is one weakness with audiovisual media as compared to the traditional stage, which media still has not overcome. This is real-time audience feedback, and the ability to adjust the performance based on differing audience reactions. This paper motivates the use of biosensors in media by highlighting the problem of signal loss due to playback technology. A metadata system is proposed that allows creatives to steer signal modifications as a function of audience emotion and cognition as determined by biosensor analysis This is needed because today's audiovisual ecosystem includes such a wide variety of playback devices that the audience's experience can differ substantially for the same source content. Metadata for narrative and emotional expectation as inserted by creatives during the content production stages combines with the assessment to adapt the rendering. As a result, the system allows for creative intent to be scalable as best as possible across many types of playback systems in a manner analogous to real-time stage presentation.

A previous model of motion perception used in engineering display design predicted the need for a 360 FPS (frames per second) and 1080-line display and image signal format system, in order to match the viewing of reality. Now that 360 FPS gaming monitors are on the market, the model is revisited with twenty years of new studies, display technology advancements such as HDR (high dynamic range), and inclusion of new viewing tasks such as performance gaming. This revised model is then used to make new predictions for display and signal format requirements to achieve distortion-free viewing of reality and synthetic signals that do not impose limits to human performance.

Conv-TasNet is a recently proposed waveform-based deep neural network that achieves state-of-the-art performance in speech source separation. Its architecture consists of a learnable encoder/decoder and a separator that operates on top of this learned space. Various improvements have been proposed to Conv-TasNet. However, they mostly focus on the separator, leaving its encoder/decoder as a (shallow) linear operator. In this paper, we conduct an empirical study of Conv-TasNet and propose an enhancement to the encoder/decoder that is based on a (deep) non-linear variant of it. In addition, we experiment with the larger and more diverse LibriTTS dataset and investigate the generalization capabilities of the studied models when trained on a much larger dataset. We propose cross-dataset evaluation that includes assessing separations from the WSJ0-2mix, LibriTTS and VCTK databases. Our results show that enhancements to the encoder/decoder can improve average SI-SNR performance by more than 1 dB. Furthermore, we offer insights into the generalization capabilities of Conv-TasNet and the potential value of improvements to the encoder/decoder.

Abstract We provide a new proof of Hua's result that every sufficiently large integer N ≡ 5 (mod 24) can be written as the sum of the five prime squares. Hua's original proof relies on the circle method and uses results from the theory of L-functions. Here, we present a proof based on the transference principle first introduced in[5]. Using a sieve theoretic approach similar to ([10]), we do not require any results related to the distributions of zeros of L- functions. The main technical difficulty of our approach lies in proving the pseudo-randomness of the majorant of the characteristic function of the W-tricked primes which requires a precise evaluation of the occurring Gaussian sums and Jacobi symbols.