An Efficient Method Using the Parameterized HRTFs for 3D Audio Real-Time Rendering on Mobile Devices

이 연구는 중앙면 상에서 주성분 분석법을 이용하여 시간 및 주파수 영역에서 머리전달함수의 모형화 기법들을 다룬다. 시간영역의 머리전달함수, 복소수 값의 머리전달함수, 확장된 머리전달함수, 로그 크기의 머리전달함수에 기반하여 각각 주성분 분석법을 수행하여 얻은 네 가지 머리전달함수 모형들에 대해서 최소자승오차 관점에서 모형화 성능을 비교하고, 모형들간의 이론적인 관계를 살펴보는 것이 이 연구의 목적이다. 모형화에 사용되는 기저함수의 수가 동일하다면, 시간영역에서의 머리전달함수 혹은 확장된 머리전달함수에 기반한 모형이 복소수 값의 머리전달함수에 기반한 모형보다 최소자승오차 관점에서 더 효율적인 모형화 성능을 지닌다. 시간영역에서의 머리전달함수에 기반한 모형과 확장된 머리전달함수에 기반한 모형은 이론적으로 동일한 모형이며 서로 푸리에 변환 관계가 있다. 로그 크기의 머리전달함수에 기반한 모형은 다른 모형들과 모형화 성능 및 이론적인 관계를 비교할 수가 없는데, 이는 로그 크기의 머리전달함수에 기반한 모형은 머리전달함수의 크기 정보만을 로그 크기로 다루는 반면에 다른 모형들은 선형 크기로 머리전달함수의 크기와 위상정보를 모두 다루기 때문이다. This study deals with modeling of head-related transfer functions(HRTFs) using principal components analysis(PCA) in the time and frequency domains. Four PCA models based on head-related impulse responses(HRIRs), complex-valued HRTFs, augmented HRTFs, and log-magnitudes of HRTFs are investigated. The objective of this study is to compare modeling performances of the PCA models in the least-squares sense and to show the theoretical relationship between the PCA models. In terms of the number of principal components needed for modeling, the PCA model based on HRIR or augmented HRTFs showed more efficient modeling performance than the PCA model based on complex-valued HRTFs. The PCA model based on HRIRs in the time domain and that based on augmented HRTFs in the frequency domain are shown to be theoretically equivalent. Modeling performance of the PCA model based on log-magnitudes of HRTFs cannot be compared with that of other PCA models because the PCA model deals with log-scaled magnitude components only, whereas the other PCA models consider both magnitude and phase components in linear scale.

A complex-valued head-related transfer function (HRTF) can be represented as a real-valued head-related impulse response (HRIR). The interaural time and level cues of HRIRs are extracted to derive the binaural model and also to normalize each measured HRIR. Using the Karhunen-Loeve expansion, normalized HRIRs are modeled as a weighted combination of a set of basis functions in a low-dimensional subspace. The basis functions and the space samples of the weights are obtained from the measured HRIR. A simple linear interpolation algorithm is employed to obtain the modeled binaural HRIRs. The modeled HRIRs are nearly identical to the measured HRIRs from an anesthetized live cat. Typical mean-square errors and cross-correlation coefficients between the 1816 measured and modeled HRIRs are 1% and 0.99, respectively. The real-valued operations and linear interpolating in the model are very effective for speeding up the model computation in real-time implementation. This approach has made it possible to simulate real free-field signals at the two eardrums of a cat via earphones and to study the neuronal responses to such a virtual acoustic space (VAR).

We proposed a new head-related transfer function (HRTF) interpolation method based on splicing. The sound spreads from sound source to listener’s ears, the wave of head-related impulse response (HRIR) clearly indicates time delay which can be symbolized by the sample number of the first obvious peak and the signal filtered by the diffraction and reflection properties of the torso, head, and pinna. Based on this feature and the importance of the time delay in interpolation, we split the measured HRIR into three parts according to the first obvious peak of the wave and the length of the useful signal. First, we used radial basis function neural network to construct regression forecast models using spatial parameters (i.e., azimuth, elevation, and distance) and the HRIRs sample number of the first obvious peak. Then, we used the tetrahedral interpolation with barycentric weights to calculate the second part of the HRIR. Finally, we spliced the forecast information and the useful signal in terms of the sample number of the first obvious peak. For an unknown position in three-dimensional space of near-field, we interpolated its HRIR by the proposed method. The Signal-to-Deviation ratio (SDR) increased 5 dB compared with the tetrahedron interpolation in the time domain.

Head-Related Transfer Function (HRTF) plays a critical role in how the auditory system perceives spatial information. The spectral cues embedded in HRTF are vital for accurately determining the elevation of sound sources. In existing approaches, deep neural networks (DNNs) have been utilized to predict the magnitude spectra of HRTF from images of the pinna, typically employing the HRTF log-magnitude as the output during training. However, HRTF encompasses the acoustic characteristics of both the head and torso, exhibiting direction-dependent patterns that pose challenges in reconstructing its spectral cues. To address this complexity, we propose an innovative method for HRTF individualization. Our model uses Pinna-Related Transfer Function (PRTF) as the output during training, which helps alleviate the impact of sound reflections from the head and torso in the head-related impulse response (HRIR). Our experimental findings, based on an HRTF dataset, illustrate that our proposed model excels in reconstructing the first and second spectral cues. Furthermore, it outperforms previous deep learning models in terms of log spectral distortion (LSD).

Known errors exist in loudspeaker array processing techniques, often degrading source localization and timbre. The goal of the present study was to use virtual loudspeaker arrays to investigate how treatment of the interaural time delay (ITD) cue from each loudspeaker impacts these errors. Virtual loudspeaker arrays rendered over headphones using head-related impulse responses (HRIRs) allow flexible control of array size. Here, three HRIR delay treatment strategies were evaluated using minimum-phase loudspeaker HRIRs: reapplying the original HRIR delays, applying the relative ITD to the contralateral ear, or separately applying the HRIR delays prior to virtual array processing. Seven array sizes were simulated, and panning techniques were used to estimate HRIRs from 3000 directions using higher-order Ambisonics, vector-base amplitude panning, and the closest loudspeaker technique. Compared to a traditional, physical array, the prior HRIR delay treatment strategy produced similar errors with a 95% reduction in the required array size. When compared to direct spherical harmonic (SH) fitting of head-related transfer functions (HRTFs), the prior delays strategy reduced errors in reconstruction accuracy of timbral and directional psychoacoustic cues. This result suggests that delay optimization can greatly reduce the number of virtual loudspeakers required for accurate rendering of acoustic scenes without SH-based HRTF representation.

Virtual Reality (VR) applications with low-latency head tracking require high-quality spatial audio effects. However, classic VR/game sound approaches cannot properly simulate the acoustic of the real world. Current audio research is moving towards 3D spatial audio to have a more realistic simulation. In 3D spatial audio, the listener has the sensation that sound comes from a particular direction in 3D space. In other words, the listener can localize a source based on audio and have a more coherent and immersive experience when paired with visual simulation. In this new context, game engines should provide sound designers with a set of 3D spatial audio tools. The following common effects are desirable in this type of toolbox: reverberations and reflections, which can be employed in the creation of caverns or churches (places with lots of echoes); modulation, which can increase the perceived variety of a recorded sound, by slightly varying its pitch (as in the sounds of footsteps); mixing and fading volumes, which can create dramatic moments in storytelling and music reproduction. In this work, we propose a realtime audio engine to spatialize sound point sources in virtual environments. This engine is an open-source architecture that provides a basic set of audio effects and an efficient way to mix and match them. We implement 3D audio spatialization by leveraging recorded head-related impulse responses (HRIRs) and we produce special sound effects with digital signal processing (DSP) techniques. Although some powerful commercial audio SDKs for Virtual Reality are currently available (e.g. Oculus), our audio engine prototype may be a flexible option when adaptation, simplification, testing, and parameter tuning are necessary.

In this paper, a fast method for the extraction of pinna spectral notches (PSN) in the median plane of a virtual spherical microphone array is discussed. In general, PSN can be extracted from the Head Related Impulse Response (HRIR) measured by a spherical array of microphones. However, the PSN extracted herein are computationally complex and also not accurate at lower elevation angles. This work proposes a novel approach to reconstruct the HRIR using microphones over the median plane of a virtual spherical array. The virtual spherical array itself is simulated using the Fourier Bessel series (FBS). Subsequently, these HRIRs are used to extract the PSN. This method is computationally efficient since it is done over the median plane rather than over the complete sphere. On the other hand, it is also accurate due to the utilization of the Fourier Bessel series in the extraction of the PSN. Experimental results obtained on the CIPIC database indicate a high degree of resemblance to the actual pinna walls, even at the lower elevation angles. The results are motivating enough for the method to be considered for resolving elevation ambiguity in 3D audio.

Currently, to obtain maximum fidelity 3D audio, an intended listener is required to undergo time consuming measurements using highly specialized and expensive equipment. Customizable Head-Related Impulse Responses (HRIRs) would remove this limitation. This paper reports our progress in the first stage of the development of customizable HRIRs. Our approach is to develop compact functional models that could be equivalent to empirically measured HRIRs but require a much smaller number of parameters, which could eventually be derived from the anatomical characteristics of a prospective listener. For this first step, HRIRs must be decomposed into multiple delayed and scaled damped sinusoids which, in turn, reveal the parameters (delay and magnitude) necessary to create an instance of the structural model equivalent to the HRIR under analysis. Previously this type of HRIR decomposition has been accomplished through an exhaustive search of the model parameters. A new method that approaches the decomposition simultaneously in the frequency (Z) and time domains is reported here.

BackgroundIn order to present virtual sound sources via headphones spatially, head-related transfer functions (HRTFs) can be applied to audio signals. In this so-called binaural virtual acoustics, the spatial perception may be degraded if the HRTFs deviate from the true HRTFs of the listener.ObjectiveIn this study, participants wearing virtual reality (VR) headsets performed a listening test on the 3D audio perception of virtual audiovisual scenes, thus enabling us to investigate the necessity and influence of the individualization of HRTFs. Two hypotheses were investigated: first, general HRTFs lead to limitations of 3D audio perception in VR and second, the localization model for stationary localization errors is transferable to nonindividualized HRTFs in more complex environments such as VR.MethodsFor the evaluation, 39 subjects rated individualized and nonindividualized HRTFs in an audiovisual virtual scene on the basis of 5 perceptual qualities: localizability, front-back position, externalization, tone color, and realism. The VR listening experiment consisted of 2 tests: in the first test, subjects evaluated their own and the general HRTF from the Massachusetts Institute of Technology Knowles Electronics Manikin for Acoustic Research database and in the second test, their own and 2 other nonindividualized HRTFs from the Acoustics Research Institute HRTF database. For the experiment, 2 subject-specific, nonindividualized HRTFs with a minimal and maximal localization error deviation were selected according to the localization model in sagittal planes.ResultsWith the Wilcoxon signed-rank test for the first test, analysis of variance for the second test, and a sample size of 78, the results were significant in all perceptual qualities, except for the front-back position between own and minimal deviant nonindividualized HRTF (P=.06).ConclusionsBoth hypotheses have been accepted. Sounds filtered by individualized HRTFs are considered easier to localize, easier to externalize, more natural in timbre, and thus more realistic compared to sounds filtered by nonindividualized HRTFs.

The understanding and simplified modeling of the Head Related Transfer Function (HRTF) is very important for applications in virtual audio reality. However HRTF depends on many factors such as the shape of head and pinnae, torso, refraction by the body, and others. In this work we investigated the influences of a thin-elastic layered structure of a head on the HRTF. The analytical solution is applied to a six-layer model of the head. The elastic properties of layers are taken into account in the modeling. The results are compared with the results of spherical rigid model and experiments. The numerical results showed the new peaks of HRTF caused by elastic scattering and re-reflecting from layers. However this effect is very small and can be neglected at high frequency.

It is commonly believed that near-field head-related transfer functions (HRTFs) provide perceptual benefits over far-field HRTFs that enhance the plausibility of binaural rendering of nearby sound sources. However, to the best of our knowledge, no study has systematically investigated whether using near-field HRTFs actually provides a perceptually more plausible virtual acoustic environment. To assess this question, we conducted two experiments in a six-degrees-of-freedom multimodal augmented reality experience where participants had to compare non-individual anechoic binaural renderings based on either synthesized near-field HRTFs or intensity-scaled far-field HRTFs and judge which of the two rendering methods led to a more plausible representation. Participants controlled the virtual sound source position by moving a small handheld loudspeaker along a prescribed trajectory laterally and frontally near the head, which provided visual and proprioceptive cues in addition to the auditory cues. The results of both experiments show no evidence that near-field cues enhance the plausibility of non-individual binaural rendering of nearby anechoic sound sources in a dynamic multimodal virtual acoustic scene as examined in this study. These findings suggest that, at least in terms of plausibility, the additional effort of including near-field cues in binaural rendering may not always be worthwhile for virtual or augmented reality applications.

Four head-related transfer function (HRTF) data sets were compared to determine the effect of HRTF measurement methodology on the localization of spatialized auditory stimuli. Spatial audio interfaces typically require HRTF data sets to generate the spatialized auditory stimuli. HRTF measurement is accomplished using a variety of techniques that can require several nearly arbitrary decisions about methodology. The effects of these choices upon the resulting spatial audio interface are unclear. Sixteen participants completed a sound localization task that included real-world, broadband stimuli spatialized at eight locations on the horizontal plane. Four different HRTF data sets were utilized to spatialize the stimuli: two publicly available HRTF data sets and two data sets obtained using different in-house measurement systems. All HRTFs were obtained from the Knowles Electronics Mannequin for Acoustics Research. Unsigned localization error and proportion of front/back reversals did not differ significantly across HRTF data sets. Poorest accuracy was observed in locations near the medial (front/back) axis of the listener, mainly because of the relatively large proportions of reversals at these locations. This study suggests that the particular generalized HRTF data set chosen for spatialization is of minimal importance to the localizability of the resulting stimuli. This result will inform the design of many spatial audio interfaces that are based upon generalized HRTFs, including wayfaring devices, communication systems, and virtual reality systems.

The extended use of mobile multimedia devices in applications like gaming, 3D video and audio reproduction, immersive teleconferencing, or virtual and augmented reality, is demanding efficient algorithms and methodologies. All these applications require real-time spatial audio engines with the capability of dealing with intensive signal processing operations while facing a number of constraints related to computational cost, latency and energy consumption. Most mobile multimedia devices include a Graphics Processing Unit (GPU) that is primarily used to accelerate video processing tasks, providing high computational capabilities due to its inherent parallel architecture. This paper describes a scalable parallel implementation of a real-time binaural audio engine for GPU-equipped mobile devices. The engine is based on a set of head-related transfer functions (HRTFs) modelled with a parametric parallel structure, allowing efficient synthesis and interpolation while reducing the size required for HRTF data storage. Several strategies to optimize the GPU implementation are evaluated over a well-known kind of processor present in a wide range of mobile devices. In this context, we analyze both the energy consumption and real-time capabilities of the system by exploring different GPU and CPU configuration alternatives. Moreover, the implementation has been conducted using the OpenCL framework, guarantying the portability of the code.

Individualization of head-related transfer functions (HRTFs) can improve the quality of binaural applications with respect to the localization accuracy, coloration, and other aspects. Using anthropometric features (AFs) of the head, neck, and pinna for individualization is a promising approach to avoid elaborate acoustic measurements or numerical simulations. Previous studies on HRTF individualization analyzed the link between AFs and technical HRTF features. However, the perceptual relevance of specific errors might not always be clear. Hence, the effects of AFs on perceived perceptual qualities with respect to the overall difference, coloration, and localization error are directly explored. To this end, a listening test was conducted in which subjects rated differences between their own HRTF and a set of nonindividual HRTFs. Based on these data, a machine learning model was developed to predict the perceived differences using ratios of a subject's individual AFs and those of presented nonindividual AFs. Results show that perceived differences can be predicted well and the HRTFs recommended by the models provide a clear improvement over generic or randomly selected HRTFs. In addition, the most relevant AFs for the prediction of each type of error were determined. The developed models are available under a free cultural license.

Head-related transfer functions (HRTFs) describe the acoustical transmission process from a point sound source to two ears in the free field. They are vital to the researches of binaural hearing and virtual auditory display. Measurement is the most important approach to obtain HRTFs, but it is time-consuming. To accurate HRTF measurement, a computer-controlling measurement system is designed. The system consists of multiple sound sources at different elevations with a finest interval of 5° and a horizontal turntable with an azimuthal resolution of 0.1°. It is able to work in non-anechoic environment and suitable for both mannequin and human subject measurement. A practical example indicates that the system allows for measuring far-field HRTFs at 493 source directions within half an hour. By using the system, a super high directional resolution HRTF database for KEMAR mannequin is established. The database includes 3889 pairs of head-related impulse responses (HRIRs, the time domain counterpart of HRTFs) at distance of 1.0 m, elevation interval of 5° (from -45° to 90° ), and azimuthal interval of 2.5°. Each HRIR is 1024-point length at 96 kHz sampling frequency and 24-bit quantization. The database is applicable to the research on binaural hearing and virtual auditory display.