A Bayes-rule based hierarchical system for binaural sound source localization

In environments with many concurrent sound sources, it is often possible to hear out individual sounds and identify their sources. To accomplish this perceptual organization of the sound sources, listeners analyze the mixture of sounds to estimate the constituent components that correspond to one acoustic sound source. Signal components that are likely to come from the same source are grouped into an auditory object, that is segregated from the acoustic complex. The grouping of signal components into sources is based on identifying shared features of the components, like common spectral cues (the same fundamental frequency or frequency modulation), temporal cues (the same temporal envelope), or spatial cues (the same interaural differences). When signal components’ onsets fall within such a time window that they are perceived as being synchronous, the components are combined to form a new auditory object, that is monitored across time on cues that sustain their grouping. Given that spatial cues are inherent to every sound and already present at the onset, and that they enhance detection of a sound by a spatial release from masking, it is unclear whether and how much temporal perception depends on the presence of spatial cues. The research presented here addressed the contribution of the binaural hearing system, which processes a sound’s spatial cues, to the processing of its temporal cues, and the contribution of the sounds’ temporal cues to the processing of their spatial cues. In order to assess the contribution of spatial cues to temporal perception, experiments were conducted in which the temporal position of a target sound’s onset was controlled by the subjects to find its desired position relative to a regular series of reference sound onsets. Manipulation of the target’s interaural differences, its sensation level, and the target and reference sounds’ durations allowed comparison of their effects on temporal perception of the target’s onset. The main finding was that temporal perception precision degraded below a critical level, depending on the absolute target and masker levels and the difference in spatial configuration between target and masker that facilitated perception at much lower signal-to-noise ratios. The implication of a sound’s signal-to-noise ratio getting below this critical level is that, when the temporal window for grouping signal components’ onsets is enlarged due to the degraded precision, there is an increased probability for signal components to be grouped incorrectly into an auditory object, or to be prevented from grouping into a new auditory object and remain part of the acoustic complex until other evidence is gathered for the existence of their source. This may be a contributing factor to the decrease of speech intelligibility in noisy environments, where precise temporal perception of transients is required for the processing of speech signals. In order to assess the contribution of temporal cues to binaural processing, thresholds were measured for discrimination between the spatial configurations of two sounds that had identical spectral range and onset and offset times, but differed in their temporal envelope structure and had equally-sized directional information to opposite sides of the medial plane. The measured minimum interaural differences in time or level at which the sounds’ spatial configurations could successfully be discriminated represented the ability to process binaural cues, and allowed comparison between the effects of the various manipulations of bandwidths, center frequencies, and temporal envelopes to the sounds. The main findings were that discrimination based on interaural level differences was good for all tested conditions (although absolute thresholds depended on the specific conditions), and that discrimination based on interaural time differences was only possible when the temporal envelopes of the sounds were sufficiently different. The binaural system must, therefore, have been capable of processing the interaural differences within short periods of time when one of the sounds was dominant in the monaural temporal envelopes of the combined signals at each ear. Analysis of the monaural temporal envelopes influences the ability to use the binaural information to separate individual signal components, and allow tracking across time of the signals components with identical spatial cues to form a perceptual unity, the auditory object. In conditions where discrimination was not possible, the temporal envelopes of the individual sounds were too similar and such periods did not occur systematically in the temporal envelopes of the combined sounds. This finding suggests that a monaural analysis of a composite signal’s temporal envelope is a prerequisite for the binaural processing of the constituent signal components’ interaural time differences. The implication is that, if the signals’ temporal envelopes are too similar, their signal components may not be correctly grouped into separate and individually lateralized auditory objects, but remain fused as a single object with an averaged spatial image.

Many kinds of sound source localization systems have been developed for detecting a direction of sound source. They are commonly using time delay of arrival (TDOA) or interaural time difference (ITD) algorithm for sound source localization where, especially, the ITD is the difference in arrival time of a sound between two ears. It is largely changed depending on frequency components of sound even though the sound source is located in the same place. In this paper we propose a binaural sound localization system using sparse coding based ITD (SITD) and self-organizing map (SOM). The sparse coding is used for decomposing given sounds into three components: time, frequency and magnitude. Moreover we estimate the azimuth angle through the SOM. This localization system is installed in our robot that has two ears, head and body. We use PeopleBot as a body of the robot.

This review article reviews the contemporary studies of localization ability for different populations in different listening environments and provides possible future research directions. The ability to accurately localize a sound source relying on three cues (interaural time difference, interaural level difference, and spectral cues) is important for communication, learning, and safety. Confounding effects including noise and reverberation, which exist in common listening environments, mask or alter localization cues and negatively affect localization performance. Hearing loss, a common public health issue, also affects localization accuracy. Although hearing devices have been developed to provide excellent audibility of speech signals, less attention has been paid to preserving and replicating crucial localization cues. Unique challenges are faced by users of various hearing devices, including hearing aids, bone-anchored hearing instruments, and cochlear implants. Hearing aids have failed to consistently improve localization performance and, in some cases, significantly impair sound localization. Bone-conduction hearing instruments show little to no benefit for sound localization performance in most cases, although some improvement is seen in binaural users. Although cochlear implants provide great hearing benefit to individuals with severe-to-profound sensorineural hearing loss, cochlear implant users have significant difficulty localizing sound, even with two implants. However, technologies in each of these areas are advancing to reduce interference with desired sound signals and preserve localization cues to help users achieve better hearing and sound localization in real-life environments.

Barn owls perform sound localization based on analyses of interaural differences in arrival time and intensity of sound. Two kinds of neural signals representing the interaural time difference (ITD) and the interaural intensity difference (IID) are processed in parallel in anatomically separate pathway. We explain briefly the neural models of pathways of ITD and IID detection which we have already presented [1,2]. We present a neural network model of ICc ls in which the signals representing ITD and IID are first combined with each other. It is shown using our neural models how the neural map, in which ITD and IID are represented along the axes being perpendicular mutually, can be generated in ICc ls by the excitatory inputs from ICc core representing ITD and the inhibitory inputs from bilateral VLVps representing IID. We show that the firing rates of ICc ls neuron are well represented by a suppressive multiplication of ITD and IID inputs. This seems to be the origin of the multiplicative binding found in ICx by Pena and Konishi [3].

Binaural sound source localization is an important technique for speech enhancement, video conferencing, and human-robot interaction, etc. However, in realistic scenarios, the reverberation and environmental noise would degrade the precision of sound direction estimation. Therefore, reliable sound localization is essential to practical applications. To deal with these disturbances, this paper presents a novel binaural sound source localization approach based on reverberation weighting and generalized parametric mapping. First, the reverberation weighting as a preprocessing stage, is used to separately suppress the early and late reverberation, while preserving interaural cues. Then, two binaural cues, i.e., interaural time and intensity differences, are extracted from the frequency-domain representations of dereverberated binaural signals for the online localization. Their corresponding templates are established using the training data. Furthermore, the generalized parametric mapping is proposed to build a generalized parametric model for describing relationships between azimuth and binaural cues analytically. Finally, a two-step sound localization process is introduced to refine azimuth estimation based on the generalized parametric model and template matching. Experiments in both simulated and real scenarios validate that the proposed method can achieve better localization performance compared to state-of-the-art methods.

Binaural sound source localization based on weighted template matching

A novel deep learning (DL) method is proposed for binaural sound source localization with low SNR. Firstly, the binaural sound signals are decomposed into several channels by using Gammatone filter. Secondly, the 4 feature parameters of Head-related Transfer Function, interaural time difference (ITD), interaural coherence (IC), interaural level difference (ILD), and interaural phase difference (IPD) are extracted. Thirdly, ITD and IC go through a Deep Belief Network (DBN) to determine the quadrant of the sound source and reduce the positioning range. Then, ITD, IC, ILD, and IPD go through a Deep Neural Network (DNN) to obtain the azimuthal angle within 90 degrees. Experimental results show that the proposed algorithm can solve the front-back confusion, and obtain a superior performance with lower complexity and higher precision under low SNR conditions.

The sound localization of barn owl with respect to horizontal and vertical direction is made based on analysis of the interaural differences in arrival time and intensity of sound, respectively. The neural information processing of detection of interaural intensity difference (IID) is made on the neural pathway parallel with the pathway of interaural time difference (ITD) detection before both the signals arrive at the shell of central nucleus of the inferior colliculus (ICc shell). We have presented a neural model of ITD detection pathway (Inoue et al, Neurocomputing 2001, vol. 38-40, pp. 675-682). We present a neuronal model of the nucleus ventralis lemnisci lateralis pars posterior (VLVp) in which the signals of sound intensities coming from both the ears are first combined to be compared with each other. We clarify the functional role of mutual inhibitory connections between both VLVps observed experimentally in detection of IID. Outputs of IID sensitive neurons in both VLVps are projected bilaterally to ICc shell and the connections are inhibitory, whereas outputs of ITD sensitive neurons in ICc core projected to ICc shell are excitatory. It is shown using our neuronal models of VLVps, ICc core, and ICc shell that the neural map, in which ITD and IID are represented along the axes perpendicular mutually, can be generated in ICc shell by the excitatory inputs from ICc core and the inhibitory inputs from bilateral VLVps, as observed experimentally. We also show that the firing rate of ICc shell neuron are well represented by a suppressive multiplication of ITD and IID inputs.

The role of interaural differences in localization of spatially segregated sound was clarified by performing listening tests on sound localization. Four subjects were asked to distinguish two sources of white noise with various interaural time differences (ITDs) or interaural level differences (ILDs). As a result, the segregated rate reached 80% when the ITD between two white noise sources corresponded to an angular difference of 22 degrees. The segregated rate degraded to 50% when the ITD between two white noise sources corresponded to an angular difference of 7 degrees. In addition, the segregated rate reached 80% when the ILD between two white noise sources corresponded to an angular difference of 30 degrees. The segregated rate degraded to 50% when the ILD between two white noise sources corresponded to an angular difference of about 15 degrees. These results suggest that sound sources could be spatially segregated using even only ITD or ILD.

This paper presents a new method of zero-crossing-based estimation for sound source directions using the spatial cues such as the inter-aural time differences (ITDs) and inter-aural intensity differences (IIDs) in reverberant conditions. The difficulty of estimating sound source directions in reverberant conditions is that the ITDs and IIDs are changed according to reflected sounds. To cope with this problem, the precedence effect in which the preceding auditory signals suppress the lagging signals unless the intensity of signals is not strong enough, is considered. This psychoacoustic phenomenon is believed to play an important role for capturing the less interrupted spatial cues. From this viewpoint, the precedence effect is implemented in zero-crossing-based sound source localization. As a result, the proposed zero-crossing time difference (ZCTD) method with the precedence effect is able to provide the robust estimation of sound source directions even in severely reverberant conditions.

Interaural time differences (ITD) and interaural level differences (ILD) both signal horizontal sound source location. To achieve a unified percept of our acoustic environment, these two cues require integration. In the present study, we tested this integration of ITD and ILD with electroencephalography (EEG) by measuring the mismatch negativity (MMN). The MMN can arise in response to spatial changes and is at least partly generated in auditory cortex. In our study, we aimed at testing for an MMN in response to stimuli with counter-balanced ITD/ILD cues. To this end, we employed a roving oddball paradigm with alternating sound sequences in two types of blocks: (a) lateralized stimuli with congruently combined ITD/ILD cues and (b) midline stimuli created by counter-balanced, incongruently combined ITD/ILD cues. We observed a significant MMN peaking at about 112–128 ms after change onset for the congruent ITD/ILD cues, for both lower (0.5 kHz) and higher carrier frequency (4 kHz). More importantly, we also observed significant MMN peaking at about 129 ms for incongruently combined ITD/ILD cues, but this effect was only detectable in the lower frequency range (0.5 kHz). There were no significant differences of the MMN responses for the two types of cue combinations (congruent/incongruent). These results suggest that—at least in the lower frequency ranges (0.5 kHz)—ITD and ILD are processed independently at the level of the MMN in auditory cortex.

1. Interaural level differences (ILDs), created by the head and pinna, have long been known to be the dominant acoustic cue for azimuthal localization of high-frequency tones. However, psychophysical experiments have demonstrated that human subjects can also lateralize complex high-frequency sounds on the basis of interaural time differences (ITDs) of the signal envelope. The lateral superior olive (LSO) is one of two pairs of binaural nuclei where the primary extraction of binaural cues for sound source location occurs. "IE" cells in LSO are inhibited by stimuli to the contralateral and excited by stimuli to the ipsilateral ear, and their response rate is therefore dependent on ILD. Anatomic specializations in the afferent pathways to the LSO suggest that this circuit also has a function in the detection of timing cues. We hypothesized that, besides ILD sensitivity, the IE property also conveys a sensitivity to ITDs of amplitude-modulated (AM) tones and could provide the physiological substrate for the psychophysical effect mentioned above. 2. In extracellular recordings from binaural LSO cells in barbiturate-anesthetized cats, response rate was a periodic function of ITDs of AM stimuli, i.e., all cells displayed ITD sensitivity. Binaural responses were smaller than responses to stimulation of the ipsilateral ear alone and were minimal when the envelopes in both ears were in-phase or nearly so. There was good correspondence between responses to ITDs and to dynamic interaural phase differences (IPDs), created by a difference in the envelope frequency to the two ears. Qualitatively, the responses were consistent with the outcome of an IE operation on temporally structured inputs. 3. To compare the relative importance of ILD and ITD, responses to combinations of the two cues were obtained. Despite robust ITD sensitivity in all binaural LSO cells encountered, the changes in response rate that would occur in response to naturally occurring ITDs were small in comparison with the changes expected for naturally occurring ILDs. The main limitation on ITD sensitivity was a steep decline in average discharge rate as the modulation frequency exceeded several hundred Hertz. 4. ITD sensitivity was also present to broadband stimuli, again with minimal rates occurring near 0 ITD. The sensitivity depended in a predictable fashion on the passband of filtered noise and was absent to binaurally uncorrelated noise bands. In response to clicks, ILDs interacted with ITD in a complicated fashion involving amplitude and latency effects. 5. Three low-characteristic frequency (CF) LSO cells were encountered that were IE and showed ITD sensitivity to the fine structure of low-frequency stimuli.(ABSTRACT TRUNCATED AT 400 WORDS)

Sound localization cues, particularly interaural level difference (ILD) cues, are known to be affected by hearing aid processing such as wide-dynamic range compression and strong directional microphones. These distorted cues may negatively impact spatial awareness and communication in complex environments, two areas of challenge for new and experienced hearing aid users. Previously, we investigated frequency specific alterations to ILD and interaural time difference (ITD) cues using linear amplification in simulated reverberant rooms and with gaged vent sizes. In reverberation, ITD became erratic and ILD reduced; minimal effects of hearing aid venting were observed (Diedesch and Stecker, Am. Aud. Soc. 2016). Here, we applied that approach to hearing aid settings more typically encountered by clinical patients. Phonak Audeo receiver-in-the-canal (RIC) hearing aids were programmed to typical open-fit hearing impairments using clinically normal settings for compression algorithms and directional microphones. Recordings, collected using an acoustic manikin, were compared across hearing aid coupling (standard open and closed domes) in anechoic and simulated rooms. Frequency-specific ITD and ILD were quantified across coupling, room, and hearing aid settings. [Work supported by the F. V. Hunt Postdoctoral Research Fellowship, NIH R01-DC011548 (GCS), R01-DC011828 (FJG), and the VA RR&D NCRAR.]

Cats are a common model for developmental studies of the psychophysical and physiological mechanisms of sound localization. Yet, there are few studies on the development of the acoustical cues to location in cats. The magnitude of the three main cues, interaural differences in time (ITDs) and level (ILDs), and monaural spectral shape cues, vary with location in adults. However, the increasing interaural distance associated with a growing head and pinnae during development will result in cues that change continuously until maturation is complete. Here, we report measurements, in cats aged 1 week to adulthood, of the physical dimensions of the head and pinnae and the localization cues, computed from measurements of directional transfer functions. At 1 week, ILD depended little on azimuth for frequencies <6–7 kHz, maximum ITD was 175 μs, and for sources varying in elevation, a prominent spectral notch was located at higher frequencies than in the older cats. As cats develop, the spectral cues and the frequencies at which ILDs become substantial (>10 dB) shift to lower frequencies, and the maximum ITD increases to nearly 370 μs. Changes in the cues are correlated with the increasing size of the head and pinnae. [Work supported by NIDCD DC05122.]

This paper presents a new method of sound source localization based on zero-crossings generated from binaural filter-bank outputs. To detect the sound source direction, in the conventional methods the inter-aural time differences (ITDs) and the inter-aural intensity differences (IIDs) are estimated using the cross-correlations of neuronal firing rates. However, these methods require high computational complexity involved in the computation of cross-correlations and they suffer from inaccuracies in estimating the ITDs and IIDs, especially in a noisy multisource environment. In this context, we propose a method using zero-crossing and peak-amplitudes (ZCPAs) of binaural filter-bank outputs for accurate and efficient estimation of ITDs and IIDs. The proposed method is able to provide accurate estimation of the sound source direction and robustness to noise while offering significantly less computational complexity compared to cross-correlation based methods.