Listen to the Brain-Auditory Sound Source Localization in Neuromorphic Computing Architectures.

In this paper, we present a comparative study of a number of features and time-frequency signal representations for the task of joint sound event detection and localization using a state-of-the-art model based on a convolutional recurrent neural network. Experiments are performed for a dataset consisting of the recordings made using a tetrahedral microphone array. Several feature inputs specific to the task of sound event detection and sound source localization are combined and subsequently tested, with the aim to achieve joint performance of both tasks for multiple overlapping sound events using a single model based on a deep neural network architecture. Apart from providing a comprehensive comparison of various state-of-the-art acoustic features such as generalized cross-correlation, and inter-channel level and phase differences, we propose new features that have not been used for this task before such as eigenvectors of the microphone covariance matrix or sines and cosines of phase differences between the channels. Results for all combinations of input features are analyzed and discussed, followed by conclusions.

Robot audition by its own ears (microphones) is essential for natural human-robot communication and interface. Since a microphone is embedded in the head of a robot, the head-related transfer function (HRTF) plays an important role in sound source localization and extraction. Usually, from binaural input, the interaural phase difference (IPD) and interaural intensity difference (IID) are calculated, and then the direction is determined by using IPD and IID with HRTF. The problem of HRTF-based sound source localization is that a HRTF should be measured for each robot in an anechoic chamber, because it depends on the shape of robot's head; HRTF should be interpolated to manipulate a moving talker, because it is available only for discrete azimuth and elevation. To cope with these problems of HRTF, we proposed the auditory epipolar geometry as a continuous function of IPD and IID to dispense with HRTF and have developed a real-time multiple-talker tracking system. This auditory epipolar geometry, however, does not give a good approximation to IID of all range and IPD of peripheral areas. In this paper, the scattering theory in physics is employed to take into consideration the diffraction of sounds around robot's head for better approximation of IID and IPD. The resulting system shows that it is efficient for localization and extraction of sound at higher frequency and from side directions.

We tackle the problem of learning object detectors without supervision. Differently from weakly-supervised object detection, we do not assume image-level class labels. Instead, we extract a supervisory signal from audio-visual data, using the audio component to "teach" the object detector. While this problem is related to sound source localisation, it is considerably harder because the detector must classify the objects by type, enumerate each instance of the object, and do so even when the object is silent. We tackle this problem by first designing a self-supervised framework with a contrastive objective that jointly learns to classify and localise objects. Then, without using any supervision, we simply use these self-supervised labels and boxes to train an image-based object detector. With this, we outperform previous unsupervised and weakly-supervised detectors for the task of object detection and sound source localization. We also show that we can align this detector to ground-truth classes with as little as one label per pseudo-class, and show how our method can learn to detect generic objects that go beyond instruments, such as airplanes and cats.

Sound source localization is essential for machine-based environmental perception when visual methods are not applicable. Current array-based localization methods often require many microphones and are heavy in computing. This article proposes an efficient algorithm for estimating the location of a single source in 3-D based on the time difference of arrival and interaural level difference (TDOA-ILD) with an array of only three microphones. The algorithm can compute the source 3-D coordinates with a closed-form analytical solution derived from geometrical modeling and identify the actual source from the cone of the confusion area through clustering methods. This article comprehensively evaluates the algorithm’s behavior for various array sizes and source locations in a wide range of signal-to-noise ratios (SNRs) with simulative models to prove that the algorithm achieves estimation errors close to its Cramer–Rao lower bound (CRLB). Experimental evaluation demonstrates the method’s localization feasibility in outdoor open environments with off-the-shelf sensing equipment. The proposed scheme is suitable for real-time 3-D sound source localization with cost-effective hardware.

Front-back reversals (FBRs) in sound-source localization tasks due to cone-of-confusion errors on the azimuth plane occur with some regularity, and their occurrence is listener-dependent. There are fewer FBRs for wideband, high-frequency sounds than for low-frequency sounds presumably because the sources of low-frequency sounds are localized on the basis of interaural differences (interaural time and level differences), which can lead to ambiguous responses. Spectral cues can aid in determining sound-source locations for wideband, high-frequency sounds, and such spectral cues do not lead to ambiguous responses. However, to what extent spectral features might aid sound-source localization is still not known. This paper explores conditions in which the spectral profile of two-octave wide noise bands, whose sources were localized on the azimuth plane, were randomly varied. The experiment demonstrated that such spectral profile randomization increased FBRs for high-frequency noise bands, presumably because whatever spectral features are used for sound-source localization were no longer as useful for resolving FBRs, and listeners relied on interaural differences for sound-source localization, which led to response ambiguities. Additionally, head rotation decreased FBRs in all cases, even when FBRs increased due to spectral profile randomization. In all cases, the occurrence of FBRs was listener-dependent.

Conclusions: Patients, who are bilaterally supplied with active middle ear implants, perform slightly better in sound localization tasks than when unilaterally aided or unaided. Objectives: To investigate the impact of bilateral use of active middle ear implants on sound source localization in the horizontal plane in patients with a sloping moderate-to-severe hearing loss. Methods: Ten adults supplied with Med-EL Vibrant Soundbridge systems (VSB) in both ears participated in the study. Four listening conditions were tested: unaided, aided with VSB on left or right ear and on both sides. In each condition the subjects had to judge the direction of broadband noises delivered randomly across a semicircular array of 11 loudspeakers arranged in an anechoic chamber. Results: When unaided or bilaterally aided, the subjects localized on average 40% of the stimuli correct; when unilaterally aided (left or right), this rate dropped to 20–30% in either condition. Precision of sound localization was highest when bilaterally aided, i.e. the mean RMS angular error was 10°, and lowest when unilaterally aided, i.e. 15°. This is in line with bilateral hearing aid users, who show similar performance in sound localization tasks.

This paper presents a new method of zero-crossing-based estimation for sound source directions using the spatial cues such as the inter-aural time differences (ITDs) and inter-aural intensity differences (IIDs) in reverberant conditions. The difficulty of estimating sound source directions in reverberant conditions is that the ITDs and IIDs are changed according to reflected sounds. To cope with this problem, the precedence effect in which the preceding auditory signals suppress the lagging signals unless the intensity of signals is not strong enough, is considered. This psychoacoustic phenomenon is believed to play an important role for capturing the less interrupted spatial cues. From this viewpoint, the precedence effect is implemented in zero-crossing-based sound source localization. As a result, the proposed zero-crossing time difference (ZCTD) method with the precedence effect is able to provide the robust estimation of sound source directions even in severely reverberant conditions.

Head-related transfer functions of rabbits within the front horizontal plane

Interaural time differences (ITD) and interaural level differences (ILD) both signal horizontal sound source location. To achieve a unified percept of our acoustic environment, these two cues require integration. In the present study, we tested this integration of ITD and ILD with electroencephalography (EEG) by measuring the mismatch negativity (MMN). The MMN can arise in response to spatial changes and is at least partly generated in auditory cortex. In our study, we aimed at testing for an MMN in response to stimuli with counter-balanced ITD/ILD cues. To this end, we employed a roving oddball paradigm with alternating sound sequences in two types of blocks: (a) lateralized stimuli with congruently combined ITD/ILD cues and (b) midline stimuli created by counter-balanced, incongruently combined ITD/ILD cues. We observed a significant MMN peaking at about 112–128 ms after change onset for the congruent ITD/ILD cues, for both lower (0.5 kHz) and higher carrier frequency (4 kHz). More importantly, we also observed significant MMN peaking at about 129 ms for incongruently combined ITD/ILD cues, but this effect was only detectable in the lower frequency range (0.5 kHz). There were no significant differences of the MMN responses for the two types of cue combinations (congruent/incongruent). These results suggest that—at least in the lower frequency ranges (0.5 kHz)—ITD and ILD are processed independently at the level of the MMN in auditory cortex.

Acoustic images constitute an emergent data modality for multimodal scene understanding. Such images have the peculiarity to distinguish the spectral signature of sounds coming from different directions in space, thus providing richer information than the one derived from mono and binaural microphones. However, acoustic images are typically generated by cumbersome microphone arrays, which are not as widespread as ordinary microphones mounted on optical cameras. To exploit this empowered modality while using standard microphones and cameras we propose to leverage the generation of synthetic acoustic images from common audio-video data for the task of audio-visual localization. The generation of synthetic acoustic images is obtained by a novel deep architecture, based on Variational Autoencoder and U-Net models, which is trained to reconstruct the ground truth spatialized audio data collected by a microphone array, from the associated video and its corresponding monaural audio signal. Namely, the model learns how to mimic what an array of microphones can produce in the same conditions. We assess the quality of the generated synthetic acoustic images on the task of unsupervised sound source localization in a qualitative and quantitative manner, while also considering standard generation metrics. Our model is evaluated by considering both multimodal datasets containing acoustic images, used for the training, and unseen datasets containing just monaural audio signals and RGB frames, showing to reach more accurate localization results as compared to the state of the art.

Sound localization is an important aspect of auditory scene analysis, allowing listeners to group acoustic components from the same location into a single stream (Bregman 1990). Binaural pitch fusion, the fusion of different frequency tones across ears, can be thought of as one type of auditory streaming. Little is known about how sound localization cues affect binaural fusion. The goal of this study was to investigate the effects of interaural level differences (ILDs), one cue for sound localization, on binaural fusion of dichotic tones. Binaural pitch fusion was measured in adult normal-hearing (NH) listeners, using five ILD conditions: ILD = 0, 1.25, 2.5, 5, and 10 dB. Fusion ranges were measured by simultaneous presentation of reference and comparison stimuli in opposite ears, and varying the comparison stimulus to find the frequency range that fused with the reference stimulus. Preliminary results (5 NH; data collection is ongoing) show that small ILDs increase fusion ranges, while larger ILDs decrease fusion ranges. These findings suggest that ILDs can affect fusion range in NH listeners, and imply that simulated sound source location may provide a top-down grouping cue for binaural fusion. [This research was funded by a NIH-NIDCD grant R01 DC013307.]Sound localization is an important aspect of auditory scene analysis, allowing listeners to group acoustic components from the same location into a single stream (Bregman 1990). Binaural pitch fusion, the fusion of different frequency tones across ears, can be thought of as one type of auditory streaming. Little is known about how sound localization cues affect binaural fusion. The goal of this study was to investigate the effects of interaural level differences (ILDs), one cue for sound localization, on binaural fusion of dichotic tones. Binaural pitch fusion was measured in adult normal-hearing (NH) listeners, using five ILD conditions: ILD = 0, 1.25, 2.5, 5, and 10 dB. Fusion ranges were measured by simultaneous presentation of reference and comparison stimuli in opposite ears, and varying the comparison stimulus to find the frequency range that fused with the reference stimulus. Preliminary results (5 NH; data collection is ongoing) show that small ILDs increase fusion ranges, while larger ILDs decrea...

SUMMARYHearing information is very important when visual information is limited. Sound source localization has been noted as particularly important in hearing information processing. Barn owls have excellent auditory processing. We propose a hardware model based on the auditory sense mechanism of barn owls. In this paper, we consider a model of the external nucleus of the inferior colliculus (ICx), which locates the source of a sound by the interaural time difference (ITD) for localization in the horizontal direction, and the interaural level difference (ILD) for localization in the vertical direction. The results show that the detection position of an ITD model and the detection position of an ILD model can converge. In addition, the output position of the ICx model changes with the position of the sound source.

Binaural sound source localization based on weighted template matching

Efficiently solving combinatorial optimization problems (COPs) such as Max-Cut is challenging because the resources required increase exponentially with the problem size. This study proposes a hardware-friendly method for solving the Max-Cut problem by implementing a spiking neural network (SNN)-based Boltzmann machine (BM) in neuromorphic hardware systems. To implement the hardware-oriented version of the spiking Boltzmann machine (sBM), the stochastic dynamics of leaky integrate-and-fire (LIF) neurons with random walk noise are analyzed, and an innovative algorithm based on overlapping time windows is proposed. The simulation results demonstrate the effective convergence and high accuracy of the proposed method for large-scale Max-Cut problems. The proposed method is validated through successful hardware implementation on a 6-transistor/2-resistor (6T2R) neuromorphic chip with phase change memory (PCM) synapses. In addition, as an expansion of the algorithm, several annealing techniques and bias split methods are proposed to improve convergence, along with circuit design ideas for efficient evaluation of sampling convergence using cell arrays and spiking systems. Overall, the results of the proposed methods demonstrate the potential of energy-efficient and hardware-implementable approaches using SNNs to solve COPs. To the best of the author's knowledge, this is the first study to solve the Max-Cut problem using an SNN neuromorphic hardware chip.

1. Interaural level differences (ILDs), created by the head and pinna, have long been known to be the dominant acoustic cue for azimuthal localization of high-frequency tones. However, psychophysical experiments have demonstrated that human subjects can also lateralize complex high-frequency sounds on the basis of interaural time differences (ITDs) of the signal envelope. The lateral superior olive (LSO) is one of two pairs of binaural nuclei where the primary extraction of binaural cues for sound source location occurs. "IE" cells in LSO are inhibited by stimuli to the contralateral and excited by stimuli to the ipsilateral ear, and their response rate is therefore dependent on ILD. Anatomic specializations in the afferent pathways to the LSO suggest that this circuit also has a function in the detection of timing cues. We hypothesized that, besides ILD sensitivity, the IE property also conveys a sensitivity to ITDs of amplitude-modulated (AM) tones and could provide the physiological substrate for the psychophysical effect mentioned above. 2. In extracellular recordings from binaural LSO cells in barbiturate-anesthetized cats, response rate was a periodic function of ITDs of AM stimuli, i.e., all cells displayed ITD sensitivity. Binaural responses were smaller than responses to stimulation of the ipsilateral ear alone and were minimal when the envelopes in both ears were in-phase or nearly so. There was good correspondence between responses to ITDs and to dynamic interaural phase differences (IPDs), created by a difference in the envelope frequency to the two ears. Qualitatively, the responses were consistent with the outcome of an IE operation on temporally structured inputs. 3. To compare the relative importance of ILD and ITD, responses to combinations of the two cues were obtained. Despite robust ITD sensitivity in all binaural LSO cells encountered, the changes in response rate that would occur in response to naturally occurring ITDs were small in comparison with the changes expected for naturally occurring ILDs. The main limitation on ITD sensitivity was a steep decline in average discharge rate as the modulation frequency exceeded several hundred Hertz. 4. ITD sensitivity was also present to broadband stimuli, again with minimal rates occurring near 0 ITD. The sensitivity depended in a predictable fashion on the passband of filtered noise and was absent to binaurally uncorrelated noise bands. In response to clicks, ILDs interacted with ITD in a complicated fashion involving amplitude and latency effects. 5. Three low-characteristic frequency (CF) LSO cells were encountered that were IE and showed ITD sensitivity to the fine structure of low-frequency stimuli.(ABSTRACT TRUNCATED AT 400 WORDS)