Unsupervised Auditory Saliency Enabled Binaural Scene Analyzer for Speaker Localization and Recognition

Efficient and robust sound source recognition and localization is one of the basic techniques for humanoid robots in terms of reaction to environments. Due to the fixed sensor arrays and limited computation resources in humanoid robots, there comes challenge for sound source recognition and localization. This article proposes a sound source recognition and localization framework to realize real-time and precise sound source recognition and localization system using humanoid robots’ sensor arrays. The type of the audio is recognized according to the cross-correlation function. And steered response power-phase transform function in discrete angle space is used to search the sound source direction. The sound source recognition and localization framework presents a new multi-robots collaboration system to get the precise three-dimensional sound source position and introduces a distance weighting revision way to optimize the localization performance. Additionally, the experiment results carried out on humanoid robot NAO demonstrate that the proposed approaches can recognize and localize the sound source efficiently and robustly.

Speaker and language recognition in noisy and degraded channel conditions continue to be a challenging problem mainly due to the mismatch between clean training and noisy test conditions. In the presence of noise, the most reliable portions of the signal are the high energy regions which can be used for robust feature extraction. In this paper, we propose a front end processing scheme based on autoregressive (AR) models that represent the high energy regions with good accuracy followed by a modulation filtering process. The AR model of the spectrogram is derived using two separable time and frequency AR transforms. The first AR model (temporal AR model) of the sub-band Hilbert envelopes is derived using frequency domain linear prediction (FDLP). This is followed by a spectral AR model applied on the FDLP envelopes. The output 2-D AR model represents a low-pass modulation filtered spectrogram of the speech signal. The band-pass modulation filtered spectrograms can further be derived by dividing two AR models with different model orders (cut-off frequencies). The modulation filtered spectrograms are converted to cepstral coefficients and are used for a speaker recognition task in noisy and reverberant conditions. Various speaker recognition experiments are performed with clean and noisy versions of the NIST-2010 speaker recognition evaluation (SRE) database using the state-of-the-art speaker recognition system. In these experiments, the proposed front-end analysis provides substantial improvements (relative improvements of up to 25%) compared to baseline techniques. Furthermore, we also illustrate the generalizability of the proposed methods using language identification (LID) experiments on highly degraded high-frequency (HF) radio channels and speech recognition experiments on noisy data.

The presence of acoustic noise in audio recordings is an ongoing issue that plagues many applications. This ambient background noise is difficult to reduce due to its unpredictable nature. Many single channel noise reduction techniques exist but are limited in that they may distort the desired speech signal due to overlapping spectral content of the speech and noise. It is therefore of interest to investigate the use of multichannel noise reduction algorithms to further attenuate noise while attempting to preserve the speech signal of interest. Specifically, this thesis looks to investigate the use of microphone arrays in conjunction with multichannel noise reduction algorithms to aid aiding in speaker identification. Recording a speaker in the presence of acoustic background noise ultimately limits the performance and confidence of speaker identification algorithms. In situations where it is impossible to control the noise environment where the speech sample is taken, noise reduction algorithms must be developed and applied to clean the speech signal in order to give speaker identification software a chance at a positive identification. Due to the limitations of single channel techniques, it is of interest to see if spatial information provided by microphone arrays can be exploited to aid in speaker identification. This thesis provides an exploration of several time domain multichannel noise reduction techniques including delay sum beamforming, multi-channel Wiener filtering, and Spatial-Temporal Prediction filtering. Each algorithm is prototyped and filter performance is evaluated using various simulations and experiments. A three-dimensional noise model is developed to simulate and compare the performance of the above methods and experimental results of three data collections are presented and analyzed. The algorithms are compared and recommendations are given for the use of each technique. Finally, ideas for future work are discussed to improve performance and implementation of these multichannel algorithms. Possible applications for this technology include audio surveillance, identity verification, video chatting, conference calling and sound source localization.

This paper presents an automatic sound source localization approach based on a combination of the basic time delay estimation sub-methods namely, Time Difference of Arrival (TDOA), and Steered Response Power (SRP) methods. The TDOA method is a fast but vulnerable approach for finding the sound source location in long distances and reverberant environments and is so sensitive in noisy situations. On the other hand, the conventional SRP method is time consuming, but a successful approach to accurately find sound source location in noisy and reverberant environments. Also, another SRP-based method, SRP Phase Transform (SRP-PHAT), has been suggested for the better noise robustness and more accuracy of sound source localization. In this paper, based on the combination of TDOA and SRP based methods, two approaches were proposed for sound source localization. In the first proposed approach called Classical TDOA-SRP, the TDOA method is used to find the approximate sound source direction and then SRP based methods were used to find the accurate location of sound source in the Field of View (FOV) which is obtained by the TDOA method. In the second proposed approach which called Optimal TDOA-SRP, for more reduction of computational processing time of SRP-based methods and better noise robustness, a new criterion has been proposed for finding the effective FOV which is obtained through the TDOA method. Experiments were carried out under different conditions confirming the validity of the purposed approaches.

Sound Source Localization (SSL) is an essential function for robot audition and yields the location and number of sound sources, which are utilized for post-processes such as sound source separation. SSL for a robot in a real environment mainly requires noise-robustness, high resolution and real-time processing. A technique using microphone array processing, that is, Multiple Signal Classification based on Standard EigenValue Decomposition (SEVD-MUSIC) is commonly used for localization. We improved its robustness against noise with high power by incorporating Generalized EigenValue Decomposition (GEVD). However, GEVD-based MUSIC (GEVD-MUSIC) has mainly two issues: 1) the resolution of pre-measured Transfer Functions (TFs) determines the resolution of SSL, 2) its computational cost is expensive for real-time processing. For the first issue, we propose a TF interpolation method integrating time-domain-based and frequency-domain-based interpolation. The interpolation achieves super-resolution SSL, whose resolution is higher than that of the pre-measured TFs. For the second issue, we propose two methods, MUSIC based on Generalized Singular Value Decomposition (GSVD-MUSIC), and Hierarchical SSL (H-SSL). GSVD-MUSIC drastically reduces the computational cost while maintaining noise-robustness in localization. H-SSL also reduces the computational cost by introducing a hierarchical search algorithm instead of using greedy search in localization. These techniques are integrated into an SSL system using a robot embedded microphone array. The experimental result showed: the proposed interpolation achieved approximately 1 degree resolution although we have only TFs at 30 degree intervals, GSVD-MUSIC attained 46.4% and 40.6% of the computational cost compared to SEVD-MUSIC and GEVD-MUSIC, respectively, H-SSL reduces 59.2% computational cost in localization of a single sound source.

One interesting aspect of sound-source localization in reverberant environments is that different stimuli (e.g., speech versus noise) can elicit different spatial sensations at the same delays between the direct sound and its reflections. For example, a given delay between a single simulated direct sound and a delayed copy presented from a different location may elicit the sensation of either a single sound located at or near its source for speech while a noise stimulus under otherwise the same conditions might be perceived as a direct sound with echoes. Furthermore, presenting stimuli over headphones versus loudspeakers can further modify spatial perception for the same stimulus conditions. This modeling study will explore reasons for these two interesting differences of behavioral outcome, using simulation tools developed in Montagne and Zhou (JASA, 2018) and Pastore and Braasch (JASA, 2020).

Sound source localization with less data is a challenging task. To address this problem, a novel sound source localization method based on compressive sensing theory is proposed in this paper. Specifically, a sparsity basis is first constructed for each microphone by shifting the audio signal recorded from one reference microphone. In this manner, the microphones except the reference one are allowed to capture audio signals under the sampling rate far below the Nyquist criterion. Next, the source positions are estimated by solving an $$l_1$$ minimization based on each frame of audio signals. Finally, a fine localization scheme is presented by fusing the estimated source positions from multiple frames. The proposed method can directly determine the number of sound sources in one step and successfully estimate the source positions in noisy and reverberant environments. Experimental results demonstrate the validity of the proposed method.

Low-frequency sound source localization in enclosed space based on time reversal method

Mobile robots communicating with people would benefit from being able to detect sound sources to help localize interesting events in real-life settings. We propose using a spherical robot with four microphones to determine the spatial locations of multiple sound sources in ordinary rooms. The arrival temporal disparities from phase difference histograms are used to calculate the time differences. A precedence effect model suppresses the influence of echoes in reverberant environments. To integrate spatial cues of different microphones, we map the correlation between different microphone pairs on a 3D map corresponding to the azimuth and elevation of sound source direction. Results of experiments indicate that our proposed system provides sound source distribution very clearly and precisely, even concurrently in reverberant environments with the Echo Avoidance (EA) model.

To demonstrate the influence of an artificial audition system on speech recognition and dialogue management for a robot, this paper presents a case study involving soft coupling of ManyEars, a sound source localization, tracking and separation system, with the CSLU Dialogue Management system. Trials were conducted in a laboratory and a cafeteria. Results indicate that preprocessing of the audio signals by ManyEars improves speech recognition and dialogue management of the system, demonstrating the feasibility and the added flexibility provided by ManyEars for a robot to interact vocally with humans in a wide variety of contexts.

The process of speaker recognition in a noisy and distant environment is a difficult task as it faces numerous challenges before clean speaker speech signal reaching the microphone. While developing a deep neural network for robust functioning in extreme conditions, the selection of a perfectly compatible optimizer, loss function, and dropout is necessary. This paper presents a comparative study of the optimization process in the neural network, how loss function effectively unites in seeking the optimizer. It emphasizes on the selection of the number of input nodes, hidden layers, and time consumed by each set of selections. This study elaborates the accuracy obtained at different combinations of parameters for robust deep neural network structure. This paper is classified under speaker and speech recognition process into improving accuracy. Experiment results shows that Adam optimizer with 150 epochs offers around 95% accuracy for speaker classification under the noisy condition at different SNR values.

Introduction Researchers at The University of Melbourne are evaluating the broader outcomes of bilateral implantation in early implanted school-age children, including effects on parent and child stress (Sarant and Garrard, 2013), language (Sarant et al., 2014), psychosocial functioning, academic achievement, and everyday listening. The Speech, Spatial, and Other Qualities of Hearing Scale (SSQ) is a three-part adult questionnaire (http://www.ihr.mrc.ac.uk/products/ display/ssq; Gatehouse and Noble, 2004). The SSQ focusses mainly on hearing functions requiring the binaural system and is therefore suitable for evaluating everyday listening performance with bilateral implants. The SSQ-P is an adapted version for use with parents of children with hearing impairment (Galvin and Noble, 2013). The first section of the SSQ focusses on speech perception in different listening scenarios, such as in group conversation in a noisy environment, or a single speaker in a reverberant environment. The second section focusses on spatial hearing, in this case the direction of movement and location of sound sources, including the speaker in a group conversation. The final section focusses on other qualities of hearing, including sound segregation, listening effort, and sound and object identification.

The Ambisonic technique has been widely used in fields such as sound-field reproduction, sound source localization, and acoustic noise control. In this paper, a recently proposed Ambisonic decoding method, the partially matching projection method (PMPD), was described in detail and extended for more practical use with near-field sources and reverberant playback environments. Several experiments were conducted to evaluate the performance of the method. In the experiments, we compared performances of PMPD method with three other Ambisonic decoding methods, “All-round Ambisonic Decoding” (AllRAD) method, mode matching decoding (MMD) method and regularized mode matching decoding (RMMD) method. Different playback conditions were considered to investigate the generalization of the method, including unevenly and evenly distributed loudspeakers, as well as free-field and reverberant playback environments. In terms of reproduction errors near sweep-spot and energy of loudspeaker gains, PMPD method worked well for different signals and playback environments. Whereas the other three methods encontered ill-posed problems or large reproduction errors in certain conditions. This means that PMPD method achieves a balence between precise reproductioin near sweet-spot and energy of loudspeaker gains with ill-posed problems. PMPD method has a more robust performance than the other three compared methods for different signals and playback environments. And we conclude that PMPD method is effective to be used as an Ambisonic decoding method, especially for different playback environments.

Multiple simultaneous sound source localization (SSL) is one of the most important applications in the speech signal processing. The one-step algorithms with the advantage of low computational complexity (and low accuracy), and the two-step methods with high accuracy (and high computational complexity) are proposed for multiple SSL. In this article, a combination of one-step-based method based on the generalized eigenvalue decomposition (GEVD), and a two-step-based method based on the adaptive generalized cross-correlation (GCC) by using the phase transform/maximum likelihood (PHAT/ML) filters along with a novel T-shaped circular distributed microphone array (TCDMA) is proposed for 3D multiple simultaneous SSL. In addition, the low computational complexity advantage of the GCC algorithm is considered in combination with the high accuracy of the GEVD method by using the distributed microphone array to eliminate spatial aliasing and thus obtain more appropriate information. The proposed T-shaped circular distributed microphone array-based adaptive GEVD and GCC-PHAT/ML algorithms (TCDMA-AGGPM) is compared with hierarchical grid refinement (HiGRID), temporal extension of multiple response model of sparse Bayesian learning with spherical harmonic (SH) extension (SH-TMSBL), sound field morphological component analysis (SF-MCA), and time-frequency mixture weight Bayesian nonparametric acoustical holography beamforming (TF-MW-BNP-AHB) methods based on the mean absolute estimation error (MAEE) criteria in noisy and reverberant environments on simulated and real data. The superiority of the proposed method is presented by showing the high accuracy and low computational complexity for 3D multiple simultaneous SSL.

A sound source localization system has been developed based on a fully decentralized sensor network. Decentralization permits all nodes in a network to handle their own processing and decision-making, and as a result, reduce network congestion and the need for a centralized processor. The system consists of an array of battery operated COTS Ethernet-ready embedded systems with an attached microphone circuit. The localization solution requires groups of at least four nodes to be active within the array to return an acceptable two-dimensional result. Sensor nodes, positioned randomly over a 10 square meter area, recorded detection times of impulsive sources with microsecond resolution. In order to achieve a scalable system, nodes were organized in groups of from 4 to 10 nodes. Grouping was determined by the selecting the nodes farthest apart from each other. A designated leader of each group analyzed the sound source arrival times and calculated the sound source location based on time-differences of arrival. Experimental results show that this approach to sound source localization can achieve accuracies of about 30 centimeters. Perhaps more importantly though, it is accomplished in a decentralized manner, which can lead to a more flexible, scalable distributed sensor network.