Cocktail Party Problem

Cocktail-Party Effect with Computational Auditory Scene Analysis — Preliminary Report —

It is relatively easy for a human listener to attend to a particular speaker at a cocktail party in the presence of other speakers, music and environmental sounds. To perform this task, the human listener needs to separate the target speech from a mixture of multiple concurrent sources reflected by various surfaces. This process is referred to as auditory scene analysis. While humans excel at this task using only two ears, machine separation based on two-microphone recordings has proven to be extremely challenging. By incorporating the mechanisms underlying the perception of sound by human listeners, computational auditory scene analysis (CASA) offers a new approach to sound segregation. Binaural hearing – hearing with two ears – employs the difference in sound source locations to improve sound segregation. In this chapter, we describe the principles of binaural processing and review the state-of-the-art in binaural CASA, particularly for speech segregation.

With the advent of hands free devices, speech recognition is of utmost importance but miserably fails to be perfect in a cock-tail party environment without speech separation or speech denoising. There are various techniques available for speech separation but the one technique used nowadays is non-negative matrix factorization (NMF). Non-negative matrix factorization decomposes the mixed signal into components namely the basis functions matrix and activation or weight matrix. Speech separation using NMF is successful, if the mixed signal consists of uncorrelated sources. In this paper we revisit the concept of decomposition using NMF for vowel utterances by various speakers and analyse them to identify correlation which may exist among the basis functions. This may be useful for formulating better training bases which eventually will lead to successful speech separation.

The term ‘cocktail party effect’ derives from acoustics and refers to the possibility to distinguish the voice of a particular speaker amid the noisy confusion produced by a plethora of overlapping voices and conversations. In this article I propose a conceptual elaboration of the term by considering the acoustic phenomenon in question, both literally and metaphorically, as one of the most characteristic conditions shaping contemporary collective and acoustic experience in environments overloaded with information. In the first part, I discuss the conditionsthat give rise to the cocktail party acoustic phenomenon, as they relate to particular types of social, communicative and listening practices. In the second part, I present a case study of the phenomenon based on the creation of a soundscape composition developed in conjunction with a written text, both occasioned by the political activity in the public space of the Syntagma Square in Athens during the summer of 2011.

Monaural speech separation is the process of separating the target speech from a noisy speech mixture recorded using single microphone. It can be used in wide range of applications including mobile telephony, hearing aid design and robust automatic speech and speaker recognition (ASR). Recently, researchers use computational auditory scene analysis (CASA) technique to successfully separate the target speech from the monaural noisy speech mixture. In CASA based monaural speech separation techniques, Ideal binary mask (IBM) and Ideal ratio mask (IRM) has been proposed as a computational goal to improve the speech intelligibility and speech quality. This paper reviews and reports various research works carried out using CASA techniques with IBM and IRM to improve speech intelligibility and quality. The experimental results show that CASA systems using IBM improves the speech intelligibility and using IRM improves the speech quality.

Auditory Scene Analysis: Computational Models

The sound of a busy environment, such as a city street, gives rise to a perception of numerous distinct events in a human listener--the 'auditory scene analysis' of the acoustic information. Recent advances in the understanding of this process from experimental psychoacoustics have led to several efforts to build a computer model capable of the same function. This work is known as 'computational auditory scene analysis'. The dominant approach to this problem has been as a sequence of modules, the output of one forming the input to the next. Sound is converted to its spectrum, cues are picked out, and representations of the cues are grouped into an abstract description of the initial input. This 'data-driven' approach has some specific weaknesses in comparison to the auditory system: it will interpret a given sound in the same way regardless of its context, and it cannot 'infer' the presence of a sound for which direct evidence is hidden by other components. The 'prediction-driven' approach is presented as an alternative, in which analysis is a process of reconciliation between the observed acoustic features and the predictions of an internal model of the sound-producing entities in the environment. In this way, predicted sound events will form part of the scene interpretation as long as they are consistent with the input sound, regardless of whether direct evidence is found. A blackboard-based implementation of this approach is described which analyzes dense, ambient sound examples into a vocabulary of noise clouds, transient clicks, and a correlogram-based representation of wide-band periodic energy called the weft. The system is assessed through experiments that firstly investigate subjects' perception of distinct events in ambient sound examples, and secondly collect quality judgments for sound events resynthesized by the system. Although rated as far from perfect, there was good agreement between the events detected by the model and by the listeners. In addition, the experimental procedure does not depend on special aspects of the algorithm (other than the generation of resyntheses), and is applicable to the assessment and comparison of other models of human auditory organization. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)

The segregation of concurrent speakers and other sound sources is an important aspect in improving the performance of audio technology, such as noise reduction and automatic speech recognition, ASR, in difficult acoustic conditions. This technology is relevant for applications like hearing aids, mobile audio devices, robotics, hands-free audio communication and speech-based computer interfaces. Computational auditory-scene analysis (CASA) techniques simulate aspects of processing properties of the human perceptual system using statistical signal-processing techniques to improve inferences about the causes of audio input received by the system. This study argues that CASA is a promising approach to achieve source separation and outlines several theoretical arguments to support this hypothesis. With a focus on computational binaural scene analysis, principles of CASA techniques are reviewed. Furthermore, in an experimental approach, the applicability of a recent model of binaural interaction to improve ASR performance in multi-speaker conditions with spatially separated moving speakers is explored. The binaural model provides input to a statistical inference filter that employs a priori information on possible movements of the sources in order to track the positions of the speakers. The tracks are used to adapt a beamformer that selects a specific speaker. The output of the beamformer is subsequently used for an ASR task. Compared to the unprocessed, that is, mixed, data in a two-speaker condition, the word recognition rates obtained with the enhanced signals based on binaural information were increased from 30.8 to 88.4 %, demonstrating the potential of the proposed CASA-based approach.

This paper introduces a speech separation system as a front-end processing step for automatic speech recognition (ASR). It employs computational auditory scene analysis (CASA) to separate the target speech from the interference speech. Specifically, the mixed speech is preprocessed based on auditory peripheral model. Then a pitch tracking is conducted and the dominant pitch is used as a main cue to find the target speech. Next, the time frequency (TF) units are merged into many segments. These segments are then combined into streams via CASA initial grouping. A regrouping strategy is employed to refine these streams via amplitude modulate (AM) cues, which are finally organized by the speaker recognition techniques into corresponding speakers. Finally, the output streams are reconstructed to compensate the missing data in the abovementioned processing steps by a cluster based feature reconstruction. Experimental results of ASR show that at low TMR (<-6dB) the proposed method offers significantly higher recognition accuracy. Index term: speech separation, CASA, speaker recognition, pitch tracking, units grouping, reconstruction

Although there are several factors causing “cocktail party effect” after more than half a century of research, the major one is considered to be the spatial separation of the target signal and the interferer. This paper will overview developments of the improvement of performance resulting from the directional separation of the target signal from interferers when listening in a field or through headphones. The basic assumption concerning the cocktail party effect is that there are one or more interfering sound sources in addition to the target signal source. In this situation it is important to remember the selective attention effect, which attenuates the interfering sound by concentrating the attention on a specific signal. Pitch of sound is the simplest cue for selective attention; however, spatial information can also be one. The latter half of this review discusses the effect of spatial filtering and an attention filter on the frequency domain.

The “cocktail party problem” challenges our ability to understand speech in noisy environments, which often include background music. Here, we explored the role of background music in speech-in-noise listening. Participants listened to an audiobook in familiar and unfamiliar music while tracking keywords in either speech or song lyrics. We used EEG to measure neural tracking of the audiobook. When speech was masked by music, the modeled peak latency at 50 ms (P1TRF) was prolonged compared to unmasked. Additionally, P1TRF amplitude was larger in unfamiliar background music, suggesting improved speech tracking. We observed prolonged latencies at 100 ms (N1TRF) when speech was not the attended stimulus, though only in less musical listeners. Our results suggest early neural representations of speech are enhanced with both attention and concurrent unfamiliar music, indicating familiar music is more distracting. One’s ability to perceptually filter “musical noise” at the cocktail party depends on objective musical abilities.

We present an algorithm for blindly recovering constituent source spectra from magnetic resonance (MR) chemical shift imaging (CSI) of the human brain. The algorithm, which we call constrained nonnegative matrix factorization (cNMF), does not enforce independence or sparsity, instead only requiring the source and mixing matrices to be nonnegative. It is based on the nonnegative matrix factorization (NMF) algorithm, extending it to include a constraint on the positivity of the amplitudes of the recovered spectra. This constraint enables recovery of physically meaningful spectra even in the presence of noise that causes a significant number of the observation amplitudes to be negative. We demonstrate and characterize the algorithm's performance using 31P volumetric brain data, comparing the results with two different blind source separation methods: Bayesian spectral decomposition (BSD) and nonnegative sparse coding (NNSC). We then incorporate the cNMF algorithm into a hierarchical decomposition framework, showing that it can be used to recover tissue-specific spectra given a processing hierarchy that proceeds coarse-to-fine. We demonstrate the hierarchical procedure on 1H brain data and conclude that the computational efficiency of the algorithm makes it well-suited for use in diagnostic work-up.

We are engaged in research on computational auditory scene analysis to attain sophisticated robot (computer) human interaction by recognizing auditory awareness. The objective of our research is the understanding of an arbitrary sound mixture including nonspeech sounds and music as well as voiced speech, obtained by robot's ears (or microphones embedded in the robot). The main issues are sound source localization, separation, and recognition at signal processing levels, and signal-to-symbol transformation at the interface level to symbol processing levels. The latter is critical in developmental communication and we are developing an automatic onomatopoeia recognition system. This paper overviews our activities in robot audition, in particular, active direction-pass filter (ADPF) that separates sounds originating from a specific direction by integrating sound source localization and visual processing. ADPF is implemented on three kinds of robots and demonstrates separating and recognizing three simultaneous speeches with a pair of microphones.

In this paper, the problem of single microphone source separation via Nonnegative Matrix Factorization (NMF) by exploiting video information is addressed. Respective audio and video modalities coming from a single human speech usually have similar time changes. It means that changes in one of them usually corresponds to changes in the other one. So it is expected that activation coefficient matrices of their NMF decomposition are similar. Based on this similarity, in this paper the activation coefficient matrix of the video modality is used as an initialization for audio source separation via NMF. In addition, the mentioned similarity is used for post-processing and for clustering the rows of the activation coefficient matrix which were resulted from randomly initialized NMF. Simulation results confirm the effectiveness of the proposed multimodal approaches in single microphone source separation.

In this paper, we propose a new time-frequency mask method for computational auditory scene analysis (CASA) based on convex optimization of the binary mask. In the proposed method, the pitch estimation and segment segregation in conventional CASA are completely replaced by the convex optimization of speech power. Considering the cross-correlation between the power spectra of noisy speech and noise in each of a Gammatone filterbank channel, the objective function of speech power used for convex optimization is built. The speech power is estimated by gradient descent method. Thus, the time-frequency units dominated by speech and noise are labeled by comparing the powers of noisy and estimated speech, and noise. The erroneous local masks are also removed by using the Teager energy of the estimated speech and time-frequency unit smoothing. The results from the average segmental signal-to-noise ratio improvement, HIT-False Alarm rate and subjective test show that the performance of the proposed method outperforms the reference methods.