Monaural speech separation system based on optimum soft mask

Monaural speech separation is the process of separating the target speech from a noisy speech mixture recorded using single microphone. It can be used in wide range of applications including mobile telephony, hearing aid design and robust automatic speech and speaker recognition (ASR). Recently, researchers use computational auditory scene analysis (CASA) technique to successfully separate the target speech from the monaural noisy speech mixture. In CASA based monaural speech separation techniques, Ideal binary mask (IBM) and Ideal ratio mask (IRM) has been proposed as a computational goal to improve the speech intelligibility and speech quality. This paper reviews and reports various research works carried out using CASA techniques with IBM and IRM to improve speech intelligibility and quality. The experimental results show that CASA systems using IBM improves the speech intelligibility and using IRM improves the speech quality.

Binary masks are essential elements to be used in monaural speech segregation and hearing aids. The performances of the ideal binary masks in terms of signal to noise ratio were evaluated in this article, and a method to predict it before application was proposed. Within the framework of the computational auditory scene analysis (CASA), Ideal binary mask (IBM) has the optimum performance in time-frequency (T-F) units. It can be used as an object goal in global level, which was confirmed by the experiments on a speech mixture database. Furthermore, energy distribution of the target and interfere signals were used together to estimate the performance of IBM in mixture separation.

Time-frequency masking is a method for noise reduction that is based on the time-frequency representation of a speech in noise signal. Depending on the estimated signal-to-noise ratio (SNR), each time-frequency unit is either attenuated or not. A special type of a time-frequency mask is the ideal binary mask (IBM), which has access to the real SNR (ideal). The IBM either retains or removes each time-frequency unit (binary mask). The IBM provides large improvements in speech intelligibility and is a valuable tool for investigating how different factors influence intelligibility. This study extends the standard outcome measure (speech intelligibility) with additional perceptual measures relevant for noise reduction: listening effort, noise annoyance, speech naturalness, and overall preference. Four types of time-frequency masking were evaluated: the original IBM, a tempered version of the IBM (called ITM) which applies limited and non-binary attenuation, and non-ideal masking (also tempered) with two different types of noise-estimation algorithms. The results from ideal masking imply that there is a trade-off between intelligibility and sound quality, which depends on the attenuation strength. Additionally, the results for non-ideal masking suggest that subjective measures can show effects of noise reduction even if noise reduction does not lead to differences in intelligibility.

Monaural speech separation is a very challenging problem. Recent studies utilize supervised learning methods to estimate the ideal binary mask (IBM) to solve the problem. In a supervised learning framework, the issue of generalization to conditions different from those used in training is paramount. This paper describes methods that require only a small training corpus but can generalize to unseen conditions. The system utilizes support vector machines to learn classification cues and then employs a rethresholding method to estimate the IBM. A distribution fitting method is used to address unseen signal-to-noise ratio conditions and an iterative voice activity detection is used to address unseen noise conditions. Systematic evaluations show that the proposed approach produces high quality IBM estimates under unseen conditions.

In this paper, we propose a novel algorithm for the separation of convolutive speech mixtures using two-microphone recordings, based on the combination of independent component analysis (ICA) and ideal binary mask (IBM), together with a post-filtering process in the cepstral domain. Essentially, the proposed algorithm consists of three steps. First, a constrained convolutive ICA algorithm is applied to separate the source signals from two-microphone recordings. In the second step, we estimate the IBM by comparing the energy of corresponding time-frequency (T-F) units from the separated sources obtained with the convolutive ICA algorithm. The last step is to reduce musical noise caused typically by T-F masking using cepstral smoothing. The performance of the proposed approach is evaluated based on both reverberant mixtures generated using a simulated room model and real recordings. The proposed algorithm offers considerably higher efficiency, together with improved speech quality while producing similar separation performance as compared with a recent approach.

Previous work has shown that a supervised-learning algorithm estimating the ideal binary mask (IBM) can improve sentence intelligibility in noise for hearing-impaired (HI) listeners from scores below 30% to above 80% [Healy et al., J. Acoust. Soc. Am. 134 (2013)]. The algorithm generates a binary mask by using a deep neural network to classify speech-dominant and noise-dominant time-frequency units. In the current study, these results are extended to consonant recognition, in order to examine the specific speech cues responsible for the observed performance improvements. Consonant recognition in speech-shaped noise or babble was examined in normal-hearing and HI listeners in three conditions: unprocessed, noise removed via the IBM, and noise removed via the classification-based algorithm. The IBM demonstrated substantial performance improvements, averaging up to 45% points. The algorithm also produced sizeable gains, averaging up to 34% points. An information-transmission analysis of cues associated with manner of articulation, place of articulation, and voicing indicated general similarity in the cues transmitted by the IBM versus the algorithm. However, important differences were observed, which may guide the future refinement of the algorithm. [Work supported by NIH.]

In this paper, we considered the problem of the speech enhancement similar to the real-world environments where several complex noise sources simultaneously degrade the quality and intelligibility of a target speech. The existing literature on the speech enhancement principally focuses on the presence of one noise source in mixture signals. However, in real-world situations, we generally face and attempt to improve the quality and intelligibility of speech where various complex stationary and nonstationary noise sources are simultaneously mixed with the target speech. Here, we have used deep learning for speech enhancement in complex-noisy environments and used ideal binary mask (IBM) as a binary classification function by using deep neural networks (DNNs). IBM is used as a target function during training and the trained DNNs are used to estimate IBM during enhancement stage. The estimated target function is then applied to the complex-noisy mixtures to obtain the target speech. The mean square error (MSE) is used as an objective cost function at various epochs. The experimental results at different input signal-to-noise ratio (SNR) showed that DNN-based complex-noisy speech enhancement outperformed the competing methods in terms of speech quality by using perceptual evaluation of speech quality (PESQ), segmental signal-to-noise ratio (SNRSeg), log-likelihood ratio (LLR), weighted spectral slope (WSS). Moreover, short-time objective intelligibility (STOI) reinforced the better speech intelligibility.

This paper introduces a speech separation system as a front-end processing step for automatic speech recognition (ASR). It employs computational auditory scene analysis (CASA) to separate the target speech from the interference speech. Specifically, the mixed speech is preprocessed based on auditory peripheral model. Then a pitch tracking is conducted and the dominant pitch is used as a main cue to find the target speech. Next, the time frequency (TF) units are merged into many segments. These segments are then combined into streams via CASA initial grouping. A regrouping strategy is employed to refine these streams via amplitude modulate (AM) cues, which are finally organized by the speaker recognition techniques into corresponding speakers. Finally, the output streams are reconstructed to compensate the missing data in the abovementioned processing steps by a cluster based feature reconstruction. Experimental results of ASR show that at low TMR (<-6dB) the proposed method offers significantly higher recognition accuracy. Index term: speech separation, CASA, speaker recognition, pitch tracking, units grouping, reconstruction

Degradation of speech signal due to adverse conditions is the major challenge for automatic speech recognition (ASR) systems. This paper introduces a novel approach to estimate an Ideal Neighbourhood Mask (INM) for speech segregation based on deep neural networks estimator. The method described here is based on the local binary patterns (LBP) technique often used in digital image processing. Ideal Neighbourhood Mask will indicate which time-frequency (T-F) units of the noisy speech are canceled. The performance assessment of the proposed application in conjunction with the traditional mask techniques, i.e., Ideal Binary Mask (IBM) and Ideal Ratio Mask (IRM), are carried out under various environments regarding the objective speech quality measures. The recognition experiments including results in the AURORA IV framework indicate that the proposed scheme, when applied in adverse environments yield significantly better performance than the conventional techniques.

In this paper, we propose a new time-frequency mask method for computational auditory scene analysis (CASA) based on convex optimization of the binary mask. In the proposed method, the pitch estimation and segment segregation in conventional CASA are completely replaced by the convex optimization of speech power. Considering the cross-correlation between the power spectra of noisy speech and noise in each of a Gammatone filterbank channel, the objective function of speech power used for convex optimization is built. The speech power is estimated by gradient descent method. Thus, the time-frequency units dominated by speech and noise are labeled by comparing the powers of noisy and estimated speech, and noise. The erroneous local masks are also removed by using the Teager energy of the estimated speech and time-frequency unit smoothing. The results from the average segmental signal-to-noise ratio improvement, HIT-False Alarm rate and subjective test show that the performance of the proposed method outperforms the reference methods.

When a target-speech/masker mixture is processed with the signal-separation technique, ideal binary mask (IBM), intelligibility of target speech is remarkably improved in both normal-hearing listeners and hearing-impaired listeners. Intelligibility of speech can also be improved by filling in speech gaps with un-modulated broadband noise. This study investigated whether intelligibility of target speech in the IBM-treated target-speech/masker mixture can be further improved by adding a broadband-noise background. The results of this study show that following the IBM manipulation, which remarkably released target speech from speech-spectrum noise, foreign-speech, or native-speech masking (experiment 1), adding a broadband-noise background with the signal-to-noise ratio no less than 4 dB significantly improved intelligibility of target speech when the masker was either noise (experiment 2) or speech (experiment 3). The results suggest that since adding the noise background shallows the areas of silence in the time-frequency domain of the IBM-treated target-speech/masker mixture, the abruption of transient changes in the mixture is smoothed and the perceived continuity of target-speech components becomes enhanced, leading to improved target-speech intelligibility. The findings are useful for advancing computational auditory scene analysis, hearing-aid/cochlear-implant designs, and understanding of speech perception under "cocktail-party" conditions.

An approach to segregate the target speech form the mixture utterance in low signal noise ratio (SNR) was proposed. Within the framework of computational auditory scene analysis (CASA), phase was the cue for segregation, and short time Fourier transforms (STFT) was used to extract the phase of the signal. Binary masking was used to group the target speech units based on the difference of phase between the mixture, clean speech and noise. The threshold of the binary masks was not linear. It adapted with the frequency change, and obtained from pretest. Experiments illustrated that the improvement of signal to noise ratio was more than 20dB in babble, m109, white and machinegun noise in -30dB to -20dB. The waveform of the result signal shown it remained most detail of the original signal, and had a well intelligibility. Phase is a robust cue in monaural speech segregation.Keywordsspeech segregationsignal phasecomputational auditory scene analysis (CASA)short time Fourier transforms (STFT)

Auditory Scene Analysis: Computational Models

Binary masking represents a powerful tool for increasing speech intelligibility in noise. An essential aspect involves the local criterion (LC), which defines the signal-to-noise ratio below which time-frequency units are discarded. But binary masking is a victim of its own success in one regard—it produces ceiling sentence intelligibility across a broad range of LC values, making the exact optimal LC value difficult to determine. Further, the optimal value for hearing-impaired (HI) listeners is largely unknown. In the current study, the optimal LC was determined in normal-hearing (NH) and HI listeners using speech materials less likely to produce ceiling effects. The CID W22 words were mixed with noise consisting of recordings from a busy hospital cafeteria, then subjected to ideal binary masking. LC values ranged from -20 to + 5 dB relative to the overall SNR of -8 dB. NH subjects were tested at 65 dBA and HI subjects were tested at 65 dBA plus NAL-RP hearing-aid gains. Preliminary results suggest that the optimal LC is similar for NH and HI listeners. Additional conditions involving different speech materials and noise types suggest that the optimal LC can vary as a function of speech and/or noise type. [Work supported by NIH.]

Cocktail-Party Effect with Computational Auditory Scene Analysis — Preliminary Report —