Review of Ideal Binary and Ratio Mask Estimation Techniques for Monaural Speech Separation

The ideal binary mask (IBM) has been one of the most successful techniques in computational auditory scene analysis (CASA) algorithms. The binary value 1 is assigned to the mask if the local signal-to-noise ratio (SNR) of a particular time-frequency (T-F) units exceeds the local criterion (LC), otherwise the value 0 is assigned to the mask. This binary weighting may discard some parts of the speech during synthesis, which leads to an unnatural sound called musical noise. This paper proposes the optimum soft mask (OSM) to reduce the musical noise, by replacing the hard limiting weights (i.e., 1 or 0) with the variable weights between 0 and 1. The Signal-to-Noise ratio is used as a performance measures to compare the performance of the proposed soft mask with IBM in the context of monaural speech separation. The experimental results show the superior performance of the proposed soft mask as compared with IBM.

IRM estimation based on data field of cochleagram for speech enhancement

Degradation of speech signal due to adverse conditions is the major challenge for automatic speech recognition (ASR) systems. This paper introduces a novel approach to estimate an Ideal Neighbourhood Mask (INM) for speech segregation based on deep neural networks estimator. The method described here is based on the local binary patterns (LBP) technique often used in digital image processing. Ideal Neighbourhood Mask will indicate which time-frequency (T-F) units of the noisy speech are canceled. The performance assessment of the proposed application in conjunction with the traditional mask techniques, i.e., Ideal Binary Mask (IBM) and Ideal Ratio Mask (IRM), are carried out under various environments regarding the objective speech quality measures. The recognition experiments including results in the AURORA IV framework indicate that the proposed scheme, when applied in adverse environments yield significantly better performance than the conventional techniques.

Time-frequency masking is a method for noise reduction that is based on the time-frequency representation of a speech in noise signal. Depending on the estimated signal-to-noise ratio (SNR), each time-frequency unit is either attenuated or not. A special type of a time-frequency mask is the ideal binary mask (IBM), which has access to the real SNR (ideal). The IBM either retains or removes each time-frequency unit (binary mask). The IBM provides large improvements in speech intelligibility and is a valuable tool for investigating how different factors influence intelligibility. This study extends the standard outcome measure (speech intelligibility) with additional perceptual measures relevant for noise reduction: listening effort, noise annoyance, speech naturalness, and overall preference. Four types of time-frequency masking were evaluated: the original IBM, a tempered version of the IBM (called ITM) which applies limited and non-binary attenuation, and non-ideal masking (also tempered) with two different types of noise-estimation algorithms. The results from ideal masking imply that there is a trade-off between intelligibility and sound quality, which depends on the attenuation strength. Additionally, the results for non-ideal masking suggest that subjective measures can show effects of noise reduction even if noise reduction does not lead to differences in intelligibility.

Hearing impaired listeners using cochlear implants (CIs) suffer from a decrease in speech intelligibility (SI) in adverse listening conditions. Time-frequency masks are often applied to perform noise suppression in an attempt to increase SI. Two important masks are the so-called ideal binary mask (IBM) with its binary weights and the ideal Wiener filter (IWF) with its continuous weights. It is unclear which of the masks has the highest potential for SI and speech quality enhancement in CI users. In this study, both approaches for SI and quality enhancement were compared. The investigations were conducted in normal-hearing (NH) subjects listening to noise vocoder CI simulations and in CI users. The potential for SI improvement was assessed in a sentence recognition task with ideal mask estimates in multitalker babble and with an interfering talker. The robustness of the approaches was evaluated with simulated estimation errors. CI users assessed the speech quality in a preference rating. The IWF outperformed the IBM in NH listeners. In contrast, no significant difference was obtained in CI users. Estimation errors degraded SI in CI users for both approaches. In terms of quality, the IWF outperformed, slightly, the IBM processed signals. The outcomes of this study suggest that the mask pattern is not that crucial for CIs. Results of speech enhancement algorithms obtained with NH subjects listening to vocoded or normally processed stimuli do not translate to CI users. This outcome means that the effect of new strategies has to be quantified with the user group considered.

Auditory Scene Analysis: Computational Models

Binary masks are essential elements to be used in monaural speech segregation and hearing aids. The performances of the ideal binary masks in terms of signal to noise ratio were evaluated in this article, and a method to predict it before application was proposed. Within the framework of the computational auditory scene analysis (CASA), Ideal binary mask (IBM) has the optimum performance in time-frequency (T-F) units. It can be used as an object goal in global level, which was confirmed by the experiments on a speech mixture database. Furthermore, energy distribution of the target and interfere signals were used together to estimate the performance of IBM in mixture separation.

In this paper, we considered the problem of the speech enhancement similar to the real-world environments where several complex noise sources simultaneously degrade the quality and intelligibility of a target speech. The existing literature on the speech enhancement principally focuses on the presence of one noise source in mixture signals. However, in real-world situations, we generally face and attempt to improve the quality and intelligibility of speech where various complex stationary and nonstationary noise sources are simultaneously mixed with the target speech. Here, we have used deep learning for speech enhancement in complex-noisy environments and used ideal binary mask (IBM) as a binary classification function by using deep neural networks (DNNs). IBM is used as a target function during training and the trained DNNs are used to estimate IBM during enhancement stage. The estimated target function is then applied to the complex-noisy mixtures to obtain the target speech. The mean square error (MSE) is used as an objective cost function at various epochs. The experimental results at different input signal-to-noise ratio (SNR) showed that DNN-based complex-noisy speech enhancement outperformed the competing methods in terms of speech quality by using perceptual evaluation of speech quality (PESQ), segmental signal-to-noise ratio (SNRSeg), log-likelihood ratio (LLR), weighted spectral slope (WSS). Moreover, short-time objective intelligibility (STOI) reinforced the better speech intelligibility.

Monaural speech separation is a very challenging problem. Recent studies utilize supervised learning methods to estimate the ideal binary mask (IBM) to solve the problem. In a supervised learning framework, the issue of generalization to conditions different from those used in training is paramount. This paper describes methods that require only a small training corpus but can generalize to unseen conditions. The system utilizes support vector machines to learn classification cues and then employs a rethresholding method to estimate the IBM. A distribution fitting method is used to address unseen signal-to-noise ratio conditions and an iterative voice activity detection is used to address unseen noise conditions. Systematic evaluations show that the proposed approach produces high quality IBM estimates under unseen conditions.

Monaural speech segregation from complex concurrent noise is an extremely challenging problem; binary mask is a method to solve this problem, however, the performance of binary mask is limited by remaining the noise in the result. In this paper, an algorithm integrated Spectral Subtraction and binary masking for speech separation and enhancement was proposed. It follows the framework of computational auditory scene analysis (CASA). The energy of time-frequency (T-F) unit was used as the clue to generate the binary mask; then the spectral subtraction algorithm was used to eliminate noise energy in original speech and an interim speech was obtained, after covered the binary mask on the interim speech, the target speech can be achieved. Systematic evaluation shows that the combined algorithm can stably improve the SNR and voice quality for noisy speech. It performs better than existing binary masking systems in most situations, especially when the noise and the speech have the similar power spectrum.

Speech separation is a process of segregating the target speech from the noisy mixture. Human auditory system naturally has the capability to separate the speech from background noise; however the machine implementation of the same has been failed. Computational Auditory Scene Analysis (CASA) is a recent approach which models the human auditory system by computational means to separate the target speech from the noisy mixture. Most of the speech separation system based on CASA uses energy as one of the reliable feature to segregate the target speech from the noisy mixture. This research work proposes a new approach using adaptive energy selection threshold to segregate the target speech. The experimental result shows significant improvement in the signal to noise ratio as compared to the conventional energy threshold method.

An approach to segregate the target speech form the mixture utterance in low signal noise ratio (SNR) was proposed. Within the framework of computational auditory scene analysis (CASA), phase was the cue for segregation, and short time Fourier transforms (STFT) was used to extract the phase of the signal. Binary masking was used to group the target speech units based on the difference of phase between the mixture, clean speech and noise. The threshold of the binary masks was not linear. It adapted with the frequency change, and obtained from pretest. Experiments illustrated that the improvement of signal to noise ratio was more than 20dB in babble, m109, white and machinegun noise in -30dB to -20dB. The waveform of the result signal shown it remained most detail of the original signal, and had a well intelligibility. Phase is a robust cue in monaural speech segregation.Keywordsspeech segregationsignal phasecomputational auditory scene analysis (CASA)short time Fourier transforms (STFT)

An improved algorithm is presented for speech enhancement via sparse representation and ideal binary mask (IBM) methods. In the traditional IBM, the basic idea is to identify voiced components as target signal and label unvoiced ones as interference noise vice versa. However, such voiced and unvoiced components still cannot be well separated in target signal and interference noise. To fully exploit the merits of sparse representation theory, we extract the exact voiced component from both the above twofold to obtain the final enhanced speech. Experimental results demonstrate the proposed method can achieve higher PESQ scores than the traditional IBM to efficiently improve speech intelligibility.

Sound source separation is a hot topic in array signal processing. In fact, it is difficult to separate multiple sound sources in reverberant environments. To solve this problem, a two-stage multiple sound source separation method is proposed based on the fully-convolutional time-domain audio separation network (Conv-TasNet) and deep neural network (DNN). In the first stage, the end-to-end separation network (i.e., the Conv-TasNet) is employed to separate the sound sources from the signal recorded in reverberant environments. We use the encoder to generate the time domain representation of the recorded signal, and the mask is obtained by the Conv-TasNet model. The recovery of each separated sound source signal is conducted by multiplying the output of the encoder by the mask. In the second stage, the separated signal is enhanced with the single DNN. The training target of the DNN is to obtain the ideal enhancement mask, which is calculated by integrating the amplitude of the frequency domain coefficients of both separated signal and clean signal. The amplitude of the frequency domain coefficients of the separated signals is used as the input of DNN in order to predict the ideal ratio mask (IRM). IRM is multiplied by the original amplitude to obtain the enhanced amplitude of the frequency domain coefficients, which is integrated with phase to obtain the enhanced source signal. The results of the subjective and objective evaluation show that, compared with the reference methods, the proposed method achieves better separation quality in both reverberant and anechoic acoustic environments.

When a target-speech/masker mixture is processed with the signal-separation technique, ideal binary mask (IBM), intelligibility of target speech is remarkably improved in both normal-hearing listeners and hearing-impaired listeners. Intelligibility of speech can also be improved by filling in speech gaps with un-modulated broadband noise. This study investigated whether intelligibility of target speech in the IBM-treated target-speech/masker mixture can be further improved by adding a broadband-noise background. The results of this study show that following the IBM manipulation, which remarkably released target speech from speech-spectrum noise, foreign-speech, or native-speech masking (experiment 1), adding a broadband-noise background with the signal-to-noise ratio no less than 4 dB significantly improved intelligibility of target speech when the masker was either noise (experiment 2) or speech (experiment 3). The results suggest that since adding the noise background shallows the areas of silence in the time-frequency domain of the IBM-treated target-speech/masker mixture, the abruption of transient changes in the mixture is smoothed and the perceived continuity of target-speech components becomes enhanced, leading to improved target-speech intelligibility. The findings are useful for advancing computational auditory scene analysis, hearing-aid/cochlear-implant designs, and understanding of speech perception under "cocktail-party" conditions.