A Parallel-Data-Free Speech Enhancement Method Using Multi-Objective Learning Cycle-Consistent Generative Adversarial Network

Speech is easily interfered by external environment in reality, which results in the loss of important features. Deep learning has become a popular speech enhancement method because of its superior potential in solving nonlinear mapping problems for complex features. However, the deficiency of traditional deep learning methods is the weak learning capability of important information from previous time steps and long-term event dependencies between the time-series data. To overcome this problem, we propose a novel speech enhancement method based on the fused features of deep neural networks (DNNs) and gated recurrent unit (GRU). The proposed method uses GRU to reduce the number of parameters of DNNs and acquire the context information of the speech, which improves the enhanced speech quality and intelligibility. Firstly, DNN with multiple hidden layers is used to learn the mapping relationship between the logarithmic power spectrum (LPS) features of noisy speech and clean speech. Secondly, the LPS feature of the deep neural network is fused with the noisy speech as the input of GRU network to compensate the missing context information. Finally, GRU network is performed to learn the mapping relationship between LPS features and log power spectrum features of clean speech spectrum. The proposed model is experimentally compared with traditional speech enhancement models, including DNN, CNN, LSTM and GRU. Experimental results demonstrate that the PESQ, SSNR and STOI of the proposed algorithm are improved by 30.72%, 39.84% and 5.53%, respectively, compared with the noise signal under the condition of matched noise. Under the condition of unmatched noise, the PESQ and STOI of the algorithm are improved by 23.8% and 37.36%, respectively. The advantage of the proposed method is that it uses the key information of features to suppress noise in both matched and unmatched noise cases and the proposed method outperforms other common methods in speech enhancement.

In this paper, a speech enhancement method based on noise compensation performed on short time magnitude spectra is presented following the geometric approach. Here the noise estimate to be subtracted from the noisy speech spectrum is proposed to be determined by exploiting the low frequency regions of noisy speech of the current frame rather than depending only on the initial silence frames. We argue that this approach of noise estimation offers the capability of tracking the time variation of the non-stationary noise. A noise estimation confidence parameter is also introduced to prevent the subtraction of the overestimated noise. A procedure is thus formulated to compensate for the distortion in the magnitude spectrum of the noisy speech signal. The noise compensated magnitude spectrum is then recombined with the noisy unchanged phase spectrum to produce a modified complex spectrum prior to synthesize an enhanced frame. Extensive simulations are carried out using NOIZEUS database in order to evaluate the performance of proposed method. It is shown in terms of objective measures, spectrogram analysis and informal subjective listening test that the proposed method consistently outperforms one of the state-of-the-art methods of speech enhancement from noisy speech corrupted by babble or car noise at high as well as very low levels of SNR.

Supervised models for speech enhancement are trained using artificially generated mixtures of clean speech and noise signals. However, the synthetic training conditions may not accurately reflect real-world conditions encountered during testing. This discrepancy can result in poor performance when the test domain significantly differs from the synthetic training domain. To tackle this issue, the UDASE task of the 7th CHiME challenge aimed to leverage real-world noisy speech recordings from the test domain for unsupervised domain adaptation of speech enhancement models. Specifically, this test domain corresponds to the CHiME-5 dataset, characterized by real multi-speaker and conversational speech recordings made in noisy and reverberant domestic environments, for which ground-truth clean speech signals are not available. In this paper, we present the objective and subjective evaluations of the systems that were submitted to the CHiME-7 UDASE task, and we provide an analysis of the results. This analysis reveals a limited correlation between subjective ratings and several supervised nonintrusive performance metrics recently proposed for speech enhancement. Conversely, the results suggest that more traditional intrusive objective metrics can be used for in-domain performance evaluation using the reverberant LibriCHiME-5 dataset developed for the challenge. The subjective evaluation indicates that all systems successfully reduced the background noise, but always at the expense of increased distortion. Out of the four speech enhancement methods evaluated subjectively, only one demonstrated an improvement in overall quality compared to the unprocessed noisy speech, highlighting the difficulty of the task. The tools and audio material created for the CHiME-7 UDASE task are shared with the community Key Terms: Speech Enhancement, CHIME-7 Challenge ,Deep Neural Networks, UDASE Task, Audio Restoration, Subjective Evaluation, objective Evaluation

The complexity of the speech enhancement problem has motivated many different solutions. However, most techniques address situations in which the target speech is fully intelligible and the background noise energy is low in comparison with that of the speech. Thus while current enhancement algorithms can improve the perceived quality, the intelligibility of the speech is not increased significantly and may even be reduced. Recent research shows that intelligibility of very noisy speech can be improved by the use of a binary mask, in which a binary weight is applied to each time-frequency bin of the input spectrogram. There are several alternative goals for the binary mask estimator, based either on the Signalto-Noise Ratio (SNR) of each time-frequency bin or on the speech signal characteristics alone. Our approach to the binary mask estimation problem aims to preserve the important speech cues independently of the noise present by identifying time-frequency regions that contain significant speech energy. The speech power spectrum varies greatly for different types of speech sound. The energy of voiced speech sounds is concentrated in the harmonics of the fundamental frequency while that of unvoiced sounds is, in contrast, distributed across a broad range of frequencies. To identify the presence of speech energy in a noisy speech signal we have therefore developed two detection algorithms. The first is a robust algorithm that identifies voiced speech segments and estimates their fundamental frequency. The second detects the presence of sibilants and estimates their energy distribution. In addition, we have developed a robust algorithm to estimate the active level of the speech. The outputs of these algorithms are combined with other features estimated from the noisy speech to form the input to a classifier which estimates a mask that accurately reflects the time-frequency distribution of speech energy even at low SNR levels. We evaluate a mask-based speech enhancer on a range of speech and noise signals and demonstrate a consistent increase in an objective intelligibility measure with respect to noisy speech.

Deep Neural Network (DNN)-based mask estimation approach is an emerging algorithm in monaural speech enhancement. It is used to enhance speech signals from the noisy background by calculating either speech or noise dominant in a particular frame of the noisy speech signal. It can construct complex models for nonlinear processing. However, the limitation of the DNN-based mask algorithm is a generalization of the targeted population. Past research works focused on their target dataset because of time consumption for the audio recording session. Thus, in this work, different recording conditions were used to study the performance of the DNN-based mask estimation approach. The findings revealed that different language test dataset, as well as different conditions, may not give large impact in speech enhancement performance since the algorithm only learn the noise information. But, the performance of speech enhancement is promising when the trained model has been designed properly, especially given the less sample variations in the input dataset involved during the training session.

Time-Frequency (T-F) domain masking is currently the dominant method for single-channel speech enhancement, while little attention has been paid to phase information. A speech enhancement method, named PHASEN-SS, is proposed in this paper. Our method is divided into two steps, first a deep neural network (DNN) with two-branch communication using a combination of mask and phase for speech enhancement, and then a data post-processing after the DNN processes the noisy speech. PHASEN-SS uses two branches to predict the amplitude mask and the phase separately, which improves the accuracy of prediction by exchanging information between two branches, and then further the enhancement by denoising the residual noise through spectral subtraction. The experiments are conducted on the publicly available Voice Bank + DEMAND dataset, as well as a noisy speech dataset is synthesized with 4 common noises in Noise92 and Voice Bank clean speech according to the specified signal-to-noise ratio (SNR). The results show that the proposed method improves on the original one, and has better robustness to speech containing babble noise at higher SNRs for different SNRs.

Dynamic controllable speech enhancement models based on quantile loss functions

This paper provides a novel deep neural networks (DNN) based speech enhancement method using multi-band excitation (MBE) model. Generally, the proposed system contains two stages, namely training stage and enhancing stage. In the training stage, two DNNs with different targets are trained. The training targets are harmonic magnitude and band difference function of clean speech, respectively. The input feature for two DNNs is log-power spectra (LPS) of noisy speech. In the enhancing stage, using the output of DNNs and online estimated pitch period, the enhanced speech can be obtained by MBE speech synthesis. Using the proposed method, the parameters of MBE model can be accurately estimated to synthesize the enhanced speech with the high quality. At the same time, the noise between the harmonics is effectively eliminated. The experiments show that the proposed method outperforms the reference methods for speech quality and intelligibility.

Conventional monaural speech enhancement methods usually enhance the magnitude spectrum of noisy speech and leave the phase unchanged. Recent studies suggest that phase is also important for both speech intelligibility and perceptual quality. Although deep learning exhibits great potential on enhancing the magnitude and phase spectra in complex spectrogram domain and waveform domain, complex spectrogram and waveform are always more difficult to predict than the magnitude spectrum due to lack of clear structure in them. In this study, a Mel-domain denoising autoencoder and a deep generative vocoder are stacked to form a joint framework for monaural speech enhancement, in which the clean speech waveform is reconstructed without using the phase. Specifically, a convolutional recurrent network (CRN) is employed as the denoising autoencoder to enhance the Mel power spectrum of noisy speech. Then, the enhanced Mel power spectrum is fed to a deep generative vocoder to synthesize the speech waveform. Furthermore, the denoising autoencoder and generative vocoder are jointly fine-tuned. Experimental results show that the proposed method significantly improves speech intelligibility and perceptual quality. More importantly, our method achieves much better generalization ability for untrained noises than previous methods.

Majority of speech processing algorithms operate only with the spectral magnitude, leaving spectral phase unstructured and unexplored. With recent advancement in deep neural networks (DNNs), the phase processing became more important as an innovative and emergent prospective of the DNN based speech enhancement. In this paper, a speech enhancement method based on DNN combined with spectral phase estimation is proposed to improve the quality and intelligibility of the noisy speech. During training, DNNs are trained to learn a mapping from the noisy speech utterances and predict the coefficient to construct an ideal ratio mask for the spectral magnitude. The temporal smoothing unwrapped spectral phase estimation is incorporated as a target and transformed into a structured spectral phase during signal reconstruction. In enhancement stage, the enhanced speech magnitude is reconstructed with estimated structured spectral phase. Experimental results demonstrate success of the proposed method for speech enhancement in terms of the speech quality and intelligibility.

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This article proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems.

This paper investigates a multi-channel denoising autoencoder (DAE)-based speech enhancement approach. In recent years, deep neural network (DNN)-based monaural speech enhancement and robust automatic speech recognition (ASR) approaches have attracted much attention due to their high performance. Although multi-channel speech enhancement usually outperforms single channel approaches, there has been little research on the use of multi-channel processing in the context of DAE. In this paper, we explore the use of several multi-channel features as DAE input to confirm whether multi-channel information can improve performance. Experimental results show that certain multi-channel features outperform both a monaural DAE and a conventional time-frequency-mask-based speech enhancement method.

In this paper, a perceptual subspace speech enhancement method using masking properties of human auditory system is proposed. Noise was estimated at every frame using Speech Presence Probability (SPP). Variance normalization was further done to remove the abrupt changes in the noisy speech signal, removing noise. A combined masking threshold is calculated from simultaneous and temporal masking, which is used to decide the gain parameters for the algorithm. Spectral Domain Constrained (SDC) estimator was employed in determining the filter coefficients. The performance evaluation was done using wcep and WSS objective measures and subjective informal listening test. The results show a superior performance of the proposed method over some of the existing speech enhancement methods.

In this paper, a hidden Markov model (HMM)-based cue parameters estimation method for single-channel speech enhancement is proposed, in which the cue parameters of binaural cue coding (BCC) are applied to single-channel speech enhancement system successfully. First, the clean speech and noise signals are considered as the left and right channels of stereo signal, respectively; and the noisy speech is treated as the down-mixed mono signal of BCC method. According to the clean speech and noise data set and the corresponding noisy speech data set, the clean cue parameters and pre-enhanced cue parameters are extracted, respectively. Then the cue HMM is trained offline, which exploits the a priori information about the clean cue parameters and the pre-enhanced cue parameters for speech enhancement. Next, using the trained cue HMM, the clean cue parameters are estimated from noisy speech online. Finally, following the synthesis principle of BCC cue parameters, the speech estimator is constructed for enhancing noisy speech. The test results demonstrate that, for the segmental signal-noise-ratio (SNR), the log spectral distortion and PESQ measures, the proposed method performs better than the reference methods.

The spectral subtraction method is a conventional approach for single channel speech enhancement. The basic principle of this method is to estimate the short-time spectral magnitude of speech by subtracting estimated noise from the noisy speech spectrum and to combines it with the phase of the noisy speech. Besides reducing the noise, this method generates an unnatural and unpleasant noise, called remnant noise. This paper proposes a novel algorithm to reduce the remnant noise, and thus improving the overall quality of the enhanced speech. In this algorithm, the output of multi-band spectral subtraction (MBSS) method is used as the input signal again for next iteration process. After the MBSS method, the additive noise is changed to remnant noise. The remnant noise is re-estimated at each iteration. The new estimated noise, furthermore, is been used to process the next MBSS. This procedure is iterated a small number of times. The simulation results as well as informal subjective evaluations prove that the speech enhanced by proposed algorithm is more pleasant to listeners than the conventional MBSS algorithm. This reveals that the proposed algorithm reduces remnant noise satisfactorily and produces good speech quality with improved signal-to-noise ratio.