Time-Frequency Conformer and Kalman Filter-Based Speech Enhancement Method

Current deep learning approaches to linear prediction coefficient (LPC) estimation for the augmented Kalman filter (AKF) produce bias estimates, due to the use of a whitening filter. This severely degrades the perceived quality and intelligibility of enhanced speech produced by the AKF. In this paper, we propose a deep learning framework that produces clean speech and noise LPC estimates with significantly less bias than previous methods, by avoiding the use of a whitening filter. The proposed framework, called DeepLPC, jointly estimates the clean speech and noise LPC power spectra. The estimated clean speech and noise LPC power spectra are passed through the inverse Fourier transform to form autocorrelation matrices, which are then solved by the Levinson-Durbin recursion to form the LPCs and prediction error variances of the speech and noise for the AKF. The performance of DeepLPC is evaluated on the NOIZEUS and DEMAND Voice Bank datasets using subjective AB listening tests, as well as seven different objective measures (CSIG, CBAK, COVL, PESQ, STOI, SegSNR, and SI-SDR). DeepLPC is compared to six existing deep learning-based methods. Compared to other deep learning approaches to clean speech LPC estimation, DeepLPC produces a lower spectral distortion (SD) level than existing methods, confirming that it exhibits less bias. DeepLPC also produced higher objective scores than any of the competing methods (with an improvement of 0.11 for CSIG, 0.15 for CBAK, 0.14 for COVL, 0.13 for PESQ, 2.66% for STOI, 1.11 dB for SegSNR, and 1.05 dB for SI-SDR over the next best method). The enhanced speech produced by DeepLPC was also the most preferred by 10 listeners. By producing less biased clean speech and noise LPC estimates, DeepLPC enables the AKF to produce enhanced speech at a higher quality and intelligibility.

Speech enhancement using Kalman filter (KF) suffers from inaccurate estimates of the noise variance and the linear prediction coefficients (LPCs) in real-life noise conditions. This causes a degraded speech enhancement performance. In this paper, a causal convolutional neural network (CCNN) model is used to more accurately estimate the noise variance and LPC parameters of the KF for speech enhancement in real-life noise conditions. Specifically, a CCNN model gives an instantaneous estimate of the noise waveform for each noisy speech frame to compute the noise variance. Each noisy speech frame is pre-whitened by a whitening filter, which is constructed with the coefficients computed from the estimated noise. The LPC parameters are computed from the pre-whitened speech. The improved noise variance and LPCs enables the KF to minimize residual noise as well as distortion in the enhanced speech. Objective and subjective testing on NOIZEUS corpus reveal that the enhanced speech produced by the proposed method exhibits higher quality and intelligibility than some benchmark methods in various noise conditions for a wide range of SNR levels.

LPC-based formant enhancement method in Kalman filtering for speech enhancement

In this paper, we considered the problem of the speech enhancement similar to the real-world environments where several complex noise sources simultaneously degrade the quality and intelligibility of a target speech. The existing literature on the speech enhancement principally focuses on the presence of one noise source in mixture signals. However, in real-world situations, we generally face and attempt to improve the quality and intelligibility of speech where various complex stationary and nonstationary noise sources are simultaneously mixed with the target speech. Here, we have used deep learning for speech enhancement in complex-noisy environments and used ideal binary mask (IBM) as a binary classification function by using deep neural networks (DNNs). IBM is used as a target function during training and the trained DNNs are used to estimate IBM during enhancement stage. The estimated target function is then applied to the complex-noisy mixtures to obtain the target speech. The mean square error (MSE) is used as an objective cost function at various epochs. The experimental results at different input signal-to-noise ratio (SNR) showed that DNN-based complex-noisy speech enhancement outperformed the competing methods in terms of speech quality by using perceptual evaluation of speech quality (PESQ), segmental signal-to-noise ratio (SNRSeg), log-likelihood ratio (LLR), weighted spectral slope (WSS). Moreover, short-time objective intelligibility (STOI) reinforced the better speech intelligibility.

The existing Kalman filter (KF) suffers from poor estimates of the noise variance and the linear prediction coefficients (LPCs) in real-world noise conditions. This results in a degraded speech enhancement performance. In this paper, a deep learning approach is used to more accurately estimate the noise variance and LPCs, enabling the KF to enhance speech in various noise conditions. Specifically, a deep learning approach to MMSE-based noise power spectral density (PSD) estimation, called DeepMMSE, is used. The estimated noise PSD is used to compute the noise variance. We also construct a whitening filter with its coefficients computed from the estimated noise PSD. It is then applied to the noisy speech, yielding pre-whitened speech for computing the LPCs. The improved noise variance and LPC estimates enable the KF to minimise the residual noise and distortion in the enhanced speech. Experimental results show that the proposed method exhibits higher quality and intelligibility in the enhanced speech than the benchmark methods in various noise conditions for a wide-range of SNR levels.

Background/Objectives: The speech signal is an attractive research issue that solves signal noises. For this reason, the paper main goal is to effectively ensure speech quality. An accurate Optimized Speech Enhancement Algorithm (OSEA) has been proposed and implemented to improve speech intelligibility and quality. The proposed approach was developed and implemented through a Discrete Wavelet Transform (DWT) and kalman filter methods. To assess speech quality, various noisy types for each SNR level were used. The obtained results were compared with a wavelet based speech enhancement using objective assessments like SNR, NRMSE and PESQ. Indeed, the obtained results demonstrated a better speech quality .All prepared tests have been implemented on a TMS320C6416 fixed –point digital signal processor which appears to be suitable platform for real-time requirements.

LPC parameters substitution for speech information hiding

Supervised models for speech enhancement are trained using artificially generated mixtures of clean speech and noise signals. However, the synthetic training conditions may not accurately reflect real-world conditions encountered during testing. This discrepancy can result in poor performance when the test domain significantly differs from the synthetic training domain. To tackle this issue, the UDASE task of the 7th CHiME challenge aimed to leverage real-world noisy speech recordings from the test domain for unsupervised domain adaptation of speech enhancement models. Specifically, this test domain corresponds to the CHiME-5 dataset, characterized by real multi-speaker and conversational speech recordings made in noisy and reverberant domestic environments, for which ground-truth clean speech signals are not available. In this paper, we present the objective and subjective evaluations of the systems that were submitted to the CHiME-7 UDASE task, and we provide an analysis of the results. This analysis reveals a limited correlation between subjective ratings and several supervised nonintrusive performance metrics recently proposed for speech enhancement. Conversely, the results suggest that more traditional intrusive objective metrics can be used for in-domain performance evaluation using the reverberant LibriCHiME-5 dataset developed for the challenge. The subjective evaluation indicates that all systems successfully reduced the background noise, but always at the expense of increased distortion. Out of the four speech enhancement methods evaluated subjectively, only one demonstrated an improvement in overall quality compared to the unprocessed noisy speech, highlighting the difficulty of the task. The tools and audio material created for the CHiME-7 UDASE task are shared with the community Key Terms: Speech Enhancement, CHIME-7 Challenge ,Deep Neural Networks, UDASE Task, Audio Restoration, Subjective Evaluation, objective Evaluation

Improvement of condition monitoring (CM) systems for wind turbines (WTs) and reduction of the cost of wind energy are possible if knowledge about the condition of different WT components is available. CM based on the WT drive train shaft torque signal can give a better understanding of the gearbox failure mechanisms as well as provide a method for detecting mass imbalance and aerodynamic asymmetry. The major obstacle preventing the industrial application of CM based on the shaft torque signal is the costly measurement equipment which is impractical for long‐term use on operating WTs. This paper suggests a novel approach for low‐cost, indirect monitoring of the shaft torque from standard WT measurements. The shaft torque is estimated recursively from measurements of generator torque, high speed shaft and low speed shaft angular speeds using the well‐known Kalman filter theory. The performance of the augmented Kalman filter with fading memory (AKFF) is compared with the augmented Kalman filter (AKF) using simulated data of the WT for different load conditions, measurement noise levels and WT fault scenarios. A multiple‐model algorithm, based on a set of different Kalman filters, is designed for practical implementation of the shaft torque estimator. Its performance is validated for a scenario where there are frequent changes of operating points. The proposed cost‐effective shaft torque estimator overcomes all major problems, which prevent the industrial application of CM systems based on shaft torque measurements. Future work will be focused on validating the method using experimental data and developing suitable signal processing algorithms for fault detection. Copyright © 2013 John Wiley & Sons, Ltd.

<p><b>Speech enhancement was, is, and will be the key technology for digital speech transmission. When developing speech enhancement algorithms, the intelligibility or the quality of the processed speech needs to be evaluated. Intelligibility is more fundamental than quality. The evaluation of speech intelligibility can be carried out through subjective listening tests and objective metrics. Carrying out subjective listening tests is time-consuming and costly. Using the objective metrics is more efficient. Thus, there has been an increasing research in speech intelligibility prediction.</b></p> <p> Speech intelligibility can be measured in terms of the information received by a listener. This thesis aims at developing a mutual information-based speech intelligibility predictor (SIP) and using the mutual information-based predictor to assist the fitting process in hearing instruments. To achieve this goal, this thesis carried out three studies. </p> <p>First, it studied the modeling of the transmitted message. For mutual information-based SIPs, the most important thing is to determine the transmitted message. This thesis studied two approaches of modeling of the message: one is the continuous-valued sound, and the other is the discrete-valued linguistic message. Two corresponding SIPs were developed. By comparing their predicted intelligibility results with the psychometric curves, which are the subjective intelligibility scores, it shows that the modeling of discrete-valued message gives a better match to the psychometric curves.</p> <p> Second, based on the modeling of the discrete-valued message, this thesis proposed a mutual information-based SIP. Since the discrete-valued message cannot be obtained from a speech signal, the proposed SIP calculates the mutual information between the clean speech and the received speech, instead of calculating the mutual information between the message and the received speech. The proposed SIP considers frequency correlation for both the clean speech and the received speech. The evaluation results show that the proposed SIP performs better than the existing state-of-the-art mutual information-based SIPs.</p> <p>Third, this thesis proposed an automatic fitting tool for the nonlinear frequency compression (NFC) operator, which is a frequency lowering operator used in hearing instruments. The automatic fitting tool adjusts the parameter in the NFC by maximizing the mutual information between the message and the frequency-lowered speech. To evaluate the automatic fitting tool, the parameter was also searched by listening tests. The results show that the parameter determined by the automatic fitting tool is consistent to the parameter determined by the listening tests.</p>

Conventional multiband speech enhancement involves splitting the spectrum into various frequency bins and performing speech enhancement in each band independently. However, owing to the pole‐interaction problem in the spectral domain, estimation of clean speech from the formants, suppressed by the influence of the formants in the neighbouring bands, may result in poor quality. To reduce the influence of stronger formants over the neighbouring bands, in the current work, clean speech is estimated by filtering unprocessed speech in the temporal domain into various equivalent rectangular bandwidth based subbands followed by discrete cosine transform (DCT) based spectral speech enhancement in each band using spectral subtraction/minimum mean square error (MMSE). To further enhance speech, a spectral subtraction‐based approach that incorporates band‐specific weighting factor obtained using respective band signal‐to‐noise ratio (SNR), and an MMSE estimator that calculates apriori speech presence/absence probability based on local and global apriori SNR rather than a fixed/equiprobable value are proposed. The performance of the algorithms is evaluated using perceptual evaluation of speech quality and composite speech quality measure. It is observed that DCT‐derived spectrum based temporal‐domain multiband speech enhancement algorithm outperforms the existing techniques for car, babble, train, white, and factory noise in the 0–10 dB SNR levels.

Combined speech enhancement and auditory modelling for robust distributed speech recognition

Wiener filtering based speech enhancement with Weighted Denoising Auto-encoder and noise classification

Recently, deep neural networks (DNNs) have become the mainstream strategy for speech enhancement task because it can achieve the higher speech quality and intelligibility than the traditional methods. However, these DNN-based methods always need a large number of parallel corpus consisting of clean speech and noise to produce noisy data for the training of the DNN in order to improve the generalization of the network. As a result, this implies that many noisy speech signals that are collected in real environment cannot be used to train the DNN because of the lack of corresponding clean speech and noise. Additionally, as we know, noise varies with the time and scenario, so we cannot obtain parallel speech and noise due to infinite noise data and some limited speech data. Thus, the network training with unparallel speech and noise data is essential for the generalization of the network. To address this problem, we propose a novel parallel-data-free speech enhancement method, in which the cycle-consistent generative adversarial network (CycleGAN) and multi-objective learning are employed. Our method is also able to make best use of the benefits of multi-objective learning. On the training stage, we utilize two different encoders to encode the features of clean speech and noisy speech, respectively. Then, two forward generators are immediately used to predict the ideal time-frequency (T-F) mask and log-power spectrum (LPS) of clean speech. Two inverse generators are applied to map the magnitude spectrum (MS) and LPS of noisy speech, respectively. In addition, four discriminators are used to distinguish the real speech features from the generated features. Two encoders, four generators and four discriminators are simultaneously trained by using adversarial, identity-mapping, latent similarity and cycle-consistent loss. On the test stage, we directly utilize the forward generators and encoders to acquire the enhanced speech. The experimental results indicate that the proposed approach is able to achieve the better speech enhancement performance than the reference methods. Moreover, the proposed method is also effective to improve speech quality and intelligibility when the networks are trained under the parallel data.

In this paper, we propose a novel speech enhancement (SE) method by exploiting the discrete wavelet transform (DWT). This new method reduces the amount of fast time-varying portion, viz. the DWT-wise detail component, in the spectrogram of speech signals so as to highlight the speech-dominant component and achieves better speech quality. A particularity of this new method is that it is completely unsupervised and requires no prior information about the clean speech and noise in the processed utterance. The presented DWT-based SE method with various scaling factors for the detail part is evaluated with a subset of Aurora-2 database, and the PESQ metric is used to indicate the quality of processed speech signals. The preliminary results show that the processed speech signals reveal a higher PESQ score in comparison with the original counterparts. Furthermore, we show that this method can still enhance the signal by totally discarding the detail part (setting the respective scaling factor to zero), revealing that the spectrogram can be down-sampled and thus compressed without the cost of lowered quality. In addition, we integrate this new method with conventional speech enhancement algorithms, including spectral subtraction, Wiener filtering, and spectral MMSE estimation, and show that the resulting integration behaves better than the respective component method. As a result, this new method is quite effective in improving the speech quality and well additive to the other SE methods.