Time-Frequency Masking Based Online Speech Enhancement with Multi-Channel Data Using Convolutional Neural Networks

Aiming at the problem that the convolutional recurrent neural network speech enhancement method cannot extract the speech harmonic information efficiently. This paper proposes a Tibetan speech enhancement method based on the Attention Mechanism Cascaded Convolutional Recurrent Neural (ACRNN) network model. Firstly, the input time domain speech signal is converted into a frequency domain signal using short-time Fourier transform. Secondly, the harmonic characteristics are extracted through the harmonic perception attention mechanism module, and the attention tensor and the original speech spectrum are spliced into the CRNN network for speech enhancement. Finally, the short-time inverse Fourier transform is used to transform the signal from the frequency domain to the time domain to generate the speech waveform after noise reduction. Experimental data shows that compared to directly using CRNN network speech noise reduction, this method has improved speech quality and intelligibility.

Speech enhancement is fundamental for many real-time speech applications and it is challenging in case of single-channel because practically only one data channel is available. Without any constraint, a countless range of solutions are possible to solve this problem. In this paper, we present a supervised learning approach to enhance a speech degraded by speech-babble noise, which is most challenging type of noise in speech enhancement systems. The proposed method is composed of deep neural networks (DNNs) and less aggressive Wiener filtering (LW) for speech enhancement, labeled as the DNN-LW. The proposed method is composed of the training and testing stages, respectively. The DNN in the training stage calculates the magnitude spectrums of noise-free speech and the noise signals, respectively from the input noise-masked speech features concurrently. The Less aggressive Wiener filter is then placed as an extra layer on top of the deep neural network to create the enhanced magnitude spectrum. Finally, the phase of noisy speech is used to restore the estimate of clean speech. During testing stage, the trained DNN is provided the features of noise-masked speech to attain the enhanced speech. The experimental results revealed that the DNN-LW approach performs significantly better against baseline speech enhancement methods.

Speech enhancement is a process which cultivates the quality of speech signal in noisy environment. It refers to removing or reducing the background noise in order to obtain an improved quality of original speech signal. Degradation of speech signal is most common problem in speech communication, so enhancement of the speech plays a vital role in improving the quality of speech signal. A number of methods are used for speech enhancement. Here we are using LPC based FEM in Wiener filter for speech enhancement. This method is then compared with several speech enhancement algorithms to obtain a better result or better speech quality. On comparing this method with other speech enhancement methods we obtain better speech performance. Here we are using NOIZEUS speech database in order to compare different speech enhancement methods. The experimental result shows that our proposed method provides better result and there is no information loss in original speech signal.

In this paper, we propose a novel speech enhancement (SE) method by exploiting the discrete wavelet transform (DWT). This new method reduces the amount of fast time-varying portion, viz. the DWT-wise detail component, in the spectrogram of speech signals so as to highlight the speech-dominant component and achieves better speech quality. A particularity of this new method is that it is completely unsupervised and requires no prior information about the clean speech and noise in the processed utterance. The presented DWT-based SE method with various scaling factors for the detail part is evaluated with a subset of Aurora-2 database, and the PESQ metric is used to indicate the quality of processed speech signals. The preliminary results show that the processed speech signals reveal a higher PESQ score in comparison with the original counterparts. Furthermore, we show that this method can still enhance the signal by totally discarding the detail part (setting the respective scaling factor to zero), revealing that the spectrogram can be down-sampled and thus compressed without the cost of lowered quality. In addition, we integrate this new method with conventional speech enhancement algorithms, including spectral subtraction, Wiener filtering, and spectral MMSE estimation, and show that the resulting integration behaves better than the respective component method. As a result, this new method is quite effective in improving the speech quality and well additive to the other SE methods.

In this paper, a speech enhancement method based on noise compensation performed on short time magnitude spectra is presented following the geometric approach. Here the noise estimate to be subtracted from the noisy speech spectrum is proposed to be determined by exploiting the low frequency regions of noisy speech of the current frame rather than depending only on the initial silence frames. We argue that this approach of noise estimation offers the capability of tracking the time variation of the non-stationary noise. A noise estimation confidence parameter is also introduced to prevent the subtraction of the overestimated noise. A procedure is thus formulated to compensate for the distortion in the magnitude spectrum of the noisy speech signal. The noise compensated magnitude spectrum is then recombined with the noisy unchanged phase spectrum to produce a modified complex spectrum prior to synthesize an enhanced frame. Extensive simulations are carried out using NOIZEUS database in order to evaluate the performance of proposed method. It is shown in terms of objective measures, spectrogram analysis and informal subjective listening test that the proposed method consistently outperforms one of the state-of-the-art methods of speech enhancement from noisy speech corrupted by babble or car noise at high as well as very low levels of SNR.

Speech is easily interfered by external environment in reality, which results in the loss of important features. Deep learning has become a popular speech enhancement method because of its superior potential in solving nonlinear mapping problems for complex features. However, the deficiency of traditional deep learning methods is the weak learning capability of important information from previous time steps and long-term event dependencies between the time-series data. To overcome this problem, we propose a novel speech enhancement method based on the fused features of deep neural networks (DNNs) and gated recurrent unit (GRU). The proposed method uses GRU to reduce the number of parameters of DNNs and acquire the context information of the speech, which improves the enhanced speech quality and intelligibility. Firstly, DNN with multiple hidden layers is used to learn the mapping relationship between the logarithmic power spectrum (LPS) features of noisy speech and clean speech. Secondly, the LPS feature of the deep neural network is fused with the noisy speech as the input of GRU network to compensate the missing context information. Finally, GRU network is performed to learn the mapping relationship between LPS features and log power spectrum features of clean speech spectrum. The proposed model is experimentally compared with traditional speech enhancement models, including DNN, CNN, LSTM and GRU. Experimental results demonstrate that the PESQ, SSNR and STOI of the proposed algorithm are improved by 30.72%, 39.84% and 5.53%, respectively, compared with the noise signal under the condition of matched noise. Under the condition of unmatched noise, the PESQ and STOI of the algorithm are improved by 23.8% and 37.36%, respectively. The advantage of the proposed method is that it uses the key information of features to suppress noise in both matched and unmatched noise cases and the proposed method outperforms other common methods in speech enhancement.

Most algorithms for speech enhancement in the spectral domain focus on the acquisition of an estimator of the clean speech parameter, such as spectrum or amplitude. Enhanced speech quality with these methods mainly depends upon the accuracy of the estimator. When the signal-to-noise ratio (SNR) becomes lower, e.g., less than 3 dB, the enhanced speech often shows unsatisfactory quality. Here, we propose a new method for speech enhancement in the spectral domain. It is based on our view that speech characteristics are perceived mainly by part of the spectral components which have higher local (or instantaneous) SNRs. These components have a special importance on speech SNR enhancement and speech understanding. Identifying these components in the noisy spectrum with a decision system and constructing enhanced speech just by the same noisy amplitude and phase, the result is similar to those popular algorithms. All that is needed is a local SNR decision for each spectral component. The threshold for the decision is derived and two popular methods for local SNR (LSNR) estimation are suggested. It shows better performance under lower SNR signals. Though the number of these kind of spectral components varies with different signal SNR, any algorithms which can pick up more than 90% of it is sufficient to match any existing methods. Primary considerations and results are shown.

Speech enhancement (SE) methods mainly focus on recovering clean speech from noisy input. In real-world speech communication, however, noises often exist in not only speaker but also listener environments. Although SE methods can suppress the noise contained in the speaker’s voice, they cannot deal with the noise that is physically present in the listener side. To address such a complicated but common scenario, we investigate a deep learning-based joint framework integrating noise reduction (NR) with listening enhancement (LE), in which the NR module first suppresses noise and the LE module then modifies the denoised speech, i.e., the output of the NR module, to further improve speech intelligibility. The enhanced speech can thus be less noisy and more intelligible for listeners. Experimental results show that our proposed method achieves promising results and significantly outperforms the disjoint processing methods in terms of various speech evaluation metrics.

In previous work, we proposed a variational autoencoder-based (VAE) Bayesian permutation training speech enhancement (SE) method (PVAE) which indicated that the SE performance of the traditional deep neural network-based (DNN) method could be improved by deep representation learning (DRL). Based on our previous work, we in this paper propose to use β-VAE to further improve PVAE's ability of representation learning. More specifically, our β-VAE can improve PVAE's capacity of disentangling different latent variables from the observed signal without the trade-off problem between disentanglement and signal reconstruction. This trade-off problem widely exists in previous β-VAE algorithms. Unlike the previous β-VAE algorithms, the proposed β-VAE strategy can also be used to optimize the DNN's structure. This means that the proposed method can not only improve PVAE's SE performance but also reduce the number of PVAE training parameters. The experimental results show that the proposed method can acquire better speech and noise latent representation than PVAE. Meanwhile, it also obtains a higher scale-invariant signal-to-distortion ratio, speech Quality, and speech intelligibility.

In this paper we propose an efficient single microphone (single channel) speech enhancement (SE) method for Quasi-Stationary noise environment. The proposed method estimates the noise by exploiting its quasi-periodic nature, followed by a statistical model based method to enhance the speech. An efficient reduced-order linear predictive error filtering is introduced to increase the signal to noise ratio (SNR) of the noisy speech. The proposed method is evaluated experimentally by considering the actual recorded Functional Magnetic Resonance Imaging (fMRI) machinery noise which is quasi periodic in nature, added in clean speech. Objective evaluation of our method shows improvement in both quality and intelligibility measures when tested with the sentences chosen from IEEE corpus added in broadband quasi periodic fMRI noise. The proposed method outperforms the standard statistical model based SE technique.

Due to the simple design pipeline, end-to-end (E2E) neural models for speech enhancement (SE) have attracted great interest. In order to improve the performance of the E2E model, the local and sequential properties of speech should be efficiently taken into account when modelling. However, in most current E2E models for SE, these properties are either not fully considered or are too complex to be realized. In this letter, we propose an efficient E2E SE model, termed WaveCRN. Compared with models based on convolutional neural networks (CNN) or long short-term memory (LSTM), WaveCRN uses a CNN module to capture the speech locality features and a stacked simple recurrent units (SRU) module to model the sequential property of the locality features. Different from conventional recurrent neural networks and LSTM, SRU can be efficiently parallelized in calculation, with even fewer model parameters. In order to more effectively suppress noise components in the noisy speech, we derive a novel restricted feature masking approach, which performs enhancement on the feature maps in the hidden layers; this is different from the approaches that apply the estimated ratio mask to the noisy spectral features, which is commonly used in speech separation methods. Experimental results on speech denoising and compressed speech restoration tasks confirm that with the SRU and the restricted feature map, WaveCRN performs comparably to other state-of-the-art approaches with notably reduced model complexity and inference time.

Deep learning-based speech enhancement (SE) methods predominantly draw upon two architectural frameworks: generative adversarial networks and diffusion models. In the realm of SE, capturing the local and global relations between signal frames is crucial for the success of these methods. These frameworks typically employ a UNet architecture as their foundational backbone, integrating Long Short-Term Memory (LSTM) networks or attention mechanisms within the UNet to effectively model both local and global signal relations. However, the coupled relation modeling way may not fully harness the potential of these relations. In this paper, we propose an innovative Association-based Fusion Speech Enhancement method (AFSE), a decoupled method. AFSE first constructs a graph that encapsulates the association between each time window of the speech signal, and then models the global relations between frames by fusing the features of these time windows in a manner akin to graph neural networks. Furthermore, AFSE leverages a UNet with dilated convolutions to model the local relations, enabling the network to maintain a high-resolution representation while benefiting from a wider receptive field. Experimental results demonstrate that the AFSE method significantly improves performance in speech enhancement tasks, validating the effectiveness and superiority of our approach. The code is available at https://github.com/jie019/AFSE_IJCAI2025.

Recently, deep neural networks (DNNs) have become the mainstream strategy for speech enhancement task because it can achieve the higher speech quality and intelligibility than the traditional methods. However, these DNN-based methods always need a large number of parallel corpus consisting of clean speech and noise to produce noisy data for the training of the DNN in order to improve the generalization of the network. As a result, this implies that many noisy speech signals that are collected in real environment cannot be used to train the DNN because of the lack of corresponding clean speech and noise. Additionally, as we know, noise varies with the time and scenario, so we cannot obtain parallel speech and noise due to infinite noise data and some limited speech data. Thus, the network training with unparallel speech and noise data is essential for the generalization of the network. To address this problem, we propose a novel parallel-data-free speech enhancement method, in which the cycle-consistent generative adversarial network (CycleGAN) and multi-objective learning are employed. Our method is also able to make best use of the benefits of multi-objective learning. On the training stage, we utilize two different encoders to encode the features of clean speech and noisy speech, respectively. Then, two forward generators are immediately used to predict the ideal time-frequency (T-F) mask and log-power spectrum (LPS) of clean speech. Two inverse generators are applied to map the magnitude spectrum (MS) and LPS of noisy speech, respectively. In addition, four discriminators are used to distinguish the real speech features from the generated features. Two encoders, four generators and four discriminators are simultaneously trained by using adversarial, identity-mapping, latent similarity and cycle-consistent loss. On the test stage, we directly utilize the forward generators and encoders to acquire the enhanced speech. The experimental results indicate that the proposed approach is able to achieve the better speech enhancement performance than the reference methods. Moreover, the proposed method is also effective to improve speech quality and intelligibility when the networks are trained under the parallel data.

It has been shown in the literature that the perceptual wavelet packet decomposition (PWPD) and the Teager energy operator (TEO) are useful for various speech processing systems and speech enhancement applications, respectively. By the use of the PWPD and the TEO, this paper presents an improved wavelet-based speech enhancement method. The main advantage of the proposed method is that the over thresholding of speech segments which is usually occurred in conventional wavelet-based speech enhancement schemes can be avoided. As a consequence, the enhanced speech quality of the proposed method can be increased substantially from those of conventional approaches. In addition, the proposed method does not require a complicated estimation of the noise level or any knowledge of the SNR. Using speech signals corrupted by additive and real noises, experimental results demonstrate that the speech enhancement method presented in this paper is capable of outperforming conventional noise cancellation schemes.

Speech enhancement based on statistical models has shown good performance, but the performance degrades when environment noise is highly non-stationary due to the stationary assumption. On the contrary, the template-based enhancement methods are more robust to non-stationary noise, but these are heavily dependent on a priori information present in training data. In order to get over both of the shortcomings, we propose a novel speech enhancement method which combines the statistical model-based enhancement scheme with the template-based enhancement. To reduce a dependency on a priori information, the speech and noise bases are updated simultaneously using the estimated speech presence probability, which is obtained from statistical model-based enhancement. Experimental results showed that the proposed method outperformed not only the statistical model-based and non-negative matrix factorization (NMF) approaches, but also their combination implemented with existing bases update rule in various kinds of noise.