Dynamic controllable speech enhancement models based on quantile loss functions

The cochlea plays a key role in the transmission from acoustic vibration to neural stimulation upon which the brain perceives the sound. A cochlear implant (CI) is an auditory prosthesis to replace the damaged cochlear hair cells to achieve acoustic-to-neural conversion. However, the CI is a very coarse bionic imitation of the normal cochlea. The highly resolved time-frequency-intensity information transmitted by the normal cochlea, which is vital to high-quality auditory perception such as speech perception in challenging environments, cannot be guaranteed by CIs. Although CI recipients with state-of-the-art commercial CI devices achieve good speech perception in quiet backgrounds, they usually suffer from poor speech perception in noisy environments. Therefore, noise suppression or speech enhancement (SE) is one of the most important technologies for CI. In this study, we introduce recent progress in deep learning (DL), mostly neural networks (NN)-based SE front ends to CI, and discuss how the hearing properties of the CI recipients could be utilized to optimize the DL-based SE. In particular, different loss functions are introduced to supervise the NN training, and a set of objective and subjective experiments is presented. Results verify that the CI recipients are more sensitive to the residual noise than the SE-induced speech distortion, which has been common knowledge in CI research. Furthermore, speech reception threshold (SRT) in noise tests demonstrates that the intelligibility of the denoised speech can be significantly improved when the NN is trained with a loss function bias to more noise suppression than that with equal attention on noise residue and speech distortion.

In this paper, we present a new Speech Enhancement (SE) technique capable of running on a smartphone, as an assistive device for hearing aids (HAs). The developed method incorporates the coherence between the speech and noise signals to obtain a SE gain function which is used in conjunction with the gain function obtained by Spectral Subtraction using adaptive gain averaging. SE using coherence based gain function is found to suppress the background noise well, while inducing speech distortion. On the other hand, SE using Spectral Subtraction improves speech quality with tolerable speech distortion, but introduces background musical noise for certain noise types. The weighted fusion of the two gain functions strikes a balance between noise suppression and speech distortion. Also it allows the user to control the weighting factor based on the noisy environment and their comfort level of hearing. The developed method is computationally fast and operates in real-time. The proposed method was evaluated for machinery, babble, and car noise types, using both objective and subjective measures for both quality and intelligibility of the enhanced speech. The results show significant improvements in comparison with stand-alone Spectral Subtraction with weighted gain averaging SE methods.

Deep Neural Network (DNN)-based mask estimation approach is an emerging algorithm in monaural speech enhancement. It is used to enhance speech signals from the noisy background by calculating either speech or noise dominant in a particular frame of the noisy speech signal. It can construct complex models for nonlinear processing. However, the limitation of the DNN-based mask algorithm is a generalization of the targeted population. Past research works focused on their target dataset because of time consumption for the audio recording session. Thus, in this work, different recording conditions were used to study the performance of the DNN-based mask estimation approach. The findings revealed that different language test dataset, as well as different conditions, may not give large impact in speech enhancement performance since the algorithm only learn the noise information. But, the performance of speech enhancement is promising when the trained model has been designed properly, especially given the less sample variations in the input dataset involved during the training session.

Recently, deep neural networks (DNNs) have become the mainstream strategy for speech enhancement task because it can achieve the higher speech quality and intelligibility than the traditional methods. However, these DNN-based methods always need a large number of parallel corpus consisting of clean speech and noise to produce noisy data for the training of the DNN in order to improve the generalization of the network. As a result, this implies that many noisy speech signals that are collected in real environment cannot be used to train the DNN because of the lack of corresponding clean speech and noise. Additionally, as we know, noise varies with the time and scenario, so we cannot obtain parallel speech and noise due to infinite noise data and some limited speech data. Thus, the network training with unparallel speech and noise data is essential for the generalization of the network. To address this problem, we propose a novel parallel-data-free speech enhancement method, in which the cycle-consistent generative adversarial network (CycleGAN) and multi-objective learning are employed. Our method is also able to make best use of the benefits of multi-objective learning. On the training stage, we utilize two different encoders to encode the features of clean speech and noisy speech, respectively. Then, two forward generators are immediately used to predict the ideal time-frequency (T-F) mask and log-power spectrum (LPS) of clean speech. Two inverse generators are applied to map the magnitude spectrum (MS) and LPS of noisy speech, respectively. In addition, four discriminators are used to distinguish the real speech features from the generated features. Two encoders, four generators and four discriminators are simultaneously trained by using adversarial, identity-mapping, latent similarity and cycle-consistent loss. On the test stage, we directly utilize the forward generators and encoders to acquire the enhanced speech. The experimental results indicate that the proposed approach is able to achieve the better speech enhancement performance than the reference methods. Moreover, the proposed method is also effective to improve speech quality and intelligibility when the networks are trained under the parallel data.

The Fast Discrete Fourier Transform is commonly analysis method for speech signal processing in frequency domain. It has the problem of adjusting window size for desired resolution. But the Fractional Fourier Transform can have both time domain and frequency domain processing capabilities. This paper performs global processing by combining Fractional Fourier Transform and wavelet multi-scale local elimination on the speech signal and enhancement experiment with RNN neural network. The experimental results show that the speech signal is processed both locally by wavelet and globally by Fractional Fourier Transform. The combination of the two allows the speech signal to take into account both local and global information in the time and frequency domain.

Although both noise suppression and speech restoration are fundamental to speech enhancement, many Deep neural network (DNN)-based approaches tend to focus disproportionately on one, often overlooking the importance of their joint handling. In this study, we propose a dual-path architecture designed to balance noise suppression and speech restoration. The main path consists of an encoder and two specialized decoders: one dedicated to estimating the clean speech spectrum and the other to predicting a noise suppression mask. To reinforce the joint modeling of noise suppression and speech restoration, we introduce an auxiliary refinement path. This path consists of a separate encoder–decoder structure and is designed to further refine the enhanced speech by incorporating complementary information, learned independently from the main path. By using this dual-path architecture, the model better preserves fine speech details while reducing residual noise. Experimental results on the VoiceBank + DEMAND dataset show that our model surpasses conventional methods across multiple evaluation metrics in the causal setup. Specifically, it achieves a PESQ score of 3.33, reflecting improved speech quality, and a CSIG score of 4.48, indicating enhanced intelligibility. Furthermore, it demonstrates superior noise suppression, achieving an SNRseg of 10.44 and a CBAK score of 3.75.

Deep Neural Networks (DNNs) have been successfully adopted as a regression model in speech enhancement. However, the performance in the battlefield environment is not always satisfactory because the noise energy is often dominating in certain speech segments causing speech distortion. For the speech enhancement in complicated battlefield environment where multiple noises can simultaneously corrupt speech, such as gunshots and explosions, we propose an enhanced method to improve the existing DNN-based speech enhancement by using Recurrent Neural Networks (RNNs). This RNN model judges whether each frame is in a low SNR state, and then fuses two DNN-based speech enhancement models. The proposed method is compared with existing DNN-based speech enhancement techniques through the perceptual evaluation of speech quality (PESQ) and the short-time objective intelligibility (STOI) scores in various noisy speech conditions. The experimental results demonstrate significant improvements over the state-of-the-art techniques and reflect the usability of the method in a real battlefield environment.KeywordsSpeech enhancementDeep Neural Networks (DNNs)Recurrent Neural Networks (RNNs)Battlefield environment

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This article proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems.

In this paper, we present a Speech Enhancement (SE) technique to improve intelligibility of speech perceived by Hearing Aid users using smartphone as an assistive device. We use the formant frequency information to improve the overall quality and intelligibility of the speech. The proposed SE method is based on new super Gaussian joint maximum a Posteriori (SGJMAP) estimator. Using the priori information of formant frequency locations, the derived gain function has " tradeoff" factors that allows the smartphone user to customize perceptual preference, by controlling the amount of noise suppression and speech distortion in real-time. The formant frequency information helps the hearing aid user to control the gains over the non-formant frequency band, allowing the HA users to attain more noise suppression while maintaining the speech intelligibility using a smartphone application. Objective intelligibility measures and subjective results reflect the usability of the developed SE application in noisy real world acoustic environment.

This study proposes a bi-directional recurrent neural network (Bi-RNN) post-processing method for speech enhancement (SE) at low signal-to noise ratios (SNR). Current speech enhancement solutions performed badly under low SNR situations. Loizou and Kim proposed a solution to reduce speech distortion errors in time-frequency (T-F) domain but it requires the knowledge of ground truth. As ground truth is unknown in real-life applications, the current study proposes to use a Bi-RNN to implement Loizou and Kim’s solution as a post-processing method for SE engines. Our solutions do not require prior knowledge of ground truth. The effectiveness of the proposed method is investigated with a spectral subtraction (SS) SE engine, a non-negative matrix factorization (NMF) SE engine, and a deep neural network ideal ratio mask (DNN-IRM) SE engine, under matched/mis-matched noise and different SNR conditions. Experimental results demonstrate that the proposed post-processing method effectively improved both perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) for all of these SE engines, especially at low SNR conditions.

To enhance noise reduction performance for digital hearing aids (DHA), this study proposes a speech-distortionless beamformer which is realized by a sequential processing of fractional delay, subtraction and integration (FDSI). A Simulink model of this beamformer was constructed and its performance of speech quality enhancement was evaluated in various noisy circumstances including babble, car and traffic noises. In this simulation, null direction of the beamformer was fixed at 120° azimuth and noise source position was altered from 100 to 140° azimuth. The distance between two microphones also varies from 8 to 20 mm. Compared to conventional beamformers, the FDSI beamformer showed higher frequency-weighted segmental SNR (fwsegSNR) and lower weighted spectral slope (WSS) while perceptual evaluation of speech quality (PESQ) was almost same in every beamformer. These results imply that speech distortion was significantly reduced. Therefore, the FDSI beamformer has great promising for speech quality enhancement in DHA.

语音增强可以改进语音质量，抑制、降低噪声干扰，提高信噪比，在手机等语音通信设备中广泛应用。近年来，由于深度神经网络学习的语音增强技术，可有效克服传统神经网络语音消噪算法易陷于局部最优的不足，取得更好的语音消噪效果，成为语音增强技术领域的研究热点。本文针对已有深度神经网络模型泛化能力较弱、存储开销较大等问题，研究提出一种基于稀疏回归深度神经网络的语音增强算法。该算法通过在预训练阶段引入丢弃法(Dropout)和稀疏约束正则化技术改进训练模型保持预训练和调优阶段模型结构一致性，提升模型泛化能力。通过权值共享和权值量化进行网络压缩，降低存储开销。用谱减法进行后处理，有效去除稳态噪声，提高语音质量。仿真实验结果表明，改进算法可达到较高的语音性能评价指标，取得较好的语音增强效果，可满足语音增强处理要求。 Speech enhancement is a mean to improve the quality and intelligibility by noise suppression and enhancing the SNR at the same time, which has been widely applied in voice communication equipments. In recent years, Deep Neural Network (DNN) has become a research hot point due to its powerful ability to avoid local optimum, which is superior to the traditional neural network. However, the existed DNN costs storage and has a bad generalization. Now, this document puts forward a sparse regression DNN model to solve the above problems. First, we will take two regularization skills called Dropout and sparsity constraint to strengthen the generalization ability of the model. Obviously, in this way, the model can reach the consistency between the pre-training model and the training model. Then network compression by weights sharing and quantization is taken to reduce storage cost. Next, spectral subtraction is used in post-processing to overcome stationary noise. The result proofs that the improved framework gets a good effect and meets the requirement of the speech processing.

Improving speech quality is becoming a basic requirement with increasing interest in speech processing applications. A lot of speech enhancement techniques are developed to reduce or completely remove listeners fatigue from various devices like smartphones and also from online communication applications. Background noise often interrupts communication, and this was solved using a hardware physical device that normally emits a negative frequency of the incoming audio noise signal to cancel out the noise. Deep learning has recently made a break-through in the speech enhancement process. This paper proposes an audio denoising model which is built on a deep neural network architecture based on spectrograms (which is a hybrid between frequency domain and time domain). The proposed deep neural network model effectively predicts the negative noise frequency for given input incoming audio file with noise. After prediction, the predicted values are then removed from the original noise audio file to create the denoised audio output.KeywordsDeep neural networkSpectrogramTransfer learningActivation functionSampling rateAudio synthesizingPreprocessing

Speech Enhancement (SE) system deals with improving the perceptual quality and preserving the speech intelligibility of the noisy mixture. The Time-Frequency (T-F) masking-based SE using the supervised learning algorithm, such as a Deep Neural Network (DNN), has outperformed the traditional SE techniques. However, the notable difference observed between the oracle mask and the predicted mask, motivates us to explore different deep learning architectures. In this paper, we propose to use a Convolutional Neural Network (CNN)-based Generative Adversarial Network (GAN) for inherent mask estimation. GAN takes an advantage of the adversarial optimization, an alternative to the other Maximum Likelihood (ML) optimization-based architectures. We also show the need for supervised T-F mask estimation for effective noise suppression. Experimental results demonstrate that the proposed T-F mask-based SE significantly outperforms the recently proposed end-to-end SEGAN and a GAN-based Pix2Pix architecture. The performance evaluation in terms of both the predicted mask and the objective measures, dictates the improvement in the speech quality, while simultaneously reducing the speech distortion observed in the noisy mixture.

In this paper, we modify the Sparse Coding Shrinkage (SCS) method with an appropriate optimal linear filter (Wiener filter) in order to improve its efficiency as a speech enhancement algorithm. SCS transform is only applicable for sparse data and speech features do not have this property in either time or frequency domains. Therefore we have used Linear Independent Component Analysis (LICA) to transfer the corrupted speech frames to the sparse code space in which noise and speech components are separated by means of a shrinkage function. Before employing SCS, Wiener filtering was applied on the ICA components to reduce noise energy and consequently the SCS shrinkage threshold. Experimental results have been obtained using connected digit database TIDIGIT contaminated with NATO RSG-10 noise data.