In-Car Speech Enhancement Based on Source Separation Technique

In this paper, we considered the problem of the speech enhancement similar to the real-world environments where several complex noise sources simultaneously degrade the quality and intelligibility of a target speech. The existing literature on the speech enhancement principally focuses on the presence of one noise source in mixture signals. However, in real-world situations, we generally face and attempt to improve the quality and intelligibility of speech where various complex stationary and nonstationary noise sources are simultaneously mixed with the target speech. Here, we have used deep learning for speech enhancement in complex-noisy environments and used ideal binary mask (IBM) as a binary classification function by using deep neural networks (DNNs). IBM is used as a target function during training and the trained DNNs are used to estimate IBM during enhancement stage. The estimated target function is then applied to the complex-noisy mixtures to obtain the target speech. The mean square error (MSE) is used as an objective cost function at various epochs. The experimental results at different input signal-to-noise ratio (SNR) showed that DNN-based complex-noisy speech enhancement outperformed the competing methods in terms of speech quality by using perceptual evaluation of speech quality (PESQ), segmental signal-to-noise ratio (SNRSeg), log-likelihood ratio (LLR), weighted spectral slope (WSS). Moreover, short-time objective intelligibility (STOI) reinforced the better speech intelligibility.

This work addresses the solution of localizing and enhancing hands-free speech inside the car environment. Cars have different types of sounds from outside, co-passengers dialogue and noise. To provide better-quality speech, a microphone array-based beamforming technique is used. This research work proposes the method for selected source localization, source separation, and enhancement. An estimation of the direction of arrival (DOA) to localize the signal direction and preferred direction is selected for speech enhancement. The spiral and sine-cosine algorithm (SSCA) algorithm is combined with an adaptive least mean square to adapt the system for different environments. The algorithm is implemented in hardware and tested in a real-time car environment. The results showed significant improvement in signal-to-noise ratio (SNR) of 5.2 dB and perceptual evaluation of speech quality (PESQ) of 2.3. Finally, the model is fine-tuned for the car to get better quality. The proposed technique is efficient, and results are compared with existing methods.

This study proposes a bi-directional recurrent neural network (Bi-RNN) post-processing method for speech enhancement (SE) at low signal-to noise ratios (SNR). Current speech enhancement solutions performed badly under low SNR situations. Loizou and Kim proposed a solution to reduce speech distortion errors in time-frequency (T-F) domain but it requires the knowledge of ground truth. As ground truth is unknown in real-life applications, the current study proposes to use a Bi-RNN to implement Loizou and Kim’s solution as a post-processing method for SE engines. Our solutions do not require prior knowledge of ground truth. The effectiveness of the proposed method is investigated with a spectral subtraction (SS) SE engine, a non-negative matrix factorization (NMF) SE engine, and a deep neural network ideal ratio mask (DNN-IRM) SE engine, under matched/mis-matched noise and different SNR conditions. Experimental results demonstrate that the proposed post-processing method effectively improved both perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) for all of these SE engines, especially at low SNR conditions.

In real-world environments, background noise significantly degrades the intelligibility and clarity of human speech. Existing audio-visual speech enhancement (AVSE) techniques often pose challenges in dynamic and noisy conditions. This study examines the inclusion of emotional features as a novel contextual cue within the AVSE framework. We analyze that incorporating emotional understanding from facial landmarks improves speech enhancement performance. We propose a deep learning-based emotion-aware audio-visual speech enhancement system (EAVSE) that uses auditory, visual, and emotional information. The proposed EAVSE extracts emotional features from facial landmarks and combines them with audio and visual modalities. Enriched multi-model data are processed by a UNet-based encoder-decoder network for joint learning and optimization. The network iteratively refines the feature representation, guided by a distortion-inspired loss function. We train and evaluate the model on the Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity dataset, known for its diverse audio-visual recordings with annotated emotions. Compared to AVSE benchmark and audio-only speech enhancement systems, the proposed model achieves significant improvements in both objective [Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI)] and subjective speech quality metrics. In particular, the scale-invariant signal-to-distortion ratio loss function demonstrates superior performance. This suggests the usefulness of the emotional contextual cues for AVSE. The experimental findings demonstrate the effectiveness of the AVSE, particularly in challenging noisy environments [signal-to-noise ratio (SNR) ≤ -7.5 dB]. The proposed model achieved Δ STOI of 7.32%, Δ PESQ of 0.33, and Δ S-SNR of 7.8 dB over noisy benchmark at 0 dB SNR.

Conventional multiband speech enhancement involves splitting the spectrum into various frequency bins and performing speech enhancement in each band independently. However, owing to the pole‐interaction problem in the spectral domain, estimation of clean speech from the formants, suppressed by the influence of the formants in the neighbouring bands, may result in poor quality. To reduce the influence of stronger formants over the neighbouring bands, in the current work, clean speech is estimated by filtering unprocessed speech in the temporal domain into various equivalent rectangular bandwidth based subbands followed by discrete cosine transform (DCT) based spectral speech enhancement in each band using spectral subtraction/minimum mean square error (MMSE). To further enhance speech, a spectral subtraction‐based approach that incorporates band‐specific weighting factor obtained using respective band signal‐to‐noise ratio (SNR), and an MMSE estimator that calculates apriori speech presence/absence probability based on local and global apriori SNR rather than a fixed/equiprobable value are proposed. The performance of the algorithms is evaluated using perceptual evaluation of speech quality and composite speech quality measure. It is observed that DCT‐derived spectrum based temporal‐domain multiband speech enhancement algorithm outperforms the existing techniques for car, babble, train, white, and factory noise in the 0–10 dB SNR levels.

The speech enhancement effect of traditional deep learning algorithms is not ideal under low signal-to-noise ratios (SNR). Skip connections-deep neural network (Skip-DNN) improves the traditional deep neural network (DNN) by adding skip connections between each layer of the neural network to solve the degradation problem of DNN. In this paper, the Multiresolution Cochleagram (MRCG) features in the gammachirp transform domain are denoised to obtain the improved MRCG (I-MRCG). The noise reduction method adopts the Minimum Mean-Square Error Short-Time Spectral Amplitude Estimator (MMSE-STSA) and takes I-MRCG as the input feature and Skip-DNN as the training network to improve the speech enhancement effect of the model. This paper also proposes an improved source-to-distortion ratio (SDR) loss function. When the loss function uses the improved SDR, it will improve the performance of Skip-DNN speech enhancement model. The experiments in this paper are performed on the Edinburgh dataset. When using I-MRCG as the input feature of Skip-DNN, the average perceptual evaluation of speech quality (PESQ) is 2.9137, and the average short-time objective intelligibility (STOI) is 0.8515. Compared with MRCG as Skip-DNN input features, the improvements are 0.91% and 0.71%, respectively. When the improved SDR is used as the loss function of the speech model, the average PESQ is 2.9699 and the average STOI is 0.8547. Compared with other loss functions, the improved SDR has a better enhancement effect when used as the loss function of the speech enhancement model.

Speech enhancement (SE) aims to improve speech quality and intelligibility by removing acoustic corruption. While various SE models using audio-only (AO) based on deep learning have been developed to achieve successful enhancement for non-speech background noise, audio-visual SE (AVSE) models have been studied to effectively remove competing speech. In this paper, we propose an AVSE model that estimates spectral masks for real and imaginary components to consider phase enhancement. It is based on the U-net structure that allows the decoder to perform information restoration by leveraging intermediate information in the encoding process and avoids the gradient vanishing problem by providing paths direct to the encoder’s layers. In the proposed model, we present early fusion to process audio and video with a single encoder that effectively generates features for the fused information easy to decode for SE with reduced parameters of the encoder and decoder. Moreover, we extend the U-net using the proposed Recurrent-Neural-Network (RNN) attention (RA) blocks and the Res paths (RPs) in the skip connections and the encoder. While the RPs are introduced to resolve the semantic gap between the low-level and high-level features, the RA blocks are developed to find efficient representations with inherent frequency-specific characteristics for speech as a type of time-series data. Experimental results on the LRS2-BBC dataset demonstrated that AV models successfully removed competing speech and our proposed model efficiently estimated complex spectral masks for SE. When compared with the conventional U-net model with a comparable number of parameters, our proposed model achieved relative improvements of about 7.23%, 5.21%, and 22.9% for the signal-to-distortion ratio, perceptual evaluation of speech quality, and FLOPS, respectively.

Aiming at the problem that the compressed sensing algorithm has poor reconstruction performance on low SNR speech, this paper introduces the OPINE-Net <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> network, which has shown good performance in image compression. An improved method is proposed on the basis of OPINE-Net <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> , which called OPINE-SNet <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> , to better adapt to the speech environment. First, the sampling network adaptively learns the sampling matrix from the speech sequence to obtain the measurement value; then, the reconstruction is completed through the initialization network and the reconstruction network in turn; finally, the independent sampling and joint reconstruction are performed using the relationship between blocks. In this paper, Perceptual evaluation of speech quality(PESQ) is selected as speech quality evaluation criteria, the results show that this method has obtained better speech quality and intelligibility.

Voice quality enhancement is a significant method for any speech communication model. Speech Enhancement (SE) and noise reduction approaches can significantly improve the perceptual voice quality of a hands-free communication system and increase the recognition rates of automatic speech recognition systems. Speech communications in real-world cases require high-performance enhancement techniques for addressing the distortions, which can corrupt the intelligibility and quality of the speech signal. Recent portable devices generally incorporate several microphones that can be easily used for improving signal quality. This paper plans to present a novel dual-channel SE model using the coherence function and heuristic concepts. The adaptive coherence function relates to the dual-microphone SE approach suitable for smartphones with primary and reference microphones. With this improved signal, the enhancement is performed by optimising denoising using Discrete Wavelet Transform (DWT) by Adaptive wind speed-based Deer Hunting Optimization Algorithm (AWS-DHOA). The considered objective function depends on the quality measure called Perceptual Evaluation of Speech Quality (PESQ) score. From the results, the RMSE of the proposed model using AWS-DHOA is 39.8%, 45.5%, 53.8% and 45.5% minimised than GWO-CFD, WOA-CFD, CSA-CFD, and RDA-CFD, respectively, on considering the babble noise. Finally, the comparative analysis confirmed that the proposed method improves speech quality and intelligibility by comparing diverse algorithms when different noise types corrupt the speech.

Speech communicated is adversely affected by environmental noise. It is important to process the speech and reduce noise for better understanding. Speech enhancement or noise reduction is useful to provide comfort for human or machine listening. Traditional algorithms provide better noise reduction and better-quality speech. Due to the non-stationary nature of noise and the quasi-stationary nature of speech, the traditional methods are proven inadequate in achieving high-quality speech. Later statistical estimators based on Gaussian, and super-Gaussian Probability Density Function (PDF) assumption further improved the enhancement performance. But still, non-stationary noise nature introduces artifacts in processed signal and results in decreased performance. It is observed that neural network approaches and the factorization approach provide better performance even under non-stationary noises by proper training and large database. Different features result in variations in output performance under unseen noise and speaker conditions. It is important to understand the importance and advantages of traditional methods, statistical estimators, and neural network approaches performances. To select the suitable method for a required application, it is essential to consider the trade-off between quality and distortion. In this work, the importance of speech enhancement methods is discussed. Performance measures used for understanding the speech enhancement like Signal to Noise Ratio (SNR), Segmental SNR, Log-Likelihood Ratio (LLR), Weighted Spectral Slope (WSS), Perceptual Evaluation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), Signal to Distortion Ratio (SDR) and Mean Opinion Score (MOS), are given. Highlights of important results are discussed for analyzing better speech enhancement methods for the required application. In this work, performance is compared using objective and subjective performance measures. Simulation results show superior performance when neural network is employed in statistical estimators.

Speech enhancement is of paramount importance in improving the intelligibility and audibility of speech signals in an uncontrolled noisy environment over wireless communication systems. However, the existing techniques used to achieve this such as spectral subtraction are characterized by remnant noise or musical note, while Kalman filter which can remove the remnant noise is associated with instability and non-linear characteristics. Therefore, this paper proposes speech enhancement model in noisy environment using hybrid spectral-Kalman filter over wireless communication system. The proposed model was carried out using acquired noisy speech signals by the Samsung Android mobile phone. The phone call was made in an uncontrolled noisy environment. The signal received was processed by the hybrid spectral-Kalman filter which was achieved by using the output speech signal from the spectral subtraction in time domain as the input of the Kalman filter. The proposed hybrid spectral-Kalman filtering technique was evaluated using Signal-to-Noise Ratio (SNR), Mean Square Error (MSE), Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI). Comparison was then performed with the existing techniques and validation was performed using corrupted speech signals obtained from the NOIZEUS corpus data set. The result revealed that the proposed technique performed better than the existing techniques in a noisy environment.

Reverberant speech separation with probabilistic time–frequency masking for B-format recordings

In this chapter, we will detail a new speech enhancement technique based on Lifting Wavelet Transform (LWT) and Artifitial Neural Network (ANN). This technique also uses the MMSE Estimate of Spectral Amplitude. It consists at the first step in applying the LWTto the noisy speech signal in order to obtain two noisy details coefficients, cD1 and cD2 and one approximation coefficient, cA2. After that, cD1 and cD2 are denoised by soft thresholding and for their thresholding, we need to use suitable thresholds, thrj,1≤j≤2. Those thresholds, thrj,1≤j≤2, are determined by using an Artificial Neural Network (ANN). The soft thresholding of those coefficients, cD1 and cD2, is performed in order to obtain two denoised coefficients, cDd1 and cDd2 . Then the denoising technique based on MMSE Estimate of Spectral Amplitude is applied to the noisy approximation cA2 in order to obtain a denoised coefficient, cAd2. Finally, the enhanced speech signal is obtained from the application of the inverse of LWT, LWT−1 to cDd1, cDd2 and cAd2. The performance of the proposed speech enhancement technique is justified by the computations of the Signal to Noise Ratio (SNR), Segmental SNR (SSNR) and Perceptual Evaluation of Speech Quality (PESQ).

Noise estimation based on optimal smoothing and minimum controlled through recursive averaging for speech enhancement

Recent work in the domain of speech enhancement has explored the use of self-supervised speech representations to aid in the training of neural speech enhancement models. However, much of this work focuses on using the deepest or final outputs of self supervised speech representation models, rather than the earlier feature encodings. The use of self supervised representations in such a way is often not fully motivated. In this work it is shown that the distance between the feature encodings of clean and noisy speech correlate strongly with psychoacoustically motivated measures of speech quality and intelligibility, as well as with human Mean Opinion Score (MOS) ratings. Experiments using this distance as a loss function are performed and improved performance over the use of STFT spectrogram distance based loss as well as other common loss functions from speech enhancement literature is demonstrated using objective measures such as perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI).