Speech enhancement by perceptual filter with sequential noise parameter estimation

Noise reduction algorithms are widely used to mitigate noise effects on speech to improve the robustness of speech technology applications. However, they inevitably cause speech distortion. The tradeoff between noise reduction and speech distortion is a key concern in designing noise reduction algorithms. This study proposes a novel framework for noise reduction by considering this tradeoff. We regard speech estimation as a function approximation problem in a regularized reproducing kernel Hilbert space (RKHS). In the estimation, the objective function is formulated to find an approximation function by controlling the tradeoff between approximation accuracy and function complexity. For noisy observations, this is equivalent to controlling the tradeoff between noise reduction and speech distortion. Since the target function is approximated in an RKHS, either a linear or nonlinear mapping function can be naturally incorporated in the estimation by a “kernel trick”. Traditional signal subspace and Wiener filtering based noise reduction can be derived as special cases when a linear kernel function is applied in this framework. We first provided a theoretical analysis of the tradeoff property of the framework in noise reduction. Then we applied our proposed noise reduction method in speech enhancement and noisy robust speech recognition experiments. Compared to several classical noise reduction methods, our proposed method showed promising advantages.

We describe an audio-visual automatic continuous speech recognition system, which significantly improves speech recognition performance over a wide range of acoustic noise levels, as well as under clean audio conditions. The system utilizes facial animation parameters (FAPs) supported by the MPEG-4 standard for the visual representation of speech. We also describe a robust and automatic algorithm we have developed to extract FAPs from visual data, which does not require hand labeling or extensive training procedures. The principal component analysis (PCA) was performed on the FAPs in order to decrease the dimensionality of the visual feature vectors, and the derived projection weights were used as visual features in the audio-visual automatic speech recognition (ASR) experiments. Both single-stream and multistream hidden Markov models (HMMs) were used to model the ASR system, integrate audio and visual information, and perform a relatively large vocabulary (approximately 1000 words) speech recognition experiments. The experiments performed use clean audio data and audio data corrupted by stationary white Gaussian noise at various SNRs. The proposed system reduces the word error rate (WER) by 20% to 23% relatively to audio-only speech recognition WERs, at various SNRs (0–30 dB) with additive white Gaussian noise, and by 19% relatively to audio-only speech recognition WER under clean audio conditions.

We present a method for sequentially estimating time-varying noise parameters. Noise parameters are sequences of time-varying mean vectors representing the noise power in the log-spectral domain. The proposed sequential Monte Carlo method generates a set of particles in compliance with the prior distribution given by clean speech models. The noise parameters in this model evolve according to random walk functions and the model uses extended Kalman filters to update the weight of each particle as a function of observed noisy speech signals, speech model parameters, and the evolved noise parameters in each particle. Finally, the updated noise parameter is obtained by means of minimum mean square error (MMSE) estimation on these particles. For efficient computations, the residual resampling and Metropolis-Hastings smoothing are used. The proposed sequential estimation method is applied to noisy speech recognition and speech enhancement under strongly time-varying noise conditions. In both scenarios, this method outperforms some alternative methods.

In a previous study, ) investigated whether pure-tone average (PTA) hearing loss and working memory capacity (WMC) modulate benefit from different binaural noise reduction (NR) settings. Results showed that listeners with smaller WMC preferred strong over moderate NR even at the expense of poorer speech recognition due to greater speech distortion (SD), whereas listeners with larger WMC did not. To enable a better understanding of these findings, the main aims of the present study were (1) to explore the perceptual consequences of changes to the signal mixture, target speech, and background noise caused by binaural NR, and (2) to determine whether response to these changes varies with WMC and PTA. As in the previous study, four age-matched groups of elderly listeners (with N = 10 per group) characterized by either mild or moderate PTAs and either better or worse performance on a visual measure of WMC participated. Five processing conditions were tested, which were based on the previously used (binaural coherence-based) NR scheme designed to attenuate diffuse signal components at mid to high frequencies. The five conditions differed in terms of the type of processing that was applied (no NR, strong NR, or strong NR with restoration of the long-term stimulus spectrum) and in terms of whether the target speech and background noise were processed in the same manner or whether one signal was left unprocessed while the other signal was processed with the gains computed for the signal mixture. Comparison across these conditions allowed assessing the effects of changes in high-frequency audibility (HFA), SD, and noise attenuation and distortion (NAD). Outcome measures included a dual-task paradigm combining speech recognition with a visual reaction time (VRT) task as well as ratings of perceived effort and overall preference. All measurements were carried out using headphone simulations of a frontal target speaker in a busy cafeteria. Relative to no NR, strong NR was found to impair speech recognition and VRT performance slightly and to improve perceived effort and overall preference markedly. Relative to strong NR, strong NR with restoration of the long-term stimulus spectrum and thus HFA did not affect speech recognition, restored VRT performance to that achievable with no NR, and increased perceived effort and reduced overall preference markedly. SD had negative effects on speech recognition and perceived effort, particularly when both speech and noise were processed with the gains computed for the signal mixture. NAD had positive effects on speech recognition, perceived effort, and overall preference, particularly when the target speech was left unprocessed. VRT performance was unaffected by SD and NAD. None of the datasets exhibited any clear signs that response to the different signal changes varies with PTA or WMC. For the outcome measures and stimuli applied here, the present study provides little evidence that PTA or WMC affect response to changes in HFA, SD, and NAD caused by binaural NR. However, statistical power restrictions suggest further research is needed. This research should also investigate whether partial HFA restoration combined with some pre-processing that reduces co-modulation distortion results in a more favorable balance of the effects of binaural NR across outcome dimensions and whether NR strength has any influence on these results.

Combined speech enhancement and auditory modelling for robust distributed speech recognition

In this paper a multi-channel speech enhancement framework for distant speech acquisition in noisy and reverberant environments for Non-negative Matrix Factorization (NMF)-based Automatic Speech Recognition (ASR) is proposed. The system is evaluated for its use in an assistive vocal interface for physically impaired and speech-impaired users. The framework utilises the Spatially Pre-processed Speech Distortion Weighted Multichannel Wiener Filter (SP-SDW-MWF) in combination with a postfilter to reduce noise and reverberation. Additionally, the estimation uncertainty of the speech enhancement framework is propagated through the Mel-Frequency Cepstrum Coefficients (MFCC) feature extraction to allow for feature compensation in a later stage. Results indicate that a) using a trade-off parameter between noise reduction and speech distortion has a positive effect on the recognition performance with respect to the well-known GSC and MWF and b) the addition of a postfilter and the feature compensation increases performance with respect to several baselines for a non-pathological and pathological speaker. Index Terms: multi-channel speech enhancement, speech recognition, uncertainty of estimation, dysarthric speech

본 논문에서는 음성 신호 대신 초음파 도플러 신호를 이용하여 음성을 인식하는 새로운 음성 인식 방법을 제안하였다. 제안된 방법은 주변 잡음에 대한 강인성과 무 접촉식 센서 사용에 따른 사용자의 불편함 감소를 포함하는 기존의 음성/무음성 인식 방법에 비해 몇 가지 장점을 갖는다. 제안된 방법에서는 40 kHz의 주파수를 갖는 초음파 신호를 입 주변에 방사하여, 반사된 신호를 취득하고, 취득된 신호의 도플러 주파수 변화를 이용하여 음성 인식을 구현하였다. 단일 채널 초음파 신호를 사용하는 기존의 연구와 달리, 다양한 위치에서의 취득된 초음파 신호를 음성 인식에 사용하기 위해 다채널 취득 장치를 고안하였다. PCA(Principal Component Analysis)특징 변수를 사용한 음성 인식에는 좌-우 모델을 갖는 은닉 마코프 모델을 사용하였다. 제안된 방법의 검증을 위해 60개의 한국어 고립어에 대해 6명의 화자로부터 취득된 초음파 도플러 신호를 인식에 사용하였으며, 기존 음성기반 음성인식 기법과 비교할 만한 수준의 인식율을 얻을 수 있었다. 또한 실험 결과 제안된 방법은 기존의 단일 채널 음성 인식 방법과 비교하여 우수한 성능을 나타내었으며, 특히 잡음 환경에서도 90 % 이상의 인식율을 얻을 수 있었다. In this paper, a new automatic speech recognition (ASR) was proposed where ultrasonic doppler signals were used, instead of conventional speech signals. The proposed method has the advantages over the conventional speech/non-speech-based ASR including robustness against acoustic noises and user comfortability associated with usage of the non-contact sensor. In the method proposed herein, 40 kHz ultrasonic signal was radiated toward to the mouth and the reflected ultrasonic signals were then received. Frequency shift caused by the doppler effects was used to implement ASR. The proposed method employed multi-channel ultrasonic signals acquired from the various locations, which is different from the previous method where single channel ultrasonic signal was employed. The PCA(Principal Component Analysis) coefficients were used as the features of ASR in which hidden markov model (HMM) with left-right model was adopted. To verify the feasibility of the proposed ASR, the speech recognition experiment was carried out the 60 Korean isolated words obtained from the six speakers. Moreover, the experiment results showed that the overall word recognition rates were comparable with the conventional speech-based ASR methods and the performance of the proposed method was superior to the conventional signal channel ASR method. Especially, the average recognition rate of 90 % was maintained under the noise environments.

Purely neural network (NN) based speech separation and enhancement methods, although can achieve good objective scores, inevitably cause nonlinear speech distortions that are harmful for the automatic speech recognition (ASR). On the other hand, the minimum variance distortionless response (MVDR) beamformer with NN-predicted masks, although can significantly reduce speech distortions, has limited noise reduction capability. In this paper, we propose a multi-tap MVDR beamformer with complex-valued masks for speech separation and enhancement. Compared to the state-of-the-art NN-mask based MVDR beamformer, the multi-tap MVDR beamformer exploits the inter-frame correlation in addition to the inter-microphone correlation that is already utilized in prior arts. Further improvements include the replacement of the real-valued masks with the complex-valued masks and the joint training of the complex-mask NN. The evaluation on our multi-modal multi-channel target speech separation and enhancement platform demonstrates that our proposed multi-tap MVDR beamformer improves both the ASR accuracy and the perceptual speech quality against prior arts.

A complex-valued multichannel speech enhancement learning algorithm for optimal tradeoff between noise reduction and speech distortion

Dynamic controllable speech enhancement models based on quantile loss functions

In this paper, we present a new Speech Enhancement (SE) technique capable of running on a smartphone, as an assistive device for hearing aids (HAs). The developed method incorporates the coherence between the speech and noise signals to obtain a SE gain function which is used in conjunction with the gain function obtained by Spectral Subtraction using adaptive gain averaging. SE using coherence based gain function is found to suppress the background noise well, while inducing speech distortion. On the other hand, SE using Spectral Subtraction improves speech quality with tolerable speech distortion, but introduces background musical noise for certain noise types. The weighted fusion of the two gain functions strikes a balance between noise suppression and speech distortion. Also it allows the user to control the weighting factor based on the noisy environment and their comfort level of hearing. The developed method is computationally fast and operates in real-time. The proposed method was evaluated for machinery, babble, and car noise types, using both objective and subjective measures for both quality and intelligibility of the enhanced speech. The results show significant improvements in comparison with stand-alone Spectral Subtraction with weighted gain averaging SE methods.

Although the existing literature has explored the link between cognitive functioning and speech recognition in noise, the specific role of fluid intelligence still needs to be studied. Given the established association between working memory capacity (WMC) and fluid intelligence and the predictive power of WMC for speech recognition in noise, we aimed to elucidate the mediating role of fluid intelligence. We used data from the n200 study, a longitudinal investigation into aging, hearing ability, and cognitive functioning. We analyzed two age-matched samples: participants with hearing aids and a group with normal hearing. WMC was assessed using the Reading Span task, and fluid intelligence was measured with Raven's Progressive Matrices. Speech recognition in noise was evaluated using Hagerman sentences presented to target 80% speech-reception thresholds in four-talker babble. Data were analyzed using mediation analysis to examine fluid intelligence as a mediator between WMC and speech recognition in noise. We found a partial mediating effect of fluid intelligence on the relationship between WMC and speech recognition in noise, and that hearing status did not moderate this effect. In other words, WMC and fluid intelligence were related, and fluid intelligence partially explained the influence of WMC on speech recognition in noise. This study shows the importance of fluid intelligence in speech recognition in noise, regardless of hearing status. Future research should use other advanced statistical techniques and explore various speech recognition tests and background maskers to deepen our understanding of the interplay between WMC and fluid intelligence in speech recognition.

This paper presents a subband adaptive filter based on wavelet filter-bank for speech enhancement. The adaptation of Kalman filter in wavelet domain has effectively reduced the non-stationary noise. A perceptual weighting filter exploiting the masking properties of psychoacoustic model is concatenated with the Kalman filter to further improve the intelligibility of speech. The proposed method owns its merits from the successful porting of Kalman filter into the wavelet domain so that speech analysis and enhancement can be carried out in time-frequency spectrum based on the human auditory model. Experimental results show that the speech enhancement system is capable of reducing noise with little speech degradation in adverse noise environments and the overall performance is superior to several competitive methods.

The application of the Kalman filter to the single-microphone speech enhancement task is presented in this chapter. Among numerous published algorithms, an important sub-group employs the estimate-maximize (EM) procedure to iteratively estimate the spectral parameters of the speech and noise signals. We elaborate on a specific member of this sub-group. In the E-step, the Kalman smoother is applied and in the M-step, a non-standard Yule-Walker equation set is solved. An approximated EM algorithm is derived by applying the gradient-descent method to the likelihood function. We obtain a sequential, computationally efficient, algorithm. It is then shown, that the sequential parameter estimation can be replaced by a Kalman filter to obtain a dual speech and parameters Kalman filter. A natural generalization to the dual scheme is an estimation scheme in which both speech and parameters are jointly estimated by applying a nonlinear extension to the Kalman filter, namely the unscented Kalman filter. Extensive experimental study, using real speech and noise signals is provided to compare the proposed methods with alternative speech enhancement algorithms. Kalman filter based algorithms are shown to maintain the natural speech quality. However, their noise reduction ability is limited.

The current study investigates whether long reverberation increases listening effort during speech recognition. Listening effort during word recognition in multi-talker babble noise was assessed with or without high levels of reverberation. A dual-task paradigm was adopted, in which the primary task was word recognition in noise at individually selected signal-to-noise ratio (SNR) that yielded an average performance level of 50% correct, and the secondary task was a visual-tracking task with individually adjusted difficulty level to yield an average performance level of 85% correct. In each 30-s trial, seven monosyllabic words were presented sequentially at a rate of four seconds per word. Young normal-hearing listeners were instructed to verbally repeat each word while performing the secondary task. In the reverberant condition, a cascade of all-pass filters was used to achieve a reverberation time of 1 s without altering the original spectrum of the speech. For the primary task in isolation, the reverberant condition required 10 dB or more in SNR to achieve the 50% target performance level. When the listeners performed the two tasks simultaneously, no consistent adverse effect of reverberation was found on the performance of the primary or secondary task compared to the no-reverberation condition.The current study investigates whether long reverberation increases listening effort during speech recognition. Listening effort during word recognition in multi-talker babble noise was assessed with or without high levels of reverberation. A dual-task paradigm was adopted, in which the primary task was word recognition in noise at individually selected signal-to-noise ratio (SNR) that yielded an average performance level of 50% correct, and the secondary task was a visual-tracking task with individually adjusted difficulty level to yield an average performance level of 85% correct. In each 30-s trial, seven monosyllabic words were presented sequentially at a rate of four seconds per word. Young normal-hearing listeners were instructed to verbally repeat each word while performing the secondary task. In the reverberant condition, a cascade of all-pass filters was used to achieve a reverberation time of 1 s without altering the original spectrum of the speech. For the primary task in isolation, the reverbe...