Speech Enhancement Using Nonnegative Matrix Factorization with Temporal Continuity

Non-negative matrix factorization for speech/music separation using source dependent decomposition rank, temporal continuity term and filtering

In this work, we will consider the dimension reduction of the set of time series, such as economic data, to find the meaningful basis vector for the set of data, and indicate which data use which basis vector. Usually each of the time series is analyzed independently in economics but here we will analyze the set of time series simultaneously. Since some of the economic data are measured as positive values and we want to decompose them as a mixture of the parts, we will apply non-negative matrix factorization to the economic data. Non-negative matrix factorization can compress dimensions by approximating a non-negative matrix with the product of two non-negative matrices. The two non-negative matrices are called the coefficient matrix and the basis matrix, and the basis matrix can be considered as a dimensionally compressed matrix. If the standard non-negative matrix factorization is used for economic data, the basis matrix may not be smooth. We think that the basis vectors should be smooth except a few special economical incidents. In the proposed method, a Graph-based non-negative matrix factorization is introduced to regularize the basis matrix of the time series. A path graph for representing the time series of economic data is incorporated into the non-negative matrix factorization as regularization. As a result, basis vectors that maintains the time series of economic data are decomposed. Furthermore, we propose to introduce a sparsity in the non-negative matrix factorization. Traditionally, the sparsity incorporated into non-negative matrix factorization has been used for basis vectors. However, the proposed method introduces the sparsity for coefficient vectors. Thus the proposed method, which simultaneously incorporates the sparsity for the coefficient vectors and the smoothness for the basis vectors, can extract the smooth basis vectors and the original economical data are approximated as the weighted sum of the few bases vectors. This allows us to discover economic trends and the best-fit trends for each data at the same time.

In order to improve the speech enhancement effect and overcome the weakness of falling into the local optimum of the traditional nonnegative matrix factorization, this paper presents speech enhancement methodology based on nonnegative matrix factorization combined with the improved genetic algorithm. The methodology consists of training stage and enhancement stage. In the training stage, instead of gradient descent, iterative learning is achieved by selecting several directions in super flat close to deflector cone as the mutation. By adding the mutation operator and simulated annealing operator to NMF, the proposed algorithm is used to acquire the accurate noise dictionary as prior information. In the de-noising stage, NMF-IGA is used to evaluate the dictionary matrix and the activation matrix of the de-noised speech. Then using the dictionary of de-noised speech to get the enhanced speech spectrogram and reconstruct the enhanced speech. It is verified that the proposed algorithm yields less residual noise and better speech quality than traditional nonnegative matrix factorization and Bayes nonnegative matrix factorization algorithm.

This study proposes a bi-directional recurrent neural network (Bi-RNN) post-processing method for speech enhancement (SE) at low signal-to noise ratios (SNR). Current speech enhancement solutions performed badly under low SNR situations. Loizou and Kim proposed a solution to reduce speech distortion errors in time-frequency (T-F) domain but it requires the knowledge of ground truth. As ground truth is unknown in real-life applications, the current study proposes to use a Bi-RNN to implement Loizou and Kim’s solution as a post-processing method for SE engines. Our solutions do not require prior knowledge of ground truth. The effectiveness of the proposed method is investigated with a spectral subtraction (SS) SE engine, a non-negative matrix factorization (NMF) SE engine, and a deep neural network ideal ratio mask (DNN-IRM) SE engine, under matched/mis-matched noise and different SNR conditions. Experimental results demonstrate that the proposed post-processing method effectively improved both perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) for all of these SE engines, especially at low SNR conditions.

Non-negative matrix factorization (NMF) has been one of the most popular methods for feature learning in the field of machine learning and computer vision. Most existing works directly apply NMF on high-dimensional image datasets for computing the effective representation of the raw images. However, in fact, the common essential information of a given class of images is hidden in their low rank parts. For obtaining an effective low-rank data representation, we in this paper propose a non-negative low-rank matrix factorization (NLMF) method for image clustering. For the purpose of improving its robustness for the data in a manifold structure, we further propose a graph regularized NLMF by incorporating the manifold structure information into our proposed objective function. Finally, we develop an efficient alternating iterative algorithm to learn the low-dimensional representation of low-rank parts of images for clustering. Alternatively, we also incorporate robust principal component analysis into our proposed scheme. Experimental results on four image datasets reveal that our proposed methods outperform four representative methods.

Speech enhancement based on statistical models has shown good performance, but the performance degrades when environment noise is highly non-stationary due to the stationary assumption. On the contrary, the template-based enhancement methods are more robust to non-stationary noise, but these are heavily dependent on a priori information present in training data. In order to get over both of the shortcomings, we propose a novel speech enhancement method which combines the statistical model-based enhancement scheme with the template-based enhancement. To reduce a dependency on a priori information, the speech and noise bases are updated simultaneously using the estimated speech presence probability, which is obtained from statistical model-based enhancement. Experimental results showed that the proposed method outperformed not only the statistical model-based and non-negative matrix factorization (NMF) approaches, but also their combination implemented with existing bases update rule in various kinds of noise.

The speech signal in general is corrupted by noise and the noise signal does not affect the speech signal uniformly over the entire spectrum. An improved Wiener filtering method is proposed in this paper for reducing background noise from speech signal in colored noise environments. In view of nonlinear variation of human ear sensibility in frequency spectrum, nonlinear multi-band Bark scale frequency spacing approach is used. The cross-correlation between the speech and noise signal is considered in the proposed method to reduce colored noise. To overcome harmonic distortion introduced in enhanced speech, in the proposed method regenerate the suppressed harmonics are regenerated. Objective and subjective tests were carried out to demonstrate improvement in the perceptual quality of speeches by the proposed technique.

The complexity of the speech enhancement problem has motivated many different solutions. However, most techniques address situations in which the target speech is fully intelligible and the background noise energy is low in comparison with that of the speech. Thus while current enhancement algorithms can improve the perceived quality, the intelligibility of the speech is not increased significantly and may even be reduced. Recent research shows that intelligibility of very noisy speech can be improved by the use of a binary mask, in which a binary weight is applied to each time-frequency bin of the input spectrogram. There are several alternative goals for the binary mask estimator, based either on the Signalto-Noise Ratio (SNR) of each time-frequency bin or on the speech signal characteristics alone. Our approach to the binary mask estimation problem aims to preserve the important speech cues independently of the noise present by identifying time-frequency regions that contain significant speech energy. The speech power spectrum varies greatly for different types of speech sound. The energy of voiced speech sounds is concentrated in the harmonics of the fundamental frequency while that of unvoiced sounds is, in contrast, distributed across a broad range of frequencies. To identify the presence of speech energy in a noisy speech signal we have therefore developed two detection algorithms. The first is a robust algorithm that identifies voiced speech segments and estimates their fundamental frequency. The second detects the presence of sibilants and estimates their energy distribution. In addition, we have developed a robust algorithm to estimate the active level of the speech. The outputs of these algorithms are combined with other features estimated from the noisy speech to form the input to a classifier which estimates a mask that accurately reflects the time-frequency distribution of speech energy even at low SNR levels. We evaluate a mask-based speech enhancer on a range of speech and noise signals and demonstrate a consistent increase in an objective intelligibility measure with respect to noisy speech.

Non-negative matrix factorization (NMF) is widely used for music transcription because of its efficiency. However, the conventional NMF-based music transcription algorithm often causes harmonic confusion errors or time split-up errors, because the NMF decomposes the time-frequency data according to the activated frequency in its time. To solve these problems, we proposed an NMF with temporal continuity and harmonicity constraints. The temporal continuity constraint prevented the time split-up of the continuous time components, and the harmonicity constraint helped to bind the fundamental with harmonic frequencies by reducing the additional octave errors. The transcription performance of the proposed algorithm was compared with that of the conventional algorithms, which showed that the proposed method helped to reduce additional false errors and increased the overall transcription performance.

Guitar string separation is a novel and complicated task. Guitar notes are not pure steady-state signals, hence, we hypothesize that neither Non-Negative Matrix Factorization (NMF) nor Non-Negative Matrix Factor Deconvolution (NMFD) are optimal for separating them. Therefore, we separate steady-state and transient parts using Harmonic-Percussive Separation (HPS) as a preprocessing step. Then, we use NMF for factorizing the harmonic part and NMFD for deconvolving the percussive part. We make use of a hexaphonic guitar dataset which allows for objective evaluation. In addition, we compare several types of time-frequency mask and introduce an intuitive way to combine a binary mask with a ratio mask. We show that the HPS mask type has an effect on source estimation. Our proposed method achieved results comparable to NMF without HPS. Finally, we show that the optimal mask at the final separation stage depends on the estimation algorithm.

Non-negative matrix factorization (NMF) has been successfully applied to speech enhancement in non-stationary noisy environments. Recently proposed online semi-supervised NMF algorithms are of particular interest as they carry the two nice properties (online and semi-supervised) of classical speech enhancement approaches. These algorithms, however, have only been evaluated using noisy mixtures shorter than 30 seconds. In this paper we find that these algorithms work well when it is run for less than 1 minute, but degradation of the enhanced speech signal starts to appear after 2 minutes. We analyze that the reason is due to the inappropriate dictionary update rule, which gradually loses its ability in updating the speech dictionary. We then propose a simple rotational reset strategy to solve the problem: Instead of continuously updating the entire speech dictionary, we periodically and rotationally select elements and reset their values to random numbers. Experiments show that this strategy successfully solves the degradation problem and the improved algorithm outperforms classical speech enhancement algorithms significantly even when they are run for 10 minutes.

This paper proposes an exemplar-based speech enhancement method based on high-resolution STFT magnitude spectrograms, where a selection of the nonnegative training data is used as the dictionary to provide a holistic nonnegative representation of the test data. We discuss how this exemplar-based model ensures that the enhanced speech signal falls on the speech manifold, which improves the quality of the enhanced speech signal. To exploit the temporal continuity, a vector autoregressive model is used to model the activations where the model parameters are learned using a new NMF-based approach. Results from several supervised and semi-supervised speech enhancement experiments indicate that the proposed exemplar-based method outperforms the considered supervised and unsupervised denoising algorithms in terms of both segmental SNR and PESQ at different input SNRs. Index Terms: nonnegative matrix factorization, exemplarbased noise reduction, overcomplete dictionary

Reducing highly non-stationary transient noise, such as keyboard typing noise, remains a challenging problem for many singlechannel speech enhancement algorithms. This paper proposes two approaches based on nonnegative matrix factorization (NMF) and probabilistic latent component analysis for transient noise reduction using a pre-trained transient noise dictionary and a universal speaker-independent speech dictionary. In addition, we develop an NMF-based speech enhancement scheme to simultaneously reduce transient and non-transient background noise, in which a low-dimensional dictionary is learnt from the noisy observations to model the background noise. We exploit the temporal dependencies of speech and background noise to design and apply informative priors via a probabilistic framework, while ignoring the temporal dynamics of the transient noise. Experimental results show that the proposed algorithms can improve the perceptual evaluation of speech quality (PESQ) up to 1.2 MOS for the keyboard typing noise.

Single‐microphone speech enhancement algorithms by using nonnegative matrix factorization can only utilize the temporal and spectral diversity of the received signal, making the performance of the noise suppression degrade rapidly in a complex environment. Microphone arrays have spatial selection and high signal gain, so it applies to the adverse noise conditions. In this paper, we present a new algorithm for speech enhancement based on two microphones with nonnegative matrix factorization. The interchannel characteristic of each nonnegative matrix factorization basis can be modeled by the adopted method, such as the amplitude ratios and the phase differences between channels. The results of the experiment confirm that the proposed algorithm is superior to other dual‐microphone speech enhancement algorithms.

An improved Wiener filtering method for reducing background noise added to speech in colored noise environments like car engine is proposed in this paper. This implementation uses the crosscorrelartion between the speech and noise signals. The noise signal does not affect the speech signal uniformly over the whole spectrum. A nonlinear multi-band Bark scale frequency spacing approach was proposed to reduce colored noise. However, classic short-time noise reduction techniques, including this Wiener filter, introduce some residual noise that remains in the enhanced speech signal. To reduce residual noise the proposed Wiener filter is combined with constrained perceptual weighting filter. Objective and subjective tests demonstrated the proposed technique can improve the perceptual quality of speeches.