Multichannel Nonnegative Matrix Factorization in Convolutive Mixtures for Audio Source Separation

In this letter, we develop a multichannel blind source separation algorithm based on a beamspace transform and the multichannel nonnegative matrix factorization (NMF) method. The conventional multichannel NMF algorithm performs well with multichannel mixing data, but there is still room for enhancement in multichannel real-world recording data. In this letter, we consider a beamspace-time-frequency domain data model for multichannel NMF method, and enhance the conventional method using a beamspace transform. Our decomposition algorithm is applied to 2-channel and 4-channel unsupervised audio source separation, using a dataset from the international Signal Separation Evaluation Campaign 2010 (SiSEC 2010). Our algorithm shows a better performance than the conventional NMF method in an evaluation results.

To improve the performance of blind audio source separation of convolutive mixtures, the local Gaussian model (LGM) having full rank covariance matrices proposed by Duong et al. is extended. The previous model basically assumes that all sources contribute to each time-frequency slot, which may fail to capture the characteristic of signals with many intermittent silent periods. A constraint on source sets that contribute to each time-frequency slot is therefore explicitly introduced. This approach can be regarded as a relaxation of the sparsity constraint in the conventional time-frequency mask. The proposed model is jointly optimized among the original local Gaussian model parameters, the relaxed version of the time-frequency mask, and a permutation alignment, leading to a robust permutation-free algorithm. We also present a novel multi-channel Wiener filter weighted by a relaxed version of the time-frequency mask. Experimental results over noisy speech signals show that the proposed model is effective compared with the original local Gaussian model and is comparable to its extension, the multi-channel nonnegative matrix factorization.

This paper presents a robust generalization of multichannel nonnegative matrix factorization (MNMF) for blind source separation of mixture audio signals recorded by a microphone array. In conventional MNMF, the complex spectra of observed mixture signals are assumed to be complex Gaussian distributed and are decomposed into the product of the power spectra, temporal activations, and spatial correlation matrices of individual sources in such a way that the complex Gaussian likelihood is maximized. Since the mixture spectra usually include outliers, we propose MNMF based on the complex Student's t likelihood, called t-MNMF, including the original MNMF as a special case. The parameters of t-MNMF can be iteratively optimized with an efficient multiplicative updating algorithm. Experiments showed that t-MNMF with a certain range of degrees of freedom tends to be insensitive to parameter initialization and outperform conventional MNMF.

This article describes a computationally-efficient blind source separation (BSS) method based on the independence, low-rankness, and directivity of the sources. A typical approach to BSS is unsupervised learning of a probabilistic model that consists of a source model representing the time-frequency structure of source images and a spatial model representing their inter-channel covariance structure. Building upon the low-rank source model based on nonnegative matrix factorization (NMF), which has been considered to be effective for inter-frequency source alignment, multichannel NMF (MNMF) assumes source images to follow multivariate complex Gaussian distributions with unconstrained full-rank spatial covariance matrices (SCMs). An effective way of reducing the computational cost and initialization sensitivity of MNMF is to restrict the degree of freedom of SCMs. While a variant of MNMF called independent low-rank matrix analysis (ILRMA) severely restricts SCMs to rank-1 matrices under an idealized condition that only directional and less-echoic sources exist, we restrict SCMs to jointly-diagonalizable yet full-rank matrices in a frequency-wise manner, resulting in FastMNMF1. To help inter-frequency source alignment, we then propose FastMNMF2 that shares the directional feature of each source over all frequency bins. To explicitly consider the directivity or diffuseness of each source, we also propose rank-constrained FastMNMF that enables us to individually specify the ranks of SCMs. Our experiments showed the superiority of FastMNMF over MNMF and ILRMA in speech separation and the effectiveness of the rank constraint in speech enhancement.

Joint-diagonalizability-constrained multichannel nonnegative matrix factorization based on time-variant multivariate complex sub-Gaussian distribution

Spatial gradients in the primary geomagnetic fields directly contribute to both the amplitudes and phases of inter-station transfer functions (IS-TFs). This suggests that, for the analysis of subsurface resistivity structures, IS-TFs should be carefully treated by checking the establishment of the plane-wave assumption. Geomagnetic time-series data include various and complicated characteristics and accordingly, time–frequency domain analysis is suitable for the discussion of spatial gradients of time-varying geomagnetic fields. However, such evaluations are complicated by the huge amount of information contained in the spectrograms from several stations. Therefore, we propose a Multi-Channel Nonnegative Matrix Factorization (MC-NMF) method that can decompose raw spectrograms into several components, allowing the spatial gradient of each geomagnetic temporal variation to be identified. We confirm that such components actually affect the estimation of IS-TFs using data acquired at the Kakioka and Memambetsu magnetic observatories in Japan. We derive the year-to-year changes in IS-TFs from each set of paired stations among Kakioka, Kanoya, and Memambetsu observatories. Although the IS-TFs should exhibit opposite polarities (a negative correlation) when the input and output observatories are swapped; surprisingly, some of them have “identical” polarities. The application of MC-NMF shows that the analyzed geomagnetic data include several components that have various spatial gradients. Although IS-TFs sometimes fail to give the expected implication regarding the spatial gradients of geomagnetic temporal variations, MC-NMF can verify whether the IS-TFs exhibit any spatial gradients. Thus, the use of IS-TFs with MC-NMF can yield better implications regarding subsurface resistivity information.

This paper addresses the problem of multichannel audio source separation in under-determined convolutive mixtures. We target a semi-blind scenario assuming that the mixing filters are known. The convolutive mixing process is exactly modeled using the time-domain impulse responses of the mixing filters. We propose a Student's t time-frequency source model based on non-negative matrix factorization (NMF). The Student's t distribution being heavy-tailed with respect to the Gaussian, it provides some flexibility in the modeling of the sources. We also study a simpler Student's t sparse source model within the same general source separation framework. The inference procedure relies on a variational expectation-maximization algorithm. Experiments show the advantage of using an NMF model compared with the sparse source model. While the Student's t NMF source model leads to slightly better results than our previous Gaussian one, we demonstrate the superiority of our method over two other approaches from the literature.

A solution for separating speech from music signal as a single channel source separation is Non-negative Matrix Factorization (NMF). In this approach spectrogram of each source signal is factorized as multiplication of two matrices which are known as basis and weight matrices. To achieve proper estimation of signal spectrogram, weight and basis matrices are updated iteratively. To estimate distance between signal and its estimation a cost function is used usually. Different cost functions have been introduced based on Kullback-Leibler (KL) and Itakura-Saito (IS) divergences. IS divergence is scale-invariant and so it is suitable for the conditions in which the coefficients of signal have a large dynamic range, for example in music short-term spectra. Based on this IS property, in this paper, we propose to use IS divergence as cost function of NMF in the training stage for music and on the other hand we suggest to use KL divergence as NMF cost function in the training stage for speech. Moreover, in the decomposition stage, we propose to use a linear combination of these two divergences in addition to a regularization term which considers temporal continuity information as a prior knowledge. Experimental results on one hour of speech and music, shows a good trade-off between signal to inference ratio (SIR) of speech and music in comparison to conventional NMF methods.

This paper describes an ultimate covariance-aware multichannel extension of nonnegative matrix factorization (NMF) for blind source separation (BSS). A typical approach to BSS is to integrate a low-rank source model with a full-rank spatial model as multichannel NMF (MNMF) based on full-rank spatial covariance matrices (CMs) or its efficient version named FastMNMF based on jointly-diagonalizable spatial CMs do. The NMF-based phase-unaware source model, however, can deal with only the positive cooccurrence relations between time-frequency bins. To overcome this limitation, we propose an efficient multichannel extension of correlated tensor factorization (CTF) named FastMCTF based on jointly-diagonalizable temporal, frequency, and spatial CMs. Integration of the jointly-diagonalizable full-rank source model proposed by FastCTF with the jointly-diagonalizable full-rank spatial model proposed by FastMNMF enables us to completely consider the positive and negative covariance relations between frequency bins, time frames, and channels. We derive a convergence-guaranteed parameter estimation algorithm based on the multiplicative update and iterative projection and experimentally show the potential of the proposed method.

This study evaluates audio source separation with Fast Independent Component Analysis (FastICA) in a fully specified, reproducible pipeline and benchmarks it against Principal Component Analysis (PCA) and Nonnegative Matrix Factorization (NMF). Three conversational recordings collected at the Department of Statistics, Pabna University of Science and Technology were canonicalized to 48 kHz WAV 96.63 s each, converted to mono, trimmed at 25 dB, RMS‐normalized, and time‐aligned. Sources were mixed with a fixed 3 × 3 row‐normalized Gaussian matrix. FastICA used parallel updates with a logcosh nonlinearity and unit‐variance whitening; PCA served as a multichannel linear baseline; NMF operated on a mono short‐time Fourier transform with KL divergence and NNDSVDA initialization, followed by soft masking and inverse transform. Performance was computed with BSS Eval after best‐permutation and scale alignment and summarized over 10 runs as mean ± SD. FastICA achieved Signal‐to‐Distortion Ratio (SDR) 53.51 ± 0.07 dB, Signal‐to‐Interference Ratio (SIR) 53.52 ± 0.07 dB, and Signal‐to‐Artifact Ratio (SAR) 79.58 ± 0.00 dB, well above a 20 dB high‐quality SDR threshold. PCA yielded SDR 2.79 ± 0.00 dB, SIR 2.79 ± 0.00 dB, SAR 80.64 ± 0.00 dB; NMF produced SDR −2.26 ± 0.00 dB, SIR 0.41 ± 0.00 dB, SAR 4.80 ± 0.00 dB. Waveform and spectrogram visualizations, together with descriptive, high‐order, and entropy statistics, corroborate these outcomes. The results establish FastICA as an effective classical approach for audio source separation and provide a transparent reference pipeline for comparative studies using SDR, SIR, and SAR.

This paper proposes visually-informed sound source separation for audio-visual understanding of indoor scenes captured by distributed microphone arrays and cameras. Our approach leverages the 3D information of sound-emitting objects, reconstructed via 3D Gaussian splatting (3DGS), to overcome a limitation of modern blind source separation methods like multichannel nonnegative matrix factorization (MNMF). While adaptable and potentially performant, the iterative optimization of MNMF often converges to poor local minima due to the highly-expressive full-rank spatial covariance matrices (SCMs) of sources. Our key idea is to treat the set of 3D Gaussians representing a sizable sound source object as a collection of sub-sources that share an audio signal but have unique emission weights, both of which are to be estimated jointly from an observed mixture. To enforce this structure, we guide MNMF by regularizing the SCM of each source object at each frequency. Specifically, we use a prior that centers the SCM estimate around a weighted sum of theoretical SCMs, which are analytically derived from the 3D Gaussian positions. Experiments with simulated data, featuring two 3D human models, demonstrated the effectiveness of the proposed method. To our knowledge, this is the first work to use 3D Gaussians as a common primitive for joint audio-visual analysis.

This paper presents a statistical method of audio source separation based on a nonparametric Bayesian extension of probabilistic latent component analysis (PLCA). A major approach to audio source separation is to use nonnegative matrix factorization (NMF) that approximates the magnitude spectrum of a mixture signal at each frame as the weighted sum of fewer source spectra. Another approach is to use PLCA that regards the magnitude spectrogram as a two-dimensional histogram of "sound quanta" and classifies each quantum into one of sources. While NMF has a physically-natural interpretation, PLCA has been used successfully for music signal analysis. To enable PLCA to estimate the number of sources, we propose Dirichlet process PLCA (DP-PLCA) and derive two kinds of learning methods based on variational Bayes and collapsed Gibbs sampling. Unlike existing learning methods for nonparametric Bayesian NMF based on the beta or gamma processes (BP-NMF and GaP-NMF), our sampling method can efficiently search for the optimal number of sources without truncating the number of sources to be considered. Experimental results showFed that DP-PLCA is superior to GaP-NMF in terms of source number estimation.

To solve the separation of multi-source signals and detect their features from a single channel, a signal separation method using multi-constraint non-negative matrix factorization (NMF) is proposed. In view of the existing NMF algorithm not performing well in the underdetermined blind source separation, the β-divergence constraints and determinant constraints are introduced in the NMF algorithm, which can enhance local feature information and reduce redundant components by constraining the objective function. In addition, the Sine-bell window function is selected as the processing method for short-time Fourier transform (STFT), and it can preserve the overall feature distribution of the original signal. The original vibration signal is first transformed into time-frequency domain with the STFT, which describes the local characteristic of the signal from the time-frequency distribution. Then, the multi-constraint NMF is applied to reduce the dimensionality of the data and separate feature components in the low dimensional space. Meanwhile, the parameter WK is constructed to filter the reconstructed signal that recombined with the feature component in the time domain. Ultimately, the separated signals will be subjected to envelope spectrum analysis to detect fault features. The simulated and experimental results indicate the effectiveness of the proposed approach, which can realize the separation of multi-source signals and their fault diagnosis of bearings. In addition, it is also confirmed that the proposed method, juxtaposed with the NMF algorithm of the traditional objective function, is more applicable for compound fault diagnosis of the rotating machinery.

We present a new probabilistic model for polyphonic audio termed Factorial Scaled Hidden Markov Model (FS-HMM), which generalizes several existing models, notably the Gaussian scaled mixture model and the Itakura-Saito Nonnegative Matrix Factorization (NMF) model. We describe two expectation-maximization (EM) algorithms for maximum likelihood estimation, which differ by the choice of complete data set. The second EM algorithm, based on a reduced complete data set and multiplicative updates inspired from NMF methodology, exhibits much faster convergence. We consider the FS-HMM in different configurations for the difficult problem of speech / music separation from a single channel and report satisfying results.

In this paper, a time-domain blind source separation algorithm for non-stationary convolutive mixtures is proposed by reprogramming the vectors of convolutive mixture model and generalizing the joint approximate diagonalization method. Firstly the sampling convolutive mixture signals are reseted for matching instantaneous mixture model, then considering non-stationarity of the sources, space whitening and joint block-diagonalization method is exploited to obtain the original signals. This algorithm simplifies the convolutive mixture problem into the instantaneous mixture problem from a new point of view, so it avoids domain transformation and convolution operation, as well as decreases the complexity. Computer simulation verifies its effectiveness and gives the analysis results about the effect on the signal to interference ratio as its parameter changes.