Joint-diagonalizability-constrained multichannel nonnegative matrix factorization based on time-variant multivariate complex sub-Gaussian distribution

This article describes a computationally-efficient blind source separation (BSS) method based on the independence, low-rankness, and directivity of the sources. A typical approach to BSS is unsupervised learning of a probabilistic model that consists of a source model representing the time-frequency structure of source images and a spatial model representing their inter-channel covariance structure. Building upon the low-rank source model based on nonnegative matrix factorization (NMF), which has been considered to be effective for inter-frequency source alignment, multichannel NMF (MNMF) assumes source images to follow multivariate complex Gaussian distributions with unconstrained full-rank spatial covariance matrices (SCMs). An effective way of reducing the computational cost and initialization sensitivity of MNMF is to restrict the degree of freedom of SCMs. While a variant of MNMF called independent low-rank matrix analysis (ILRMA) severely restricts SCMs to rank-1 matrices under an idealized condition that only directional and less-echoic sources exist, we restrict SCMs to jointly-diagonalizable yet full-rank matrices in a frequency-wise manner, resulting in FastMNMF1. To help inter-frequency source alignment, we then propose FastMNMF2 that shares the directional feature of each source over all frequency bins. To explicitly consider the directivity or diffuseness of each source, we also propose rank-constrained FastMNMF that enables us to individually specify the ranks of SCMs. Our experiments showed the superiority of FastMNMF over MNMF and ILRMA in speech separation and the effectiveness of the rank constraint in speech enhancement.

This paper proposes visually-informed sound source separation for audio-visual understanding of indoor scenes captured by distributed microphone arrays and cameras. Our approach leverages the 3D information of sound-emitting objects, reconstructed via 3D Gaussian splatting (3DGS), to overcome a limitation of modern blind source separation methods like multichannel nonnegative matrix factorization (MNMF). While adaptable and potentially performant, the iterative optimization of MNMF often converges to poor local minima due to the highly-expressive full-rank spatial covariance matrices (SCMs) of sources. Our key idea is to treat the set of 3D Gaussians representing a sizable sound source object as a collection of sub-sources that share an audio signal but have unique emission weights, both of which are to be estimated jointly from an observed mixture. To enforce this structure, we guide MNMF by regularizing the SCM of each source object at each frequency. Specifically, we use a prior that centers the SCM estimate around a weighted sum of theoretical SCMs, which are analytically derived from the 3D Gaussian positions. Experiments with simulated data, featuring two 3D human models, demonstrated the effectiveness of the proposed method. To our knowledge, this is the first work to use 3D Gaussians as a common primitive for joint audio-visual analysis.

This paper describes an ultimate covariance-aware multichannel extension of nonnegative matrix factorization (NMF) for blind source separation (BSS). A typical approach to BSS is to integrate a low-rank source model with a full-rank spatial model as multichannel NMF (MNMF) based on full-rank spatial covariance matrices (CMs) or its efficient version named FastMNMF based on jointly-diagonalizable spatial CMs do. The NMF-based phase-unaware source model, however, can deal with only the positive cooccurrence relations between time-frequency bins. To overcome this limitation, we propose an efficient multichannel extension of correlated tensor factorization (CTF) named FastMCTF based on jointly-diagonalizable temporal, frequency, and spatial CMs. Integration of the jointly-diagonalizable full-rank source model proposed by FastCTF with the jointly-diagonalizable full-rank spatial model proposed by FastMNMF enables us to completely consider the positive and negative covariance relations between frequency bins, time frames, and channels. We derive a convergence-guaranteed parameter estimation algorithm based on the multiplicative update and iterative projection and experimentally show the potential of the proposed method.

This paper presents a robust generalization of multichannel nonnegative matrix factorization (MNMF) for blind source separation of mixture audio signals recorded by a microphone array. In conventional MNMF, the complex spectra of observed mixture signals are assumed to be complex Gaussian distributed and are decomposed into the product of the power spectra, temporal activations, and spatial correlation matrices of individual sources in such a way that the complex Gaussian likelihood is maximized. Since the mixture spectra usually include outliers, we propose MNMF based on the complex Student's t likelihood, called t-MNMF, including the original MNMF as a special case. The parameters of t-MNMF can be iteratively optimized with an efficient multiplicative updating algorithm. Experiments showed that t-MNMF with a certain range of degrees of freedom tends to be insensitive to parameter initialization and outperform conventional MNMF.

In this letter, we develop a multichannel blind source separation algorithm based on a beamspace transform and the multichannel nonnegative matrix factorization (NMF) method. The conventional multichannel NMF algorithm performs well with multichannel mixing data, but there is still room for enhancement in multichannel real-world recording data. In this letter, we consider a beamspace-time-frequency domain data model for multichannel NMF method, and enhance the conventional method using a beamspace transform. Our decomposition algorithm is applied to 2-channel and 4-channel unsupervised audio source separation, using a dataset from the international Signal Separation Evaluation Campaign 2010 (SiSEC 2010). Our algorithm shows a better performance than the conventional NMF method in an evaluation results.

To improve the performance of blind audio source separation of convolutive mixtures, the local Gaussian model (LGM) having full rank covariance matrices proposed by Duong et al. is extended. The previous model basically assumes that all sources contribute to each time-frequency slot, which may fail to capture the characteristic of signals with many intermittent silent periods. A constraint on source sets that contribute to each time-frequency slot is therefore explicitly introduced. This approach can be regarded as a relaxation of the sparsity constraint in the conventional time-frequency mask. The proposed model is jointly optimized among the original local Gaussian model parameters, the relaxed version of the time-frequency mask, and a permutation alignment, leading to a robust permutation-free algorithm. We also present a novel multi-channel Wiener filter weighted by a relaxed version of the time-frequency mask. Experimental results over noisy speech signals show that the proposed model is effective compared with the original local Gaussian model and is comparable to its extension, the multi-channel nonnegative matrix factorization.

We consider inference in a general data-driven object-based model of multichannel audio data, assumed generated as a possibly underdetermined convolutive mixture of source signals. We work in the short-time Fourier transform (STFT) domain, where convolution is routinely approximated as linear instantaneous mixing in each frequency band. Each source STFT is given a model inspired from nonnegative matrix factorization (NMF) with the Itakura-Saito divergence, which underlies a statistical model of superimposed Gaussian components. We address estimation of the mixing and source parameters using two methods. The first one consists of maximizing the exact joint likelihood of the multichannel data using an expectation-maximization (EM) algorithm. The second method consists of maximizing the sum of individual likelihoods of all channels using a multiplicative update algorithm inspired from NMF methodology. Our decomposition algorithms are applied to stereo audio source separation in various settings, covering blind and supervised separation, music and speech sources, synthetic instantaneous and convolutive mixtures, as well as professionally produced music recordings. Our EM method produces competitive results with respect to state-of-the-art as illustrated on two tasks from the international Signal Separation Evaluation Campaign (SiSEC 2008).

Underdetermined blind source separation using normalized spatial covariance matrix and multichannel nonnegative matrix factorization

Conventionally, there is a method to use auscultation when medical doctors diagnose a disease from heartbeats with heart noise. However, the respiratory sound and the heart sound are mixed. Further, it is difficult for the desired heart sound to be heard. For solving this problem, we propose a method to separate respiratory sound and heart sound based on blind source separation (BSS). In BSS method, we use a non-linear function corresponding to the probability distribution of source signals in order to separate. A distribution of the respiratory sound is a high-order sub-Gaussian distribution close to uniform distribution, a distribution of the heart sound is a super Gaussian distribution. In the conventional method, there is no non-linear function corresponding to both distribution. Therefore, in this paper, in order to separate the respiratory sound and the heart sound, we propose new non-linear function accurately separated from super Gaussian distribution through sub-Gaussian distribution to uniform distribution. Then, we build the BSS method based on the proposed non-linear function. We evaluate the proposed method by computer simulation.

In the recent past, blind source separation (BSS) algorithms using multivariate statistical data analysis technique have been successfully used for source identification and separation in the field of biomedical and statistical signal processing. Recently numbers of different BSS techniques have been developed. With BSS methods being the feasible method for source separation and decomposition of biosignals, it is important to compare the different techniques and determine the most suitable method for the applications. This paper presents the performance of five BSS algorithms (SOBI, TDSEP, FastICA, JADE and Infomax) for decomposition of sEMG to identify subtle finger movements. It is observed that BSS algorithms based on second-order statistics (SOBI and TDSEP) gives better performance compared to algorithms based on higher-order statistics (FastICA, JADE and infomax).

Blind single-channel source separation is a long-standing machine learning and signal processing problem. Traditional blind source separation (BSS) algorithms were proposed to solve this task utilizing multiple signal constraints. Generative adversarial network (GAN) are free from statistical constraints and samples, but the role of adversarial training in the task of BSS has not been fully demonstrated. Therefore, a new separation network model that enables to learn the known separated signal distribution from the stepwise fine estimation of the unknown mixture distribution was presented in this paper, and a self-attention mechanism was introduced to solve the problem of the blurring details of the generated image by the generator which preserves image details in the process of image separation. Compared with the existing single-channel blind source separation algorithm based on generative adversarial network-neural egg separation (NES), the detailed information of this new separation algorithm is more prominent, and the source signal in the mixed image has been separated more effectively, and has better separation performance than the classic blind source separation algorithms.

During the last decade, distributed microphone arrays have been proposed in order to increase accuracy and spatial coverage of speaker localization systems operating in large and reverberant rooms. In principle, the framework provided by a distributed microphone network can also be applied effectively when using Blind Source Separation (BSS). Separation is commonly performed by processing the signals sampled at closely spaced microphones in a single adaptation step, for example by means of Independent Component Analysis (ICA). When the microphone spacing or the distance between source and microphones increase, the separation performance reduces due to spatial aliasing effects and to a reduced spatial coherence at microphones. In this paper we propose a new method, here referred to as Cooperative Wiener ICA (CW-ICA), which is able to apply BSS to signals acquired by a network of distributed microphone arrays. Different ICA adaptations are applied to the signals recorded by each array and are interconnected in order to constrain each adaptation to converge to a solution related to the same physical interpretation. A preliminary analysis on a network of two arrays shows that the proposed method can be applied successfully to source separation and localization tasks.

This paper proposes a novel method for simultaneously solving the problems of underdetermined blind source separation (BSS), source activity detection, dereverberation and direction-of-arrival (DOA) estimation by introducing an extension of the "multichannel factorial hidden Markov model (MFH-MM)." The MFHMM is an extension of the multichannel non-negative matrix factorization (NMF) modeL in which the basis spectra are allowed to vary over time according to the transitions of the hidden states. This model has allowed us to perform source separation, source activity detection and dereverberation in a unified manner. In our previous model, the spatial covariance of each source has been treated as a model parameter. This has led the entire generative model to have an unnecessarily high degree of freedom, and thus the parameter inference has been prone to getting trapped into undesired local optima. To reasonably restrict the solution space of the spatial covariance matrix of each source, we propose to describe it as a weighted sum of the fixed spatial covariance matrix corresponding to the discrete set of DOAs. Through the parameter inference, the proposed model allows us to simultaneously solve the problems of underdetermined BSS, source activity detection, dereverberation and DOA estimation. Experimental results revealed that the proposed method was superior to a previous method in terms of the signal-to-distortion ratios of separated signals.

This paper presents a new low-latency online blind source separation (BSS) algorithm. Although algorithmic delay of a frequency domain online BSS can be reduced simply by shortening the short-time Fourier transform (STFT) frame length, it degrades the source separation performance in the presence of reverberation. This paper proposes a method to solve this problem by integrating BSS with Weighted Prediction Error (WPE) based dereverberation. Although a simple cascade of online BSS after online WPE upgrades the separation performance, the overall optimality is not guaranteed. Instead, this paper extends a recently proposed batch processing algorithm that can jointly optimize dereverberation and separation so that it can perform online processing with low computational cost and little processing delay (< 12 ms). The results of a source separation experiment in a noisy car environment suggest that the proposed online method has better separation performance than the simple cascaded methods.

A new blind image source separation algorithm based on feedback sparse component analysis