TERM Model: Tensor Ring Mixture Model for Density Estimation

Sum-Product Networks (SPNs) are expressive probabilistic models that provide exact, tractable inference. They achieve this efficiency by making use of local independence. On the other hand, mixtures of exchangeable variable models (MEVMs) are a class of tractable probabilistic models that make use of exchangeability of discrete random variables to render inference tractable. Exchangeability, which arises naturally in relational domains, has not been considered for efficient representation and inference in SPNs yet. The contribution of this paper is a novel probabilistic model which we call Exchangeability-Aware Sum-Product Networks (XSPNs). It contains both SPNs and MEVMs as special cases, and combines the ability of SPNs to efficiently learn deep probabilistic models with the ability of MEVMs to efficiently handle exchangeable random variables. We introduce a structure learning algorithm for XSPNs and empirically show that they can be more accurate than conventional SPNs when the data contains repeated, interchangeable parts.

This chapter introduces several fundamental models and algorithms for probabilistic clustering, including mixture models, Expectation-Maximization (EM) algorithm, and probabilistic topic models. It focuses on general framework of modeling, the probabilistic explanation, the standard algorithms to learn the model, and its applications. Mixture models are probabilistic models which are increasingly used to find the clusters for univariate and multivariate data. The chapter begins the discussion of mixture models, in which the values of the discrete latent variables can be interpreted as the assignments of data points to specific components of the mixture. It uses the Gaussian mixture model to motivate the EM algorithm in an informal way, and then gives a more general view of the EM algorithm, which is a standard learning algorithm for many probabilistic models. The chapter presents two popular probabilistic topic models, i.e., probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA), for document clustering and analysis.

Voice based computing applications, such as phone communication and speech recognition, use microphone arrays to capture voice from a human speaker. In many environments of interest, however, sounds from other sources interfere with the speaker's voice, posing severe problems for subsequent processing. This paper describes a new framework for treating this problem, and presents and demonstrates a new algorithm for the cancellation of interfering sounds. Our framework combines techniques from statistical machine learning with ideas from speech and audio processing. An important feature involves training rich probabilistic models on data from different types of relevant sound sources. Those source models are then incorporated into a larger probabilistic model of the observed microphone data. Using that model we derive our algorithm, which is of the expectation-maximization type and infers from data the clean sound of separate individual sources. We report very good results on data recorded in different environments

SummaryMaximality of ancillaries is important in both conditional inference without nuisance parameters and marginal inference with nuisance parameters. We extend results of Basu (1959) to the more classical former case and discuss the different nature of ancillaries in these two contexts. We apply the results to a general ancillary independent of a sufficient statistic. Finally, we discuss difficulties in finding necessary conditions for maximality.

Conditional inference has an ease of implementation that is generally unavailable with marginal inference. The main patterns for conditional inference are provided by the location and transformation families as initiated by Fisher, and by the exponential patterns as initiated by Neyman and Pearson; these are surveyed briefly together with some discussion as to how and why conditioning should be used in inference for them.

Loosely speaking, clustering refers to the problem of grouping data, and plays a central role in statistical machine learning and data analysis. One way in which to formalize the clustering problem is in terms of a mixture model, where the mixture components represent clusters within the data. We consider a semi-parametric formulation, in which a random vector X isin Ropf <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sup> is modeled as a distribution Fx(x)=Sigma <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha=1</sub> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">m</sup> omega <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> (x-mu <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> ). (1) with m components. Here omega <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> isin (0, 1) is the weight on mixture component alpha. The mean vectors mu <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> isin Ropf <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">d</sup> are the (parametric) component of interest, whereas the dispersion distributions F <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> are a non-parametric nuisance component, on which we impose only tail conditions (e.g., sub-Gaussian or sub-exponential tail decay). Given n independent and identically distributed samples from the mixture model (1), we consider the problem of "learning" the mixture. More formally, for parameters (delta, epsi) isin (0, 1) times (0, 1), we say that a method (epsi, delta)-learns the mixture if it correctly determines the mixture label of all n samples with probability greater than 1 - delta, and estimates the mean vectors to accuracy epsi with high probability. This conference abstract provides an overview of the results in our full-length paper. We derive both necessary and sufficient conditions on the scaling of the sample size n as a function of the ambient dimension d, the minimum separation r(d) = min <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alphanebeta</sub> parmu <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> - mu <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">beta</sub> par <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> between mixture components, and tail decay parameters. All of our analysis is high-dimensional in nature, meaning that we allow the sample size n, ambient dimension d and other parameters to scale in arbitrary ways. Our necessary conditions are information-theoretic in nature, and provide lower bounds on the performance of any algorithm, regardless of its computational complexity. Our sufficient conditions are based on analyzing a particular form of spectral clustering. For mixture models without any constraints on the mean vectors mu <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">alpha</sub> we show that standard spectral clustering-that is, based on sample means and covariance matrices-can achieve the information-theoretic limits. We also analyze mixture models in which the mean vectors are ldquosparse", and derive information-theoretic lower bounds. For such models, spectral clustering based on sample means/covariances is highly sub-optimal, but modified spectral clustering algorithms using thresholding estimators are nearly optimal.

Catastrophic forgetting is a prominent challenge in machine learning. It denotes the phenomenon wherein models undergo a significant loss of previously acquired knowledge upon learning new information. Supervised Continual Learning (SCL) has emerged as a promising approach to mitigate this issue by enabling models to adapt to non-stationary data distributions while leveraging labeled data. However, practical limitations arise for SCL in real-world settings, where labeled data is scarce. Conversely, Unsupervised Continual Learning (UCL) aims to mitigate forgetting without the need for manual annotations. Still, many previous UCL methods rely on replay-based strategies, which may not be viable in contexts where storing training data is impractical. Additionally, replay methods may face challenges related to overfitting and representation drifts when the buffer size is limited. To overcome the limitations of replay strategies, we propose an approach based on parameter isolation, combined with gradient projection. Specifically, we use task-specific hard attention and gradient projection to constrain updates to parameters crucial for previous tasks. Furthermore, our approach offers advantages over architecture-based methodologies by avoiding the need for network expansion and allowing for sequential learning within a predefined network architecture. Contrastive learning-based unsupervised methods are effective in preserving representation continuity. However, the loss in contrastive learning may suffer from a decrease in the diversity of negative samples. To address this, we incorporate both direct instance grouping and discrimination at the cross-level for actual and pseudo groups to contrastively learn unsupervised representations.We employ evidential deep learning (EDL) on rotation augmentation-based pseudo groups to effectively identify OOD instances and learn distinct class representations. Through comprehensive experiments, we demonstrate that the proposed model achieves an overall-average accuracy of 77.82% for task incremental learning (TIL), and 67.39% for class incremental learning (CIL) across various benchmark datasets, encompassing both short and long sequences of tasks. Notably, our model achieves almost zero forgetting, outperforming state-of-the-art (SOTA) baseline accuracies of 73.33% and 60.8% for TIL and CIL, respectively. Additionally, our model demonstrates a significant improvement over the SOTA SCL baseline, achieving a 2.82% and 3.68% increase in average TIL and CIL accuracy, respectively, while substantially reducing forgetting from approximately 12.66% and 19.56% to nearly zero.

The advancement of intelligent multimedia computing, particularly through deep neural networks (DNNs), has gained remarkable attention across a wide array of research domains and practical applications, such as audio-visual joint processing, image classification, video computing, computer vision, etc. However, the concern for sustainability and inherent black-box characteristics of DNNs have emerged as significant challenges, especially when dealing with multiple data sources. To tackle these challenges, alternative learning models have been actively explored in recent years, with green learning and interpretable learning standing out as two key directions. In this paper, a discriminative correlation perceptron-style neural network (DC-PNN) is proposed for feature representation learning with application to multimodal information computing. By leveraging the joint strength of statistics machine learning (SML) principles and the cascade architecture of a perceptron-style neural network (PNN), the DC-PNN model is green in nature with interpretable properties. To evaluate its effectiveness and generalizability, we conduct experiments on six benchmark databases with different scales and then apply to a real world problem, Glaucoma image classification. The results demonstrate that the DC-PNN model outperforms state-of-the-art (SOTA) methods. In addition, the parameters and FLOP analysis are implemented, validating the advantage of the model in terms of efficiency and resource requirements.

Machine learning using Bernoulli mixture models: Clustering, rule extraction and dimensionality reduction

Probabilistic models are receiving increasing attention as a complementary alternative to more widespread machine learning approaches such as neural networks. One particularly interesting class of models, so-called Sum-Product Networks (SPNs), combine the expressiveness of probabilistic models with tractable inference, making them an interesting candidate for use in real-world applications. Previously, inference in SPNs has successfully been accelerated by fully pipelined FPGA-based hardware. However, with these approaches, the maximum size of the SPN for FPGA acceleration has effectively been limited by the fully spatial mapping of arithmetic operations into hardware and the number of available resources in the FPGA. In this work, we present an extended and specialized modulo scheduling algorithm based on Integer Linear Programming (ILP) for time-multiplexed sharing of hardware arithmetic operators in the SPN inference accelerator. In addition and in order to scale the scheduling to large SPN graphs, we combine the scheduling algorithm with a graph-partitioning heuristic, exploiting the graph structure of SPNs. The combination of heuristic graph partitioning and ILP-based scheduling allows generating pipelined accelerators with the best possible initiation interval, while limiting the resource utilization to pre-set bounds. The evaluation discusses the effect different parameters have on convergence time and solution quality. A performance comparison shows that the FPGA improves the inference throughput over a comparable CPU- and GPU platform by a factor (geo.-mean) of 4.4x and 1.7x, respectively.

Submodular and supermodular functions have found wide applicability in machine learning, capturing notions such as diversity and regularity, respectively. These notions have deep consequences for optimization, and the problem of (approximately) optimizing submodular functions has received much attention. However, beyond optimization, these notions allow specifying expressive probabilistic models that can be used to quantify predictive uncertainty via marginal inference. Prominent, well-studied special cases include Ising models and determinantal point processes, but the general class of log-submodular and log-supermodular models is much richer and little studied. In this paper, we investigate the use of Markov chain Monte Carlo sampling to perform approximate inference in general log-submodular and log-supermodular models. In particular, we consider a simple Gibbs sampling procedure, and establish two sufficient conditions, the first guaranteeing polynomial-time, and the second fast (O(n log n)) mixing. We also evaluate the efficiency of the Gibbs sampler on three examples of such models, and compare against a recently proposed variational approach.

The mixture model is known as model-based clustering that is used to model a mixture of unknown distributions. The clustering of mixture model is based on four important criteria, including the number of components in the mixture model, clustering kernel (such as Gaussian mixture models, Dirichlet, etc.), estimation methods, and dimensionality (Lai et al., 2019). Finite mixture model is a finite dimensional of a hierarchical model. It is useful in modeling the data with outliers, non-normal distributed or heavy tails. Furthermore, finite mixture model is flexible when fitted with the models that have multiple modes or skewed distribution. The flexibility depends on the increasing number of parameters with the existence of a number of components. The finite mixture model is a flexible model family and widely applied for large heterogeneous datasets. In addition, the finite mixture model is a probabilistic model that is used to examine the presence of unobserved situations or groups and to measure the distinct parameters or distribution. The situations, such as trend, seasoning, crisis time, normal situation, etc., might affect the number of components that exist for a probabilistic distribution. Furthermore, the finite mixture model is essential for time series data because these data exhibit nonlinearity properties and may have missing data or a jump-diffusion situation (Gensler, 2017; McLachlan and Lee, 2019). Keywords: Bayesian method; Finite Mixture Model; Maximum Likelihood Estimation; Prior distribution; Likelihood Function.

Comparing conventional statistical models and machine learning in a small cohort of South African cardiac patients

Ensemble learning is a very prevalent method employed in machine learning. The relative success of ensemble methods is attributed to their ability to tackle a wide range of instances and complex problems that require different low-level approaches. However, ensemble methods are relatively less popular in reinforcement learning owing to the high sample complexity and computational expense involved in obtaining a diverse ensemble. We present a novel training and model selection framework for model-free reinforcement algorithms that use ensembles of policies obtained from a single training run. These policies are diverse in nature and are learned through directed perturbation of the model parameters at regular intervals. We show that learning and selecting an adequately diverse set of policies is required for a good ensemble while extreme diversity can prove detrimental to overall performance. Selection of an adequately diverse set of policies is done through our novel policy selection framework. We evaluate our approach on challenging discrete and continuous control tasks and also discuss various ensembling strategies. Our framework is substantially sample efficient, computationally inexpensive and is seen to outperform state-of-the-art (SOTA) scores in Atari 2600 and Mujoco.

In this paper we examine estimation in a mixture model under various assumptions for the underlying distribution. We suggest that statistical inference should be carried out using the conditional distribution given the appropriate ancillary statistics. The conditional likelihood approach used here can be justified by Cox's (1958) criterion about ancillarity in the presence of a nuisance parameter, and in the spirit of Godambe's (1976) conditional likelihood optimum estimating equation. The analysis is known to be easier and more stable than the unconditional likelihood function approach. The probability distributions in the analysis of count data discussed here include Poisson, binomial, geometric and negative binomial distributions. The information loss incurred by using the conditional approach is discussed