Clustering with mixed variables

Technological advances led to the generation of large scale complex data. Thus, extraction and retrieval of information to automatically discover latent pattern have been largely studied in the various domains of science and technology. Consequently, machine learning experienced tremendous development and various statistical approaches have been suggested. In particular, data clustering has received a lot of attention. Finite mixture models have been revealed to be one of the flexible and popular approaches in data clustering. Considering mixture models, three crucial aspects should be addressed. The first issue is choosing a distribution which is flexible enough to fit the data. In this paper, a model based on multivariate Beta distributions is proposed. The two other challenges in mixture models are estimation of model's parameters and model complexity. To tackle these challenges, variational inference techniques demonstrated considerable robustness. In this paper, two methods are studied, namely, batch and online variational inferences and the models are evaluated on four medical applications including image segmentation of colorectal cancer, multi‐class colon tissue analysis, digital imaging in skin lesion diagnosis and computer aid detection of Malaria.

WINNER OF THE 2007 DEGROOT PRIZE! The prominence of finite mixture modelling is greater than ever. Many important statistical topics like clustering data, outlier treatment, or dealing with unobserved heterogeneity involve finite mixture models in some way or other. The area of potential applications goes beyond simple data analysis and extends to regression analysis and to non-linear time series analysis using Markov switching models. For more than the hundred years since Karl Pearson showed in 1894 how to estimate the five parameters of a mixture of two normal distributions using the method of moments, statistical inference for finite mixture models has been a challenge to everybody who deals with them. In the past ten years, very powerful computational tools emerged for dealing with these models which combine a Bayesian approach with recent Monte simulation techniques based on Markov chains. This book reviews these techniques and covers the most recent advances in the field, among them bridge sampling techniques and reversible jump Markov chain Monte Carlo methods. It is the first time that the Bayesian perspective of finite mixture modelling is systematically presented in book form. It is argued that the Bayesian approach provides much insight in this context and is easily implemented in practice. Although the main focus is on Bayesian inference, the author reviews several frequentist techniques, especially selecting the number of components of a finite mixture model, and discusses some of their shortcomings compared to the Bayesian approach. The aim of this book is to impart the finite mixture and Markov switching approach to statistical modelling to a wide-ranging community. This includes not only statisticians, but also biologists, economists, engineers, financial agents, market researcher, medical researchers or any other frequent user of statistical models. This book should help newcomers to the field to understand how finite mixture and Markov switching models are formulated, what structures they imply on the data, what they could be used for, and how they are estimated. Researchers familiar with the subject also will profit from reading this book. The presentation is rather informal without abandoning mathematical correctness. Previous notions of Bayesian inference and Monte Carlo simulation are useful but not needed.

Finite mixture models are an important tool in the statistical analysis of data, for example in data clustering. The optimal parameters of a mixture model are usually computed by maximizing the log-likelihood functional via the Expectation-Maximization algorithm. We propose an alternative approach based on the theory of Mean Field Games, a class of differential games with an infinite number of agents. We show that the solution of a finite state space multi-population Mean Field Games system characterizes the critical points of the log-likelihood functional for a Bernoulli mixture. The approach is then generalized to mixture models of categorical distributions. Hence, the Mean Field Games approach provides a method to compute the parameters of the mixture model, and we show its application to some standard examples in cluster analysis.

This chapter introduces several fundamental models and algorithms for probabilistic clustering, including mixture models, Expectation-Maximization (EM) algorithm, and probabilistic topic models. It focuses on general framework of modeling, the probabilistic explanation, the standard algorithms to learn the model, and its applications. Mixture models are probabilistic models which are increasingly used to find the clusters for univariate and multivariate data. The chapter begins the discussion of mixture models, in which the values of the discrete latent variables can be interpreted as the assignments of data points to specific components of the mixture. It uses the Gaussian mixture model to motivate the EM algorithm in an informal way, and then gives a more general view of the EM algorithm, which is a standard learning algorithm for many probabilistic models. The chapter presents two popular probabilistic topic models, i.e., probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA), for document clustering and analysis.

Using general regression with local tuning for learning mixture models from incomplete data sets

Variational learning for finite Beta-Liouville mixture models

The expectation-maximization (EM) algorithm is a popular approach for parameter estimation of finite mixture model (FMM). A drawback of this approach is that the number of components of the finite mixture model is not known in advance, nevertheless, it is a key issue for EM algorithms. In this paper, a penalized minimum matching distance-guided EM algorithm is discussed. Under the framework of Greedy EM, a fast and accurate algorithm for estimating the number of components of the Gaussian mixture model (GMM) is proposed. The performance of this algorithm is validated via simulative experiments of univariate and bivariate Gaussian mixture models.

Mixture modeling is a popular technique for identifying unobserved subpopulations (e.g., components) within a data set, with Gaussian (normal) mixture modeling being the form most widely used. Generally, the parameters of these Gaussian mixtures cannot be estimated in closed form, so estimates are typically obtained via an iterative process. The most common estimation procedure is maximum likelihood via the expectation-maximization (EM) algorithm. Like many approaches for identifying subpopulations, finite mixture modeling can suffer from locally optimal solutions, and the final parameter estimates are dependent on the initial starting values of the EM algorithm. Initial values have been shown to significantly impact the quality of the solution, and researchers have proposed several approaches for selecting the set of starting values. Five techniques for obtaining starting values that are implemented in popular software packages are compared. Their performances are assessed in terms of the following four measures: (1) the ability to find the best observed solution, (2) settling on a solution that classifies observations correctly, (3) the number of local solutions found by each technique, and (4) the speed at which the start values are obtained. On the basis of these results, a set of recommendations is provided to the user.

Many scientific works on grouping mixed data have chosen different experimental scenarios to structure their findings, which involve deciding from the programming language to the use of real-world and/or simulated datasets and performance measures, for example. Due to these characteristics directly influence the conclusion of the studies and the way new scientific works are proposed, it would be useful to have a wide map with the main choices that have been done by the authors of high-impact scientific documents so that the community can reflect on the thematic direction, identify best practices, and propose new paths for future research. To the best of our knowledge, such a map does not exist, neither a methodological procedure to build it. Therefore, this paper proposes a systematic methodology to reach such maps and provides a wide and in-depth map of the main choices the authors of high-impact scientific documents on mixed data clustering and surrounding studies have done in their experiments. As a result, 160 documents were systematically selected and classified into one of the six class of data clustering approaches, besides individually tabulated. From the tables for each class, we found, for instance, that real-world datasets are used more frequently than simulated ones, the documents used more external indices, followed by internal and relative ones, it is not common for the authors to inform the programming language they have used, except in the partitional class. We also provided the address of the algorithms’ code when they are made available by the authors.

Clustering algorithms play a pivotal role in data mining, offering powerful tools for uncovering hidden patterns and structures within datasets. These algorithms aim to divide data points into coherent groups based on similarities or dissimilarities, making it easier to explore and understand complex data. Clustering algorithms typically rely on similarity measures to assess the likeness between data points. Consequently, selecting a suitable similarity measure is crucial for achieving satisfactory clustering outcomes. However, this decision can pose significant challenges, especially for non-experts, given the plethora of similarity measures available in the literature and their performance which is closely linked to the specific dataset, clustering algorithm, and cluster validity index employed. This difficulty is even more important when considering mixed data clustering. Mixed data refers to heterogeneous data characterized by both numerical and categorical attributes. In such a context, the same similarity measure cannot be used for both types of attributes due to their different nature. Commonly, two similarity measures are combined, one for numerical attributes and one for categorical attributes. This adds a layer of complexity to the problem since it requires the selection of two similarity measures instead of just one. This paper introduces SIMREC, a similarity measure recommendation system for mixed data clustering. The system uses meta-learning to mine the relationship between dataset characteristics and similarity measures performances for different mixed data clustering algorithms and cluster validity indices. Therefore, given a mixed dataset, a mixed data clustering algorithm, and a cluster validity index, the system can recommend suitable pairs of numerical and categorical similarity measures based on the characteristics of the dataset. We implemented the proposed system using 130 pairs of similarity measures (10 numerical and 13 categorical), 4 commonly used mixed data clustering algorithms (K-Prototypes, LSH-K-Prototypes, K-Medoids, and Hierarchical Clustering), and three cluster validity indices (Silhouette, Clustering Accuracy, and Adjusted Rand Index). Our experiments on 185 publicly available mixed datasets show that the pairs of similarity measures recommended by SIMREC outperform the baseline pairs, including classically used pairs of similarity measures in the literature.

The mixture model is known as model-based clustering that is used to model a mixture of unknown distributions. The clustering of mixture model is based on four important criteria, including the number of components in the mixture model, clustering kernel (such as Gaussian mixture models, Dirichlet, etc.), estimation methods, and dimensionality (Lai et al., 2019). Finite mixture model is a finite dimensional of a hierarchical model. It is useful in modeling the data with outliers, non-normal distributed or heavy tails. Furthermore, finite mixture model is flexible when fitted with the models that have multiple modes or skewed distribution. The flexibility depends on the increasing number of parameters with the existence of a number of components. The finite mixture model is a flexible model family and widely applied for large heterogeneous datasets. In addition, the finite mixture model is a probabilistic model that is used to examine the presence of unobserved situations or groups and to measure the distinct parameters or distribution. The situations, such as trend, seasoning, crisis time, normal situation, etc., might affect the number of components that exist for a probabilistic distribution. Furthermore, the finite mixture model is essential for time series data because these data exhibit nonlinearity properties and may have missing data or a jump-diffusion situation (Gensler, 2017; McLachlan and Lee, 2019). Keywords: Bayesian method; Finite Mixture Model; Maximum Likelihood Estimation; Prior distribution; Likelihood Function.

Pre-analysis of Multi-batch Bioprocesses Data with Finite Mixture Models in the Reduced Feature Subspace

Data mining discloses hidden, previously unknown, and potentially useful information from large amounts of data. As comparison to the traditional statistical and machine learning data analysis techniques, data mining emphasizes to provide a convenient and complete environment for the data analysis. Data mining has become a popular technology in analyzing complex data. Clustering is one of the data mining core techniques. In the field of data mining and data clustering, it is a highly desirable task to perform cluster analysis on large data sets with mixed numeric, categorical, ordinal, and ratio-scaled with binary and nominal values. However, most already available data merging and grouping through clustering algorithms are effective for the numeric data rather than the mixed data set. For this purpose, this paper makes efforts to present a new amalgamation algorithm for these mixed data sets by modifying the common cost function, trace of the within cluster dispersion matrix. The genetic algorithm (GA) is used to optimize the new cost function to obtain valid clustering result. We can compare and analyze that the GA-based clustering algorithm is feasible for the high-dimensional data sets with mixed data values that are obtained in real life results. Core Idea of Our Paper: By this paper, we try to describe a technique for estimating the cost function metrics from mixed numeric, categorical and other type databases by using an uncertain grade-of-membership clustering model with the efficiency of Genetic Algorithm. This technique can be applied to the problem of opportunity analysis for business decision-making. This general approach could be adapted to many other applications where a decision agent needs to assess the value of items from a set of opportunities with respect to a reference set representing its business. For processing numeric attributes, instead of generalizing them, a prototype may be developed for experiments with synthetic and real data sets, and comparison with those of the traditional approaches. The results confirmed the feasibility of the framework and the superiority of the extended techniques.

We show that the identification results of finite mixture and misclassification models are equivalent in a widely used scenario except for an extra ordering assumption. In the misclassification model, an ordering condition is imposed to pin down the precise values of the latent variable, which are also of interest to researchers and need to be identified. In contrast, finite mixture models are usually identified up to permutations of a latent index, which results in local identification. This local identification is satisfactory because the latent index does not convey any economic meaning. However, reaching global identification is important for estimation, especially when researchers use bootstrap to estimate standard errors. This is because standard errors approximated by bootstrap may be incorrect without a global estimator. We demonstrate that games with multiple equilibria fit in our framework and the global estimator with ordering conditions provides more reliable estimates.

Using incremental general regression neural network for learning mixture models from incomplete data