A Bayesian Approach to Unsupervised Feature Selection and Density Estimation Using Expectation Propagation

Expectation propagation (EP) is a deterministic approximation algorithm that is often used to perform approximate Bayesian parameter learning. EP approximates the full intractable posterior distribution through a set of local approximations that are iteratively refined for each datapoint. EP can offer analytic and computational advantages over other approximations, such as Variational Inference (VI), and is the method of choice for a number of models. The local nature of EP appears to make it an ideal candidate for performing Bayesian learning on large models in large-scale dataset settings. However, EP has a crucial limitation in this context: the number of approximating factors needs to increase with the number of data-points, N, which often entails a prohibitively large memory overhead. This paper presents an extension to EP, called stochastic expectation propagation (SEP), that maintains a global posterior approximation (like VI) but updates it in a local way (like EP). Experiments on a number of canonical learning problems using synthetic and real-world datasets indicate that SEP performs almost as well as full EP, but reduces the memory consumption by a factor of $N$. SEP is therefore ideally suited to performing approximate Bayesian learning in the large model, large dataset setting.

Learning statistical models successfully is both an essential and a challenging task for various pattern recognition and knowledge discovery applications. In particular, generative models such as finite and infinite mixture models have demonstrated to be efficient in terms of overall performance. In this paper, a robust framework based on an expectation propagation (EP) inference is developed to learn inverted Beta-Liouville (IBL) mixture models which is proper choice for positive data classification. Within the proposed EP learning method, the full posterior distribution is estimated accurately, the model complexity and all related parameters are evaluated simultaneously in a single optimization scheme. Extensive experiments using challenging real-world applications including recognition of facial expression, automatic human action categorization, and hand gesture recognition show the merit of our approach in terms of achieving better results than comparable techniques.

We present in this paper a new approach for unsupervised feature selection for non Gaussian data controlled by a finite mixture of generalized Dirichlet distributions. We model each feature by a mixture of two Beta distributions: one relevant and depends on component labels while the second distribution is uninformative for the clustering. The relevance of each feature is then quantified by the mixture weight associated to the relevant Beta distribution. Experiments in summarizing image collections have shown the merits of our approach.

Multilevel logistic regression models are increasingly being used to analyze clustered data in medical, public health, epidemiological, and educational research. Procedures for estimating the parameters of such models are available in many statistical software packages. There is currently little evidence on the minimum number of clusters necessary to reliably fit multilevel regression models. We conducted a Monte Carlo study to compare the performance of different statistical software procedures for estimating multilevel logistic regression models when the number of clusters was low. We examined procedures available in BUGS, HLM, R, SAS, and Stata. We found that there were qualitative differences in the performance of different software procedures for estimating multilevel logistic models when the number of clusters was low. Among the likelihood-based procedures, estimation methods based on adaptive Gauss-Hermite approximations to the likelihood (glmer in R and xtlogit in Stata) or adaptive Gaussian quadrature (Proc NLMIXED in SAS) tended to have superior performance for estimating variance components when the number of clusters was small, compared to software procedures based on penalized quasi-likelihood. However, only Bayesian estimation with BUGS allowed for accurate estimation of variance components when there were fewer than 10 clusters. For all statistical software procedures, estimation of variance components tended to be poor when there were only five subjects per cluster, regardless of the number of clusters.

Unsupervised feature selection based on minimum-redundant subspace learning with self-weighted adaptive graph

Text clustering is suitable for dividing many text documents into distinct groups. The size of the documents has an impact on the performance of text clustering, reducing its effectiveness. Text documents often include sparse and uninformative characteristics, which can negatively impact the efficiency of the text clustering technique and increase the computational time required. Feature selection is a crucial strategy in unsupervised learning that involves choosing a subset of informative text features to enhance the efficiency of text clustering and decrease computing time. This work presents a novel approach based on an improved Prairie dog algorithm to solve the feature selection problem. K-means clustering is employed to assess the efficacy of the acquired subgroups of features. The proposed algorithm is being compared to other algorithms published in the literature. The feature selection strategy ultimately promotes the clustering algorithm to get precise clusters.

Ensemble clustering is known as a challenging research direction in data mining. The results of several individual clustering methods are combined to produce higher quality final clusters. This study introduces a parallel hierarchical clustering approach based on the divide-and-conquer strategy, which is an attempt to realize faster and more efficient ensemble clustering. Here, we propose a cluster consensus selection approach that selects a subset of meriting primary clusters to participate in the final consensus. Considering the sample-cluster and cluster–cluster similarity on the selected primary clusters, we form the final clusters based on the clusters clustering technique as a consensus function. In addition, the proposed scheme is equipped with an unsupervised feature selection approach to remove features that do not contribute significantly to clustering. Extensive evaluations have been performed on datasets of different dimensions from the University of California Irvine (UCI) machine learning repository. The simulation results guarantee the efficiency of the proposed scheme and improves the average performance between 6% and 24% compared to the state-of-the-art clustering methods.

Density estimation for pre-binned data is challenging due to the loss of exact position information of the original observations. Traditional kernel density estimation methods cannot be applied when data are pre-binned in unequally spaced bins or when one or more bins are semi-infinite intervals. We propose a novel density estimation approach using the generalized lambda distribution (GLD) for data that have been pre-binned over a sequence of consecutive bins. This method enjoys the high power of the parametric model and the great shape flexibility of the GLD. The performances of the proposed estimators are benchmarked via simulation studies. Both simulation results and a real data application show that the proposed density estimators work well for data of moderate or large sizes.

This paper addresses the parameter estimation problem for lithium-ion battery pack models comprising cells in series. This valuable information can be exploited in fault diagnostics to estimate the number of cells that are exhibiting abnormal behaviour, e.g. large resistances or small capacities. In particular, we use a Bayesian approach to estimate the parameters of a two-cell arrangement modelled using equivalent circuits. Although our modeling framework has been extensively reported in the literature, its structural identifiability properties have not been reported yet to the best of the authors’ knowledge. Moreover, most contributions in the literature tackle the estimation problem through point-wise estimates assuming Gaussian noise using e.g. least-squares methods (maximum likelihood estimation) or Kalman filters (maximum a posteriori estimation). In contrast, we apply methods that are suitable for nonlinear and non-Gaussian estimation problems and estimate the full posterior probability distribution of the parameters. We study how the model structure, available measurements and prior knowledge of the model parameters impact the underlying posterior probability distribution that is recovered for the parameters. For two cells in series, a bimodal distribution is obtained whose modes are centered around the real values of the parameters for each cell. Therefore, bounds on the model parameters for a battery pack can be derived.

A Poisson ridge regression estimator

Recently, Jewson and Rossell (2022) proposed a new approach for kernel density estimation using an exponentiated form of kernel density estimators. The density estimator contained two hyperparameters that flexibly controls the smoothness of the resulting density. We tune them in a data-driven manner by minimizing an objective function based on the Hyvärinen score to avoid the optimization involving the intractable normalizing constant caused by the exponentiation. We show the asymptotic properties of the proposed estimator and emphasize the importance of including the two hyperparameters for flexible density estimation. Our simulation studies and application to income data show that the proposed density estimator is promising when the underlying density is multi-modal or when observations contain outliers.

We study the problem of downlink channel estimation in multi-user massive multiple input multiple output (MIMO) systems. To this end, we consider the use of Bayesian compressive sensing approach in which the clustered sparse structure of the channel in the angular domain is employed to reduce the pilot overhead. To capture the clustered structure, we employ a conditionally independent identically distributed Bernoulli-Gaussian prior on the sparse vector representing the channel, and a Markov prior on its support vector. An expectation propagation (EP) algorithm is developed to approximate the intractable joint distribution of the sparse vector and its support with a distribution from an exponential family. The approximate distribution is then used for direct estimation of the channel. The EP algorithm assumes that the model parameters are known a priori. Since these parameters are unknown, we estimate these parameters using the expectation maximization (EM) algorithm. The combination of EM and EP referred to as EM-EP algorithm is reminiscent of the variational EM approach. Simulation results show that the proposed EM-EP algorithm outperforms several recently-proposed algorithms in the literature.

The density estimation of diverse sensor types in a heterogeneous sensor network is a useful and challenging service that can be applied in clustering schemes, node redeployment, and sleep mode scheduling. Energy efficiency is one of the main requirements for any wireless sensor network service. Besides, the service has to provide a fresh version of the estimation to each node. Network dynamics, especially node mobility, introduce new challenges. Moreover, churn makes the problem even more complicated. In this paper we introduce a gossip-based approach for the density estimation of sensor diversity in clustered dynamic networks. The devised method supports node mobility and churn, as well as redeployment of new nodes. It is fully distributed and adaptive to network dynamics. We analyze the effect of mobility as well as scalability in the number of clusters and the quantity of nodes. Our algorithm has a fast convergence speed and provides more accurate estimation compared to similar approaches.

A number of clustering algorithms can be employed to find clusters in multivariate data. However, the effectiveness and efficiency of the existing algorithms are limited, since the respective data has high dimension, contain large amount of noise and consist of clusters with arbitrary shapes and densities. In this paper, a new kernel density-based clustering algorithm, called Local Triangular Kernel-based Clustering (LTKC), is proposed to deal with these conditions. LTKC is based on combination of k-nearest-neighbor density estimation and triangular kernel density-based clustering. The advantages of our LTKC approach are: (1) it has a firm mathematical basis; (2) it requires only one parameter, number of neighbors; (3) it defines the number of cluster automatically; (4) it allows discovering clusters with arbitrary shapes and densities ;and (5) it is significantly faster than existing algorithms. LTKC is tested using artificial data and applied to some UCI data. A comparison with k-means, KFCM and well known density-based clustering algorithms including ILGC, DBSCAN, and DENCLUE shows the superiority of our proposed LTKC algorithm.

Bayesian estimation of stable CARMA spot models for electricity prices