Unsupervised outlier detection using random subspace and subsampling ensembles of Dirichlet process mixtures

The aim of this study was to implement a Dirichlet process mixture (DPM) model for automatic tumor edge identification on (18)F-fluorodeoxyglucose positron emission tomography ((18)F-FDG PET) images by optimizing the parameters on which the algorithm depends, to validate it experimentally, and to test its robustness. The DPM model belongs to the class of the Bayesian nonparametric models and uses the Dirichlet process prior for flexible nonparametric mixture modeling, without any preliminary choice of the number of mixture components. The DPM algorithm implemented in the statistical software package R was used in this work. The contouring accuracy was evaluated on several image data sets: on an IEC phantom (spherical inserts with diameter in the range 10-37 mm) acquired by a Philips Gemini Big Bore PET-CT scanner, using 9 different target-to-background ratios (TBRs) from 2.5 to 70; on a digital phantom simulating spherical/uniform lesions and tumors, irregular in shape and activity; and on 20 clinical cases (10 lung and 10 esophageal cancer patients). The influence of the DPM parameters on contour generation was studied in two steps. In the first one, only the IEC spheres having diameters of 22 and 37 mm and a sphere of the digital phantom (41.6 mm diameter) were studied by varying the main parameters until the diameter of the spheres was obtained within 0.2% of the true value. In the second step, the results obtained for this training set were applied to the entire data set to determine DPM based volumes of all available lesions. These volumes were compared to those obtained by applying already known algorithms (Gaussian mixture model and gradient-based) and to true values, when available. Only one parameter was found able to significantly influence segmentation accuracy (ANOVA test). This parameter was linearly connected to the uptake variance of the tested region of interest (ROI). In the first step of the study, a calibration curve was determined to automatically generate the optimal parameter from the variance of the ROI. This "calibration curve" was then applied to contour the whole data set. The accuracy (mean discrepancy between DPM model-based contours and reference contours) of volume estimation was below (1 ± 7)% on the whole data set (1 SD). The overlap between true and automatically segmented contours, measured by the Dice similarity coefficient, was 0.93 with a SD of 0.03. The proposed DPM model was able to accurately reproduce known volumes of FDG concentration, with high overlap between segmented and true volumes. For all the analyzed inserts of the IEC phantom, the algorithm proved to be robust to variations in radius and in TBR. The main advantage of this algorithm was that no setting of DPM parameters was required in advance, since the proper setting of the only parameter that could significantly influence the segmentation results was automatically related to the uptake variance of the chosen ROI. Furthermore, the algorithm did not need any preliminary choice of the optimum number of classes to describe the ROIs within PET images and no assumption about the shape of the lesion and the uptake heterogeneity of the tracer was required.

In the Bayesian mixture modeling framework it is possible to infer the necessary number of components to model the data and therefore it is unnecessary to explicitly restrict the number of components. Nonparametric mixture models sidestep the problem of finding the "correct" number of mixture components by assuming infinitely many components. In this paper Dirichlet process mixture (DPM) models are cast as infinite mixture models and inference using Markov chain Monte Carlo is described. The specification of the priors on the model parameters is often guided by mathematical and practical convenience. The primary goal of this paper is to compare the choice of conjugate and non-conjugate base distributions on a particular class of DPM models which is widely used in applications, the Dirichlet process Gaussian mixture model (DPGMM). We compare computational efficiency and modeling performance of DPGMM defined using a conjugate and a conditionally conjugate base distribution. We show that better density models can result from using a wider class of priors with no or only a modest increase in computational effort.

In machine learning and data mining tasks, Clustering is considered to be one of the most important techniques. The same sorts of documents are grouped by performing clustering techniques. Similarity measuring is used to determine transaction relationships. Hierarchical clustering model generates tree structured results. Partitioned based clustering produces the result in grid format. Text documents are unstructured data values with high dimensional attributes. Document clustering group transforms unlabeled text documents into meaningful clusters. In the event of document grouping process, traditional clustering methods require cluster count (K). Clustering accuracy degrades drastically with reference to the unsuitable cluster count. It is observed that document features are automatically partitioned into two groups namely — discriminative words and non-discriminative words. In particular, discriminative words are only useful for grouping documents. The involvement of non-discriminative words confuses the clustering process and leads to poor clustering solution in return. A variation inference algorithm is used to infer the document collection structure and partition of document words simultaneously. Dirichlet Process Mixture (DPM) model is used to partition documents in a way utilizing both the data likelihood and the clustering property of the Dirichlet Process (DP). Dirichlet Process Mixture Model for Feature Extraction (DPMFE) is used to discover the latent cluster structure based on the DPM model and it is performed without involving the number of clusters as input.

This paper presents an empirical study of endogenous treatment effects in the presence of heterogeneous responses. We estimate the incentive and selection effects of having prescription drug coverage on total drug expenditures using a sample from the Medicare Current Beneficiary Survey (MCBS). A Dirichlet process mixture (DPM) model is used to model the heterogeneity in treatment effects. Rather than estimating a finite mixture model with a fixed number of components, we specify a DP prior on the parameters, thus allowing the data and prior knowledge to determine the number of components. Endogeneity is modeled by the covariance of the error terms of the selection and the outcome equations in a two-equation selection model. We find that there are strong incentive and advantageous selection effects, with the average treatment effect and the average treatment effect for the treated estimated at $1132 and $858, respectively. Substantial heterogeneity is found to exist in the selection effects, particularly for those having lower drug expenditures. Copyright © 2014 John Wiley & Sons, Ltd.

Hidden Markov random field (HMRF) models are widely used for image segmentation, as they appear naturally in problems where a spatially constrained clustering scheme is asked for. A major limitation of HMRF models concerns the automatic selection of the proper number of their states, i.e., the number of region clusters derived by the image segmentation procedure. Existing methods, including likelihood- or entropy-based criteria, and reversible Markov chain Monte Carlo methods, usually tend to yield noisy model size estimates while imposing heavy computational requirements. Recently, Dirichlet process (DP, infinite) mixture models have emerged in the cornerstone of nonparametric Bayesian statistics as promising candidates for clustering applications where the number of clusters is unknown a priori; infinite mixture models based on the original DP or spatially constrained variants of it have been applied in unsupervised image segmentation applications showing promising results. Under this motivation, to resolve the aforementioned issues of HMRF models, in this paper, we introduce a nonparametric Bayesian formulation for the HMRF model, the infinite HMRF model, formulated on the basis of a joint Dirichlet process mixture (DPM) and Markov random field (MRF) construction. We derive an efficient variational Bayesian inference algorithm for the proposed model, and we experimentally demonstrate its advantages over competing methodologies.

Dirichlet process mixture models (or mixture of Dirichlet process [MDP]) are Bayesian non‐parametric mixture models that can solve the problem of determining the number of components in mixture models by assuming infinitely many components. We propose a new approach for MDP by using a matrix‐generalized half‐ distribution as a non‐informative prior for the covariance in the base distribution. This new approach aims to improve the performance of MDP in cases where conjugacy constraints cause underperformance of current priors, such as Wishart distribution or inverse‐Wishart. The effectiveness of this approach is demonstrated through simulations and by comparing its performance on the Old Faithful geyser dataset with existing MDP specifications.

ABSTRACTA natural Bayesian approach for mixture models with an unknown number of components is to take the usual finite mixture model with symmetric Dirichlet weights, and put a prior on the number of components—that is, to use a mixture of finite mixtures (MFM). The most commonly used method of inference for MFMs is reversible jump Markov chain Monte Carlo, but it can be nontrivial to design good reversible jump moves, especially in high-dimensional spaces. Meanwhile, there are samplers for Dirichlet process mixture (DPM) models that are relatively simple and are easily adapted to new applications. It turns out that, in fact, many of the essential properties of DPMs are also exhibited by MFMs—an exchangeable partition distribution, restaurant process, random measure representation, and stick-breaking representation—and crucially, the MFM analogues are simple enough that they can be used much like the corresponding DPM properties. Consequently, many of the powerful methods developed for inference in DPMs can be directly applied to MFMs as well; this simplifies the implementation of MFMs and can substantially improve mixing. We illustrate with real and simulated data, including high-dimensional gene expression data used to discriminate cancer subtypes. Supplementary materials for this article are available online.

As a powerful nonparametric Bayesian model, the infinite mixture model has been successfully used in machine learning and computer vision. The success of the infinite mixture model owes to the capability clustering and density estimation. In this paper, we propose a nonparametric Bayesian model for single-image super-resolution. Specifically, we combine the Dirichlet process and Gaussian process regression for estimating the distribution of the training patches and modeling the relationship between the low-resolution and high-resolution patches: 1) the proposed method groups the training patches by utilizing the clustering property of Dirichlet process; 2) the proposed method relates the low-resolution and high-resolution patches by predicting the property of Gaussian process; and 3) the mentioned two points are not independent but jointly learned. Hence, the proposed method can make full use of the nonparametric Bayesian model. First, the Dirichlet process mixture model is used to obtain more accurate clusters for training patches. Second, Gaussian process regression is established on each cluster, and this directly reduces the computational complexity. Finally, the two procedures are learned simultaneously to gain the suitable clusters with the ability of prediction. The parameters can be inferred simply via the Gibbs sampling technique. Thorough super-resolution experiments on various images demonstrate that the proposed method is superior to some state-of-the-art methods.

In this work we present and apply a new Hyperspectral image classification method based on Dirichlet Process Gaussian mixture models (also known as infinite Gaussian mixture models - IGMMs). Since this approach is a non-parametric Bayesian method, we circumvent the problem of model selection which is unavoidable and often difficult when employing traditional parametric Gaussian mixture models (GMM). In order to infer model parameters from observations, we adopt a Gibbs Sampling method to sample the posterior distributions of these parameters. In the preprocessing step, we use Local Fisher's Discriminant Analysis (LFDA) for dimension reduction since we expect it to preserve the multi-modal, non-Gaussian structure of the hyperspectral data. We compared our proposed IGMM based classification method to the existing state-of-the-art classification methods using popular hyperspectral imagery datasets. To simulate challenging real-world sensing environment, we mix every pixel with background and study the performance of these classification approaches. Our experiments show that the proposed LFDA-IGMM method and GMM method have almost the same performance for most cases, given “pure” data - with mixed pixels, LFDA-IGMM outperformed GMM for sub-pixel classification. Both methods outperformed the other commonly used classification approaches when the training sample size is relatively large. Also, the IGMM based method is less sensitive to pixel mixing than other methods.

This paper studies convergence behavior of latent mixing measures that arise in finite and infinite mixture models, using transportation distances (i.e., Wasserstein metrics). The relationship between Wasserstein distances on the space of mixing measures and f-divergence functionals such as Hellinger and Kullback-Leibler distances on the space of mixture distributions is investigated in detail using various identifiability conditions. Convergence in Wasserstein metrics for discrete measures implies convergence of individual atoms that provide support for the measures, thereby providing a natural interpretation of convergence of clusters in clustering applications where mixture models are typically employed. Convergence rates of posterior distributions for latent mixing measures are established, for both finite mixtures of multivariate distributions and infinite mixtures based on the Dirichlet process.

Outlier detection is to separate anomalous data from inliers in the dataset. Recently, the most deep learning methods of outlier detection leverage an auxiliary reconstruction task by assuming that outliers are more difficult to recover than normal samples (inliers). However, it is not always true in deep auto-encoder (AE) based models. The auto-encoder based detectors may recover certain outliers even if outliers are not in the training data, because they do not constrain the feature learning. Instead, we think outlier detection can be done in the feature space by measuring the distance between outliers' features and the consistency feature of inliers. To achieve this, we propose an unsupervised outlier detection method using a memory module and a contrastive learning module (MCOD). The memory module constrains the consistency of features, which merely represent the normal data. The contrastive learning module learns more discriminative features, which boosts the distinction between outliers and inliers. Extensive experiments on four benchmark datasets show that our proposed MCOD performs well and outperforms eleven state-of-the-art methods.

Unsupervised outlier detection using neural network-based mixtures of probabilistic principal component analyzers for building chiller fault diagnosis

Three-way decision-based co-detection for outliers

Anomaly detection is a critical issue in Network Intrusion Detection Systems (NIDSs). Most anomaly based NIDSs employ supervised algorithms, whose performances highly depend on attack-free training data. However, this kind of training data is difficult to obtain in real world network environment. Moreover, with changing network environment or services, patterns of normal traffic will be changed. This leads to high false positive rate of supervised NIDSs. Unsupervised outlier detection can overcome the drawbacks of supervised anomaly detection. Therefore, we apply one of the efficient data mining algorithms called random forests algorithm in anomaly based NIDSs. Without attack-free training data, random forests algorithm can detect outliers in datasets of network traffic. In this paper, we discuss our framework of anomaly based network intrusion detection. In the framework, patterns of network services are built by random forests algorithm over traffic data. Intrusions are detected by determining outliers related to the built patterns. We present the modification on the outlier detection algorithm of random forests. We also report our experimental results over the KDD'99 dataset. The results show that the proposed approach is comparable to previously reported unsupervised anomaly detection approaches evaluated over the KDD' 99 dataset.

Outlier detection is a very important task in many fields like network intrusion detection, credit card fraud detection, stock market analysis, detecting outlying cases in medical data etc. Outlier detection in streaming data is very challenging because streaming data cannot be scanned multiple times and also new concepts may keep evolving in coming data over time. Irrelevant attributes can be termed as noisy attributes and such attributes further magnify the challenge of working with data streams. In this paper, we propose an unsupervised outlier detection scheme for streaming data. This scheme is based on clustering as clustering is an unsupervised data mining task and it does not require labeled data. In proposed scheme both density based and partitioning clustering method are combined to take advantage of both density based and distance based outlier detection. Proposed scheme also assigns weights to attributes depending upon their respective relevance in mining task and weights are adaptive in nature. Weighted attributes are helpful to reduce or remove the effect of noisy attributes. Keeping in view the challenges of streaming data, the proposed scheme is incremental and adaptive to concept evolution. Experimental results on synthetic and real world data sets show that our proposed approach outperforms other existing approach (CORM) in terms of outlier detection rate, false alarm rate, and increasing percentages of outliers.