On model selection for dense stochastic block models

We give a brief introduction to the minimum description length (MDL) principle. The MDL principle is a mathematical formulation of Occam’s razor. It says ‘simple explanations of a given phenomenon are to be preferred over complex ones.’ This is recognized as one of basic stances of scientists, and plays an important role in statistics and machine learning. In particular, Rissanen proposed MDL criterion for statistical model selection based on information theory in 1978. After that, much literature has been published and the notion of MDL principle was founded in the 1990s. In this article, we review some important results on the MDL principle.KeywordsBayes mixtureLaplace estimatorMDLModel selectionMinimax regretUniversal code

This paper proposes a new method for detecting the uncertainty of a latent structure. We consider the case where the latent structure of dataset changes gradually over time, with the goal of selecting the optimal model at any given time. In selecting the optimal model, we use the minimum description length (MDL) principle, specifically the normalized maximum likelihood (NML), which is the optimal code length in the sense of Shtarkov’s minimax regret. To detect the uncertainty of a latent structure, the main idea proposed here is that the uncertainty of the model selection will increase at the initial stage when the model change occurs. Here, we propose a new indicator called structural entropy (SE), which defines model selection uncertainty based on the MDL principle. We use several models for model selection, including the clustering structures of the Gaussian mixture model and Poisson mixture model, and a time-dependent structure model such as the autoregression model. We show the behavior of the proposed indicator (SE) using an artificial dataset and a real marketing dataset.

This paper presents a new spatially adaptive wavelet denoising method. Based on a doubly stochastic process model of wavelet coefficients, the method gives a new threshold, which varies spatially according to the variances of the coefficients, using the minimum description length (MDL) principle. The new threshold is not only easier to analyze since it is in a closed form, but also provides more facility for future compression than several other methods, almost without deteriorating mean square error risk.

Bayesian belief network (BBN) is a framework for representation/inference of some knowledge with uncertainty. Since the process of constructing a BBN manually by experts is time-consuming in general, some method supporting the task is needed. We proposed an algorithm for acquiring some BBN automatically from finite examples based on minimum description length (MDL) principle. This paper addresses an improvement which relaxes a constraint that the original scheme held on the representation. In BBNs, attributes and stochastic dependencies between them are expressed as nodes and directed links connecting them, respectively, where each attribute may be a predicate, a numerical data, etc., and each dependence is numerically expressed as the conditional probability of one attribute given other attributes if their dependence exists. Therefore, in general, BBNs are represented in terms of the network structure and the conditional probabilities.

This paper presents a new method for estimating the number of clusters and cluster structure in microarray data sets, based on minimum description length (MDL) principle and normalized maximum likelihood (NML) model for linear regression. The cluster-structure models for microarray data are compared based on the MDL principle. The performance of the new method is studied using simulated and real microarray data sets.

An approach to the problem of constructing tree structured classifiers that is based on Rissanen's (1983) minimum description length (MDL) principle is presented. Simple and efficient rules for sequential growing and pruning of the tree are derived using this approach. The two rules are derived from a single MDL-based criterion. These splitting and pruning rules are intuitively pleasing and computationally simple. The computational load of the pruning rule is substantially smaller than the alternative pruning schemes of A. Mabbet et al. (1980) and L. Breiman et al. (1984). The extension of these splitting and pruning criteria to the case of multiple classes is straightforward. Experimental results illustrating the performance of this technique in automatic character recognition are provided.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Luckiness and Regret in Minimum Description Length Inference

Query-based text summarization is aimed at extracting essential information that answers the query from original text. The answer is presented in a minimal, often predefined, number of words. In this paper we introduce a new unsupervised approach for query-based extractive summarization, based on the minimum description length (MDL) principle that employs Krimp compression algorithm (Vreeken et al., 2011). The key idea of our approach is to select frequent word sets related to a given query that compress document sentences better and therefore describe the document better. A summary is extracted by selecting sentences that best cover query-related frequent word sets. The approach is evaluated based on the DUC 2005 and DUC 2006 datasets which are specifically designed for query-based summarization (DUC, 2005 2006). It competes with the best results.

The advance in the global environment, rapidly changing markets, and information technology has created a new stage for design. In such an environment, one strategy for success is the Collaborative Product Development (CPD). Organizing people effectively is the goal of Collaborative Product Development, and it solves the problem with certain foreseeability. The development group activities are influenced not only by the methods and decisions available, but also by correlation among personnel. Grouping the personnel according to their correlation intensity is defined as collaboration space division (CSD). Upon establishment of a correlation matrix (CM) of personnel and an analysis of the collaboration space, the genetic algorithm (GA) and minimum description length (MDL) principle may be used as tools in optimizing collaboration space. The MDL principle is used in setting up an object function, and the GA is used as a methodology. The algorithm encodes spatial information as a chromosome in binary. After repetitious crossover, mutation, selection and multiplication, a robust chromosome is found, which can be decoded into an optimal collaboration space. This new method can calculate the members in sub-spaces and individual groupings within the staff. Furthermore, the intersection of sub-spaces and public persons belonging to all sub-spaces can be determined simultaneously.

The responses of visual cortical neurons during fixation tasks can be significantly modulated by stimuli from beyond the classical receptive field. Modulatory effects in neural responses have also been recently reported in a task where a monkey freely views a natural scene. In this article, we describe a hierarchical network model of visual recognition that explains these experimental observations by using a form of the extended Kalman filter as given by the minimum description length (MDL) principle. The model dynamically combines input-driven bottom-up signals with expectation-driven top-down signals to predict current recognition state. Synaptic weights in the model are adapted in a Hebbian manner according to a learning rule also derived from the MDL principle. The resulting prediction-learning scheme can be viewed as implementing a form of expectation-maximization (EM) algorithm. The architecture of the model posits an active computational role of the reciprocal connections between adjoining visual cortical areas in determining neural response properties. In particular, the model demonstrates the possible role of feedback from higher cortical areas in mediating neurophysiological effects due to stimuli from beyond the classical receptive field. Simulations of the model are provided that help explain the experimental observations regarding neural responses in both free viewing and fixation conditions.

We review the principles of minimum description length and stochastic complexity as used in data compression and statistical modeling. Stochastic complexity is formulated as the solution to optimum universal coding problems extending Shannon's basic source coding theorem. The normalized maximized likelihood, mixture, and predictive codings are each shown to achieve the stochastic complexity to within asymptotically vanishing terms. We assess the performance of the minimum description length criterion both from the vantage point of quality of data compression and accuracy of statistical inference. Context tree modeling, density estimation, and model selection in Gaussian linear regression serve as examples.

Integrative mining of heterogeneous data is one of the major challenges for data mining in the next decade. We address the problem of integrative clustering of data with mixed type attributes. Most existing solutions suffer from one or both of the following drawbacks: Either they require input parameters which are difficult to estimate, or/and they do not adequately support mixed type attributes. Our technique INTEGRATE is a novel clustering approach that truly integrates the information provided by heterogeneous numerical and categorical attributes. Originating from information theory, the Minimum Description Length (MDL) principle allows a unified view on numerical and categorical information and thus naturally balances the influence of both sources of information in clustering. Moreover, supported by the MDL principle, parameter-free clustering can be performed which enhances the usability of INTEGRATE on real world data. Extensive experiments demonstrate the effectiveness of INTEGRATE in exploiting numerical and categorical information for clustering. As an efficient iterative algorithm INTEGRATE is scalable to large data sets.

Reference assisted assembly requires the use of a reference sequence, as a model, to assist in the assembly of the novel genome. The standard method for identifying the best reference sequence for the assembly of a novel genome aims at counting the number of reads that align to the reference sequence, and then choosing the reference sequence which has the highest number of reads aligning to it. This article explores the use of minimum description length (MDL) principle and its two variants, the two-part MDL and Sophisticated MDL, in identifying the optimal reference sequence for genome assembly. The article compares the MDL based proposed scheme with the standard method coming to the conclusion that “counting the number of reads of the novel genome present in the reference sequence” is not a sufficient condition. Therefore, the proposed MDL scheme includes within itself the standard method of “counting the number of reads that align to the reference sequence” and also moves forward towards looking at the model, the reference sequence, as well, in identifying the optimal reference sequence. The proposed MDL based scheme not only becomes the sufficient criterion for identifying the optimal reference sequence for genome assembly but also improves the reference sequence so that it becomes more suitable for the assembly of the novel genome.

The structure and biological behavior of a cell are determined by the pattern of gene expressions within that cell. The so-called gene prediction problem refers to finding rules, or sets of possible rules, on how certain genes expressions determine the expression level of a given target gene. In this paper, we investigate the gene prediction problem and propose the use of new predictors, selected according to the minimum description length (MDL) principle. We compare the use of Boolean predictors, ternary predictors and perceptron predictors. We resort to MDL as a tool for selecting the proper size of the prediction window. MDL is also well suited for comparing predictors having different complexities. We show that the best description can be achieved by the Boolean and ternary predictors, since they obtain better fitting of the data with a lower complexity of the model. To illustrate the comparison, both synthetic and experimental data are used.

A consistent model selection algorithm is presented for superimposed signal models. The proposed method is motivated by the Wald statistic and reduces the computational complexity of procedures based on the minimum description length (MDL) principle. The procedure is suggested when a noncyclostationary signal model or short data length prevents use of covariance rank test. For maximum model-order K, the procedure provides O(K) computational savings over an MDL test. Additionally, a proof establishes the consistency of a least-squares estimator using overparametrized. models. Finite sample performance of the proposed model selection method is studied via Monte Carlo simulations for estimating the multipath delays and amplitudes of a chirp signal.