The minimum description length principle in coding and modeling

Modern techniques invented for data compression provide efficient automated algorithms for the modeling of the observed symbolic dynamics. We demonstrate the relationship between coding and modeling, motivating the well-known minimum description length (MDL) principle, and give concrete demonstrations of the "context-tree weighting" and "context-tree maximizing" algorithms. The predictive modeling technique obviates many of the technical difficulties traditionally associated with the correct MDL analyses. These symbolic models, representing the symbol generating process as a finite-state automaton with probabilistic emission probabilities, provide excellent and reliable entropy estimations. The resimulations of estimated tree models satisfying the MDL model-selection criterion are faithful to the original in a number of measures. The modeling suggests that the automated context-tree model construction could replace fixed-order word lengths in many traditional forms of empirical symbolic analysis of the data. We provide an explicit pseudocode for implementation of the context-tree weighting and maximizing algorithms, as well as for the conversion to an equivalent Markov chain.

This paper proposes a new method for detecting the uncertainty of a latent structure. We consider the case where the latent structure of dataset changes gradually over time, with the goal of selecting the optimal model at any given time. In selecting the optimal model, we use the minimum description length (MDL) principle, specifically the normalized maximum likelihood (NML), which is the optimal code length in the sense of Shtarkov’s minimax regret. To detect the uncertainty of a latent structure, the main idea proposed here is that the uncertainty of the model selection will increase at the initial stage when the model change occurs. Here, we propose a new indicator called structural entropy (SE), which defines model selection uncertainty based on the MDL principle. We use several models for model selection, including the clustering structures of the Gaussian mixture model and Poisson mixture model, and a time-dependent structure model such as the autoregression model. We show the behavior of the proposed indicator (SE) using an artificial dataset and a real marketing dataset.

We give a brief introduction to the minimum description length (MDL) principle. The MDL principle is a mathematical formulation of Occam’s razor. It says ‘simple explanations of a given phenomenon are to be preferred over complex ones.’ This is recognized as one of basic stances of scientists, and plays an important role in statistics and machine learning. In particular, Rissanen proposed MDL criterion for statistical model selection based on information theory in 1978. After that, much literature has been published and the notion of MDL principle was founded in the 1990s. In this article, we review some important results on the MDL principle.KeywordsBayes mixtureLaplace estimatorMDLModel selectionMinimax regretUniversal code

Bayesian belief network (BBN) is a framework for representation/inference of some knowledge with uncertainty. Since the process of constructing a BBN manually by experts is time-consuming in general, some method supporting the task is needed. We proposed an algorithm for acquiring some BBN automatically from finite examples based on minimum description length (MDL) principle. This paper addresses an improvement which relaxes a constraint that the original scheme held on the representation. In BBNs, attributes and stochastic dependencies between them are expressed as nodes and directed links connecting them, respectively, where each attribute may be a predicate, a numerical data, etc., and each dependence is numerically expressed as the conditional probability of one attribute given other attributes if their dependence exists. Therefore, in general, BBNs are represented in terms of the network structure and the conditional probabilities.

This paper presents a new method for estimating the number of clusters and cluster structure in microarray data sets, based on minimum description length (MDL) principle and normalized maximum likelihood (NML) model for linear regression. The cluster-structure models for microarray data are compared based on the MDL principle. The performance of the new method is studied using simulated and real microarray data sets.

An approach to the problem of constructing tree structured classifiers that is based on Rissanen's (1983) minimum description length (MDL) principle is presented. Simple and efficient rules for sequential growing and pruning of the tree are derived using this approach. The two rules are derived from a single MDL-based criterion. These splitting and pruning rules are intuitively pleasing and computationally simple. The computational load of the pruning rule is substantially smaller than the alternative pruning schemes of A. Mabbet et al. (1980) and L. Breiman et al. (1984). The extension of these splitting and pruning criteria to the case of multiple classes is straightforward. Experimental results illustrating the performance of this technique in automatic character recognition are provided.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Multinomial processing tree (MPT) modeling has been widely and successfully applied as a statistical methodology for measuring hypothesized latent cognitive processes in selected experimental paradigms. In this article, we address the problem of selecting the best MPT model from a set of scientifically plausible MPT models, given observed data. We introduce a minimum description length (MDL) based model-selection approach that overcomes the limitations of existing methods such as the G(2)-based likelihood ratio test, the Akaike information criterion, and the Bayesian information criterion. To help ease the computational burden of implementing MDL, we provide a computer program in MATLAB that performs MDL-based model selection for any MPT model, with or without inequality constraints. Finally, we discuss applications of the MDL approach to well-studied MPT models with real data sets collected in two different experimental paradigms: source monitoring and pair clustering. The aforementioned MATLAB program may be downloaded from http://pbr.psychonomic-journals.org/content/supplemental.

This paper presents a new spatially adaptive wavelet denoising method. Based on a doubly stochastic process model of wavelet coefficients, the method gives a new threshold, which varies spatially according to the variances of the coefficients, using the minimum description length (MDL) principle. The new threshold is not only easier to analyze since it is in a closed form, but also provides more facility for future compression than several other methods, almost without deteriorating mean square error risk.

Query-based text summarization is aimed at extracting essential information that answers the query from original text. The answer is presented in a minimal, often predefined, number of words. In this paper we introduce a new unsupervised approach for query-based extractive summarization, based on the minimum description length (MDL) principle that employs Krimp compression algorithm (Vreeken et al., 2011). The key idea of our approach is to select frequent word sets related to a given query that compress document sentences better and therefore describe the document better. A summary is extracted by selecting sentences that best cover query-related frequent word sets. The approach is evaluated based on the DUC 2005 and DUC 2006 datasets which are specifically designed for query-based summarization (DUC, 2005 2006). It competes with the best results.

The advance in the global environment, rapidly changing markets, and information technology has created a new stage for design. In such an environment, one strategy for success is the Collaborative Product Development (CPD). Organizing people effectively is the goal of Collaborative Product Development, and it solves the problem with certain foreseeability. The development group activities are influenced not only by the methods and decisions available, but also by correlation among personnel. Grouping the personnel according to their correlation intensity is defined as collaboration space division (CSD). Upon establishment of a correlation matrix (CM) of personnel and an analysis of the collaboration space, the genetic algorithm (GA) and minimum description length (MDL) principle may be used as tools in optimizing collaboration space. The MDL principle is used in setting up an object function, and the GA is used as a methodology. The algorithm encodes spatial information as a chromosome in binary. After repetitious crossover, mutation, selection and multiplication, a robust chromosome is found, which can be decoded into an optimal collaboration space. This new method can calculate the members in sub-spaces and individual groupings within the staff. Furthermore, the intersection of sub-spaces and public persons belonging to all sub-spaces can be determined simultaneously.

The responses of visual cortical neurons during fixation tasks can be significantly modulated by stimuli from beyond the classical receptive field. Modulatory effects in neural responses have also been recently reported in a task where a monkey freely views a natural scene. In this article, we describe a hierarchical network model of visual recognition that explains these experimental observations by using a form of the extended Kalman filter as given by the minimum description length (MDL) principle. The model dynamically combines input-driven bottom-up signals with expectation-driven top-down signals to predict current recognition state. Synaptic weights in the model are adapted in a Hebbian manner according to a learning rule also derived from the MDL principle. The resulting prediction-learning scheme can be viewed as implementing a form of expectation-maximization (EM) algorithm. The architecture of the model posits an active computational role of the reciprocal connections between adjoining visual cortical areas in determining neural response properties. In particular, the model demonstrates the possible role of feedback from higher cortical areas in mediating neurophysiological effects due to stimuli from beyond the classical receptive field. Simulations of the model are provided that help explain the experimental observations regarding neural responses in both free viewing and fixation conditions.

A consistent model selection algorithm is presented for superimposed signal models. The proposed method is motivated by the Wald statistic and reduces the computational complexity of procedures based on the minimum description length (MDL) principle. The procedure is suggested when a noncyclostationary signal model or short data length prevents use of covariance rank test. For maximum model-order K, the procedure provides O(K) computational savings over an MDL test. Additionally, a proof establishes the consistency of a least-squares estimator using overparametrized. models. Finite sample performance of the proposed model selection method is studied via Monte Carlo simulations for estimating the multipath delays and amplitudes of a chirp signal.

Luckiness and Regret in Minimum Description Length Inference

This paper studies estimation of stochastic block models with Rissanen’s minimum description length (MDL) principle in the dense graph asymptotics. We focus on the problem of model specification, i.e., identification of the number of blocks. Refinements of the true partition always decrease the code part corresponding to the edge placement, and thus a respective increase of the code part specifying the model should overweight that gain in order to yield a minimum at the true partition. The balance between these effects turns out to be delicate. We show that the MDL principle identifies the true partition among models whose relative block sizes are bounded away from zero. The results are extended to models with Poisson-distributed edge weights.

In this chapter, we investigate refinements to the structure selection method used in our recently developed minimum description length (MDL) method for interpreting clumps of nuclei in histological images. We start from the SNEF method, which fits elliptical shapes to the clump image based on the extracted contours and on the image gradient information. Introducing some variability in the parameters of the algorithm, we obtain a number of competing interpretations and we select the least redundant interpretation based on the MDL principle, where the description codelengths are evaluated by a simple implementable coding scheme. We investigate in this paper two ways for allowing additional variability in the basic SNEF method: first by utilizing a pre-processing stage of smoothing the original image using various degrees of smoothing and second by using re-scaling of the original image at various downsizing scales. Both transformations have the potential to hide artifacts and features of the original image that prevented the proper interpretation of the nuclei shapes, and we show experimentally that the set of candidate segmentations obtained will contain variants with better MDL values than the MDL of the initial SNEF segmentations. We compare the results of the automatic interpretation algorithm against the ground truth defined by annotations of human subjects.