Query-based summarization using MDL principle

We give a brief introduction to the minimum description length (MDL) principle. The MDL principle is a mathematical formulation of Occam’s razor. It says ‘simple explanations of a given phenomenon are to be preferred over complex ones.’ This is recognized as one of basic stances of scientists, and plays an important role in statistics and machine learning. In particular, Rissanen proposed MDL criterion for statistical model selection based on information theory in 1978. After that, much literature has been published and the notion of MDL principle was founded in the 1990s. In this article, we review some important results on the MDL principle.KeywordsBayes mixtureLaplace estimatorMDLModel selectionMinimax regretUniversal code

Bayesian belief network (BBN) is a framework for representation/inference of some knowledge with uncertainty. Since the process of constructing a BBN manually by experts is time-consuming in general, some method supporting the task is needed. We proposed an algorithm for acquiring some BBN automatically from finite examples based on minimum description length (MDL) principle. This paper addresses an improvement which relaxes a constraint that the original scheme held on the representation. In BBNs, attributes and stochastic dependencies between them are expressed as nodes and directed links connecting them, respectively, where each attribute may be a predicate, a numerical data, etc., and each dependence is numerically expressed as the conditional probability of one attribute given other attributes if their dependence exists. Therefore, in general, BBNs are represented in terms of the network structure and the conditional probabilities.

This paper presents a new method for estimating the number of clusters and cluster structure in microarray data sets, based on minimum description length (MDL) principle and normalized maximum likelihood (NML) model for linear regression. The cluster-structure models for microarray data are compared based on the MDL principle. The performance of the new method is studied using simulated and real microarray data sets.

An approach to the problem of constructing tree structured classifiers that is based on Rissanen's (1983) minimum description length (MDL) principle is presented. Simple and efficient rules for sequential growing and pruning of the tree are derived using this approach. The two rules are derived from a single MDL-based criterion. These splitting and pruning rules are intuitively pleasing and computationally simple. The computational load of the pruning rule is substantially smaller than the alternative pruning schemes of A. Mabbet et al. (1980) and L. Breiman et al. (1984). The extension of these splitting and pruning criteria to the case of multiple classes is straightforward. Experimental results illustrating the performance of this technique in automatic character recognition are provided.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

This paper presents a new spatially adaptive wavelet denoising method. Based on a doubly stochastic process model of wavelet coefficients, the method gives a new threshold, which varies spatially according to the variances of the coefficients, using the minimum description length (MDL) principle. The new threshold is not only easier to analyze since it is in a closed form, but also provides more facility for future compression than several other methods, almost without deteriorating mean square error risk.

Luckiness and Regret in Minimum Description Length Inference

This paper proposes a new method for detecting the uncertainty of a latent structure. We consider the case where the latent structure of dataset changes gradually over time, with the goal of selecting the optimal model at any given time. In selecting the optimal model, we use the minimum description length (MDL) principle, specifically the normalized maximum likelihood (NML), which is the optimal code length in the sense of Shtarkov’s minimax regret. To detect the uncertainty of a latent structure, the main idea proposed here is that the uncertainty of the model selection will increase at the initial stage when the model change occurs. Here, we propose a new indicator called structural entropy (SE), which defines model selection uncertainty based on the MDL principle. We use several models for model selection, including the clustering structures of the Gaussian mixture model and Poisson mixture model, and a time-dependent structure model such as the autoregression model. We show the behavior of the proposed indicator (SE) using an artificial dataset and a real marketing dataset.

The advance in the global environment, rapidly changing markets, and information technology has created a new stage for design. In such an environment, one strategy for success is the Collaborative Product Development (CPD). Organizing people effectively is the goal of Collaborative Product Development, and it solves the problem with certain foreseeability. The development group activities are influenced not only by the methods and decisions available, but also by correlation among personnel. Grouping the personnel according to their correlation intensity is defined as collaboration space division (CSD). Upon establishment of a correlation matrix (CM) of personnel and an analysis of the collaboration space, the genetic algorithm (GA) and minimum description length (MDL) principle may be used as tools in optimizing collaboration space. The MDL principle is used in setting up an object function, and the GA is used as a methodology. The algorithm encodes spatial information as a chromosome in binary. After repetitious crossover, mutation, selection and multiplication, a robust chromosome is found, which can be decoded into an optimal collaboration space. This new method can calculate the members in sub-spaces and individual groupings within the staff. Furthermore, the intersection of sub-spaces and public persons belonging to all sub-spaces can be determined simultaneously.

The responses of visual cortical neurons during fixation tasks can be significantly modulated by stimuli from beyond the classical receptive field. Modulatory effects in neural responses have also been recently reported in a task where a monkey freely views a natural scene. In this article, we describe a hierarchical network model of visual recognition that explains these experimental observations by using a form of the extended Kalman filter as given by the minimum description length (MDL) principle. The model dynamically combines input-driven bottom-up signals with expectation-driven top-down signals to predict current recognition state. Synaptic weights in the model are adapted in a Hebbian manner according to a learning rule also derived from the MDL principle. The resulting prediction-learning scheme can be viewed as implementing a form of expectation-maximization (EM) algorithm. The architecture of the model posits an active computational role of the reciprocal connections between adjoining visual cortical areas in determining neural response properties. In particular, the model demonstrates the possible role of feedback from higher cortical areas in mediating neurophysiological effects due to stimuli from beyond the classical receptive field. Simulations of the model are provided that help explain the experimental observations regarding neural responses in both free viewing and fixation conditions.

Integrative mining of heterogeneous data is one of the major challenges for data mining in the next decade. We address the problem of integrative clustering of data with mixed type attributes. Most existing solutions suffer from one or both of the following drawbacks: Either they require input parameters which are difficult to estimate, or/and they do not adequately support mixed type attributes. Our technique INTEGRATE is a novel clustering approach that truly integrates the information provided by heterogeneous numerical and categorical attributes. Originating from information theory, the Minimum Description Length (MDL) principle allows a unified view on numerical and categorical information and thus naturally balances the influence of both sources of information in clustering. Moreover, supported by the MDL principle, parameter-free clustering can be performed which enhances the usability of INTEGRATE on real world data. Extensive experiments demonstrate the effectiveness of INTEGRATE in exploiting numerical and categorical information for clustering. As an efficient iterative algorithm INTEGRATE is scalable to large data sets.

The structure and biological behavior of a cell are determined by the pattern of gene expressions within that cell. The so-called gene prediction problem refers to finding rules, or sets of possible rules, on how certain genes expressions determine the expression level of a given target gene. In this paper, we investigate the gene prediction problem and propose the use of new predictors, selected according to the minimum description length (MDL) principle. We compare the use of Boolean predictors, ternary predictors and perceptron predictors. We resort to MDL as a tool for selecting the proper size of the prediction window. MDL is also well suited for comparing predictors having different complexities. We show that the best description can be achieved by the Boolean and ternary predictors, since they obtain better fitting of the data with a lower complexity of the model. To illustrate the comparison, both synthetic and experimental data are used.

Presents a new formulation of moving object segmentation and motion estimation to quantify the potential gain in using object-oriented motion compensation in image sequence coding. Motivated by Rissanen's (1983) minimum description length (MDL) principle that allows unified treatment of coding model complexity and parameter values, an objective function is constructed in terms of an ideal coding length function for each motion-compensated frame. Based on this new objective function, a procedure to segment and estimate moving objects in a scene is proposed. Each object in the scene is represented by its boundaries, motion parameters, and motion-compensated prediction error (MCPE). A number of experimental comparisons between block-oriented and object oriented coding schemes suggest that significant potential coding gain using the new MDL-based criterion is possible. >

Reference assisted assembly requires the use of a reference sequence, as a model, to assist in the assembly of the novel genome. The standard method for identifying the best reference sequence for the assembly of a novel genome aims at counting the number of reads that align to the reference sequence, and then choosing the reference sequence which has the highest number of reads aligning to it. This article explores the use of minimum description length (MDL) principle and its two variants, the two-part MDL and Sophisticated MDL, in identifying the optimal reference sequence for genome assembly. The article compares the MDL based proposed scheme with the standard method coming to the conclusion that “counting the number of reads of the novel genome present in the reference sequence” is not a sufficient condition. Therefore, the proposed MDL scheme includes within itself the standard method of “counting the number of reads that align to the reference sequence” and also moves forward towards looking at the model, the reference sequence, as well, in identifying the optimal reference sequence. The proposed MDL based scheme not only becomes the sufficient criterion for identifying the optimal reference sequence for genome assembly but also improves the reference sequence so that it becomes more suitable for the assembly of the novel genome.

We observed sea clutter using a millimeter wave radar having a frequency 34.86GHz, a beamwidth 0.25°, and a pulsewidth 30ns. To determine the sea clutter amplitude, we introduce the Minimum Description Length (MDL) principle, which is more rigorous fit of the distribution to the data than the Akaike Information Criterion (AIC). It is discovered that the sea clutter amplitudes obey the log-Weibull distribution with the shape parameter of 19.19 for entire data and the Weibull and log-Weibull distributions with the shape parameters of 4.81 to 5.17 and 17.72 to 20.99, respectively, for data within the beam width of an antenna.

In this chapter, we investigate refinements to the structure selection method used in our recently developed minimum description length (MDL) method for interpreting clumps of nuclei in histological images. We start from the SNEF method, which fits elliptical shapes to the clump image based on the extracted contours and on the image gradient information. Introducing some variability in the parameters of the algorithm, we obtain a number of competing interpretations and we select the least redundant interpretation based on the MDL principle, where the description codelengths are evaluated by a simple implementable coding scheme. We investigate in this paper two ways for allowing additional variability in the basic SNEF method: first by utilizing a pre-processing stage of smoothing the original image using various degrees of smoothing and second by using re-scaling of the original image at various downsizing scales. Both transformations have the potential to hide artifacts and features of the original image that prevented the proper interpretation of the nuclei shapes, and we show experimentally that the set of candidate segmentations obtained will contain variants with better MDL values than the MDL of the initial SNEF segmentations. We compare the results of the automatic interpretation algorithm against the ground truth defined by annotations of human subjects.