A Maximum Entropy Procedure to Solve Likelihood Equations

A maximum entropy approach to adaptive statistical language modelling

We present an extension to Jaynes’ maximum entropy principle that incorporates latent variables. The principle oflatent maximum entropywe propose is different from both Jaynes’ maximum entropy principle and maximum likelihood estimation, but can yield better estimates in the presence of hidden variables and limited training data. We first show that solving for a latent maximum entropy model poses a hard nonlinear constrained optimization problem in general. However, we then show that feasible solutions to this problem can be obtained efficiently for the special case of log-linear models---which forms the basis for an efficient approximation to the latent maximum entropy principle. We derive an algorithm that combines expectation-maximization with iterative scaling to produce feasible log-linear solutions. This algorithm can be interpreted as an alternating minimization algorithm in the information divergence, and reveals an intimate connection between the latent maximum entropy and maximum likelihood principles. To select a final model, we generate a series of feasible candidates, calculate the entropy of each, and choose the model that attains the highest entropy. Our experimental results show that estimation based on the latent maximum entropy principle generally gives better results than maximum likelihood when estimating latent variable models on small observed data samples.

The variations of the b coefficient in the frequency-magnitude relationship for earthquakes which occurred at Mt. Etna from 01/01/90 to 31/12/92 are analyzed; the completeness threshold for our earthquakes catalogue is M = 2.30. The b values calculated using the Maximum Entropy Principle (MEP) are compared to those obtained by conventional methods of Least Squares (LS) and Maximum Likelihood (ML). All the differences among the b values computed using these methods, and the reasons for these differences, are discussed and examined. In particular, our results show that the b values obtained by MEP are lower than the others calculated using LS and ML; this implies that, on the average, LS and ML underestimate the seismic hazard at Mt. Etna. Moreover, temporal variations of bMEP are more evident than the corresponding ones of bLS and bML; indeed, in some cases the trend of bMEP variation is opposite those of bLS and bML. A significant correlation between temporal variations of the volcanic activity and the b values is evident only if the MEP is used; this means that, if b temporal variations are analyzed in order to detect changes in the volcano dynamics and predict the eruptions, the maximum entropy approach should be preferred. Finally, the observed pattern of bMEP temporal variations with regard to the changes in the volcanic activity is consistent with the hypothesis of a compressive stress field acting on Mt. Etna.

A new multi-directional search approach that aims at maximizing the flow entropy of water distribution systems is investigated. The aim is to develop an efficient and practical maximum entropy based approach. The resulting optimization problem has four objectives, and the merits of objective reduction in the computational solution of the problem are investigated also. The relationship between statistical flow entropy and hydraulic reliability/failure tolerance is not monotonic. Consequently, a large number of maximum flow entropy solutions must be investigated to strike a balance between cost and hydraulic reliability. A multi-objective evolutionary optimization model is developed that generates simultaneously a wide range of maximum entropy values along with clusters of maximum and near-maximum entropy solutions. Results for a benchmark network and a real network in the literature are included that demonstrate the effectiveness of the procedure.

One well-recognized problem in three-dimensional (3D) confocal microscopic images is that the intensities in deeper slices are generally weaker than those in shallower slices. The loss of intensity with depth hampers both qualitative observation and quantitative measurement of specimens. Two major types of methods exist to compensate for this intensity loss: the first is based on the geometrical optics inside the specimen, and the second applies an empirical parametric intensity decay function (IDF) of depth. A common feature shared by both methods is that they are parameter-dependent. However, for the optics-based method there are as yet no fully automated parameter-setting approaches; and for the IDF method the traditional profile-fitting approach cannot provide proper parameters if the presumed IDF model does not match the experimental intensity-depth profile of the 3D image. In this paper, we propose a novel maximum-entropy (ME) approach to fully automated parameter-setting. In principle the ME approach is suitable for any compensation method as long as it is parameter-dependent. The basic assumption is that without intensity loss an ideal 3D image should be generally homogeneous with respect to depth and this axial homogeneity can be represented by the entropy of a normalized intensity-depth profile. Experiments on real confocal images showed that such a profile was consistent with visual evaluation of axial intensity homogeneity and that the ME approach could provide proper parameters for both compensation methods mentioned above. Moreover, for the IDF method, experiments on both real and simulated data showed that the ME approach could provide more precise parameters than with traditional profile-fitting. The Appendix provides a proof that under certain conditions the global maximization of the profile-entropy is guaranteed.

In this paper, we present a maximum entropy (maxent) approach to the fusion of experts opinions, or classifiers outputs, problem. The maxent approach is quite versatile and allows us to express in a clear, rigorous, way the a priori knowledge that is available on the problem. For instance, our knowledge about the reliability of the experts and the correlations between these experts can be easily integrated: Each piece of knowledge is expressed in the form of a linear constraint. An iterative scaling algorithm is used in order to compute the maxent solution of the problem. The maximum entropy method seeks the joint probability density of a set of random variables that has maximum entropy while satisfying the constraints. It is therefore the ”most honest” characterization of our knowledge given the available facts (constraints). In the case of conflicting constraints, we propose to minimise the ”lack of constraints satisfaction” or to relax some constraints and recompute the maximum entropy solution. The maxent fusion rule is illustrated by some simulations.

The classical maximum entropy (ME) approach to estimating the unknown parameters of a multinomial discrete choice problem, which is equivalent to the maximum likelihood multinomial logit (ML) estimator, is generalized. The generalized maximum entropy (GME) model includes noise terms in the multinomial information constraints. Each noise term is modeled as the mean of a finite set of a priori known points in the interval [−1,1] with unknown probabilities where no parametric assumptions about the error distribution are made. A GME model for the multinomial probabilities and for the distributions associated with the noise terms is derived by maximizing the joint entropy of multinomial and noise distributions, under the assumption of independence. The GME formulation reduces to the ME in the limit as the sample grows large or when no noise is included in the entropy maximization. Further, even though the GME and the logit estimators are conceptually different, the dual GME model is related to a gener...

The Maximum Likelihood (ML) and Best Linear Unbiased (BLU) estimators of the location and scale parameters of an extreme value distribution (Lawless [1982]) are compared under conditions of small sample sizes and Type I censorship. The comparisons were made in terms of the mean square error criterion. According to this criterion, the ML estimator of σ in the case of very small sample sizes (n < 10) and heavy censorship (low censoring time) proved to be more efficient than the corresponding BLU estimator. However, the BLU estimator for σ attains parity with the corresponding ML estimator when the censoring time increases even for sample sizes as low as 10. The BLU estimator of σ attains equivalence with the ML estimator when the sample size increases above 10, particularly when the censoring time is also increased. The situation is reversed when it came to estimating the location parameter μ, as the BLU estimator was found to be consistently more efficient than the ML estimator despite the improved performance of the ML estimator when the sample size increases. However, computational ease and convenience favor the ML estimators. Keywords: maximum likelihood estimatorsbest linear unbiased estimatorslife-testingreliability

Location‐only and use‐availability data: analysis methods converge

During the MaxEnt 2002 workshop in Moscow, Idaho, Tony Vignaux asked again a few simple questions about using Maximum Entropy or Bayesian approaches for the famous Dice problems which have been analyzed many times through this workshop and also in other places. Here, there is another analysis of these problems. I hope that, this paper will answer a few questions of Tony and other participants of the workshop on the situations where we can use Maximum Entropy or Bayesian approaches or even the cases where we can actually use both of them.

Price responsiveness of herbicide demand in corn for farmers in Indiana's White River Basin using cross-section data from individual farms is estimated. Particular attention is paid to appropriate treatment of binding nonnegativity constraints. Estimation was first attempted using an approach to demand systems estimation suggested by Lee and Pitt. However, analytical and computational difficulties effectively preclude estimation by the Lee and Pitt approach. As an alternative, a maximum entropy (ME) approach is presented and discussed. Results from the ME estimator tentatively indicate limited response of herbicide demand to changes in own prices. The maximum entropy approach to demand systems estimation appears to have merit and warrants further attention.

The Pareto distribution has been introduced in Chapter 19. Also discussed there is a brief review of literature and methods of estimating its parameters. Further elaboration of the distribution is given in Chapter 20. Methods of parameter estimation were reviewed by Hosking and Wallis (1987). The methods of moments (MOM), maximum likelihood estimation (MLE) and probability weighted moments (PWM) were included in the review. Guo and Singh (1992) and Singh and Guo (1995) employed the principle of maximum entropy (POME) to develop a new competitive method of parameter estimation (Singh and Rajagopal, 1986) for the 3-parameter generalized Pareto (GP3) distribution and compared it with MOM, MLE and PWM using Monte Carlo simulated data The parameter estimates yielded by POME were either superior or comparable for high skewness.KeywordsRoot Mean Square ErrorMaximum Likelihood EstimationPareto DistributionGeneralize Pareto DistributionQuantile EstimationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Understanding how ensembles of neurons collectively interact will be a key step in developing a mechanistic theory of cognitive processes. Recent progress in multineuron recording and analysis techniques has generated tremendous excitement over the physiology of living neural networks. One of the key developments driving this interest is a new class of models based on the principle of maximum entropy. Maximum entropy models have been reported to account for spatial correlation structure in ensembles of neurons recorded from several different types of data. Importantly, these models require only information about the firing rates of individual neurons and their pairwise correlations. If this approach is generally applicable, it would drastically simplify the problem of understanding how neural networks behave. Given the interest in this method, several groups now have worked to extend maximum entropy models to account for temporal correlations. Here, we review how maximum entropy models have been applied to neuronal ensemble data to account for spatial and temporal correlations. We also discuss criticisms of the maximum entropy approach that argue that it is not generally applicable to larger ensembles of neurons. We conclude that future maximum entropy models will need to address three issues: temporal correlations, higher-order correlations, and larger ensemble sizes. Finally, we provide a brief list of topics for future research.

The Multiple Comparison Procedures with Modeling Techniques (MCP-Mod) framework has been recently approved by the U.S. Food, Administration, and European Medicines Agency as fit-for-purpose for phase II studies. Nonetheless, this approach relies on the asymptotic properties of Maximum Likelihood (ML) estimators, which might not be reasonable for small sample sizes. In this paper, we derived improved ML estimators and correction for their covariance matrices in the censored Weibull regression model based on the corrective and preventive approaches. We performed two simulation studies to evaluate ML and improved ML estimators with their covariance matrices in (i) a regression framework (ii) the Multiple Comparison Procedures with Modeling Techniques framework. We have shown that improved ML estimators are less biased than ML estimators yielding Wald-type statistics that controls type I error without loss of power in both frameworks. Therefore, we recommend the use of improved ML estimators in the MCP-Mod approach to control type I error at nominal value for sample sizes ranging from 5 to 25 subjects per dose.

For a balanced two‐way mixed model, the maximum likelihood (ML) and restricted ML (REML) estimators of the variance components were obtained and compared under the non‐negativity requirements of the variance components by Lee and Kapadia (1984). In this note, for a mixed (random blocks) incomplete block model, explicit forms for the REML estimators of variance components are obtained. They are always non‐negative and have smaller mean squared error (MSE) than the analysis of variance (AOV) estimators. The asymptotic sampling variances of the maximum likelihood (ML) estimators and the REML estimators are compared and the balanced incomplete block design (BIBD) is considered as a special case. The ML estimators are shown to have smaller asymptotic variances than the REML estimators, but a numerical result in the randomized complete block design (RCBD) demonstrated that the performances of the REML and ML estimators are not much different in the MSE sense.