Maximum Entropy Principle in Reliability Analysis

We present an extension to Jaynes’ maximum entropy principle that incorporates latent variables. The principle oflatent maximum entropywe propose is different from both Jaynes’ maximum entropy principle and maximum likelihood estimation, but can yield better estimates in the presence of hidden variables and limited training data. We first show that solving for a latent maximum entropy model poses a hard nonlinear constrained optimization problem in general. However, we then show that feasible solutions to this problem can be obtained efficiently for the special case of log-linear models---which forms the basis for an efficient approximation to the latent maximum entropy principle. We derive an algorithm that combines expectation-maximization with iterative scaling to produce feasible log-linear solutions. This algorithm can be interpreted as an alternating minimization algorithm in the information divergence, and reveals an intimate connection between the latent maximum entropy and maximum likelihood principles. To select a final model, we generate a series of feasible candidates, calculate the entropy of each, and choose the model that attains the highest entropy. Our experimental results show that estimation based on the latent maximum entropy principle generally gives better results than maximum likelihood when estimating latent variable models on small observed data samples.

The set of solutions inferred by the generic maximum entropy (MaxEnt) or maximum relative entropy (MaxREnt) principles of Jaynes - considered as a function of the moment constraints or their conjugate Lagrangian multipliers - is endowed with a Riemannian geometric description, based on the second differential tensor of the entropy or its Legendre transform (negative Massieu function). The analysis provides a generalised {\it least action bound} applicable to all Jaynesian systems, which provides a lower bound to the cost (in generic entropy units) of a transition between inferred positions along a specified path, at specified rates of change of the control parameters. The analysis therefore extends the concepts of "finite time thermodynamics" to the generic Jaynes domain, providing a link between purely static (stationary) inferred positions of a system, and dynamic transitions between these positions (as a function of time or some other coordinate). If the path is unspecified, the analysis gives an absolute lower bound for the cost of the transition, corresponding to the geodesic of the Riemannian hypersurface. The analysis is applied to (i) an equilibrium thermodynamic system subject to mean internal energy and volume constraints, and (ii) a flow system at steady state, subject to constraints on the mean heat, mass and momentum fluxes and chemical reaction rates. The first example recovers the {\it minimum entropy cost} of a transition between equilibrium positions, a widely used result of finite-time thermodynamics. The second example leads to a new {\it minimum entropy production principle}, for the cost of a transition between steady state positions of a flow system.

The principle of maximum entropy and a generalization, the principle of minimum cross entropy, are prescriptions for solving problems of the following sort, which are encountered in a remarkable number of different fields. Namely, some system may be in any one of a given set of states; the probabilities of its being in the various states are not specified, but information about the probability distribution is available in the form of expectation values of given functions on the set of states. It is required to assign an optimal, or minimally prejudiced, set of state probabilities consistent with the given information. This paper describes the principles and presents APL functions that use the Newton-Raphson method to compute maximum-entropy and minimum-cross-entropy probability distributions for arbitrary sets of given expectation values. Example calculations are given.

The methods for determining OWA operator weights have aroused wide attention. We first review the main existing methods for determining OWA operator weights. We next introduce the principle of maximum entropy for setting up probability distributions on the basis of partial knowledge and prove that Xu's normal distribution-based method obeys the principle of maximum entropy. Finally, we propose an argument-dependent approach based on normal distribution, which assigns very low weights to these “false” or “biased” opinions and can relieve the influence of the unfair arguments. A numerical example is provided to illustrate the application of the proposed approach. © 2007 Wiley Periodicals, Inc. Int J Int Syst 22: 209–221, 2007.

Conventional methods for assessing the impacts of climate change on crop production are often unable to account for the impacts of extreme weather events and therefore underestimate the impacts of climate change. Risk assessment allows for the inclusion of inconsistent impact assessment results in the risk assessment framework and thus provides a qualitative or quantitative assessment of possible risks suffered by crops amid climate change. Due to the subjective assumptions on prior distributions (e.g., emission scenarios and climate model performance) and the assumption that variables are independent of one another, conventional risk assessment methods for crop production amid climate change could produce relatively large errors. In this study, a probability function for future weather scenarios is established based on the principle of maximum entropy using future weather scenario data from the Intergovernmental Panel on Climate Change. In addition, by linking future weather scenarios and winter wheat yields, the risks of winter wheat yield reduction caused by high temperatures as well as the risks of a decrease in rainfall on the North China Plain (NCP) amid climate change are systematically investigated. The results show the following. The risks of winter wheat yield reduction caused by high temperatures will be higher in the north than in the south of the NCP in 2030, 2050 and 2080. In particular, the probabilities of winter wheat yield reduction will be relatively high in central Hebei and northwestern Shandong. In addition, the risks of a decrease in rainfall during the winter wheat season will be higher in the northern NCP than in the southern NCP in 2030, 2050 and 2080. The probabilities of a decrease in rainfall will be relatively high in northern Hebei, Beijing and Tianjin in 2030, and the risks of a decrease in rainfall are gradually increasing. In this study, the principle of maximum entropy is successfully introduced into the field of risk assessments for crop production amid climate change and used to assess the risks to winter wheat production. The method used in this study could enrich the theories and technical methods for assessing crop production risks amid climate change. The results could provide a theoretical basis for developing measures and techniques for adapting winter wheat production to climate change.

We describe the most likely forms of the probability distributions of bed load particle velocities, accelerations, hop distances, and travel times, in a manner that formally appeals to inferential statistics while honoring mechanical and kinematic constraints imposed by equilibrium transport conditions. The analysis is based on E. Jaynes's elaboration of the implications of the similarity between the Gibbs entropy in statistical mechanics and the Shannon entropy in information theory. By maximizing the information entropy of a distribution subject to known constraints on its moments, our choice of the form of the distribution is unbiased. The analysis suggests that particle velocities and travel times are exponentially distributed and that particle accelerations follow a Laplace distribution with zero mean. Particle hop distances, viewed alone, ought to be distributed exponentially. However, the covariance between hop distances and travel times precludes this result. Instead, the covariance structure suggests that hop distances follow a Weibull distribution. These distributions are consistent with high‐resolution measurements obtained from high‐speed imaging of bed load particle motions. The analysis brings us closer to choosing distributions based on our mechanical insight.

Principle of maximum entropy in hydrologic frequency analysis

Minimum Cross Entropy and Informational Approaches for Spectrum Estimation

Entropy methods enable a convenient general approach to providing a probability distribution with partial information. The minimum cross-entropy principle selects the distribution that minimizes the Kullback–Leibler divergence subject to the given constraints. This general principle encompasses a wide variety of distributions, and generalizes other methods that have been proposed independently. There remains, however, some confusion about the breadth of entropy methods in the literature. In particular, the asymmetry of the Kullback–Leibler divergence provides two important special cases when the target distribution is uniform: the maximum entropy method and the maximum log-probability method. This paper compares the performance of both methods under a variety of conditions. We also examine a generalized maximum log-probability method as a further demonstration of the generality of the entropy approach.

According to Jaynes (1963), statistical mechanics can be interpreted as a special type of statistical inference based on the principle of maximum entropy. The result of such an inference depends on the available information about a given physical system, but the principle itself does not decide what kind of information is essential and what is not. The well known Gibbs canonical state results from the principle when the statistical mean value of energy is supposed to be known. The maximum entropy inference employing higher-order moments of energy is examined from the point of view of the thermodynamic limit. It is shown that the contribution to the total entropy, arising from the extra information corresponding to the higher-order moments, is o(N), when N to infinity , N/V=constant (N is the number of particles and V is the volume). Consequently, from a purely thermodynamic point of view, this extra information is non-essential and can be neglected in the maximum entropy procedure. These conclusions also characterise the maximum entropy inference employing higher-order moments of (almost) arbitrary extensive quantities. One can say that the maximum entropy inference has a certain 'stability' property with respect to information corresponding to higher-order moments of extensive physical quantities.

Shannon quantifies the information-carrying capacity of a channel. He connects this with the probabilities of the next letter being a or b, … and shows that Gibbs’s entropy of this probability distribution is the correct measure of the channel’s capacity. Probability distributions should be chosen to maximize entropy subject to available data. A probability distribution is a subjective thing, which implies that its entropy is subjective. But for a chemist entropy is an objective thing—how is this conflict to be resolved? The connection between the rates of thermal processes and entropy is explored. The ergodic hypothesis is explained and rejected.

The maximum entropy principle for determining probabilities is discussed. Jaynes'program concerning inductive inference and the foundations of statistical mechanics is reviewed and taken as a guide for the present paper. Attention is devoted to the historical and philosophical aspects of classical statistical mechanics, such as the assumption of equal a priori probability of microstates. The alleged subjectivity of entropy in connection with information theory and irreversible processes is also discussed. Finally, the paper deals with the birth of quantum statistics with emphasis to statistical dependence and indistinguishability in the Bose-Einstein counting procedure. The conclusions suggest that the way of counting elementary cases as assumed as equiprobable is an empirical hypothesis rather than the result of a rational a priori analysis supported by principles such as that if indifference or maximum entropy.

This work introduces the concept of deterministic annealing (DA) as a useful approach to clustering and other related optimization problems. It is strongly motivated by analogies to statistical physics, but is formally derived within information theory and probability theory. This approach enables escaping local optima that plague traditional techniques, without the extremely slow schedules typically required by stochastic methods. The clustering solutions obtained by DA are totally independent of the choice of initial configuration. A probabilistic framework is constructed, which is based on the principle of maximum entropy. The association probabilities at a given average distortion are Gibbs distributions parametrized by the corresponding Lagrange multiplier $\beta$, which is inversely proportional to the temperature in the physical analogy. By computing marginal probabilities, an effective cost is obtained, which is minimized to find the most probable set of cluster representatives at a given temperature. This effective cost is the free energy in statistical mechanics, which is indeed optimized at isothermal, stochastic equilibrium. Within the probabilistic framework, annealing is introduced by controlling the Lagrange multiplier $\beta$. This annealing is interpreted as gradually reducing the fuzziness of the associations. Phase transitions are identified in the process, which are, in fact, cluster splits. A sequence of phase transitions produces a hierarchy of fuzzy-clustering solutions. Critical $\beta$ are computed exactly for the first phase transition and approximately for the following ones. Specific algorithms are derivable from the general approach, to address different aspects of clustering. From the experimental results it appears that DA is substantially superior to traditional techniques. The approach is extended to deal with a larger family of optimization problems that can be reformulated as constrained clustering. A probabilistic framework for constrained clustering is derived. Three examples are discussed. Mass-constrained clustering yields an improvement of the clustering procedure. The process is now independent of the number and multiplicities of representatives. The travelling salesman problem is reformulated as constrained clustering, yielding the elastic net approach. A second Lagrange multiplier is identified and is controlled in the process. Finally, this approach is suggested for self-organization of neural networks.

Neighborhood effect within polycrystalline materials: Elastoplastic micromechanical analysis by cellular automaton and finite element

Although the concept of thermodynamic entropy due to Clausius dates back to the early 1850s, the mathematical theory of informational entropy was not developed until the pioneering work of Shannon in 1948, the development of principle of maximum entropy (POME) and theorem of concentration by Jaynes in 1957, principle of minimum cross entropy by Kullback and Leibler in 1959, and the formulation of entropy in frequency domain by Burg in 1967. The concept of informational entropy is more intuitive, because it is a measure of information or uncertainty which is encountered in daily life. Hence, its application is ubiquitous. If we peruse hydrologic problems, it becomes clear that their solutions involve either measurement of information through data collection, or extraction of information through data analysis, or maximization or minimization of information by optimization, or prediction of information through modeling, or analysis and synthesis of information by simulation, or weighing of information for decision making. Thus, solutions of hydrologic problems may involve the direct application of entropy, examples of which are monitoring network evaluation and design, water resources allocation, and model selection. Solutions of some problems involve the application of the POME, such as derivation of frequency distributions and parameter estimation, whereas solutions of other problems may involve the POME and a flux‐concentration type relation, such as modeling of hydrologic processes. There seems hardly any area in hydrology where entropy cannot be gainfully applied. This paper discusses basic ingredients for the application of entropy theory.