Information About the Moments or the Likelihood Model Parameters? A Chicken and Egg Problem

We present an extension to Jaynes’ maximum entropy principle that incorporates latent variables. The principle oflatent maximum entropywe propose is different from both Jaynes’ maximum entropy principle and maximum likelihood estimation, but can yield better estimates in the presence of hidden variables and limited training data. We first show that solving for a latent maximum entropy model poses a hard nonlinear constrained optimization problem in general. However, we then show that feasible solutions to this problem can be obtained efficiently for the special case of log-linear models---which forms the basis for an efficient approximation to the latent maximum entropy principle. We derive an algorithm that combines expectation-maximization with iterative scaling to produce feasible log-linear solutions. This algorithm can be interpreted as an alternating minimization algorithm in the information divergence, and reveals an intimate connection between the latent maximum entropy and maximum likelihood principles. To select a final model, we generate a series of feasible candidates, calculate the entropy of each, and choose the model that attains the highest entropy. Our experimental results show that estimation based on the latent maximum entropy principle generally gives better results than maximum likelihood when estimating latent variable models on small observed data samples.

The similarity between maximum entropy (MaxEnt) and minimum relative entropy (MRE) allows recent advances in probabilistic inversion to obviate some of the shortcomings in the former method. The purpose of this paper is to review and extend the theory and practice of minimum relative entropy. In this regard, we illustrate important philosophies on inversion and the similarly and differences between maximum entropy, minimum relative entropy, classical smallest model (SVD) and Bayesian solutions for inverse problems. MaxEnt is applicable when we are determining a function that can be regarded as a probability distribution. The approach can be extended to the case of the general linear problem and is interpreted as the model which fits all the constraints and is the one model which has the greatest multiplicity or “spreadout” that can be realized in the greatest number of ways. The MRE solution to the inverse problem differs from the maximum entropy viewpoint as noted above. The relative entropy formulation provides the advantage of allowing for non-positive models, a prior bias in the estimated pdf and `hard' bounds if desired. We outline how MRE can be used as a measure of resolution in linear inversion and show that MRE provides us with a method to explore the limits of model space. The Bayesian methodology readily lends itself to the problem of updating prior probabilities based on uncertain field measurements, and whose truth follows from the theorems of total and compound probabilities. In the Bayesian approach information is complete and Bayes' theorem gives a unique posterior pdf. In comparing the results of the classical, MaxEnt, MRE and Bayesian approaches we notice that the approaches produce different results. In␣comparing MaxEnt with MRE for Jayne's die problem we see excellent comparisons between the results. We compare MaxEnt, smallest model and MRE approaches for the density distribution of an equivalent spherically-symmetric earth and for the contaminant plume-source problem. Theoretical comparisons between MRE and Bayesian solutions for the case of the linear model and Gaussian priors may show different results. The Bayesian expected-value solution approaches that of MRE and that of the smallest model as the prior distribution becomes uniform, but the Bayesian maximum aposteriori (MAP) solution may not exist for an underdetermined case with a uniform prior.

A crucial decision in designing a spatial sample for soil survey is the number of sampling locations required to answer, with sufficient accuracy and precision, the questions posed by decision makers at different levels of geographic aggregation. In the Indian Soil Health Card (SHC) scheme, many thousands of locations are sampled per district. In this paper the SHC data are used to estimate the mean of a soil property within a defined study area, e.g., a district, or the areal fraction of the study area where some condition is satisfied, e.g., exceedence of a critical level. The central question is whether this large sample size is needed for this aim. The sample size required for a given maximum length of a confidence interval can be computed with formulas from classical sampling theory, using a prior estimate of the variance of the property of interest within the study area. Similarly, for the areal fraction a prior estimate of this fraction is required. In practice we are uncertain about these prior estimates, and our uncertainty is not accounted for in classical sample size determination (SSD). This deficiency can be overcome with a Bayesian approach, in which the prior estimate of the variance or areal fraction is replaced by a prior distribution. Once new data from the sample are available, this prior distribution is updated to a posterior distribution using Bayes’ rule. The apparent problem with a Bayesian approach prior to a sampling campaign is that the data are not yet available. This dilemma can be solved by computing, for a given sample size, the predictive distribution of the data, given a prior distribution on the population and design parameter. Thus we do not have a single vector with data values, but a finite or infinite set of possible data vectors. As a consequence, we have as many posterior distribution functions as we have data vectors. This leads to a probability distribution of lengths or coverages of Bayesian credible intervals, from which various criteria for SSD can be derived. Besides the fully Bayesian approach, a mixed Bayesian-likelihood approach for SSD is available. This is of interest when, after the data have been collected, we prefer to estimate the mean from these data only, using the frequentist approach, ignoring the prior distribution. The fully Bayesian and mixed Bayesian-likelihood approach are illustrated for estimating the mean of log-transformed Zn and the areal fraction with Zn-deficiency, defined as Zn concentration <0.9 mg kg −1, in the thirteen districts of Andhra Pradesh state. The SHC data from 2015–2017 are used to derive prior distributions. For all districts the Bayesian and mixed Bayesian-likelihood sample sizes are much smaller than the current sample sizes. The hyperparameters of the prior distributions have a strong effect on the sample sizes. We discuss methods to deal with this. Even at the mandal (sub-district) level the sample size can almost always be reduced substantially. Clearly SHC over-sampled, and here we show how to reduce the effort while still providing information required for decision-making. R scripts for SSD are provided as supplementary material.

At the start of new operation, a production process is commonly unstable. Then, a process condition is into being stable gradually over time. For the purpose of shifting to mass production, the stability in a process should be evaluated. This paper proposes a method of evaluating the process stability based on Bayesian statistics and information theory. In Bayesian statistics, the knowledge for a process is renewed by deriving posterior distribution based on current prior distribution and observations. Here, equivalency between prior and posterior distributions could be considered to be a criterion for the stability in a process. It is needed to define difference between prior and posterior distributions in order to evaluate the equivalency between prior and posterior distributions. We first formulate the relation between the prior distribution and the posterior distribution for process mean using Bayesian statistics. Secondly, we evaluate the difference between the both distributions based on Kullback‐Leibler (K‐L) divergence in information theory. Finally, some numerical examples in the method of evaluating the stability in process mean are illustrated. Also, we discuss about a decision rule for the stability in process mean through the numerical investigation.

Conventional methods for assessing the impacts of climate change on crop production are often unable to account for the impacts of extreme weather events and therefore underestimate the impacts of climate change. Risk assessment allows for the inclusion of inconsistent impact assessment results in the risk assessment framework and thus provides a qualitative or quantitative assessment of possible risks suffered by crops amid climate change. Due to the subjective assumptions on prior distributions (e.g., emission scenarios and climate model performance) and the assumption that variables are independent of one another, conventional risk assessment methods for crop production amid climate change could produce relatively large errors. In this study, a probability function for future weather scenarios is established based on the principle of maximum entropy using future weather scenario data from the Intergovernmental Panel on Climate Change. In addition, by linking future weather scenarios and winter wheat yields, the risks of winter wheat yield reduction caused by high temperatures as well as the risks of a decrease in rainfall on the North China Plain (NCP) amid climate change are systematically investigated. The results show the following. The risks of winter wheat yield reduction caused by high temperatures will be higher in the north than in the south of the NCP in 2030, 2050 and 2080. In particular, the probabilities of winter wheat yield reduction will be relatively high in central Hebei and northwestern Shandong. In addition, the risks of a decrease in rainfall during the winter wheat season will be higher in the northern NCP than in the southern NCP in 2030, 2050 and 2080. The probabilities of a decrease in rainfall will be relatively high in northern Hebei, Beijing and Tianjin in 2030, and the risks of a decrease in rainfall are gradually increasing. In this study, the principle of maximum entropy is successfully introduced into the field of risk assessments for crop production amid climate change and used to assess the risks to winter wheat production. The method used in this study could enrich the theories and technical methods for assessing crop production risks amid climate change. The results could provide a theoretical basis for developing measures and techniques for adapting winter wheat production to climate change.

ABSTRACTTrusted quality of Web service (QoWS) issue is critical for e‐commerce applications. However, many existing studies have little work in situations that have insufficient or no historical information regarding QoWS data. In this study, we propose a trusted QoWS metric approach, that is, Bayesian Approach with Maximum Entropy Principle. The key of our proposed approach is to extract QoWS prior distribution of Web service by using Maximum Entropy Principle and then to infer QoWS posterior distribution of Web service by using Bayesian Approach. On the basis of the obtained QoWS posterior distribution, trusted QoWS can be measured. We conduct extensive simulations to evaluate our proposed approach. The simulation results demonstrate that our proposed approach can obtain trusted QoWS effectively. Copyright © 2012 John Wiley &amp; Sons, Ltd.

The set of solutions inferred by the generic maximum entropy (MaxEnt) or maximum relative entropy (MaxREnt) principles of Jaynes - considered as a function of the moment constraints or their conjugate Lagrangian multipliers - is endowed with a Riemannian geometric description, based on the second differential tensor of the entropy or its Legendre transform (negative Massieu function). The analysis provides a generalised {\it least action bound} applicable to all Jaynesian systems, which provides a lower bound to the cost (in generic entropy units) of a transition between inferred positions along a specified path, at specified rates of change of the control parameters. The analysis therefore extends the concepts of "finite time thermodynamics" to the generic Jaynes domain, providing a link between purely static (stationary) inferred positions of a system, and dynamic transitions between these positions (as a function of time or some other coordinate). If the path is unspecified, the analysis gives an absolute lower bound for the cost of the transition, corresponding to the geodesic of the Riemannian hypersurface. The analysis is applied to (i) an equilibrium thermodynamic system subject to mean internal energy and volume constraints, and (ii) a flow system at steady state, subject to constraints on the mean heat, mass and momentum fluxes and chemical reaction rates. The first example recovers the {\it minimum entropy cost} of a transition between equilibrium positions, a widely used result of finite-time thermodynamics. The second example leads to a new {\it minimum entropy production principle}, for the cost of a transition between steady state positions of a flow system.

This work presents an application of the forward and inverse predictive modeling methodology of Cacuci and Ionescu-Bujor (2010) in the inverse mode to determine, within a tight a priori specified convergence criterion and overall accuracy, an unknown time-dependent boundary condition (specifically, the time-dependent inlet acid concentration) for a dissolver model case study by using measurements of the state function (specifically, the time-dependent acid concentration) at a specified location (specifically, in the dissolver’s compartment farthest from the inlet). The unknown time-dependent boundary condition is described by 635 unknown discrete scalar parameters. This forward and inverse predictive modeling methodology uses the maximum entropy principle to construct an optimal approximation of the unknown a priori distribution by using the a priori known mean values and uncertainties characterizing the model parameters, along with the computed and experimentally measured model responses and their covariances. This a priori distribution is subsequently combined using Bayes’ theorem with the likelihood provided by the computational model. The first-order response sensitivities serve as weighting functions in this objective combination of computational and experimental information.The use of the maximum entropy principle enables the forward and inverse predictive modeling of Cacuci and Ionescu-Bujor (2010) to construct an intrinsic regularizing metric for solving any inverse problem. In the present dissolver case study, the unknown time-dependent boundary condition is predicted by the methodology within an a priori selected convergence criterion, without user intervention and/or introduction of arbitrary regularization parameters, as the currently popular procedures need to do. This predictive modeling methodology yields optimally calibrated values for all model parameters, with reduced predicted uncertainties, as well as optimal (best-estimate) predicted values for the model responses (in this case study, the time-dependent acid concentrations in the dissolver’s compartments), also with reduced predicted uncertainties. Notably, even though the experimental data pertain solely to the compartment farthest from the inlet (where the data were measured), the application of this predictive modeling methodology actually improves the predicted values and reduces their predicted uncertainties not only in the compartment in which the data were actually measured but also throughout the entire dissolver, including the compartment farthest from the measurements (i.e., at the inlet). This is because this forward and inverse predictive modeling methodology combines and transmits information simultaneously over the entire phase-space, comprising all time steps and spatial locations.These results underscore the importance of this work in presenting the objective resolution (i.e., resolution in the absence of user-defined subjective adjustment of arbitrary regularization parameters) of a time-dependent inverse case study of potential importance to diversion activities associated with proliferation and international safeguards. The results obtained in this work establish confidence in the dissolver model’s accuracy for simulating the acid concentrations required to dissolve used nuclear fuel. In turn, these results will be used to generate source terms for key reprocessing facility components downstream and to support material accountability for nuclear safeguards.

The maximum entropy principle for determining probabilities is discussed. Jaynes'program concerning inductive inference and the foundations of statistical mechanics is reviewed and taken as a guide for the present paper. Attention is devoted to the historical and philosophical aspects of classical statistical mechanics, such as the assumption of equal a priori probability of microstates. The alleged subjectivity of entropy in connection with information theory and irreversible processes is also discussed. Finally, the paper deals with the birth of quantum statistics with emphasis to statistical dependence and indistinguishability in the Bose-Einstein counting procedure. The conclusions suggest that the way of counting elementary cases as assumed as equiprobable is an empirical hypothesis rather than the result of a rational a priori analysis supported by principles such as that if indifference or maximum entropy.

The Maximum Entropy Principle has been used to model complex chemical reaction processes. The maximum entropy principle has been employed by the Rate-Controlled Constrained-Equilibrium (RCCE) method to determine concentration of different species during non-equilibrium combustion process. In this model, it is assumed that the system evolves through constrained equilibrium states where entropy of the mixture is maximized subject to constraints. Mixture composition is determined by integrating set of differential equations of constraints rather than integration of differential equations for species as is done with detailed kinetics techniques. Since the number of constraints is much smaller than the number of species present, the number of rate equations required to describe the time evolution of the system is considerably reduced. This method has been used to model the stoichiometric mixture of the formaldehyde-oxygen combustion process. In this study 29 species and 139 reactions has been used, while keeping the energy and volume of the system constant. Calculations have been done at different sets of pressures and temperatures, ranging from 1 atm to 100 atm, and from 900 K to 1500 K respectively. Three fixed elemental constraints: conservation of elemental carbon, elemental oxygen and elemental hydrogen and from one to six variable constraints were used. The four to nine rate equations for the constraint potentials (Lagrange multipliers conjugate to the constraints) were integrated and as expected, RCCE calculations gave correct equilibrium values in all cases. Only 8 constraints were required to give very good agreement with detailed calculations. Ignition delay times and major species concentrations were within 0.5% to 5% of the values predicted by detailed chemistry calculations. Adding more constraints improved the accuracy of the mole fractions of minor species at early times, but had only a little effect on the ignition delay times. Rate-Controlled Constrained-Equilibrium calculations reduced the computation time by 50% when using eight constraints.

A Bayesian parameter estimation method applied to a marine ecosystem model for the coastal Gulf of Alaska

This article introduces two new maximum entropy (ME) methods for modeling the distribution of time to an event. One method is within the classical ME framework and provides characterizations of change point models such as the piecewise exponential distribution. The second method uses the entropy of the equilibrium distribution (ED) for the objective function and provides new characterizations of the exponential, Weibull, Pareto, and uniform distributions. With the same moment constraints, the classical ME and the maximum ED entropy algorithms generate different models for the interarrival time. © 2014 Wiley Periodicals, Inc. Naval Research Logistics 61: 427–434, 2014

Bayesian system reliability assessment method with maximum entropy as prior distribution is developed in the present analysis. The system reliability assessment model is established, and the method and process of system reliability assessment are put forward. The method is based on the theory of information entropy. The probability density function and distribution function of system reliability parameters are given by using maximum entropy method. Prior distribution of Bayesian analysis method is the distribution function. Likelihood function is given by a small amount of field test data. According to Bayesian methods, posterior distribution is determined by using prior distribution and likelihood function. The point estimation and interval estimation of system reliability parameters are calculated in order to evaluate the system reliability. The method solves better the problem of system reliability assessment for small sample cases, and is an optimum method for reliability assessment of large and complex systems.

A probabilistic multiscale model for poroelastic properties of organic-rich shale is extended to include the effect of microcracks. The approach is based on physics-based modeling of poroelastic properties and uncertainty propagation from nano- to macro-scale using homogenization schemes. The multiscale thought model used in this work improves on an existing thought model by adding another level to model embedded microcracks. From the scale of single clay particle to the scale of embedded microcracks, the model parameters are calibrated using experimental characterization at different length scales. To quantify the crack density and microcrack orientations, an inverse optimization approach is utilized which uses the UPV measurements at the macro-scale. Major model parameters representing compositional and mechanical properties at each length scale are modeled as a random variable. Probabilistic description of both scalar random variables and matrix-valued random variables is constructed using the maximum entropy principle (MaxEnt). Using such statistical description of model input parameters with Monte Carlo simulation, probabilistic descriptions of model outputs are obtained at each length scale. The results of stochastic upscaling are validated against available experimental values. Finally, a global sensitivity analysis is performed to identify the model input parameters which are most influential to the statistical fluctuation of upscaled poroelastic properties. The presented multiscale framework provides an enhanced understating regarding the effect of uncertainties associated with microcrack density and orientation, and other subscale features on homogenized poroelastic properties, thus further improves predictive capability for shale multiscale behavior.

The pioneering work of Jaynes in Bayesian/maximum entropy methods has been successfully explored in many disciplines. The principle of maximum entropy (PME) is a powerful and versatile tool of inferring a probability distribution from constraints that do not completely characterize that distribution. Minimum relative entropy (MRE) is a method which has all the important attributes of the maximum entropy approach with the advantage that prior information may be easily included. In this paper we use MRE to determine the prior probability density function (pdf) of a set of model parameters based on limited information. The resulting pdf is used in Monte Carlo simulations to provide expected values in field variables such as concentration, and confidence limits. We compare the probabilistic results from a traditional advection‐dispersion (ADE) model based on volumetric averaging concepts with that of a model based on the assumption that the hydraulic conductivity is a stationary stochastic process. The results suggest that although Naff's (1990) model satisfies the observed data to a better degree than ADE model, the upper and lower confidence bands about the mean value are larger than the ADE results. This result we attribute to the fact that Naff's (1990) model simply contains more parameters, each of which is unknown and has to be estimated. There is no statistical difference between the expected values of second‐spatial moments for the two models. The examples presented in this paper illustrate problems associated with assigning Gaussian pdfs as priors in a probabilistic model. First, such an assumption for the input parameters actually injects more information into the problem than may actually exist, whether consciously or unconsciously. This fact is borne out by comparison with minimum relative entropy theory. Second, the output parameters as suggested from the Monte Carlo analysis cannot be assumed to be Gaussian distributed even when the prior pdf is Gaussian in form. In a practical setting, the significance of this result and the approximation of Gaussian form would depend on the toxicity and environmental standards that apply to the problem.