Minimum relative entropy: Forward probabilistic modeling

The similarity between maximum entropy (MaxEnt) and minimum relative entropy (MRE) allows recent advances in probabilistic inversion to obviate some of the shortcomings in the former method. The purpose of this paper is to review and extend the theory and practice of minimum relative entropy. In this regard, we illustrate important philosophies on inversion and the similarly and differences between maximum entropy, minimum relative entropy, classical smallest model (SVD) and Bayesian solutions for inverse problems. MaxEnt is applicable when we are determining a function that can be regarded as a probability distribution. The approach can be extended to the case of the general linear problem and is interpreted as the model which fits all the constraints and is the one model which has the greatest multiplicity or “spreadout” that can be realized in the greatest number of ways. The MRE solution to the inverse problem differs from the maximum entropy viewpoint as noted above. The relative entropy formulation provides the advantage of allowing for non-positive models, a prior bias in the estimated pdf and `hard' bounds if desired. We outline how MRE can be used as a measure of resolution in linear inversion and show that MRE provides us with a method to explore the limits of model space. The Bayesian methodology readily lends itself to the problem of updating prior probabilities based on uncertain field measurements, and whose truth follows from the theorems of total and compound probabilities. In the Bayesian approach information is complete and Bayes' theorem gives a unique posterior pdf. In comparing the results of the classical, MaxEnt, MRE and Bayesian approaches we notice that the approaches produce different results. In␣comparing MaxEnt with MRE for Jayne's die problem we see excellent comparisons between the results. We compare MaxEnt, smallest model and MRE approaches for the density distribution of an equivalent spherically-symmetric earth and for the contaminant plume-source problem. Theoretical comparisons between MRE and Bayesian solutions for the case of the linear model and Gaussian priors may show different results. The Bayesian expected-value solution approaches that of MRE and that of the smallest model as the prior distribution becomes uniform, but the Bayesian maximum aposteriori (MAP) solution may not exist for an underdetermined case with a uniform prior.

The modeling of complex service systems entails capturing many sub-components of the system, and the dependencies that exist among them in the form of a joint probability distribution. Two common methods for constructing joint probability distributions from experts using partial information include maximum entropy methods and copula methods. In this paper we explore the performance of these methods in capturing the dependence between random variables using correlation coefficients and lower-order pairwise assessments. We focus on the case of discrete random variables, and compare the performance of these methods using a Monte Carlo simulation when the variables exhibit both independence and non-linear dependence structures. We show that the maximum entropy method with correlation coefficients and the Gaussian copula method perform similarly, while the maximum entropy method with pairwise assessments performs better particularly when the variables exhibit non-linear dependence.

Entropy methods enable a convenient general approach to providing a probability distribution with partial information. The minimum cross-entropy principle selects the distribution that minimizes the Kullback–Leibler divergence subject to the given constraints. This general principle encompasses a wide variety of distributions, and generalizes other methods that have been proposed independently. There remains, however, some confusion about the breadth of entropy methods in the literature. In particular, the asymmetry of the Kullback–Leibler divergence provides two important special cases when the target distribution is uniform: the maximum entropy method and the maximum log-probability method. This paper compares the performance of both methods under a variety of conditions. We also examine a generalized maximum log-probability method as a further demonstration of the generality of the entropy approach.

A MATLAB implementation of the minimum relative entropy method for linear inverse problems

A Bayesian approach was initiated to estimate the crack size probability density function, pdf corresponding to the traditional conservative engineering end of service life limit, known as the 100% fatigue life expended (FLE) limit. The initial application of the Bayesian approach was based on fleet findings for a particular location on 25 inspected aircraft. The FLE values corresponding to the particular airframe location exceeded 100%. First a prior crack size probability density function, pdf, at FLE of 100% was estimated in line with underlying FLE presumptions and whatever information could be found in the literature. Second, for all the inspected aircraft at a FLE greater than 100%, the load histories were taken from the respective onboard structural data recording system (SDRS) datasets. The prior crack size pdf corresponding to the FLE values of each inspection was then simulated by a Monte Carlo technique starting with the prior pdf at 100% FLE. Since loads for the aircraft were known the only simulation uncertainty was in the material properties for crack growth, assuming no model uncertainty. Next the simulated prior crack size pdf at the inspected FLE and the corresponding observed cracks were combined via Bayesian updating to compute the posterior crack size pdf at the inspected FLE. The posterior crack size pdf at the inspected FLE was used to back-calculate the corresponding posterior crack size pdf that needed to be at the FLE of 100%. Simple (equal weights) averaging combined the resulting back-calculate the corresponding posterior crack size pdf functions at FLE of 100%. The computations resulted in an updated posterior crack size pdf at FLE equaling 100% representing the inventory population. In turn, the posterior pdf at FLE equaling 100% serves as the prior pdf for the next set of inspection results. Likewise, the PDF at FLE equaling 100% is used to project the current state of damage of any aircraft at some FLE greater than 100% that has not undergone inspection. In other words, the posterior pdf at FLE equal to 100% is used along with the known (measured) loads to estimate the current crack size pdf for an aircraft at FLE greater than 100%. Subsequently, the computed current crack size pdf is the basis for projection of the aircraft's crack size pdf corresponding to future usage

We propose a probabilistic model for estimating word confidence by fusing predictor features. Starting from the maximum entropy (ME) method, we first prove that ME model is equivalent to the best model with certain form to the minimum expected cross entropy (MECE) criterion. Under the MECE criterion, We extend the form of ME model by introducing a hidden state. We call the new model hidden-state maximum entropy (HSME) model. In a keyword-spotting task, we combine predictor features from both phonetic and word-level systems. Compared to lattice posterior alone, recall at 80% precision is improved from 38.1% to 49.5% on voicemail and from 37.1% to 51.9% on Switchboard. Compared with other fusion methods, HSME consistently outperforms decision tree, and most cases SVM.

This article introduces two new maximum entropy (ME) methods for modeling the distribution of time to an event. One method is within the classical ME framework and provides characterizations of change point models such as the piecewise exponential distribution. The second method uses the entropy of the equilibrium distribution (ED) for the objective function and provides new characterizations of the exponential, Weibull, Pareto, and uniform distributions. With the same moment constraints, the classical ME and the maximum ED entropy algorithms generate different models for the interarrival time. © 2014 Wiley Periodicals, Inc. Naval Research Logistics 61: 427–434, 2014

Two Chebyshev recursion methods are presented for calculations with very large sparse Hamiltonians, the kernel polynomial method (KPM) and the maximum entropy method (MEM). They are applicable to physical properties involving large numbers of eigenstates such as densities of states, spectral functions, thermodynamics, total energies for Monte Carlo simulations and forces for tight binding molecular dynamics. this paper emphasizes efficient algorithms.

Summary It is common practice to fit mathematical models to radionuclide activity–depth profiles in soils in order to quantify rates of vertical migration through the soil profile. We have fitted six such models to 21 different activity–depth profiles of radiocaesium ( 137 Cs) derived from Chernobyl and determined relations between the models and the values of their parameters. The advection and dispersion parameters obtained using four solutions to the advection–dispersion equation (each based on different initial and boundary conditions or different simplifications) are in good agreement. We further develop a relation between parameter values obtained using the advection–dispersion models and those determined by a simpler exponential function of the form A exp(– Bt ) where t is the time and A and B are parameters to be estimated. One of the advection–dispersion models proved to be significantly better than the others in terms of goodness‐of‐fit, versatility and ease of use. A simple model, using calculations based on measured characteristics of the activity–depth profile, was shown to accord well with parameters derived from more complex models based on statistical curve fitting. We have also evaluated the ‘residence time’ or ‘compartmental’ model approach to characterizing radionuclide activity–depth profiles. We relate such models to a numerical solution of a simple advection equation, and we show that apparent dispersion in compartmental models is an artefact of numerical dispersion, which can be quantified by the Courant condition. For activity profiles that have a significant advection component, using solutions to the advection–dispersion equation, we have observed a strong positive correlation between advection and dispersion in the profile.

Nonlocal and localized analyses of nonreactive solute transport in bounded randomly heterogeneous porous media: Computational analysis

Background and Aim:African swine fever (ASF) is an infectious disease and a major viral pig disease that threatens pork production in several locations globally. The mortality rate of ASF in domestic pigs is very high, causing a decrease in pig populations and significant economic losses for farmers. Environmental or ecological risk factors are the most important associated with the spread of the ASF virus. Environmental (or ecological) niche models are commonly used to estimate the probability of an event using the maximum entropy (Maxent) method. This study aimed to estimate the probability risk of future ASF outbreaks in North Sumatra, Indonesia.Materials and Methods:Secondary data from the National Animal Health System Database (iSIKHNAS), including data on the ASF outbreaks of 2019–2020 in North Sumatra, Indonesia, were used in this study. The first analysis performed involved the identification of environmental risk factors using multiple regression analysis. The second analysis performed was the estimation of probability risk for future ASF outbreaks in North Sumatra, Indonesia, using the Maxent method. Data processing was performed using Microsoft Excel, ArcGIS version 10.5 software (ESRI, California, United States), Maxent version 3.4.4 software, and Rstudio (http://www.r-project.org/).Results:The Maxent method was found to be highly accurate with a statistically significant area under the curve value of 0.860. The greatest contributing environmental factor identified by the model was the harbor, which contributed 57%. The range of high probability risk of future ASF outbreaks was found to be 0.723–0.84.Conclusion:The estimation of the highest probability risk of future ASF outbreaks in North Sumatra, Indonesia, was 0.723–0.84. The most contributing environmental factor identified using the Maxent method was harbors, at 57%. This methodology can be used to carry out subsequent ASF analyses and contribute to developing prevention and control strategies in this area.

In the context of Bayesian inversion for scientific and engineering modeling, Markov chain Monte Carlo (MCMC) sampling strategies have become the benchmark due to their flexibility and robustness in dealing with arbitrary posterior probability density functions (PDFs). However, these algorithms have been shown to be inefficient when sampling from high-dimensional posterior distributions or exhibit multimodality and/or strong parameter correlations. In such contexts, transitional MCMC (TMCMC) provides a more efficient alternative. Despite the recent applicability for Bayesian updating and model selection across a variety of disciplines, TMCMC may require a prohibitive number of tempering stages when the prior pdf is significantly different from the target posterior. Furthermore, the need to start with an initial set of samples from the prior distribution may present a challenge when dealing with implicit priors, e.g., based on feasible regions. Finally, TMCMC cannot be used for inverse problems with improper prior PDFs that represent a lack of prior knowledge on all or a subset of parameters. A generalization of TMCMC is proposed that alleviates such challenges and limitations toward providing a more robust and efficient tempering sampling strategy. We present convergence analysis, proving that the distance between the intermediate distributions and the target posterior distribution monotonically decreases as the algorithm proceeds. We also demonstrate the advantages of the proposed generalization through a series of test problems and an engineering application in the oil and gas industry.

We discuss how the method of maximum entropy, MaxEnt, can be extended beyond its original scope, as a rule to assign a probability distribution, to a full‐fledged method for inductive inference. The main concept is the (relative) entropy S[p|q] which is designed as a tool to update from a prior probability distribution q to a posterior probability distribution p when new information in the form of a constraint becomes available. The extended method goes beyond the mere selection of a single posterior p, but also addresses the question of how much less probable other distributions might be. Our approach clarifies how the entropy S[p|q] is used while avoiding the question of its meaning. Ultimately, entropy is a tool for induction which needs no interpretation. Finally, being a tool for generalization from special examples, we ask whether the functional form of the entropy depends on the choice of the examples and we find that it does. The conclusion is that there is no single general theory of inductive inference and that alternative expressions for the entropy are possible.

Bayesian system reliability assessment method with maximum entropy as prior distribution is developed in the present analysis. The system reliability assessment model is established, and the method and process of system reliability assessment are put forward. The method is based on the theory of information entropy. The probability density function and distribution function of system reliability parameters are given by using maximum entropy method. Prior distribution of Bayesian analysis method is the distribution function. Likelihood function is given by a small amount of field test data. According to Bayesian methods, posterior distribution is determined by using prior distribution and likelihood function. The point estimation and interval estimation of system reliability parameters are calculated in order to evaluate the system reliability. The method solves better the problem of system reliability assessment for small sample cases, and is an optimum method for reliability assessment of large and complex systems.

Conventional methods for assessing the impacts of climate change on crop production are often unable to account for the impacts of extreme weather events and therefore underestimate the impacts of climate change. Risk assessment allows for the inclusion of inconsistent impact assessment results in the risk assessment framework and thus provides a qualitative or quantitative assessment of possible risks suffered by crops amid climate change. Due to the subjective assumptions on prior distributions (e.g., emission scenarios and climate model performance) and the assumption that variables are independent of one another, conventional risk assessment methods for crop production amid climate change could produce relatively large errors. In this study, a probability function for future weather scenarios is established based on the principle of maximum entropy using future weather scenario data from the Intergovernmental Panel on Climate Change. In addition, by linking future weather scenarios and winter wheat yields, the risks of winter wheat yield reduction caused by high temperatures as well as the risks of a decrease in rainfall on the North China Plain (NCP) amid climate change are systematically investigated. The results show the following. The risks of winter wheat yield reduction caused by high temperatures will be higher in the north than in the south of the NCP in 2030, 2050 and 2080. In particular, the probabilities of winter wheat yield reduction will be relatively high in central Hebei and northwestern Shandong. In addition, the risks of a decrease in rainfall during the winter wheat season will be higher in the northern NCP than in the southern NCP in 2030, 2050 and 2080. The probabilities of a decrease in rainfall will be relatively high in northern Hebei, Beijing and Tianjin in 2030, and the risks of a decrease in rainfall are gradually increasing. In this study, the principle of maximum entropy is successfully introduced into the field of risk assessments for crop production amid climate change and used to assess the risks to winter wheat production. The method used in this study could enrich the theories and technical methods for assessing crop production risks amid climate change. The results could provide a theoretical basis for developing measures and techniques for adapting winter wheat production to climate change.