Relative Entropy and Inductive Inference

The similarity between maximum entropy (MaxEnt) and minimum relative entropy (MRE) allows recent advances in probabilistic inversion to obviate some of the shortcomings in the former method. The purpose of this paper is to review and extend the theory and practice of minimum relative entropy. In this regard, we illustrate important philosophies on inversion and the similarly and differences between maximum entropy, minimum relative entropy, classical smallest model (SVD) and Bayesian solutions for inverse problems. MaxEnt is applicable when we are determining a function that can be regarded as a probability distribution. The approach can be extended to the case of the general linear problem and is interpreted as the model which fits all the constraints and is the one model which has the greatest multiplicity or “spreadout” that can be realized in the greatest number of ways. The MRE solution to the inverse problem differs from the maximum entropy viewpoint as noted above. The relative entropy formulation provides the advantage of allowing for non-positive models, a prior bias in the estimated pdf and `hard' bounds if desired. We outline how MRE can be used as a measure of resolution in linear inversion and show that MRE provides us with a method to explore the limits of model space. The Bayesian methodology readily lends itself to the problem of updating prior probabilities based on uncertain field measurements, and whose truth follows from the theorems of total and compound probabilities. In the Bayesian approach information is complete and Bayes' theorem gives a unique posterior pdf. In comparing the results of the classical, MaxEnt, MRE and Bayesian approaches we notice that the approaches produce different results. In␣comparing MaxEnt with MRE for Jayne's die problem we see excellent comparisons between the results. We compare MaxEnt, smallest model and MRE approaches for the density distribution of an equivalent spherically-symmetric earth and for the contaminant plume-source problem. Theoretical comparisons between MRE and Bayesian solutions for the case of the linear model and Gaussian priors may show different results. The Bayesian expected-value solution approaches that of MRE and that of the smallest model as the prior distribution becomes uniform, but the Bayesian maximum aposteriori (MAP) solution may not exist for an underdetermined case with a uniform prior.

Bayesian system reliability assessment method with maximum entropy as prior distribution is developed in the present analysis. The system reliability assessment model is established, and the method and process of system reliability assessment are put forward. The method is based on the theory of information entropy. The probability density function and distribution function of system reliability parameters are given by using maximum entropy method. Prior distribution of Bayesian analysis method is the distribution function. Likelihood function is given by a small amount of field test data. According to Bayesian methods, posterior distribution is determined by using prior distribution and likelihood function. The point estimation and interval estimation of system reliability parameters are calculated in order to evaluate the system reliability. The method solves better the problem of system reliability assessment for small sample cases, and is an optimum method for reliability assessment of large and complex systems.

When analyzing the behavior pattern of the object that is composed of a set of various samples, it is an efficient way to analyze its pattern based on the features' probability distribution among a set of samples. Contrast to the traditional ways that focus on each sample's feature patterns and then build a model to differentiate this one from another one, the probability distribution based pattern recognition can conclude a set of samples' features by analyzing its sample probability distribution model and then differentiate this set of samples from another one. By this way, not only can we save a lot of time and resource, but also it is more representative to display the group features of a set of samples. This paper makes use of this special advantage of the probability distribution based pattern recognition to detect the network anomalies by analyzing the different guideline such as Probability Distribution, Relative Entropy and Normalized Relative Entropy. In addition, this paper also analyzes the efficiency among the different algorithm implementations, CPU based serial algorithm, GPU based parallel algorithm and GPU based MapReduce parallel algorithms.

Thermodynamics can be developed from a microscopic starting point in terms of entropy and the maximum entropy principle. We investigate here to what extent one can replace entropy with relative entropy which has several advantages, for example, in the context of local quantum field theory. We find that the principle of maximum entropy can be replaced by a principle of minimum expected relative entropy. Various ensembles and their thermodynamic potentials can be defined through relative entropy. We also show that thermal fluctuations are in fact governed by a relative entropy. Furthermore, we reformulate the third law of thermodynamics using relative entropy only.

The main idea of this work is an application of relative entropy in the numerical analysis of probabilistic divergence between original material tensors of the composite constituents and its effective tensor in the presence of material uncertainties. The homogenization method is based upon the deformation energy of the representative volume elements for the fiber‐reinforced and particulate composites and uncertainty propagation begins with elastic moduli of the fibers, particles, and composite matrices. Relative entropy follows a mathematical model originating from Bhattacharyya probabilistic divergence and has been applied here for Gaussian distributions. The semi‐analytical probabilistic method based on analytical integration of polynomial bases obtained via the least squares method fittings enables for determination of the basic probabilistic characteristics of the effective tensor and the relative entropies. The methodology invented in this work may be extended toward other probability distributions and relative entropies, for homogenization of nonlinear composites and also accounting for some structural interface defects.

After a brief expository account of the Shannon-Jaynes principle of maximum entropy (POME) for discrete and continuous variables, we give here an account of some recent research work which (i) a “histogram” method to contrast the discrete and continuous modes computation and the role of histogram in actual practice when dealing with continuous probability distributions. (ii) The idea of mean logarithmic decrement associated with a probability distribution is introduced and is shown to be related to the concept of differential entropy. The mean with respect to an arbitrary probability distribution of the logarithm of the ratio of the probability density function for an exponential distribution is discussed in the context of hydrological investigations. Unlike the entropy of a continuous probability distribution introduced in (i), this quantity which is an example of the Kullback-Leibler (KL) Information, is always positive and invariant under coordinate transformation. (iii) The constraints entering into POME as well as a minimum K L information are identified as a class of sufficient statistics which determine the unknown parameters in the probability density functions that occur in the most commonly used hydrological models. (iv) An example of (iii) where only the first two moments in a semi-infinite domain are givens is discussed to shed light on the limitations of POME, recentky recognized by Wragg and coworkers, and is made relevant to the work of Sonuga on rainfall-run off relationship. Finally, (v) a method of generating probability distributions starting from one basic distribution employing coordinate transformations is given. This in conjuction with (iii) leads to the notions of “Physical constraints” in contrast to the “mathematical constraints” in examing parameter estimation.

Entropy methods enable a convenient general approach to providing a probability distribution with partial information. The minimum cross-entropy principle selects the distribution that minimizes the Kullback–Leibler divergence subject to the given constraints. This general principle encompasses a wide variety of distributions, and generalizes other methods that have been proposed independently. There remains, however, some confusion about the breadth of entropy methods in the literature. In particular, the asymmetry of the Kullback–Leibler divergence provides two important special cases when the target distribution is uniform: the maximum entropy method and the maximum log-probability method. This paper compares the performance of both methods under a variety of conditions. We also examine a generalized maximum log-probability method as a further demonstration of the generality of the entropy approach.

The maximum entropy principle for determining probabilities is discussed. Jaynes'program concerning inductive inference and the foundations of statistical mechanics is reviewed and taken as a guide for the present paper. Attention is devoted to the historical and philosophical aspects of classical statistical mechanics, such as the assumption of equal a priori probability of microstates. The alleged subjectivity of entropy in connection with information theory and irreversible processes is also discussed. Finally, the paper deals with the birth of quantum statistics with emphasis to statistical dependence and indistinguishability in the Bose-Einstein counting procedure. The conclusions suggest that the way of counting elementary cases as assumed as equiprobable is an empirical hypothesis rather than the result of a rational a priori analysis supported by principles such as that if indifference or maximum entropy.

The pioneering work of Jaynes in Bayesian/maximum entropy methods has been successfully explored in many disciplines. The principle of maximum entropy (PME) is a powerful and versatile tool of inferring a probability distribution from constraints that do not completely characterize that distribution. Minimum relative entropy (MRE) is a method which has all the important attributes of the maximum entropy approach with the advantage that prior information may be easily included. In this paper we use MRE to determine the prior probability density function (pdf) of a set of model parameters based on limited information. The resulting pdf is used in Monte Carlo simulations to provide expected values in field variables such as concentration, and confidence limits. We compare the probabilistic results from a traditional advection‐dispersion (ADE) model based on volumetric averaging concepts with that of a model based on the assumption that the hydraulic conductivity is a stationary stochastic process. The results suggest that although Naff's (1990) model satisfies the observed data to a better degree than ADE model, the upper and lower confidence bands about the mean value are larger than the ADE results. This result we attribute to the fact that Naff's (1990) model simply contains more parameters, each of which is unknown and has to be estimated. There is no statistical difference between the expected values of second‐spatial moments for the two models. The examples presented in this paper illustrate problems associated with assigning Gaussian pdfs as priors in a probabilistic model. First, such an assumption for the input parameters actually injects more information into the problem than may actually exist, whether consciously or unconsciously. This fact is borne out by comparison with minimum relative entropy theory. Second, the output parameters as suggested from the Monte Carlo analysis cannot be assumed to be Gaussian distributed even when the prior pdf is Gaussian in form. In a practical setting, the significance of this result and the approximation of Gaussian form would depend on the toxicity and environmental standards that apply to the problem.

Previously1 we introduced the use of the maximum entropy principle (MEP) for image reconstruction of moment-coded images. The MEP and the minimum relative principle (MREP) are used in reconstructing images previously compressed by retaining a subset of the coefficients of their discrete cosine transform (DCT), a popular method of image compression since it approximates closely the ideal Karhunen-Loeve transform compression. The normal way to reconstruct such compressed images is by using the inverse DCT. The reconstructed image under the MEP is the one that maximizes the entropy of the image subject to constraints reflecting the retained DCT coefficients. Under the MREP, the reconstructed image is obtained by successive minimization of its relative entropy subject to one constraint at a time, with each solution serving as a prior for the next minimization. Both methods were applied to images compressed by DCT, and the results were compared to normal inverse DCT reconstruction. It is concluded that MEP and MREP are better than IDCT, and the improvement is attributed to the fact that these methods make no assumptions about the value of the unretained coefficients, while IDCT assumes them equal to zero.

Inference of Hydraulically Fractured Reservoir Properties from Production Data Using the Indicator-Based Probability Perturbation Assisted History-Matching Method

With the processing of data streams, come inevitable challenges, such as changes in the prior (class drift) and posterior (concept drift) probability distribution over the processing time. Both these phenomena have a negative impact on the quality of the classification. Heavily imbalanced problems, which are often typical for real-world applications, bring additional processing difficulties. Classifiers are often biased towards the majority class and have difficulty identifying instances of categories described with a lower number of objects. The following article proposes a Prior Probability Assisted Classifier (2PAC), a method aiming to improve the classification quality of heavily imbalanced data streams with dynamic changes by using the estimated prior probability value and the correction of the classifier's decision for batch predictions. Presented extensive computer experiments, supported by statistical analysis, show the ability to improve the classification quality using the proposed method.

We present an approach to detect stylistic variation across social variables (here: gender and social class), considering also diachronic change in language use. For detection of stylistic variation, we use relative entropy, measuring the difference between probability distributions at different linguistic levels (here: lexis and grammar). In addition, by relative entropy, we can determine which linguistic units are related to stylistic variation.

The modeling of complex service systems entails capturing many sub-components of the system, and the dependencies that exist among them in the form of a joint probability distribution. Two common methods for constructing joint probability distributions from experts using partial information include maximum entropy methods and copula methods. In this paper we explore the performance of these methods in capturing the dependence between random variables using correlation coefficients and lower-order pairwise assessments. We focus on the case of discrete random variables, and compare the performance of these methods using a Monte Carlo simulation when the variables exhibit both independence and non-linear dependence structures. We show that the maximum entropy method with correlation coefficients and the Gaussian copula method perform similarly, while the maximum entropy method with pairwise assessments performs better particularly when the variables exhibit non-linear dependence.

پدیده یخبندان دیررس رخدادی فرین و زیانبخش جوی بوده که در شاخه های مختلف مهندسی مانند کشاورزی، باغبانی و غیره بررسی می شود. تحلیل احتمالی این رخداد و به کارگیری نتایج آن می تواند آسیب های وارده به بخش کشاورزی، باغبانی و غیره را کاهش دهد. موضوع این تحقیق تحلیل آخرین روز یخبندان کلیه ایستگاه های همدید خراسان رضوی است. هشت توزیع احتمالی نرمال، گامبل نوع 1، گامای2 پارامتری، لوگ نرمال 2 و 3 پارامتری، پارتوی تعمیم یافته، حدی تعمیم یافته و پیرسون نوع 3 بر داده ها برازش و پارامترها با هفت روش (5 روش گشتاوری، حداکثر درست نمائی و آنتروپی) برآورد شد. بهترین تابع برازشی برای هر ایستگاه با آزمون K-S انتخاب و آخرین روز یخبندان در دوره های بازگشت مختلف با توزیع برتر پیش بینی شد. بهترین تابع برازشی در ایستگاه ها متفاوت و عمدتا شامل پارتوی تعمیم یافته، پیرسون نوع 3 و لوگ نرمال 3 پارامتری است. بیشترین کاربرد روش برآورد پارامترها مربوط به انواع گشتاورها است. بنابراین نمی توان یک تابع برازشی با یک روش خاص برآورد پارامترها را در بررسی این پدیده پیشنهاد داد. همچنین بررسی روابط رگرسیونی بین دوره های بازگشت مختلف آخرین روز یخبندان و شاخص های دمائی سالانه هر ایستگاه نشان داد که می توان آخرین روز یخبندان را براساس متوسط دمای کمینه هر ایستگاه با دقت مناسب برآورد کرد.