Minimum relative entropy and probabilistic inversion in groundwater hydrology

This work develops iterative algorithms for decoding cascade-coded images by Relative Entropy (RE) minimization. In cascade coding, blocks of an image ar first transform-coded and then the retained coefficients are transmitted by using moment-preserving Block Truncation Coding (BTC). The BTC coding introduces a quantization error in the values of the retained coefficients. Upon reception,t he distorted coefficients are used in reconstructing the image by the inverse transform, with the unretained coefficients set equal to zero. The proposed algorithms construct the original image from the distorted coefficients by minimizing the RE of the image, with the coefficients used as constraints. In addition, the error introduced by the BTC coding is used as an additional constraint, since it is known to the receiver by the nature of the BTC coding. The iterative nature of the algorithm pertains to the way the algorithm uses the constraints, i.e. one at a time, with each reconstruction used as a prior for the next RE minimization. This is the first time the RE minimization with errors in the constraints has been used in image decompression even though it is common in spectrum estimation when there are errors in the correlation measurements.

The pioneering work of Jaynes in Bayesian/maximum entropy methods has been successfully explored in many disciplines. The principle of maximum entropy (PME) is a powerful and versatile tool of inferring a probability distribution from constraints that do not completely characterize that distribution. Minimum relative entropy (MRE) is a method which has all the important attributes of the maximum entropy approach with the advantage that prior information may be easily included. In this paper we use MRE to determine the prior probability density function (pdf) of a set of model parameters based on limited information. The resulting pdf is used in Monte Carlo simulations to provide expected values in field variables such as concentration, and confidence limits. We compare the probabilistic results from a traditional advection‐dispersion (ADE) model based on volumetric averaging concepts with that of a model based on the assumption that the hydraulic conductivity is a stationary stochastic process. The results suggest that although Naff's (1990) model satisfies the observed data to a better degree than ADE model, the upper and lower confidence bands about the mean value are larger than the ADE results. This result we attribute to the fact that Naff's (1990) model simply contains more parameters, each of which is unknown and has to be estimated. There is no statistical difference between the expected values of second‐spatial moments for the two models. The examples presented in this paper illustrate problems associated with assigning Gaussian pdfs as priors in a probabilistic model. First, such an assumption for the input parameters actually injects more information into the problem than may actually exist, whether consciously or unconsciously. This fact is borne out by comparison with minimum relative entropy theory. Second, the output parameters as suggested from the Monte Carlo analysis cannot be assumed to be Gaussian distributed even when the prior pdf is Gaussian in form. In a practical setting, the significance of this result and the approximation of Gaussian form would depend on the toxicity and environmental standards that apply to the problem.

In this paper the principle of minimum relative entropy (PMRE) is proposed as a fundamental principle and idea that can be used in the field of AGI. It is shown to have a very strong mathematical foundation, that it is even more fundamental then Bayes rule or MaxEnt alone and that it can be related to neuroscience. Hierarchical structures, hierarchies in timescales and learning and generating sequences of sequences are some of the aspects that Friston (Fri09) described by using his free-energy principle. These are aspects of cognitive architectures that are in agreement with the foundations of hierarchical memory prediction frameworks (GH09). The PMRE is very similar and often equivalent to Friston's free-energy principle (Fri09), however for actions and the defi nitions of surprise there is a diff erence. It is proposed to use relative entropy as the standard definition of surprise. Experiments have shown that this is currently the best indicator of human surprise (IB09). The learning rate or interestingness can be defi ned as the rate of decrease of relative entropy, so curiosity can then be implemented as looking for situations with the highest learning rate.

A MATLAB implementation of the minimum relative entropy method for linear inverse problems

The set of solutions inferred by the generic maximum entropy (MaxEnt) or maximum relative entropy (MaxREnt) principles of Jaynes - considered as a function of the moment constraints or their conjugate Lagrangian multipliers - is endowed with a Riemannian geometric description, based on the second differential tensor of the entropy or its Legendre transform (negative Massieu function). The analysis provides a generalised {\it least action bound} applicable to all Jaynesian systems, which provides a lower bound to the cost (in generic entropy units) of a transition between inferred positions along a specified path, at specified rates of change of the control parameters. The analysis therefore extends the concepts of "finite time thermodynamics" to the generic Jaynes domain, providing a link between purely static (stationary) inferred positions of a system, and dynamic transitions between these positions (as a function of time or some other coordinate). If the path is unspecified, the analysis gives an absolute lower bound for the cost of the transition, corresponding to the geodesic of the Riemannian hypersurface. The analysis is applied to (i) an equilibrium thermodynamic system subject to mean internal energy and volume constraints, and (ii) a flow system at steady state, subject to constraints on the mean heat, mass and momentum fluxes and chemical reaction rates. The first example recovers the {\it minimum entropy cost} of a transition between equilibrium positions, a widely used result of finite-time thermodynamics. The second example leads to a new {\it minimum entropy production principle}, for the cost of a transition between steady state positions of a flow system.

Inverse methods can be used to reconstruct the release history of a known source of groundwater contamination from concentration data describing the present‐day spatial distribution of the contaminant plume. Using hypothetical release history functions and contaminant plumes, we evaluate the relative effectiveness of two proposed inverse methods, Tikhonov regularization (TR) and minimum relative entropy (MRE) inversion, in reconstructing the release history of a conservative contaminant in a one‐dimensional domain [Skaggs and Kabala, 1994; Woodbury and Ulrych, 1996]. We also address issues of reproducibility of the solution and the appropriateness of models for simulating random measurement error. The results show that if error‐free plume concentration data are available, both methods perform well in reconstructing a smooth source history function. With error‐free data the MRE method is more robust than TR in reconstructing a nonsmooth source history function; however, the TR method is more robust if the data contain measurement error. Two error models were evaluated in this study, and we found that the particular error model does not affect the reliability of the solutions. The results for the TR method have somewhat greater reproducibility because, in some cases, its input parameters are less subjective than those of the MRE method; however, the MRE solution can identify regions where the data give little or no information about the source history function, while the TR solution cannot.

We propose a practical algorithm for the calculation of the relative entropy of entanglement (REE), defined as the minimum relative entropy between a state and the set of states with positive partial transpose. Our algorithm is based on a practical semidefinite cutting plane approach. In low dimensions the implementation of the algorithm in matlab provides an estimation for the REE with an absolute error smaller than ${10}^{\ensuremath{-}3}$.

The minimal relative entropy criterion for the selection of an equivalent martingale measure in an incomplete market seems to still hold some mystique in its financial interpretation. In this paper we work toward this interpretation by suggesting and exploring the idea of relating a martingale measure selection criterion to a CAPM-like pricing scheme. We examine this idea in the case of the minimal relative entropy criterion and we present some preliminary results. We work within a one-period financial market and show that the minimal relative entropy pricing criterion is equivalent to some CAPM-like pricing scheme where the classical beta coefficient formula has been replaced by some “entropic beta” and the market portfolio by some “appropriate” reference portfolio. Furthermore, we show that if the assets involved have returns that are jointly normal, then this “entropic beta” formula coincides with the classical beta coefficient. Additionally and for comparison reasons, we briefly illustrate that if our criterion for the choice of the martingale measure was the minimization of the variance of the Radon–Nikodym derivative, then the resulting martingale pricing and the pricing implied by the classical CAPM scheme would be the same.

Coarse-grained (CG)models parametrized using atomistic referencedata, i.e., “bottom up” CG models, have proven usefulin the study of biomolecules and other soft matter. However, the constructionof highly accurate, low resolution CG models of biomolecules remainschallenging. We demonstrate in this work how virtual particles, CGsites with no atomistic correspondence, can be incorporated into CGmodels within the context of relative entropy minimization (REM) aslatent variables. The methodology presented, variational derivativerelative entropy minimization (VD-REM), enables optimization of virtualparticle interactions through a gradient descent algorithm aided bymachine learning. We apply this methodology to the challenging caseof a solvent-free CG model of a 1,2-dioleoyl-sn-glycero-3-phosphocholine(DOPC) lipid bilayer and demonstrate that introduction of virtualparticles captures solvent-mediated behavior and higher-order correlationswhich REM alone cannot capture in a more standard CG model based onlyon the mapping of collections of atoms to the CG sites.

Geometry of distributions associated with Tsallis statistics and properties of relative entropy minimization

A recursive algorithm, which appears to be new, is presented for estimating the amplitude and phase of a wave field from intensity-only measurements on two or more scan planes at different axial positions. The problem is framed as a nonlinear optimization, in which the angular spectrum of the complex field model is adjusted in order to minimize the relative entropy, or Kullback-Leibler divergence, between the measured and reconstructed intensities. The most common approach to this so-called phase retrieval problem is a variation of the well-known Gerchberg-Saxton algorithm devised by Misell (J. Phys. D6, L6, 1973), which is efficient and extremely simple to implement. The new algorithm has a computational structure that is very similar to Misell's approach, despite the fundamental difference in the optimization criteria used for each. Based upon results from noisy simulated data, the new algorithm appears to be more robust than Misell's approach and to produce better results from low signal-to-noise ratio data. The convergence of the new algorithm is examined.

In this paper, we present a maximum entropy (maxent) approach to the fusion of experts opinions, or classifiers outputs, problem. The maxent approach is quite versatile and allows us to express in a clear, rigorous, way the a priori knowledge that is available on the problem. For instance, our knowledge about the reliability of the experts and the correlations between these experts can be easily integrated: Each piece of knowledge is expressed in the form of a linear constraint. An iterative scaling algorithm is used in order to compute the maxent solution of the problem. The maximum entropy method seeks the joint probability density of a set of random variables that has maximum entropy while satisfying the constraints. It is therefore the ”most honest” characterization of our knowledge given the available facts (constraints). In the case of conflicting constraints, we propose to minimise the ”lack of constraints satisfaction” or to relax some constraints and recompute the maximum entropy solution. The maxent fusion rule is illustrated by some simulations.

ABSTRACTTrusted quality of Web service (QoWS) issue is critical for e‐commerce applications. However, many existing studies have little work in situations that have insufficient or no historical information regarding QoWS data. In this study, we propose a trusted QoWS metric approach, that is, Bayesian Approach with Maximum Entropy Principle. The key of our proposed approach is to extract QoWS prior distribution of Web service by using Maximum Entropy Principle and then to infer QoWS posterior distribution of Web service by using Bayesian Approach. On the basis of the obtained QoWS posterior distribution, trusted QoWS can be measured. We conduct extensive simulations to evaluate our proposed approach. The simulation results demonstrate that our proposed approach can obtain trusted QoWS effectively. Copyright © 2012 John Wiley & Sons, Ltd.

In digital communications systems the error probability can often be expressed as the expectation of a function of an interference random variable. To approximate such expectations a number of methods have been proposed such as series expansion methods, Gauss quadrature rules and maximum entropy and minimum relative entropy methods. The methods proposed to approximate the error probability expectations typically require knowledge of a large number of moments of the interference random variable which may not be always available. In this paper, we use the Pearson system of distributions to approximate the distribution of the interference random variable and we then use the approximate distribution to compute the error probability expectations. This system of distributions has been used in a variety of situations to model distributions whose behaviour deviates from Gaussian behaviour. Very good agreement is observed between the Pearson approximation and simulation in the intersymbol interference case. Very good agreement was also observed between the Pearson approximation and simulation in the intermodulation distortion case.

Thermodynamics can be developed from a microscopic starting point in terms of entropy and the maximum entropy principle. We investigate here to what extent one can replace entropy with relative entropy which has several advantages, for example, in the context of local quantum field theory. We find that the principle of maximum entropy can be replaced by a principle of minimum expected relative entropy. Various ensembles and their thermodynamic potentials can be defined through relative entropy. We also show that thermal fluctuations are in fact governed by a relative entropy. Furthermore, we reformulate the third law of thermodynamics using relative entropy only.