Entropy in Hydrology

Minimum Cross Entropy and Informational Approaches for Spectrum Estimation

The principle of maximum entropy and a generalization, the principle of minimum cross entropy, are prescriptions for solving problems of the following sort, which are encountered in a remarkable number of different fields. Namely, some system may be in any one of a given set of states; the probabilities of its being in the various states are not specified, but information about the probability distribution is available in the form of expectation values of given functions on the set of states. It is required to assign an optimal, or minimally prejudiced, set of state probabilities consistent with the given information. This paper describes the principles and presents APL functions that use the Newton-Raphson method to compute maximum-entropy and minimum-cross-entropy probability distributions for arbitrary sets of given expectation values. Example calculations are given.

Probability theory provides a sound and convenient machinery to be used for knowledge representation and automated reasoning (see, for instance, [Cox46, DPT90, DP91b, LS88, Pea88, TGK92]). In many cases, only relatively few relationships between relevant variables are known, due to incomplete information. Or maybe, an abstractional representation is intended, incorporating only fundamental relationships. In both cases, the knowledge explicitly stated is not sufficient to determine uniquely a probability distribution. One way to cope with this indetermination is to calculate upper and lower bounds for probabilities (cf. [Nil86, TGK92, DPT90]). This method, however, brings about two problems: Sometimes the inferred bounds are quite bad, and one has to handle intervals instead of single values.An alternative way that provides best expectation values for the unknown probabilities and guarantees a logically sound reasoning is to use the principle of maximum entropy resp. the principle of minimum cross entropy to represent all available probabilistic knowledge by a unique distribution (see Section 2.5; cf. [Sho86, Kul68, Jay83a, GHK94]). Here we assume the available knowledge to constitute of a (consistent) set R of conditionals, each equipped with a probability, usually providing only incomplete probabilistic knowledge.The aim of this chapter is to establish a direct and constructive link between probabilistic conditionals and their suitable representation via distributions, taking prior knowledge into account if necessary.We develope the following four principles which mark the corner-stones for using quantified conditionals consistently for probabilistic knowledge representation and updating: (P1) The principle of conditional preservation: this is to express that prior conditional dependencies shall be preserved “as far as possible” under adaptation; (P2) the idea of a functional concept which underlies the adaptation and which allows us to calculate a posterior distribution from prior and new knowledge (P3) the principle of logical coherence 1: posterior distributions shall be used coherently as priors for further inferences (P4) the principle of representation invariance: the resulting distribution shall not depend upon the actual probabilistic representation of the new information. KeywordsWeight FactorPosterior DistributionPrior DistributionRepresentation InvariancePositive DistributionThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We present an extension to Jaynes’ maximum entropy principle that incorporates latent variables. The principle oflatent maximum entropywe propose is different from both Jaynes’ maximum entropy principle and maximum likelihood estimation, but can yield better estimates in the presence of hidden variables and limited training data. We first show that solving for a latent maximum entropy model poses a hard nonlinear constrained optimization problem in general. However, we then show that feasible solutions to this problem can be obtained efficiently for the special case of log-linear models---which forms the basis for an efficient approximation to the latent maximum entropy principle. We derive an algorithm that combines expectation-maximization with iterative scaling to produce feasible log-linear solutions. This algorithm can be interpreted as an alternating minimization algorithm in the information divergence, and reveals an intimate connection between the latent maximum entropy and maximum likelihood principles. To select a final model, we generate a series of feasible candidates, calculate the entropy of each, and choose the model that attains the highest entropy. Our experimental results show that estimation based on the latent maximum entropy principle generally gives better results than maximum likelihood when estimating latent variable models on small observed data samples.

Detailed evaluation of a proposed engineering change (EC) or its effects is a time-consuming process requiring considerable user experience and expertise. Therefore, enterprises plan detailed evaluation of only those EC effects that might have a significant impact. Since similar ECs are likely to have similar effects and impacts, past EC knowledge can be utilized for determining whether the proposed EC effect has significant impact. This paper presents an approach for predicting the impact of proposed EC effect based on past ECs that are similar to it. Our approach accounts for the differences in context of impact between attribute values in two changes. The Bayes’ rule is utilized to determine differences in impact value based on the differences in attribute values. The probability values required in Bayes’ rule are determined based on the principle of minimum cross entropy. An example EC knowledge base is created and utilized to evaluate our approach against two state-of-the-art approaches, namely k-nearest neighbor (NN) and regularized local similarity discriminant analysis (SDA). The success rate in predicting impact is used as an evaluation metric. The results show that there is a statistically significant improvement in success rate obtained using our approach as compared to those obtained using the two state-of-the-art approaches. The results also show that for a very large number of proposed ECs, i.e., N > 100, the success rate in predicting impact using our approach shall be greater than that obtained using the two state-of-the-art approaches.

The set of solutions inferred by the generic maximum entropy (MaxEnt) or maximum relative entropy (MaxREnt) principles of Jaynes - considered as a function of the moment constraints or their conjugate Lagrangian multipliers - is endowed with a Riemannian geometric description, based on the second differential tensor of the entropy or its Legendre transform (negative Massieu function). The analysis provides a generalised {\it least action bound} applicable to all Jaynesian systems, which provides a lower bound to the cost (in generic entropy units) of a transition between inferred positions along a specified path, at specified rates of change of the control parameters. The analysis therefore extends the concepts of "finite time thermodynamics" to the generic Jaynes domain, providing a link between purely static (stationary) inferred positions of a system, and dynamic transitions between these positions (as a function of time or some other coordinate). If the path is unspecified, the analysis gives an absolute lower bound for the cost of the transition, corresponding to the geodesic of the Riemannian hypersurface. The analysis is applied to (i) an equilibrium thermodynamic system subject to mean internal energy and volume constraints, and (ii) a flow system at steady state, subject to constraints on the mean heat, mass and momentum fluxes and chemical reaction rates. The first example recovers the {\it minimum entropy cost} of a transition between equilibrium positions, a widely used result of finite-time thermodynamics. The second example leads to a new {\it minimum entropy production principle}, for the cost of a transition between steady state positions of a flow system.

Diverse forms of nonlinearity within stochastic equations give rise to varying dynamics in processes, which may influence the behavior of extreme values. This study focuses on two nonlinear models of the discrete Langevin equation: one with a fixed diffusion function (M1) and the other with a fixed marginal distribution (M2), both characterized by a nonlinearity parameter. Extremes are defined according to the run theory with thresholds based on percentiles. The behavior of inter-extreme times and run lengths is examined by employing Fisher's Information Measure and the Shannon Entropy. Our findings reveal a clear relationship between the entropic and informational measures and the nonlinearity of model M1-these measures decrease as the nonlinearity parameter increases. Similar relationships are evident for the M2 model, albeit to a lesser extent, even though the background data's marginal distribution remains unaffected by this parameter. As thresholds increase, both the values of Fisher's Information Measure and the Shannon Entropy also increase.

In the continuing quest for operational definitions of information, this paper analyzes samples of data and information measurements used in biological systems at the genetic, cellular, and organ levels. The transduction of energy into information in biological systems yields information in the form of genetic code words (condons), bit sequences, analog codes, and paired chemical markers. Pragmatic information measures applicable to many artificial systems (robots, energy information systems, realtime systems) can be expressed in terms of international system (SI) metric units or the economic productivity units of time or money. Natural and artificial systems have in common identifiable sets of subsystems which are dedicated to processing information: input transducers, internal transducers, channels and nets, decoders, associators, memory, deciders, encoders, and output transducers. Transducers convert energy into information, or conversely. Intelligent systems can substitute informationintensive behavior for energy-intensive behavior, or for capital- or time-intensive behavior. Ratios of energy used by systems to process information tend to be very small fractions of the total energy used by these systems. However, controversy exists between proponents of the information-communications paradigm and those who regard information as a highly refined form of system control energy. Fresh approaches to information measurement are needed. New generations of real-time systems should provide opportunities to measure relationships between real-time information flows and energy flows in object systems or processes.

Entropy and the principle of maximum entropy are being increasingly applied to a wide range of problems in hydrology and water resources. This paper reviews some of the hydrologic applications of entropy, and comments on the entropy-based approaches. Entropy is a powerful model-building tool, and its potential in explaining the great many hydrologic processes remains largely untapped.KeywordsWater ResourceMaximum EntropySediment YieldWater Resource ResearchConditional EntropyThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The fuzzy entropy is used to express the mathematical values of the fuzziness of fuzzy sets and is defined by using the concept of membership function. The maximum entropy principle attempts to choose the membership function with a finite number of the fuzzy values subject to constraints generated by given moment vector functions that have maximum entropy value. On the other hand, the minimum cross-entropy principle tells us that out of all membership functions satisfying given moment constraints, select the one that is closest to the given a priori membership function. This study is connected with new Generalized Minimum Cross Fuzzy Entropy Methods (GMinx(F)EntM) in the form of MinMinx(F)Ent and MaxMinx(F)Ent methods. The aim of this study consists of applying GMinx(F)EntM on given simulated fuzzy data. The performances of distributions (MinMinx(F)Ent)m and (MaxMinx(F)Ent)m are compared by Chi-Square, Root Mean Square Error, coefficient of determination criteria and fuzzy cross entropy measure. The obtained results show that Generalized Minimum Cross Fuzzy Entropy Optimization distributions give significant results in the data modeling for fuzzy data analysis.KeywordsMinimum cross fuzzy entropy principleFuzzy entropy measureSimulation of fuzzy random variables

In the field of Natural Language Processing, one of the very important research areas of Information Extraction (IE) comes in Named Entity Recognition (NER). NER is a subtask of IE that seeks to identify and classify the predefined categories of named entities in text documents. Considerable amount of work has been done on NER in recent years due to the increasing demand of automated texts and the wide availability of electronic corpora. While it is relatively easy and natural for a human reader to read and understand the context of a given article, getting a machine to understand and differentiate between words is a big challenge. For instance, the word ‘brown’ may refer to a person called Mr. Brown, or the colour of an item which is brown. Human readers can easily discern the meaning of the word by looking at the context of that particular sentence, but it would be almost impossible for a computer to interpret it without any additional information. To deal with the issue, researchers in NER field have proposed various rule-based systems (Wakao, Gaizauskas & Wilks, 1996; Krupka & Hausman, 1998; Maynard, Tablan, Ursu, Cunningham & Wilks, 2001). These systems are able to achieve high accuracy in recognition with the help of some lists of known named entities called gazetteers. The problem with rule-based approach is that it lacks the robustness and portability. It incurs steep maintenance cost especially when new rules need to be introduced for some new information or new domains. A better option is thus to use machine learning approach that is trainable and adaptable. Three wellknown machine learning approaches that have been used extensively in NER are Hidden Markov Model (HMM), Maximum Entropy Model (MEM) and Decision Tree. Many of the existing machine learning-based NER systems (Bikel, Schwartz & Weischedel, 1999; Zhou & Su, 2002; Borthwick, Sterling, Agichten & Grisham, 1998; Bender, Och & Ney, 2003; Chieu & Ng, 2002; Sekine, Grisham & Shinnou, 1998) are able to achieve near-human performance for named entity tagging, even though the overall performance is still about 2% short from the rule-based systems. There have also been many attempts to improve the performance of NER using a hybrid approach with the combination of handcrafted rules and statistical models (Mikheev, Moens & Grover, 1999; Srihari & Li, 2000; Seon, Ko, Kim & Seo, 2001). These systems can achieve relatively good performance in the targeted domains owing to the comprehensive handcrafted rules. Nevertheless, the portability problem still remains unsolved when it comes to dealing with NER in various domains. As such, this article presents a hybrid machine learning approach using MEM and HMM successively. The reason for using two statistical models in succession instead of one is due to the distinctive nature of the two models. HMM is able to achieve better performance than any other statistical models, and is generally regarded as the most successful one in machine learning approach. However, it suffers from sparseness problem, which means considerable amount of data is needed for it to achieve acceptable performance. On the other hand, MEM is able to maintain reasonable performance even when there is little data available for training purpose. The idea is therefore to walkthrough the testing corpus using MEM first in order to generate a temporary tagging result, while this procedure can be simultaneously used as a training process for HMM. During the second walkthrough, the corpus uses HMM for the final tagging. In this process, the temporary tagging result generated by MEM will be used as a reference for subsequent error checking and correction. In the case when there is little training data available, the final result can still be reliable based on the contribution of the initial MEM tagging result.

Conventional methods for assessing the impacts of climate change on crop production are often unable to account for the impacts of extreme weather events and therefore underestimate the impacts of climate change. Risk assessment allows for the inclusion of inconsistent impact assessment results in the risk assessment framework and thus provides a qualitative or quantitative assessment of possible risks suffered by crops amid climate change. Due to the subjective assumptions on prior distributions (e.g., emission scenarios and climate model performance) and the assumption that variables are independent of one another, conventional risk assessment methods for crop production amid climate change could produce relatively large errors. In this study, a probability function for future weather scenarios is established based on the principle of maximum entropy using future weather scenario data from the Intergovernmental Panel on Climate Change. In addition, by linking future weather scenarios and winter wheat yields, the risks of winter wheat yield reduction caused by high temperatures as well as the risks of a decrease in rainfall on the North China Plain (NCP) amid climate change are systematically investigated. The results show the following. The risks of winter wheat yield reduction caused by high temperatures will be higher in the north than in the south of the NCP in 2030, 2050 and 2080. In particular, the probabilities of winter wheat yield reduction will be relatively high in central Hebei and northwestern Shandong. In addition, the risks of a decrease in rainfall during the winter wheat season will be higher in the northern NCP than in the southern NCP in 2030, 2050 and 2080. The probabilities of a decrease in rainfall will be relatively high in northern Hebei, Beijing and Tianjin in 2030, and the risks of a decrease in rainfall are gradually increasing. In this study, the principle of maximum entropy is successfully introduced into the field of risk assessments for crop production amid climate change and used to assess the risks to winter wheat production. The method used in this study could enrich the theories and technical methods for assessing crop production risks amid climate change. The results could provide a theoretical basis for developing measures and techniques for adapting winter wheat production to climate change.

Principle of maximum entropy in hydrologic frequency analysis

BackgroundUsing information measures to infer biological regulatory networks can capture nonlinear relationships between variables. However, it is computationally challenging, and there is a lack of convenient tools.ResultsWe introduce Informeasure, an R package designed to quantify nonlinear dependencies in biological regulatory networks from an information theory perspective. This package compiles a comprehensive set of information measurements, including mutual information, conditional mutual information, interaction information, partial information decomposition, and part mutual information. Mutual information is used for bivariate network inference, while the other four estimators are dedicated to trivariate network analysis.ConclusionsInformeasure is a turnkey solution, allowing users to utilize these information measures immediately upon installation. Informeasure is available as an R/Bioconductor package at https://bioconductor.org/packages/Informeasure.

After a brief expository account of the Shannon-Jaynes principle of maximum entropy (POME) for discrete and continuous variables, we give here an account of some recent research work which (i) a “histogram” method to contrast the discrete and continuous modes computation and the role of histogram in actual practice when dealing with continuous probability distributions. (ii) The idea of mean logarithmic decrement associated with a probability distribution is introduced and is shown to be related to the concept of differential entropy. The mean with respect to an arbitrary probability distribution of the logarithm of the ratio of the probability density function for an exponential distribution is discussed in the context of hydrological investigations. Unlike the entropy of a continuous probability distribution introduced in (i), this quantity which is an example of the Kullback-Leibler (KL) Information, is always positive and invariant under coordinate transformation. (iii) The constraints entering into POME as well as a minimum K L information are identified as a class of sufficient statistics which determine the unknown parameters in the probability density functions that occur in the most commonly used hydrological models. (iv) An example of (iii) where only the first two moments in a semi-infinite domain are givens is discussed to shed light on the limitations of POME, recentky recognized by Wragg and coworkers, and is made relevant to the work of Sonuga on rainfall-run off relationship. Finally, (v) a method of generating probability distributions starting from one basic distribution employing coordinate transformations is given. This in conjuction with (iii) leads to the notions of “Physical constraints” in contrast to the “mathematical constraints” in examing parameter estimation.