Inequalities in Information Theory

: Calculating the amount of information about a random function contained in another random function has important uses in communication theory. An expression for the mutual information for continuous time random processes has been given by Gelfand and Yaglom, Chiang, and Perez by generalizing Shannon's result in a natural way. Under a condition of absolute continuity of measures the continuous time expression has the same form as Shannon's result. For two Gaussian processes Gelfand and Yaglom express the mutual information in terms of a mean square estimation error. We generalize this result to diffusion processes and express the solution in a different form which is more naturally related to a corresponding filtering problem. We also use these results to calculate some information rates.

This paper presents a maximum entropy framework for the aggregation of expert opinions where the expert opinions concern the prediction of the outcome of an uncertain event. The event to be predicted and individual predictions rendered are assumed to be discrete random variables. A measure of expert competence is defined using a distance metric between the actual outcome of the event and each expert's predicted outcome. Following Levy and Delic (Levy, W. B., H. Delic. 1994. Maximum entropy aggregation of individual opinions. IEEE Trans. Sys. Man & Cybernetics 24 606–613.), we use Shannon's information measure (Shannon [Shannon, C. E. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27 379–423.], Jaynes [Jaynes, E. T. 1957. Information theory and statistical mechanics. Phys. Rev. 106 Part I: 620–630, 108 Part II: 171–190.]) to derive aggregation rules for combining two or more expert predictions into a single aggregated prediction that appropriately calibrates different degrees of expert competence and reflects any dependence that may exist among the expert predictions. The resulting maximum entropy aggregated prediction is least prejudiced in the sense that it utilizes all information available but remains maximally non committal with regard to information not available. Numerical examples to illuminate the implications of maximum entropy aggregation are also presented.

Many scholars researched social informatization from different perspectives, including profession, economy and technology. In this thesis, the author will studied on social informatization from the perspective of basic characteristics of information. Firstly, the author will describe other scholars’ social informationization theories. Secondly, the definition of information and basic characteristics of information will be given. Then, the production mode informatization and life-style informatization from the perspective of basic characteristics of information will be studied on in this paper. Lastly, the conclusion will be drawn.

Information inequalities form the most important set of tools for proving converse coding theorems in information theory problems. They are sometimes referred to as the Laws of Information Theory, because they govern the impossibilities in information theory. For a long time, all information inequalities we knew were nothing but simple consequences of the nonnegativity of Shannon's information measures. Owing to the recent discovery of a few so-called non-Shannon-type information inequalities, it is now known that there are laws in information theory beyond those laid down by Shannon. In this paper, we show that the unconditional inequality discovered by the authors (1998) in fact implies a class of 2/sup 14/ non-Shannon-type inequalities, and we show possible applications of these inequalities in information theory problems.

Philosophers have used information theoretic concepts and theorems for philosophical purposes since the publication of Shannon’s seminal work, “The Mathematical Theory of Communication”. The efforts of different philosophers led to the formation of Philosophy of Information as a subfield of philosophy in the late 1990s (Floridi, in press). Although a significant part of those efforts was devoted to the mathematical formalism of information and communication theory, a thorough analysis of the fundamental mathematical properties of information-carrying relations has not yet been done. The point here is that a thorough analysis of the fundamental properties of information-carrying relations will shed light on some important controversies. The overall aim of this chapter is to begin this process of elucidation. It therefore includes a detailed examination of three semantic theories of information: Dretske’s entropy-based framework, Harms’ theory of mutual information and Cohen and Meskin’s counterfactual theory. These three theories are selected because they represent all lines of reasoning available in the literature in regard to the relevance of Shannon’s mathematical theory of information for philosophical purposes. Thus, the immediate goal is to cover the entire landscape of the literature with respect to this criterion. Moreover, this chapter offers a novel analysis of the transitivity of information-carrying relations.

Given n discrete random variables /spl Omega/={X/sub 1/,...,X/sub n/}, associated with any subset /spl alpha/ of {1,2,...,n}, there is a joint entropy H(X/sub /spl alpha//) where X/sub /spl alpha//={X/sub i/: i/spl isin//spl alpha/}. This can be viewed as a function defined on 2/sup {1,2,...,n}/ taking values in [0, +/spl infin/). We call this function the entropy function of /spl Omega/. The nonnegativity of the joint entropies implies that this function is nonnegative; the nonnegativity of the conditional joint entropies implies that this function is nondecreasing; and the nonnegativity of the conditional mutual information implies that this function is two-alternative. These properties are the so-called basic information inequalities of Shannon's information measures. An entropy function can be viewed as a 2/sup n/-1-dimensional vector where the coordinates are indexed by the subsets of the ground set {1,2,...,n}. As introduced by Yeng (see ibid., vol.43, no.6, p.1923-34, 1997) /spl Gamma//sub n/ stands for the cone in IR(2/sup n/-1) consisting of all vectors which have all these properties. Let /spl Gamma//sub n/* be the set of all 2/sup n/-1-dimensional vectors which correspond to the entropy functions of some sets of n discrete random variables. A fundamental information-theoretic problem is whether or not /spl Gamma/~/sub n/*=/spl Gamma//sub n/. Here /spl Gamma/~/sub n/* stands for the closure of the set /spl Gamma//sub n/*. We show that /spl Gamma/~/sub n/* is a convex cone, /spl Gamma//sub 2/*=/spl Gamma//sub 2/, /spl Gamma//sub 3/*/spl ne//spl Gamma//sub 3/, but /spl Gamma/~/sub 3/*=/spl Gamma//sub 3/. For four random variables, we have discovered a conditional inequality which is not implied by the basic information inequalities of the same set of random variables. This lends an evidence to the plausible conjecture that /spl Gamma/~/sub n/*/spl ne//spl Gamma//sub n/ for n>3.

60 years ago the American mathematician and electrical engineer Claude Elwood Shannon (1916-2001) published "A Mathematical Theory of Communication" and nobody could survey this theory's enormous consequences for science and technology. Shannon's article appeared in two parts in the July and October 1948 editions of the Bell System Technical Journal. However, it is very probable that this article wouldn't have become famous without the help of the American mathematician and Science administrator Warren Weaver (1894-1978), whose popular text "The Mathematics of communication" re-interpreted Shannon's work for broader scientific audiences. Weaver's "preface" and Shannon's article were published together in the book The Mathematical Theory of Communication (1949) that represents the beginning of the then so-called "Information theory".

The goal of this work is to present information theory, specifically Claude Shannon's mathematical theory of communication, in a clinical context and elucidate its potential contributions to understanding the process of diagnostic inference. We use probability theory, information theory, and clinical examples to develop information theory as a means to examine uncertainty in diagnostic testing situations. We begin our discussion with a brief review of probability theory as it relates to diagnostic testing. An outline of Shannon's theory of communication theory and how it directly translates to the medical diagnostic process serves as the essential justification for this article. Finally, we introduce the mathematical tools of information theory that allow for an understanding of diagnostic uncertainty and test effectiveness in a variety of contexts. We show that information theory provides a quantitative framework for understanding uncertainty that readily extends to medical diagnostic contexts.

Proving linear inequalities and identities of Shannon’s information measures, possibly with linear constraints on the information measures, is an important problem in information theory. For this purpose, ITIP and other variant algorithms have been developed and implemented, which are all based on solving a linear program (LP). In particular, an identity f = 0 is verified by solving two LPs, one for f ≥ 0 and one for f ≤ 0. In this paper, we develop a set of algorithms that can be implemented by symbolic computation. Based on these algorithms, procedures for verifying linear information inequalities and identities are devised. Compared with LP-based algorithms, our procedures can produce analytical proofs that are both human-verifiable and free of numerical errors. Our procedures are also more efficient computationally. For constrained inequalities, by taking advantage of the algebraic structure of the problem, the size of the LP that needs to be solved can be significantly reduced. For identities, instead of solving two LPs, the identity can be verified directly with very little computation.

The concept of “information” in association with music recently re-emerged in music psychology and cognition in the form of statistical learning and as the basis for models of music perception. The first time music and information were associated in a systematic manner was in the 1950s and 60s when, particularly in North American music theory, researchers applied concepts from information theory to analysis and composition. Among the work of the 1950s was Edgar Coons’ and David Kraehenbuehl’s Information as a Measure of Structure in Music (1958). The authors suggested that information could be developed into a useful compositional tool that would aid decisionmaking in terms of form, but would not hamper the composer’s personal style or freedom of choice with regards to musical materials. In this article, the fundamental concepts behind the development of a musical information theory are outlined, giving some simple examples of its possibilities and proposing directions for future development. The outline embraces the necessity for a contemporary musical theory to combine formal, scientific methods and an awareness of contexts and relationships inspired by the New Musicology and the study of music as a cultural activity. As a theory of communication, information theory is presented as a far more subjective and pliable concept than is implied in Claude E. Shannon’s Mathematical Theory of Communication (1948), the work upon which many of the musical information theories in the 1950s were based. A flexible view of information theory and the concept of information must be maintained to enable them to take part in contemporary musicological discourse.

In everyday usage, information is knowledge or facts acquired or derived from study, instruction or observation. Information is presumed to be both meaningful and veridical, and to have some appropriate connection to its object. Information might be misleading, but it can never be false. Standard information theory, on the other hand, as developed for communications [1], measurement [2] induction [3; 4] and computation [5; 6], entirely ignores the semantic aspects of information. Thus it might seem to have little relevance to our common notion of information. This is especially true considering the range of applications of information theory found in the literature of a variety of fields. Assuming, however, that the mind works computationally and can get information about things via physical channels, then technical accounts of information strongly restrict any plausible account of the vulgar notion. Some more recent information-oriented approaches to epistemology [7] and semantics [8] go further, though my introduction to the ideas was through Michael Arbib, Michael Scriven and Kenneth Sayre in the profoundly inventive late 60s and early 70s. In this talk I will look at how the world must be in order for us to have information about it. This will take three major sections: 1) intrinsic information -- there is a unique information in any structure that can be determined using group theory, 2) the physical world (including our minds) must have specific properties in order for us to have information about the world, and 3) the nature of information channels that can convey information to us for evaluation and testing. In the process I will outline theories of physical information and semantic information. Much of the talk will be an, I hope simplified, version of [9] and [10], and other sources on my web page, and the book, Every Thing Must Go [10]. Acknowledgments I acknowledge the support of the National Research Council of South Africa. References and Notes Shannon, C.E. and Weaver, W. 1949. The Mathematical Theory of Communication. Urbana, University of Illinois Press. Brillouin, L 1962. Science and Information Theory, 2nd edition. New York, Academic Press. Solomonoff, R. 1964. A formal theory of inductive inference, Part I.Information and Control, Vol 7, No. 1: 1-22. Solomonoff, R. 1964. A formal theory of inductive inference, Part II.Information and Control, Vol 7, No. 2: 224-254. Kolmogorov, A.N. 1965. Three approaches to the quantitative definition of information. Problems of Inform. Transmission 1: 1-7. Chaitin, G.J. A theory of program size formally identical to information theory. J. ACM 22: 329-340. Dretske, F. 1981. Knowledge and the Flow of Information. Cambridge, MA, MIT Press. Barwise, Jon and John Perry. 1983. Situations and Attitudes. Cambridge, MA, MIT Press. Collier, John 1990. Intrinsic information. in Philip Hanson (ed) Information, Language and Cognition: Vancouver Studies in Cognitive Science, Vol. 1. University of British Columbia Press, now by Oxford University Press: 390-409. Collier, John. 2012. Information, causation and computation.Information and Computation: Essays on Scientific and Philosophical Understanding of Foundations of Information and Computation. Gordana Dodig Crnkovic and Mark Burgin (eds), Singapore, World Scientific: 89-106. Ladyman, J., Ross, D., with Collier, J., Spurrett, D. 2007. Every Thing Must Go. Oxford, Oxford University Press.

Capacity is an important property for QoS support in Mobile Ad Hoc Networks (MANETs) and has recently attracted considerable attention in research. However, most approaches rely on simplified models (infinite network, isotropic radio propagation, unidirectional links, perfect scheduling, etc.) and provide only asymptotic bounds or integer linear programming equations. In this paper we take a probabilistic approach and model capacity as a random variable that depends on node distribution, communication pattern, radio propagation and channel assignment. Expected values for the random variable at a given point within the parameter space are then computed using Monte-Carlo simulation. The modularity of the approach allows for capacity analysis under more realistic network models. We demonstrate the potential of such a model by showing that for acknowledgement-based networks throughput capacity increases in the presence of shadowing, even above the critical node density.

A probabilistic graphical model approach to stochastic multiscale partial differential equations

Proving or disproving an information inequality is a crucial step in establishing the converse results in coding theorems. However, an information inequality involving more than a few random variables is difficult to be proved or disproved manually. In 1997, Yeung developed a framework that uses linear programming for verifying linear information inequalities. Under the framework, this paper considers a few other problems that can be solved by using Lagrange duality and convex approximation. We will demonstrate how linear programming can be used to find an analytic proof of an information inequality or an analytic counterexample to disprove it if the inequality is not true in general. The way to automatically find a shortest proof or a smallest counterexample is explored. When a given information inequality cannot be proved, the sufficient conditions for a counterexample to disprove the information inequality are found by linear programming. Lastly, we propose a scalable algorithmic framework based on the alternating direction method of multipliers to accelerate solving a multitude of user-specific problems whose overall computational cost can be amortized with the number of users, and present its publicly-available software implementation for large-scale problems.

The well-studied Vamos matroid has provided a wealth of interesting theoretical results in matroid theory. We use the Vamos matroid to construct a new network, which we call the Vamos network. We then exploit the Vamos network to answer in the negative the open question as to whether Shannon-type information inequalities are in general sufficient for computing network coding capacities. To accomplish this, we first determine the smallest coding capacity upper bound that can be obtained for the Vaamos network using only Shannon-type information inequalities. Then, we prove that a smaller capacity upper bound for the Vamos network can be obtained by using a non-Shannon-type information inequality discovered in 1998 by Zhang and Yeung. This is the first published application of a non-Shannon-type inequality to network coding. Finally, we demonstrate that one can compute the exact routing capacity and linear coding capacity of the Vamos network.