Information Divergence Geometry and the Application to Statistical Machine Learning

A statistical transformation model consists of a smooth data manifold, on which a Lie group smoothly acts, together with a family of probability density functions on the data manifold parametrized by elements in the Lie group. For such a statistical transformation model, the Fisher–Rao semi-definite metric and the Amari–Chentsov cubic tensor are defined in the Lie group. If the family of probability density functions is invariant with respect to the Lie group action, the Fisher–Rao semi-definite metric and the Amari–Chentsov tensor are left-invariant, and hence we have a left-invariant structure of a statistical manifold. In the present work, the general framework of statistical transformation models is explained. Then, the left-invariant geodesic flow associated with the Fisher–Rao metric is considered for two specific families of probability density functions on the Lie group. The corresponding Euler–Poincaré and the Lie–Poisson equations are explicitly found in view of geometric mechanics. Related dynamical systems over Lie groups are also mentioned. A generalization in relation to the invariance of the family of probability density functions is further studied.

This paper looks at graphical models and discusses the quality of tree approximations by formulating the problem as a detection problem. One of the widely used algorithms for tree-structured approximation and modeling is the Chow-Liu algorithm. While this algorithm is optimal for Gaussian distributions in the sense of the Kullback-Leibler (KL) divergence, it is not optimal when compared with other information divergences and criteria such as Area Under the Curve (AUC). In this paper, we discuss the quality of tree approximation using the AUC. We define the correlation approximation matrix (CAM) and show that the KL divergence and AUC depend on the eigenvalues of the CAM. Examples show that the quality of tree approximations is in general not good for both information divergences and AUC when the number of nodes is large.

This tutorial is concerned with applications of information theory concepts in statistics, in the finite alphabet setting. The information measure known as information divergence or Kullback-Leibler distance or relative entropy plays a key role, often with a geometric flavor as an analogue of squared Euclidean distance, as in the concepts of I-projection, I-radius and I-centroid. The topics covered include large deviations, hypothesis testing, maximum likelihood estimation in exponential families, analysis of contingency tables, and iterative algorithms with an “information geometry” background. Also, an introduction is provided to the theory of universal coding, and to statistical inference via the minimum description length principle motivated by that theory.

Many real processes have stochastic features which seem to be representable in some intuitive sense as `close to Poisson’, `nearly random’, `nearly uniform’ or with binary variables `nearly independent’. Each of those particular reference states, defined by an equation, is unstable in the formal sense, but it is passed through or hovered about by the observed process. Information geometry gives precise meaning for nearness and neighbourhood in a state space of processes, naturally quantifying proximity of a process to a particular state via an information theoretic metric structure on smoothly parametrized families of probability density functions. We illustrate some aspects of the methodology through case studies: inhomogeneous statistical evolutionary rate processes for epidemics, amino acid spacings along protein chains, constrained disordering of crystals, distinguishing nearby signal distributions and testing pseudorandom number generators.

Independent component analysis based on marginal density estimation using weighted Parzen windows

Formulation of the Tree Approximation Problem as a Detection Problem and Relation between the AUC and Information Theory Divergences

Label distribution learning (LDL) is the state-of-the-art approach to dealing with a number of real-world applications, such as chronological age estimation from a face image, where there is an inherent similarity among adjacent age labels. LDL takes into account the semantic similarity by assigning a label distribution to each instance. The well-known Kullback-Leibler (KL) divergence is the widely used loss function for the LDL framework. However, the KL divergence does not fully and effectively capture the semantic similarity among age labels, thus leading to suboptimal performance. In this article, we propose a novel loss function based on the optimal transport theory for the LDL-based age estimation. A ground metric function plays an important role in the optimal transport formulation. It should be carefully determined based on the underlying geometric structure of the label space of the application in-hand. The label space in the age estimation problem has a specific geometric structure, that is, closer ages have more inherent semantic relationships. Inspired by this, we devise a novel ground metric function, which enables the loss function to increase the influence of highly correlated ages; thus exploiting the semantic similarity among ages more effectively than the existing loss functions. We then use the proposed loss function, namely, γ -Wasserstein loss, for training a deep neural network (DNN). This leads to a notoriously computationally expensive and nonconvex optimization problem. Following the standard methodology, we formulate the optimization function as a convex problem and then use an efficient iterative algorithm to update the parameters of the DNN. Extensive experiments in age estimation on different benchmark datasets validate the effectiveness of the proposed method, which consistently outperforms state-of-the-art approaches.

A mathematical framework called "information geometry" investigates the geometrical attributes and features of probability distributions and statistical models. In a variety of domains, including as machine learning, optimisation, and inference, it offers a potent toolkit for analysing and optimising complicated systems. Information geometry and its uses in optimisation and inference are examined in this work.First, we give a general overview of information geometry's foundational ideas and principles, including topics like the Fisher information metre, divergence measures, and exponential families. We go over how to quantify the geometric links between probability distributions and derive practical geometric structures using these ideas.The use of information geometry in optimisation issues is what we investigate next. We show how the Fisher information metric can direct effective search strategies and convergence analysis in optimisation algorithms by utilising its geometric characteristics. We go over the benefits of applying information geometry to a variety of optimisation tasks, including parameter estimation, model choice, and neural network training. We also look into how information geometry affects statistical inference. We emphasise how the development of effective and reliable inference algorithms is made possible by the geometric structures of exponential families. We go over the use of divergence measures to quantify the differences between distributions, making tasks like model comparison and hypothesis testing easier.We also review current developments in information geometry, especially its application to probabilistic programming and deep learning. We go over how information geometry can improve deep neural networks' capacity for generalisation, interpretation, and uncertainty estimation.In this study, information geometry and its uses in optimisation and inference are thoroughly studied. Information geometry provides useful insights and methods for resolving challenging issues in a variety of fields by taking advantage of the geometric aspects of probability distributions.

We adapt supervised statistical machine learning methods to regularize noisy unsupervised feature vectors. Theorems on two graph-based denoising approaches taken from numerical analysis and harmonic spectral methods are discussed. A feature vector x = (x1, …, x p ) = {x q } q = 1 p is viewed as a function f(q) on its index set. This function can be regularized or smoothed using a graph/metric structure on the index set. This smoothing can involve a penalty functional on feature vectors analogous to those in statistical learning. Our regularization of feature vectors is independent of their role in subsequent supervised learning tasks. An application is given to cancer prediction/classification in computational biology.

The strength/intensity of the stimulus in the random dot motion task (RDMT) [1] is determined by the percentage of dots in the kinematogram moving towards a saccadic target, a. Due to the uncertainty in the stimuli, neurons in sensory systems have evolved to transform environmental information, comprising evidence upon which a decision can be made (e.g. saccading to a). The neurons in the middle-temporal area (MT) appear to produce such evidence during the RDMT, given their tuning to a 'preferred' direction of visual motion. If the dots move predominantly in the preferred direction of an MT neuron, it generates inter-spike intervals (ISI) supporting a saccade to a. These ISIs seem randomly sampled from a distribution, fa, with mean, μa. Otherwise, the ISIs follow another distribution, fb, with mean, μb, where μb is larger than μa and this difference increases with stimulus strength. The accuracy vs motion-strength function of an ideal observer provided with empirical distributions like fa and fb, from a single MT neuron, approximates the subject's psychometric function (at the behavioral level) [1]. The distributions fa and fb are non-negative, positively skewed and have a mode larger than 0 (figure ​(figure1A),1A), as is typical for neural events recorded in many brain areas. Here we investigate why this is advantageous for decision-making. As theoretical decision-making units, we produced 5 new instantiations of the multi-hypothesis sequential probability ratio test (MSPRT) [2]. Each unit assumes its stream of input evidence to follow 1 of 5 probability density functions (PDF) whose compatibility with the empirical distribution of ISIs varies (figure ​(figure1A).1A). These include the Inverse Gaussian, Lognormal, Gamma, Inverse-Gamma and Exponential PDFs (the latter is the distribution of the inter-event times in the oft-used Poisson process). Under equal and appropriate conditions, we then compared their mean decision sample with that of an MSPRT instantiation that assumes Gaussian inputs proposed in [3] and discussed in general in [4], as exemplified in figure ​figure1B.1B. The mean decision sample is the mean number of observations required by a unit to identify which of N parallel information sources supports saccading to a, with a given accuracy. This decision sample is a model of the 'neural decision time'; the psychophysical reaction time also includes sensory and motor delays. The pattern of our results is explicable using a measure of the discrimination information between fa and fb, i.e. the Kullback-Leibler divergence (KLD). We found that, the mean decision sample decreases with increasing fa to fb KLD and, crucially, this follows a power law (figure ​(figure1C).1C). At the behavioral level, Pieron [5] reported the mean reaction time to the presentation of a stimulus (go/no-go decision-making) being shorter for more intense stimuli, and that a power law relates these measures. The universality of Pieron's law indicates that it can inform us of something fundamental about sensorimotor decision-making. Our results suggest that its explanation could lie in the power law relationship between the mean neural decision time and the discrimination information (KLD) among the distributions of sensory evidence. Figure 1 (A) PDFs of interest fitted to ISIs (grey bars) recorded in [1]from the MT during the RDMT. (B) mean decision sample for each MSPRT realisation (accuracy 95% over 1000 trials). (C) the values in (B) vs the fa to fb KLD. The dashed line is a fitted power ...

This paper investigates and discusses the use of information divergence, through the widely used Kullback–Leibler (KL) divergence, under the multivariate (generalized) $$\gamma $$-order normal distribution ($$\gamma $$-GND). The behavior of the KL divergence, as far as its symmetricity is concerned, is studied by calculating the divergence of $$\gamma $$-GND over the Student’s multivariate t-distribution and vice versa. Certain special cases are also given and discussed. Furthermore, three symmetrized forms of the KL divergence, i.e., the Jeffreys distance, the geometric-KL as well as the harmonic-KL distances, are computed between two members of the $$\gamma $$-GND family, while the corresponding differences between those information distances are also discussed.

High viscosity of quench oil is a critical problem of quench system in ethylene cracking furnaces in petrochemical plant, due to its influences on the safety and stability of equipments, meanwhile, the variety of viscosity of quench oil has a negative impact on yield of ethylene and other chemical products. This paper presents a new statistical learning model to forecast the real-time variety of quench oil viscosity based on statistical algorithm and machine learning method. Firstly, statistical algorithm is applied to reduce dimension of parameters, secondly, fitting real-time predictive model through machine learning method. The simulation results shows that this model can monitor the variety of viscosity per hour according to identified controllable parameters which are highly correlated with viscosity of quench oil.

Pythagoras theorem in information geometry and applications to generalized linear models

The goal of independent component analysis (ICA) is to decompose observed signals into components as independent as possible. In linear instantaneous blind source separation, ICA is used to separate linear instantaneous mixtures of source signals into signals that are as close as possible to the original signals. In the estimation of the so-called demixing matrix one has to distinguish two different factors: 1. Variance of the estimated inverse mixing matrix in the noiseless case due to randomness of the sources. 2. Bias of the demixing matrix from the inverse mixing matrix: This chapter studies both factors for circular and noncircular complex mixtures. It is important to note that the complex case is not directly equivalent to the real case of twice larger dimension. In the derivations, we aim to clearly show the connections and differences between the complex and real cases. In the first part of the chapter, we derive a closed-form expression for the CRB of the demixing matrix for instantaneous noncircular complex mixtures. We also study the CRB numerically for the family of noncircular complex generalized Gaussian distributions (GGD) and compare it to simulation results of several ICA estimators. In the second part, we consider a linear noisy noncircular complex mixing model and derive an analytic expression for the demixing matrix of ICA based on the Kullback-Leibler divergence (KLD). We show that for a wide range of both the shape parameter and the noncircularity index of the GGD, the signal-to-interference-plus-noise ratio (SINR) of KLD-based ICA is close to that of linear MMSE estimation. Furthermore, we show how to extend our derivations to the overdetermined case (\(M>N\)) with circular complex noise.

In this paper, an optimal control scheme for reducing the fluctuation of residual gas fraction (RGF) under variational operating condition is developed by combining stochastic logical system approach with statistical learning method. The method estimating RGF from measured in-cylinder pressure is introduced firstly. Then, the stochastic properties of the RGF are analyzed according to statistical data captured by conducting experiments on a test bench equipped with a L4 internal combustion engine. The influences to the probability distribution of the RGF from both control input and environment parameters are also analyzed. Based on the statistical analysis, a stochastic logical transient model is adopted for describing cyclic behavior of the RGF. Optimal control policy maps for different fixed operating conditions are calculated then. Besides, a statistical learning-based method is applied to learn the probability density function (PDF) of RGF in the real-time which is used to adjust the control MAP based on logical optimization. The whole optimal control policy map is obtained based on Gaussian process regression with consideration of statistical information of RGF. Finally, the performance of the proposed method is experimentally validated.