Neural Dissimilarity Indices That Predict Oddball Detection in Behaviour

Reliability and Representational Bandwidth in the Auditory Cortex

The entropy rates of Markov chains (random walks) defined on connected undirected graphs are well studied in many surveys. We study the entropy rates related to the first-passage time probability distributions of fair random walks, their relative (Kullback–Leibler) entropies, and the entropy related to two biased random walks – with the random absorption of walkers and the shortest paths random walks. We show that uncertainty of first-passage times quantified by the entropy rates characterizes the connectedness of the graph. The relative entropy derived for the biased random walks estimates the level of uncertainty between connectivity and connectedness – the local and global properties of nodes in the graph.

The main issue in this work is to study the limit functions necessary for the reliability assessment of structural steel with the use of the relative entropy apparatus. This will be done using a few different mathematical theories relevant to this relative entropy, namely those proposed by Bhattacharyya, Kullback–Leibler, Jeffreys, and Hellinger. Probabilistic analysis in the presence of uncertainty in material characteristics will be delivered using three different numerical strategies—Monte Carlo simulation, the stochastic perturbation method, as well as the semi-analytical approach. All of these methods are based on the weighted least squares method approximations of the structural response functions versus the given uncertainty source, and they allow efficient determination of the first two probabilistic moments of the structural responses including stresses, displacements, and strains. The entire computational implementation will be delivered using the finite element method system ABAQUS and computer algebra program MAPLE, where relative entropies, as well as polynomial response functions, will be determined. This study demonstrates that the relative entropies may be efficiently used in reliability assessment close to the widely engaged first-order reliability method (FORM). The relative entropy concept enables us to study the probabilistic distance of any two distributions, so that structural resistance and extreme effort in elastoplastic behavior need not be restricted to Gaussian distributions.

In this article, a general goodness of fit test is developed by using a new estimate of Kullback-Leibler (KL) information based on Type-II censored data. The proposed test is consistent, and the test statistic is nonnegative, just like KL information. Then, the test statistic is used to test for exponentiality based on Type-II censored data. Through a simulation study, power values of the proposed test are compared with some prominent existing tests. A real-life data analysis is finally presented for illustrative purpose.

We prove characterization theorems for relative entropy (also known as Kullback–Leibler divergence), q-logarithmic entropy (also known as Tsallis entropy), and q-logarithmic relative entropy. All three have been characterized axiomatically before, but we show that earlier proofs can be simplified considerably, at the same time relaxing some of the hypotheses.

In this paper, we employ the Kullback-Leibler Divergence (KD) measure from information theory within the framework of Self Organizing Maps applied to network planning and dimensioning. Here, the KD measure estimates the closeness among probability densities (or relative Entropy), and is used instead of the distance or dot product measures. Applied to a real network planning under the supervision of link budget calculations, the new method shows improved performance and a promise of wider network applications.

A Quarter of a Century of Place Cells

The problems of multiple-input multiple-output (MIMO) radar adaptive waveform design in additive white Gaussian noise channels and multitarget recognition based on sequential likelihood ratio test are jointly addressed in this paper. Two information-theoretic waveform design strategies, namely, the optimal waveform for maximizing the mutual information (MI) between the extended target impulse response and the target echoes and the optimal waveform for maximizing the Kullback-Leibler (KL) divergence (or relative entropy), are applied in the multitarget recognition application. For multitarget case, two adaptive waveform design methods for all possible targets based on the current knowledge of each hypothesis are proposed. Method 1 is the probability weighted waveform method. Method 2 is the probability weighted target signature method. The optimal waveform is transmitted and adaptively changed such that a decision is made based on the likelihood ratio after several illuminations. Numerical results demonstrate that the best waveform is the KL divergence-based optimal waveform using Method 1 as it has the lowest average illumination number and the highest correct decision rate for target recognition. By optimally designing and adaptively changing the transmitted waveform, the average number of illuminations required for multitarget recognition can be much reduced.

Even if the entropy of a finite, discrete density is always positive, in the case of continuous density the entropy is not always positive. This drawback can be corrected by introducing another concept, which measures the relative entropy between two given densities. This chapter studies the Kullback–Leibler relative entropy (known also as the Kullback–Leibler divergence) between two probability densities in both discrete and continuous cases.

The impact of model risk on dynamic portfolio selection under multi-period mean-standard-deviation criterion

The purpose of this paper is twofold. On a technical side, we propose an extension of the Hausdorff distance from metric spaces to spaces equipped with asymmetric distance measures. Specifically, we focus on extending it to the family of Bregman divergences, which includes the popular Kullback–Leibler divergence (also known as relative entropy). The resulting dissimilarity measure is called a Bregman–Hausdorff divergence and compares two collections of vectors—without assuming any pairing or alignment between their elements. We propose new algorithms for computing Bregman–Hausdorff divergences based on a recently developed Kd-tree data structure for nearest neighbor search with respect to Bregman divergences. The algorithms are surprisingly efficient even for large inputs with hundreds of dimensions. As a benchmark, we use the new divergence to compare two collections of probabilistic predictions produced by different machine learning models trained using the relative entropy loss. In addition to the introduction of this technical concept, we provide a survey. It outlines the basics of Bregman geometry, and motivated the Kullback–Leibler divergence using concepts from information theory. We also describe computational geometric algorithms that have been extended to this geometry, focusing on algorithms relevant for machine learning.

This study examines an adaptive log-density estimation method with an $$\ell _1$$ -type penalty. The proposed estimator is guaranteed to be a valid density in the sense that it is positive and integrates to one. The smoothness of the estimator is controlled in a data-adaptive way via $$\ell _1$$ penalization. The advantages of the penalized log-density estimator are discussed with an emphasis on wavelet estimators. Theoretical properties of the estimator are studied when the quality of fit is measured by the Kullback–Leibler divergence (relative entropy). A nonasymptotic oracle inequality is obtained assuming a near orthogonality condition on the given dictionary. Based on the oracle inequality, selection consistency and minimax adaptivity are proved under some regularity conditions. The proposed method is implemented with a coordinate descent algorithm. Numerical illustrations based on the periodized Meyer wavelets are performed to demonstrate the finite sample performance of the proposed estimator.

In this paper, we show, discuss, and compare the effects of symmetrization in two calculation subroutines of the Fuzzy Oil Drop model, a coarse-grained model of density of hydrophobicity in proteins. In the FOD model, an input structure is enclosed in an axis-aligned ellipsoid called a drop. Two profiles of hydrophobicity are then calculated for its residues: theoretical (based on the 3D Gauss function) and observed (based on pairwise hydrophobic interactions). Condition of the hydrophobic core is revealed by comparing those profiles through relative entropy, while analysis of their local differences allows, in particular, determination of the starting location for the search for protein–protein and protein–ligand interaction areas. Here, we improve the baseline workflow of the FOD model by introducing symmetry to the hydrophobicity profile comparison and ellipsoid bounding procedures. In the first modification (FOD–JS), Kullback–Leibler divergence is enhanced with its Jensen–Shannon variant. In the second modification (FOD-PCA), the molecule is optimally aligned with the axes of the coordinate system via principal component analysis, and the size of its drop is determined by the standard deviation of all its effective atoms, making it less susceptible to structural outliers. Tests on several molecules with various shapes and functions confirm that the proposed modifications improve the accuracy, robustness, speed, and usability of Gauss function-based modeling of the density of hydrophobicity in protein structures.

The Shannon entropy and the associated Kullback—Leibler divergence measure (relative entropy) between probability measures are fundamental from the applications point of view, and arise naturally from statistical concepts. While restricted to information theory and statistics in the work of Shannon [38] and Kullback—Leibler [31] in the early 1950s, the concept of entropy started to be used in optimization modeling for various problems of engineering and management science. An early work on entropy optimization problems over linear constraints sets (equality or inequality) was studied by Chames and Cooper [16] via convex programming techniques. Many other useful applications in a diversity of problems such as traffic engineering, game theory, information theory, and marketing were developed by Chames et al. (see, e.g., [15,17-19] and the references therein). For further general information and applications, we refer the reader to Frieden [23] and Kay and Marple [30] for engineering problems and to Lev and Theil [32] and more recently to Theil and Feibig [39] for economic and finance models.

A biologically plausible model of myelin plasticity demonstrates how oligodendrocytes can selectively adjust conduction delays in axon bundles to allow robust communication between brain regions through long-range synchronization.