Relative Entropy Application to Study the Elastoplastic Behavior of S235JR Structural Steel

The entropy rates of Markov chains (random walks) defined on connected undirected graphs are well studied in many surveys. We study the entropy rates related to the first-passage time probability distributions of fair random walks, their relative (Kullback–Leibler) entropies, and the entropy related to two biased random walks – with the random absorption of walkers and the shortest paths random walks. We show that uncertainty of first-passage times quantified by the entropy rates characterizes the connectedness of the graph. The relative entropy derived for the biased random walks estimates the level of uncertainty between connectivity and connectedness – the local and global properties of nodes in the graph.

The main aim of this paper was to propose a new reliability index for steel structure assessment and to check it using the example of a popular Pratt truss girder. Structural analysis was completed in the finite element method system Autodesk ROBOT, and probabilistic analysis was implemented in the computer algebra software MAPLE. The stochastic finite element method (SFEM) was contrasted here with the Monte Carlo simulation and the girder span was selected as the input structural uncertainty source. Both methods were based on the same structural polynomial response functions determined for extreme deformation, for extreme stresses and also for the structural joint exhibiting the largest effort. These polynomials were statistically optimized during the additional least squares method experiments. The first four basic probabilistic characteristics of the structural responses, the first-order reliability method (FORM) index, and as the new proposition for this index were computed and presented. This new index formula follows the relative probabilistic entropy model proposed by Bhattacharyya. The computer analysis results presented here show a very strong coincidence of both probabilistic numerical techniques and confirms the applicability of the new reliability index for the input coefficient of variation not larger than 0.15. These studies should be continued for other engineering systems’ reliability and, particularly, for large-scale and multiscale computer simulations. The results presented in this paper may serve in different applied sciences, from biology through to econometrics, experimental physics and, of course, various branches of engineering.

Neuroscientists have recently shown that images that are difficult to find in visual search elicit similar patterns of firing across a population of recorded neurons. The $L^{1}$ distance between firing rate vectors associated with two images was strongly correlated with the inverse of decision time in behavior. But why should decision times be correlated with $L^{1}$ distance? What is the decision-theoretic basis? In our decision theoretic formulation, we model visual search as an active sequential hypothesis testing problem with switching costs. Our analysis suggests an appropriate neuronal dissimilarity index, which correlates equally strongly with the inverse of decision time as the $L^{1}$ distance. We also consider a number of other possibilities, such as the relative entropy (Kullback–Leibler divergence) and the Chernoff entropy of the firing rate distributions. A more stringent test of equality of means, which would have provided a strong backing for our modeling, fails for our proposed as well as the other already discussed dissimilarity indices. However, test statistics from the equality of means test, when used to rank the indices in terms of their ability to explain the observed results, places our proposed dissimilarity index at the top followed by relative entropy, Chernoff entropy, and the $L^{1}$ indices. Computations of the different indices require an estimate of the relative entropy between two Poisson point processes. An estimator is developed and is shown to have near unbiased performance for almost all operating regions.

We prove characterization theorems for relative entropy (also known as Kullback–Leibler divergence), q-logarithmic entropy (also known as Tsallis entropy), and q-logarithmic relative entropy. All three have been characterized axiomatically before, but we show that earlier proofs can be simplified considerably, at the same time relaxing some of the hypotheses.

In this paper, we employ the Kullback-Leibler Divergence (KD) measure from information theory within the framework of Self Organizing Maps applied to network planning and dimensioning. Here, the KD measure estimates the closeness among probability densities (or relative Entropy), and is used instead of the distance or dot product measures. Applied to a real network planning under the supervision of link budget calculations, the new method shows improved performance and a promise of wider network applications.

The problems of multiple-input multiple-output (MIMO) radar adaptive waveform design in additive white Gaussian noise channels and multitarget recognition based on sequential likelihood ratio test are jointly addressed in this paper. Two information-theoretic waveform design strategies, namely, the optimal waveform for maximizing the mutual information (MI) between the extended target impulse response and the target echoes and the optimal waveform for maximizing the Kullback-Leibler (KL) divergence (or relative entropy), are applied in the multitarget recognition application. For multitarget case, two adaptive waveform design methods for all possible targets based on the current knowledge of each hypothesis are proposed. Method 1 is the probability weighted waveform method. Method 2 is the probability weighted target signature method. The optimal waveform is transmitted and adaptively changed such that a decision is made based on the likelihood ratio after several illuminations. Numerical results demonstrate that the best waveform is the KL divergence-based optimal waveform using Method 1 as it has the lowest average illumination number and the highest correct decision rate for target recognition. By optimally designing and adaptively changing the transmitted waveform, the average number of illuminations required for multitarget recognition can be much reduced.

Even if the entropy of a finite, discrete density is always positive, in the case of continuous density the entropy is not always positive. This drawback can be corrected by introducing another concept, which measures the relative entropy between two given densities. This chapter studies the Kullback–Leibler relative entropy (known also as the Kullback–Leibler divergence) between two probability densities in both discrete and continuous cases.

The impact of model risk on dynamic portfolio selection under multi-period mean-standard-deviation criterion

The purpose of this paper is twofold. On a technical side, we propose an extension of the Hausdorff distance from metric spaces to spaces equipped with asymmetric distance measures. Specifically, we focus on extending it to the family of Bregman divergences, which includes the popular Kullback–Leibler divergence (also known as relative entropy). The resulting dissimilarity measure is called a Bregman–Hausdorff divergence and compares two collections of vectors—without assuming any pairing or alignment between their elements. We propose new algorithms for computing Bregman–Hausdorff divergences based on a recently developed Kd-tree data structure for nearest neighbor search with respect to Bregman divergences. The algorithms are surprisingly efficient even for large inputs with hundreds of dimensions. As a benchmark, we use the new divergence to compare two collections of probabilistic predictions produced by different machine learning models trained using the relative entropy loss. In addition to the introduction of this technical concept, we provide a survey. It outlines the basics of Bregman geometry, and motivated the Kullback–Leibler divergence using concepts from information theory. We also describe computational geometric algorithms that have been extended to this geometry, focusing on algorithms relevant for machine learning.

This study examines an adaptive log-density estimation method with an $$\ell _1$$ -type penalty. The proposed estimator is guaranteed to be a valid density in the sense that it is positive and integrates to one. The smoothness of the estimator is controlled in a data-adaptive way via $$\ell _1$$ penalization. The advantages of the penalized log-density estimator are discussed with an emphasis on wavelet estimators. Theoretical properties of the estimator are studied when the quality of fit is measured by the Kullback–Leibler divergence (relative entropy). A nonasymptotic oracle inequality is obtained assuming a near orthogonality condition on the given dictionary. Based on the oracle inequality, selection consistency and minimax adaptivity are proved under some regularity conditions. The proposed method is implemented with a coordinate descent algorithm. Numerical illustrations based on the periodized Meyer wavelets are performed to demonstrate the finite sample performance of the proposed estimator.

In this paper, we show, discuss, and compare the effects of symmetrization in two calculation subroutines of the Fuzzy Oil Drop model, a coarse-grained model of density of hydrophobicity in proteins. In the FOD model, an input structure is enclosed in an axis-aligned ellipsoid called a drop. Two profiles of hydrophobicity are then calculated for its residues: theoretical (based on the 3D Gauss function) and observed (based on pairwise hydrophobic interactions). Condition of the hydrophobic core is revealed by comparing those profiles through relative entropy, while analysis of their local differences allows, in particular, determination of the starting location for the search for protein–protein and protein–ligand interaction areas. Here, we improve the baseline workflow of the FOD model by introducing symmetry to the hydrophobicity profile comparison and ellipsoid bounding procedures. In the first modification (FOD–JS), Kullback–Leibler divergence is enhanced with its Jensen–Shannon variant. In the second modification (FOD-PCA), the molecule is optimally aligned with the axes of the coordinate system via principal component analysis, and the size of its drop is determined by the standard deviation of all its effective atoms, making it less susceptible to structural outliers. Tests on several molecules with various shapes and functions confirm that the proposed modifications improve the accuracy, robustness, speed, and usability of Gauss function-based modeling of the density of hydrophobicity in protein structures.

Sensitivity analysis of vertically loaded pile reliability

The Shannon entropy and the associated Kullback—Leibler divergence measure (relative entropy) between probability measures are fundamental from the applications point of view, and arise naturally from statistical concepts. While restricted to information theory and statistics in the work of Shannon [38] and Kullback—Leibler [31] in the early 1950s, the concept of entropy started to be used in optimization modeling for various problems of engineering and management science. An early work on entropy optimization problems over linear constraints sets (equality or inequality) was studied by Chames and Cooper [16] via convex programming techniques. Many other useful applications in a diversity of problems such as traffic engineering, game theory, information theory, and marketing were developed by Chames et al. (see, e.g., [15,17-19] and the references therein). For further general information and applications, we refer the reader to Frieden [23] and Kay and Marple [30] for engineering problems and to Lev and Theil [32] and more recently to Theil and Feibig [39] for economic and finance models.

Mapping probability of liquefaction using geostatistics and first order reliability method based on CPTU measurements

Attempts at generalizing Shannon entropy and Kullback-Leibler divergence (relative entropy) led to a plenthora of deformation models in theoretical physics, including q-model, κ-model, etc. Naudts and Zhang (Inf. Geom., 1 (2018) 79) established that these models can be unified under two notions: deformed ϕ-exponential family (Naudts, J., J. Inequal. Pure Appl. Math., 5 (2004) 102) and conjugate -embedding (Zhang J., Neural Comput., 16 (2004) 159) of probability functions. Conjugate -embedding has a gauge freedom which, upon its fixing, subsumes the U-model of Eguchi (Sugaku Expositions, 19 (2006) 197) proposed in a statistical machine learning context. The generalization by -entropy, -cross-entropy, -divergence, when applied to the ϕ-exponential family, yields either a Hessian structure or a conformal Hessian structure under different gauge selections —this “splitting” is the hallmark when deforming the exponential family with its dually flat (Hessian) geometry. This letter provides a unified information geometric perspective of deformation of the exponential model, with calculations for Tsallis q-model.