New maximum entropy methods for modeling lifetime distributions

In this paper, Maximum Entropy (ME) framework is used to classify text documents. The ME framework has a lot of advantages when compared with other supervised learning algorithms, such as naive Bayes classifier. For example, it makes no inherent conditional independence assumptions between terms. With four labeled data sets, extensive experiments are made to compare the accuracy of ME algorithm with those of naive Bayes and Support Vector Machine (SVM), which are two popular and accurate algorithms in the domain of text classification. The final result is that ME method consistently outperforms naive Bayes and SVM algorithms in accuracy. On the WebKB and Industry Vector data sets, the accuracy of ME algorithm increases from 81.38% to 85.52% and from 85.73% to 89.78% respectively. On the third 20 Newsgroups data set, our experimental result is opposite to that of Nigam et al. For the last Reuters-21578 data set, the accuracy of ME algorithm increases from 94.76% to 96.16%.

We derive general distribution tests based on the method of maximum entropy (ME) density. The proposed tests are derived from maximizing the differential entropy subject to given moment constraints. By exploiting the equivalence between the ME and maximum likelihood (ML) estimates for the general exponential family, we can use the conventional likelihood ratio (LR), Wald, and Lagrange multiplier (LM) testing principles in the maximum entropy framework. In particular, we use the LM approach to derive tests for normality. Monte Carlo evidence suggests that the proposed tests are compatible with and sometimes outperform some commonly used normality tests. We show that the proposed tests can be extended to tests based on regression residuals and non-i.i.d. data in a straightforward manner. An empirical example on production function estimation is presented.

Intrusion detection is a critical component of secure information systems. Current intrusion detection systems (IDS) especially NIDS (Network Intrusion Detection System) examine all data features to detect intrusions. However, some of the features may be redundant or contribute little to the detection process and therefore they have great impact on the system performance. This paper proposes a lightweight intrusion detection model that is computationally efficient and effective based on feature selection and Maximum Entropy (ME) model. Firstly, the issue of identifying important input features is addressed. Since elimination of the insignificant and/or useless inputs leads to a simplification of the problem, therefore results to faster and more accurate detection. Secondly, classic ME model is used to learn and detect intrusions using the selected important features. Experimental results on the well-known KDD 1999 dataset show the proposed model is effective and can be applied to real-time intrusion detection environments.

The maximum fuzzy entropy algorithm is one of the effective methods of image segmentation, researchers often combine maximum fuzzy entropy and genetic algorithm (GA), but the algorithms have the defects of complexity encoding and unstable search, etc. To improve the algorithms, artificial fish school algorithm (AFSA) is introduced to more rapid optimization of maximum fuzzy entropy, this paper proposes a novel maximum fuzzy entropy image segmentation algorithm based on AFSA (AFSA-based MFEISA). Firstly, the histogram of the image and the probability of each gray level are counted up, and then the multiple threshold integer coding is designed and the maximum fuzzy entropy is converted to the objective function of coding, finally, the maximum fuzzy entropy and its corresponding thresholds are obtained by the improved AFSA and the image is segmented. The experimental results show that the proposed algorithm can achieve better segmentation effect than OTSU algorithm and maximum entropy algorithms, at the same time, it not only has more simple and effective coding scheme and better segmentation effect than the algorithms based on fuzzy entropy and GA, etc, but also it shows better stability and convergence.

Entropy methods enable a convenient general approach to providing a probability distribution with partial information. The minimum cross-entropy principle selects the distribution that minimizes the Kullback–Leibler divergence subject to the given constraints. This general principle encompasses a wide variety of distributions, and generalizes other methods that have been proposed independently. There remains, however, some confusion about the breadth of entropy methods in the literature. In particular, the asymmetry of the Kullback–Leibler divergence provides two important special cases when the target distribution is uniform: the maximum entropy method and the maximum log-probability method. This paper compares the performance of both methods under a variety of conditions. We also examine a generalized maximum log-probability method as a further demonstration of the generality of the entropy approach.

The classical maximum entropy (ME) approach to estimating the unknown parameters of a multinomial discrete choice problem, which is equivalent to the maximum likelihood multinomial logit (ML) estimator, is generalized. The generalized maximum entropy (GME) model includes noise terms in the multinomial information constraints. Each noise term is modeled as the mean of a finite set of a priori known points in the interval [−1,1] with unknown probabilities where no parametric assumptions about the error distribution are made. A GME model for the multinomial probabilities and for the distributions associated with the noise terms is derived by maximizing the joint entropy of multinomial and noise distributions, under the assumption of independence. The GME formulation reduces to the ME in the limit as the sample grows large or when no noise is included in the entropy maximization. Further, even though the GME and the logit estimators are conceptually different, the dual GME model is related to a gener...

The modeling of complex service systems entails capturing many sub-components of the system, and the dependencies that exist among them in the form of a joint probability distribution. Two common methods for constructing joint probability distributions from experts using partial information include maximum entropy methods and copula methods. In this paper we explore the performance of these methods in capturing the dependence between random variables using correlation coefficients and lower-order pairwise assessments. We focus on the case of discrete random variables, and compare the performance of these methods using a Monte Carlo simulation when the variables exhibit both independence and non-linear dependence structures. We show that the maximum entropy method with correlation coefficients and the Gaussian copula method perform similarly, while the maximum entropy method with pairwise assessments performs better particularly when the variables exhibit non-linear dependence.

After a brief expository account of the Shannon-Jaynes principle of maximum entropy (POME) for discrete and continuous variables, we give here an account of some recent research work which (i) a “histogram” method to contrast the discrete and continuous modes computation and the role of histogram in actual practice when dealing with continuous probability distributions. (ii) The idea of mean logarithmic decrement associated with a probability distribution is introduced and is shown to be related to the concept of differential entropy. The mean with respect to an arbitrary probability distribution of the logarithm of the ratio of the probability density function for an exponential distribution is discussed in the context of hydrological investigations. Unlike the entropy of a continuous probability distribution introduced in (i), this quantity which is an example of the Kullback-Leibler (KL) Information, is always positive and invariant under coordinate transformation. (iii) The constraints entering into POME as well as a minimum K L information are identified as a class of sufficient statistics which determine the unknown parameters in the probability density functions that occur in the most commonly used hydrological models. (iv) An example of (iii) where only the first two moments in a semi-infinite domain are givens is discussed to shed light on the limitations of POME, recentky recognized by Wragg and coworkers, and is made relevant to the work of Sonuga on rainfall-run off relationship. Finally, (v) a method of generating probability distributions starting from one basic distribution employing coordinate transformations is given. This in conjuction with (iii) leads to the notions of “Physical constraints” in contrast to the “mathematical constraints” in examing parameter estimation.

Reconstruction of calmodulin single-molecule FRET states, dye interactions, and CaMKII peptide binding by MultiNest and classic maximum entropy

We propose a probabilistic model for estimating word confidence by fusing predictor features. Starting from the maximum entropy (ME) method, we first prove that ME model is equivalent to the best model with certain form to the minimum expected cross entropy (MECE) criterion. Under the MECE criterion, We extend the form of ME model by introducing a hidden state. We call the new model hidden-state maximum entropy (HSME) model. In a keyword-spotting task, we combine predictor features from both phonetic and word-level systems. Compared to lattice posterior alone, recall at 80% precision is improved from 38.1% to 49.5% on voicemail and from 37.1% to 51.9% on Switchboard. Compared with other fusion methods, HSME consistently outperforms decision tree, and most cases SVM.

Uniform distribution is the simplest statistical distribution. Although there is hardly any hydrologic variable that follows a uniform probability distribution, it is invoked in a variety of applications. For example, in Bayesian statistical modeling in hydrology it is frequently used as a prior distribution. In systems hydrology, uniform distribution is the pulse function obtained by subtracting two step functions lagged by the length of the uniform distribution. The pulse function is a key to deriving the unit hydrograph theory. The instantaneous unit hydrograph of the rational method, used in urban hydrology, is a uniform distribution (Singh, 1988). Of all the statistical distributions, uniform distribution has the highest entropy. In river morphology, when a river approaches equilibrium or dynamic equilibrium, its characteristics tend to follow a uniform distribution. Under equilibrium, rivers follow the minimum rate of energy dissipation. Furthermore, a river constantly adjusts its cross-sectional geometry and longitudinal profile to accommodate the influx of water and sediment coming from its drainage basin, and this adjustment is in accordance with the principle of maximum entropy. Thus, there is a close link between equilibrium and uniform distribution and then between maximum entropy (uniform distribution) and minimum rate of energy dissipation. This link plays a fundamental role in river engineering and training works, river morphology, evolution of deltas, etc.

A maximum entropy approach to adaptive statistical language modelling

The set of solutions inferred by the generic maximum entropy (MaxEnt) or maximum relative entropy (MaxREnt) principles of Jaynes - considered as a function of the moment constraints or their conjugate Lagrangian multipliers - is endowed with a Riemannian geometric description, based on the second differential tensor of the entropy or its Legendre transform (negative Massieu function). The analysis provides a generalised {\it least action bound} applicable to all Jaynesian systems, which provides a lower bound to the cost (in generic entropy units) of a transition between inferred positions along a specified path, at specified rates of change of the control parameters. The analysis therefore extends the concepts of "finite time thermodynamics" to the generic Jaynes domain, providing a link between purely static (stationary) inferred positions of a system, and dynamic transitions between these positions (as a function of time or some other coordinate). If the path is unspecified, the analysis gives an absolute lower bound for the cost of the transition, corresponding to the geodesic of the Riemannian hypersurface. The analysis is applied to (i) an equilibrium thermodynamic system subject to mean internal energy and volume constraints, and (ii) a flow system at steady state, subject to constraints on the mean heat, mass and momentum fluxes and chemical reaction rates. The first example recovers the {\it minimum entropy cost} of a transition between equilibrium positions, a widely used result of finite-time thermodynamics. The second example leads to a new {\it minimum entropy production principle}, for the cost of a transition between steady state positions of a flow system.

Two Chebyshev recursion methods are presented for calculations with very large sparse Hamiltonians, the kernel polynomial method (KPM) and the maximum entropy method (MEM). They are applicable to physical properties involving large numbers of eigenstates such as densities of states, spectral functions, thermodynamics, total energies for Monte Carlo simulations and forces for tight binding molecular dynamics. this paper emphasizes efficient algorithms.

Background and Aim:African swine fever (ASF) is an infectious disease and a major viral pig disease that threatens pork production in several locations globally. The mortality rate of ASF in domestic pigs is very high, causing a decrease in pig populations and significant economic losses for farmers. Environmental or ecological risk factors are the most important associated with the spread of the ASF virus. Environmental (or ecological) niche models are commonly used to estimate the probability of an event using the maximum entropy (Maxent) method. This study aimed to estimate the probability risk of future ASF outbreaks in North Sumatra, Indonesia.Materials and Methods:Secondary data from the National Animal Health System Database (iSIKHNAS), including data on the ASF outbreaks of 2019–2020 in North Sumatra, Indonesia, were used in this study. The first analysis performed involved the identification of environmental risk factors using multiple regression analysis. The second analysis performed was the estimation of probability risk for future ASF outbreaks in North Sumatra, Indonesia, using the Maxent method. Data processing was performed using Microsoft Excel, ArcGIS version 10.5 software (ESRI, California, United States), Maxent version 3.4.4 software, and Rstudio (http://www.r-project.org/).Results:The Maxent method was found to be highly accurate with a statistically significant area under the curve value of 0.860. The greatest contributing environmental factor identified by the model was the harbor, which contributed 57%. The range of high probability risk of future ASF outbreaks was found to be 0.723–0.84.Conclusion:The estimation of the highest probability risk of future ASF outbreaks in North Sumatra, Indonesia, was 0.723–0.84. The most contributing environmental factor identified using the Maxent method was harbors, at 57%. This methodology can be used to carry out subsequent ASF analyses and contribute to developing prevention and control strategies in this area.