Yet Another Method for Combining Classifiers Outputs: A Maximum Entropy Approach

Although many different structural models have been proposed for perfluorosulfonic acid (PFSA) ionomers such as Nafion, the very high degree of disorder in these materials makes it difficult, if not impossible, to deduce their morphology unambiguously from small angle X-ray and neutron scattering (SAXS/SANS) experiments alone. We present a combination of a model-independent procedure for obtaining structural information from SAXS patterns based on a Maximum Entropy (MaxEnt) approach coupled with mesoscale simulations of the morphology of Nafion using Dissipative Particle Dynamics (DPD) parameterized with atomistic calculations and density functional theory. These two methods show that the nanoscale ionic clustering in PFSAs is intimately linked to, but spatially separate from, the larger scale organization of the fluorocarbon backbone. Although we are unable to directly observe crystallization of the backbone in the DPD simulations, the high density regions of fluorocarbon segments correspond exactly to those regions where the density of ionic clusters is lowest, each forming an independent bicontinuous domain. We are thus able to demonstrate a unified morphological description of PFSAs based on both statistical (MaxEnt) and thermodynamic (DPD) descriptions, which broadly favours a bicontinuous network of ionic clusters embedded in a matrix of fluorocarbon chains.

Projections of anthropogenically-induced global climate change and its impacts on potential distributions of invasive species are crucial for implementing effective conservation and management strategies. Lantana camara L., a popular ornamental plant native to tropical America, has become naturalized in some 50 countries and is considered one of the world’s worst weeds. To increase our understanding of its potential extent of spread and examine the responses of global geographic distribution, predictive models incorporating global distribution data of L. camara were generated. These models were used to identify areas of environmental suitability and project the effects of future climate change based on an ensemble of the four global climate models (GCMs) within the Inter-Sectoral Impact Model Intercomparis on Project (ISI-MIP). Each model was run under the four emission scenarios (Representative Concentration Pathways, RCPs) using the Maximum entropy (Maxent) approach. Future model predictions through 2050 indicated an overall expansion of L. camara, despite future suitability varying considerably among continents. Under the four RCP scenarios, the range of L. camara expanded further inland in many regions (e.g. Africa, Australia), especially under the RCP85 emission scenario. The global distribution of L. camara, though restricted within geographical regions of similar latitude as at present (35°N ~ 35°S), was projected to expand equator-ward in response to future climate conditions. Considerable discrepancy in predicted environmental suitability for L. camara among GCMs highlights the complexities of the likely effects of climate change on its potential distribution and the need to improve the reliability of predictions in novel climates.

A maximum entropy approach to adaptive statistical language modelling

The similarity between maximum entropy (MaxEnt) and minimum relative entropy (MRE) allows recent advances in probabilistic inversion to obviate some of the shortcomings in the former method. The purpose of this paper is to review and extend the theory and practice of minimum relative entropy. In this regard, we illustrate important philosophies on inversion and the similarly and differences between maximum entropy, minimum relative entropy, classical smallest model (SVD) and Bayesian solutions for inverse problems. MaxEnt is applicable when we are determining a function that can be regarded as a probability distribution. The approach can be extended to the case of the general linear problem and is interpreted as the model which fits all the constraints and is the one model which has the greatest multiplicity or “spreadout” that can be realized in the greatest number of ways. The MRE solution to the inverse problem differs from the maximum entropy viewpoint as noted above. The relative entropy formulation provides the advantage of allowing for non-positive models, a prior bias in the estimated pdf and `hard' bounds if desired. We outline how MRE can be used as a measure of resolution in linear inversion and show that MRE provides us with a method to explore the limits of model space. The Bayesian methodology readily lends itself to the problem of updating prior probabilities based on uncertain field measurements, and whose truth follows from the theorems of total and compound probabilities. In the Bayesian approach information is complete and Bayes' theorem gives a unique posterior pdf. In comparing the results of the classical, MaxEnt, MRE and Bayesian approaches we notice that the approaches produce different results. In␣comparing MaxEnt with MRE for Jayne's die problem we see excellent comparisons between the results. We compare MaxEnt, smallest model and MRE approaches for the density distribution of an equivalent spherically-symmetric earth and for the contaminant plume-source problem. Theoretical comparisons between MRE and Bayesian solutions for the case of the linear model and Gaussian priors may show different results. The Bayesian expected-value solution approaches that of MRE and that of the smallest model as the prior distribution becomes uniform, but the Bayesian maximum aposteriori (MAP) solution may not exist for an underdetermined case with a uniform prior.

In this article, we provide initial findings regarding the problem of solving likelihood equations by means of a maximum entropy (ME) approach. Unlike standard procedures that require equating the score function of the maximum likelihood problem at zero, we propose an alternative strategy where the score is instead used as an external informative constraint to the maximization of the convex Shannon’s entropy function. The problem involves the reparameterization of the score parameters as expected values of discrete probability distributions where probabilities need to be estimated. This leads to a simpler situation where parameters are searched in smaller (hyper) simplex space. We assessed our proposal by means of empirical case studies and a simulation study, the latter involving the most critical case of logistic regression under data separation. The results suggested that the maximum entropy reformulation of the score problem solves the likelihood equation problem. Similarly, when maximum likelihood estimation is difficult, as is the case of logistic regression under separation, the maximum entropy proposal achieved results (numerically) comparable to those obtained by the Firth’s bias-corrected approach. Overall, these first findings reveal that a maximum entropy solution can be considered as an alternative technique to solve the likelihood equation.

One well-recognized problem in three-dimensional (3D) confocal microscopic images is that the intensities in deeper slices are generally weaker than those in shallower slices. The loss of intensity with depth hampers both qualitative observation and quantitative measurement of specimens. Two major types of methods exist to compensate for this intensity loss: the first is based on the geometrical optics inside the specimen, and the second applies an empirical parametric intensity decay function (IDF) of depth. A common feature shared by both methods is that they are parameter-dependent. However, for the optics-based method there are as yet no fully automated parameter-setting approaches; and for the IDF method the traditional profile-fitting approach cannot provide proper parameters if the presumed IDF model does not match the experimental intensity-depth profile of the 3D image. In this paper, we propose a novel maximum-entropy (ME) approach to fully automated parameter-setting. In principle the ME approach is suitable for any compensation method as long as it is parameter-dependent. The basic assumption is that without intensity loss an ideal 3D image should be generally homogeneous with respect to depth and this axial homogeneity can be represented by the entropy of a normalized intensity-depth profile. Experiments on real confocal images showed that such a profile was consistent with visual evaluation of axial intensity homogeneity and that the ME approach could provide proper parameters for both compensation methods mentioned above. Moreover, for the IDF method, experiments on both real and simulated data showed that the ME approach could provide more precise parameters than with traditional profile-fitting. The Appendix provides a proof that under certain conditions the global maximization of the profile-entropy is guaranteed.

We present a novel application of the maximum entropy (MaxEnt) method for solving the double-inverse problems of removing instrument broadening from X-ray diffraction profiles and calculating the column-length distribution of the crystallites. The MaxEnt approach is shown to have compelling advantages over the conventional methods it replaces: it is stable and robust, incorporates noise anda prioriinformation into the solution, preserves positivity of the solution, and can be applied successively. We also show how uncertainties in the derived profiles and column distributions can be determined and used in subsequent calculations, including integral breadth, Fourier coefficients, column-length distributions and apparent particle sizes. Calculations are performed on simulated X-ray diffraction profiles for a range of particle sizes, with a detailed study of the sensitivity of the results to background-level estimates and the use of an incorrect instrument response function.

During the MaxEnt 2002 workshop in Moscow, Idaho, Tony Vignaux asked again a few simple questions about using Maximum Entropy or Bayesian approaches for the famous Dice problems which have been analyzed many times through this workshop and also in other places. Here, there is another analysis of these problems. I hope that, this paper will answer a few questions of Tony and other participants of the workshop on the situations where we can use Maximum Entropy or Bayesian approaches or even the cases where we can actually use both of them.

We present an extension to Jaynes’ maximum entropy principle that incorporates latent variables. The principle oflatent maximum entropywe propose is different from both Jaynes’ maximum entropy principle and maximum likelihood estimation, but can yield better estimates in the presence of hidden variables and limited training data. We first show that solving for a latent maximum entropy model poses a hard nonlinear constrained optimization problem in general. However, we then show that feasible solutions to this problem can be obtained efficiently for the special case of log-linear models---which forms the basis for an efficient approximation to the latent maximum entropy principle. We derive an algorithm that combines expectation-maximization with iterative scaling to produce feasible log-linear solutions. This algorithm can be interpreted as an alternating minimization algorithm in the information divergence, and reveals an intimate connection between the latent maximum entropy and maximum likelihood principles. To select a final model, we generate a series of feasible candidates, calculate the entropy of each, and choose the model that attains the highest entropy. Our experimental results show that estimation based on the latent maximum entropy principle generally gives better results than maximum likelihood when estimating latent variable models on small observed data samples.

Location‐only and use‐availability data: analysis methods converge

A new multi-directional search approach that aims at maximizing the flow entropy of water distribution systems is investigated. The aim is to develop an efficient and practical maximum entropy based approach. The resulting optimization problem has four objectives, and the merits of objective reduction in the computational solution of the problem are investigated also. The relationship between statistical flow entropy and hydraulic reliability/failure tolerance is not monotonic. Consequently, a large number of maximum flow entropy solutions must be investigated to strike a balance between cost and hydraulic reliability. A multi-objective evolutionary optimization model is developed that generates simultaneously a wide range of maximum entropy values along with clusters of maximum and near-maximum entropy solutions. Results for a benchmark network and a real network in the literature are included that demonstrate the effectiveness of the procedure.

The classical maximum entropy (ME) approach to estimating the unknown parameters of a multinomial discrete choice problem, which is equivalent to the maximum likelihood multinomial logit (ML) estimator, is generalized. The generalized maximum entropy (GME) model includes noise terms in the multinomial information constraints. Each noise term is modeled as the mean of a finite set of a priori known points in the interval [−1,1] with unknown probabilities where no parametric assumptions about the error distribution are made. A GME model for the multinomial probabilities and for the distributions associated with the noise terms is derived by maximizing the joint entropy of multinomial and noise distributions, under the assumption of independence. The GME formulation reduces to the ME in the limit as the sample grows large or when no noise is included in the entropy maximization. Further, even though the GME and the logit estimators are conceptually different, the dual GME model is related to a gener...

Price responsiveness of herbicide demand in corn for farmers in Indiana's White River Basin using cross-section data from individual farms is estimated. Particular attention is paid to appropriate treatment of binding nonnegativity constraints. Estimation was first attempted using an approach to demand systems estimation suggested by Lee and Pitt. However, analytical and computational difficulties effectively preclude estimation by the Lee and Pitt approach. As an alternative, a maximum entropy (ME) approach is presented and discussed. Results from the ME estimator tentatively indicate limited response of herbicide demand to changes in own prices. The maximum entropy approach to demand systems estimation appears to have merit and warrants further attention.

Understanding how ensembles of neurons collectively interact will be a key step in developing a mechanistic theory of cognitive processes. Recent progress in multineuron recording and analysis techniques has generated tremendous excitement over the physiology of living neural networks. One of the key developments driving this interest is a new class of models based on the principle of maximum entropy. Maximum entropy models have been reported to account for spatial correlation structure in ensembles of neurons recorded from several different types of data. Importantly, these models require only information about the firing rates of individual neurons and their pairwise correlations. If this approach is generally applicable, it would drastically simplify the problem of understanding how neural networks behave. Given the interest in this method, several groups now have worked to extend maximum entropy models to account for temporal correlations. Here, we review how maximum entropy models have been applied to neuronal ensemble data to account for spatial and temporal correlations. We also discuss criticisms of the maximum entropy approach that argue that it is not generally applicable to larger ensembles of neurons. We conclude that future maximum entropy models will need to address three issues: temporal correlations, higher-order correlations, and larger ensemble sizes. Finally, we provide a brief list of topics for future research.

Spectroscopy and imaging are widely used to characterise systems structurally and functionally, and this information allows for the rational development of novel drugs and theranostics in many diverse areas of medicine. This thesis focuses on applying Electron Paramagnetic Resonance (EPR) techniques to determine the structure and function of protein molecules. A feasibility study to develop paramagnetic probes for EPR imaging is also presented.In EPR the probe is a paramagnetic centre and information is obtained by measuring interactions of this probe with its environment. In this thesis continuous wave (CW) EPR and Double Electron Electron Resonance (DEER) in conjunction with a site-directed spin labelling (SDSL) were employed to study structure and function of spin-labelled protein molecules. The techniques allow examination of molecular systems that cannot be crystallised or are too big for efficient NMR investigation. DEER provides information on distance distributions between two spin labels located on Cys-mutated residues of the protein molecule and can access longer distances in comparison with NMR. CW EPR provides information about the mobility of a single spin label which can be used to characterise protein dynamics. Structural modelling was used to combine the crystal structures and Molecular Dynamics (MD) simulations with DEER distance constraints to present a DEER-based structural model of the conformational variability of the protein molecule.Chapters 2 and 3 focus on characterising two metal ion substrate binding proteins (SBP), Zn2+-binding AdcA and Mn2+-binding PsaA, using DEER and CW EPR. The ATP-Binding Cassette (ABC) permeases, with which the SBPs are associated, are a primary importer used by bacteria to scavenge the essential first-row metal ions (e.g. iron, zinc, manganese) from a host environment. The process is essential for bacterial survival and propagation. Understanding the metal ion acquisition mechanisms by these SBPs can provide new opportunities for targeted drug development. Collectively, the CW EPR and DEER data along with crystal structures, differential scanning fluorimetry (DSF), Molecular Dynamics simulation, and smFRET microscopy, were able to determine a structural model for Zn2+-binding in the AdcA protein referred to as “trap-door” mechanism. Our data show that the “trap-door” mechanism employed by AdcA is different from the “spring-hammer” mechanism employed by PsaA.Non-Ribosomal Peptide Synthetase (NRPS) are the topic of chapter 4. NRPS’s are a family of mega enzymes that produce a diverse range of pharmaceuticals, yet how these molecular machines operate to produce such complex chemicals is very poorly understood. This project will investigate module 7 of the teicoplanin NRPS, a 200 kDa protein, which is the last step in the production of the glycopeptide antibiotics (GPA). A comprehensive structural and functional understanding of the teicoplanin NRPS machinery will provide a paradigm for this enzyme family, enabling the tailored re-engineering of the in vivo NRPS biosynthesis for the development of new antibiotics and pharmaceuticals. This thesis concentrates on a di-domain complex PCP-X from module 7 (Tcp12) of teicoplanin NRPS which contains the peptidyl carrier protein (PCP) domain and the X-domain. These two domains are joined by a long linker which is thought to be flexible. The goal is to investigate conformational dynamics of the PCP-X construct in different states. This includes the PCP domain in loaded and unloaded states, and how the PCP-X construct interacts with an OxyB (P450) protein. Using DEER data from a large number of double spin-labelled PCP-X mutants, rigid-body structural models based on the domain’s crystal structures were investigated to characterise the mobility of the PCP-X di-domain and compare it to very limited crystal structure data of the larger domain construct.Chapter 5 describes the application of the Maximum Entropy (MaxEnt) approach to 2D HYSCORE image reconstruction and its comparison with the conventionally utilised Discrete Fourier Transformation (DFT). HYSCORE is a widely used electron spin echo envelope (ESEEM) experiment used for measuring the nuclear quadrupole and hyperfine couplings. While EPR experiments are generally very sensitive, the time required for two and higher dimensional techniques are a significant limitation. The results show that the MaxEnt algorithm allows sampling of data non-uniformity, provides better sensitivity and an increase in resolution compared to DFT.Lastly, chapter 6 describes the development of PEG-based hyperbranched polymers and pPEGMA-coated gold nanoparticles as EPR imaging agents and sensors. Both nanoparticles have been shown as promising platforms that can be functionalized with targeting ligands and various imaging probes (i.e. a paramagnetic spin label) for the purpose of using multiple molecular imaging modalities for in vivo theranostics. The choice of a spin label dictates the possible application of the final agent, either for EPR imaging or EPR spectroscopy. The preliminary study assessed the sensitivity of commercially available EPR equipment and showed that nitroxides are more suitable for use as a sensor rather than an imaging probe. Future work based on the results in the thesis are discussed at the end of this chapter.