A probabilistic graphical model approach to stochastic multiscale partial differential equations

In numerous real world applications, from sensor networks to computer vision to natural text processing, one needs to reason about the system in question in the face of uncertainty. A key problem in all those settings is to compute the probability distribution over the variables of interest (the query) given the observed values of other random variables (the evidence). Probabilistic graphical models (PGMs) have become the approach of choice for representing and reasoning with high-dimensional probability distributions. However, for most models capable of accurately representing real-life distributions, inference is fundamentally intractable. As a result, optimally balancing the expressive power and inference complexity of the models, as well as designing better approximate inference algorithms, remain important open problems with potential to significantly improve the quality of answers to probabilistic queries. This thesis contributes algorithms for learning and approximate inference in probabilistic graphical models that improve on the state of the art by emphasizing the computational aspects of inference over the representational properties of the models. Our contributions fall into two categories: learning accurate models where exact inference is tractable and speeding up approximate inference by focusing computation on the query variables and only spending as much effort on the remaining parts of the model as needed to answer the query accurately. First, for a case when the set of evidence variables is not known in advance and a single model is needed that can be used to answer any query well, we propose a polynomial time algorithm for learning the structure of tractable graphical models with quality guarantees, including PAC learnability and graceful degradation guarantees. Ours is the first efficient algorithm to provide this type of guarantees. A key theoretical insight of our approach is a tractable upper bound on the mutual information of arbitrarily large sets of random variables that yields exponential speedups over the exact computation. Second, for a setting where the set of evidence variables is known in advance, we propose an approach for learning tractable models that tailors the structure of the model for the particular value of evidence that become known at test time. By avoiding a commitment to a single tractable structure during learning, we are able to expand the representation power of the model without sacrificing efficient exact inference and parameter learning. We provide a general framework that allows one to leverage existing structure learning algorithms for discovering high-quality evidence-specific structures. Empirically, we demonstrate state of the art accuracy on real-life datasets and an order of magnitude speedup. Finally, for applications where the intractable model structure is a given and approximate inference is needed, we propose a principled way to speed up convergence of belief propagation by focusing the computation on the query variables and away from the variables that are of no direct interest to the user. We demonstrate significant speedups over the state of the art on large-scale relational models. Unlike existing approaches, ours does not involve model simplification, and thus has an advantage of converging to the fixed point of the full model. More generally, we argue that the common approach of concentrating on the structure of representation provided by PGMs, and only structuring the computation as representation allows, is suboptimal because of the fundamental computational problems. It is the computation that eventually yields answers to the queries, so directly focusing on structure of computation is a natural direction for improving the quality of the answers. The results of this thesis are a step towards adapting the structure of computation as a foundation of graphical models.

Probabilistic graphical models are a fundamental tool in probabilistic modeling, machine learning and artificial intelligence. They allow us to integrate in a natural way expert knowledge, physical modeling, heterogeneous and correlated data and quantities of interest. For exactly this reason, multiple sources of model uncertainty are inherent within the modular structure of the graphical model. In this paper we develop information-theoretic, robust uncertainty quantification methods and non-parametric stress tests for directed graphical models to assess the effect and the propagation through the graph of multi-sourced model uncertainties to quantities of interest. These methods allow us to rank the different sources of uncertainty and correct the graphical model by targeting its most impactful components with respect to the quantities of interest. Thus, from a machine learning perspective, we provide a mathematically rigorous approach to correctability that guarantees a systematic selection for improvement of components of a graphical model while controlling potential new errors created in the process in other parts of the model. We demonstrate our methods in two physico-chemical examples, namely quantum scale-informed chemical kinetics and materials screening to improve the efficiency of fuel cells.

Probabilistic graphical models (PGM) are one of the rich frameworks. These models are used over complex domains for coding probability distributions. The joint distributions interact with each other over large numbers of random variables and are the combination of statistics and computer science. These concepts are dependent on theories such as probability theory, graph algorithms, machine learning, which make a basic tool in devising many machine learning problems. These are the origin for the contemporary methods in an extensive range of applications. These applications range as medical diagnosis, image understanding, speech recognition, natural language processing, etc. Graphical models are one of dominant tools for handling image processing applications. On the other hand, the volume of image data gives rise to a problem. The representation of all possible graphical model node variables with that of discrete states heads to the number of states for the model. This leads to interpretation computationally obstinate. Many projects involve a human intervention or an automated system to obtain the consensus established on existing information. The PGM, discussed in this chapter, offers a variety of approaches. The approach is based on models and allows interpretable models to be built which then is employed by reasoning algorithms. These models are also studied significantly from data and allow the approaches for cases where the model is manually built. Most real-world applications are of uncertain data which makes a model building more challenging. This chapter emphasizes on PGM where the uncertainty of data is obvious. PGM provides models that are more realistic. These are extended from Bayesian networks, undirected Markov networks, discrete and continuous models, and extensions to deal with dynamical systems and relational data also. For each class of models, the chapter describes the fundamental bases: representation, inference, and learning. Finally, the chapter considers the decision making under the uncertainty of the data.

Graphical modelling is an important tool for the efficient representation and analysis of uncertain information in knowledge-based systems. While Bayesian networks and Markov networks from probabilistic graphical modelling are well-known for a couple of years, the field of possibilistic graphical modelling occurs as a new promising area of research. Possibilistic networks provide an alternative approach compared to probabilistic networks, whenever it is necessary to model uncertainty and imprecision as two different kinds of imperfect information. Imprecision in the sense of set-valued data has often to be considered in situations where data are obtained from human observations or non-precise measurement units. In this contribution we present a comparison of the background and perspectives of probabilistic and possibilistic graphical models, and give an overview on the current state of the art of possibilistic networks with respect to propagation and learning algorithms, applicable to data mining and data fusion problems.

As genomics becomes increasingly relevant to medicine, medical informatics and bioinformatics are gradually converging into a larger field that we call computational biomedicine. Developing a computational framework that is common to the different disciplines that compose computational biomedicine will be a major enabler of the further development and integration of this research domain. Probabilistic graphical models such as Hidden Markov Models, belief networks, and missing-data models together with computational methods such as dynamic programming, Expectation-Maximization, data-augmentation Gibbs sampling, and the Metropolis-Hastings algorithm provide the tools for an integrated probabilistic approach to computational biomedicine. We show how graphical models have already found a broad application in different fields composing computational biomedicine. We also indicate several challenges that lie at the interface between medical informatics, statistical genomics, and bioinformatics. We also argue that graphical models offer a unified framework making it possible to integrate in a statistically meaningful way multiple models ranging from the molecular level to cellular and to clinical levels. Because of their versatility and firm statistical underpinning, we assert that probabilistic graphical models can serve as the lingua franca for many computationally intensive approaches to biology and medicine. As such, graphical models should be a foundation of the curriculum of students in these fields. From such a foundation, students could then build towards specific computational methods in medical informatics, medical image analysis, statistical genetics, or bioinformatics while keeping the communication open between these areas.

In continuous domain, how to efficiently learn the complex probabilistic graphical model is a bottleneck problem for estimation of distribution algorithms (EDAs). The predominant researches focus on Gaussian probabilistic model instead of histogram distribution model because of its comparative superiority in the computational complexity. In this paper, however, we find that using the histogram model does not necessarily bring into exponential computational complexity. Based on the fact many bins are zero-height, we propose a novel method that can learn the multivariate dependency histogram based probabilistic graphical model with acceptable polynomial computational complexity. Several strategies previously used in the HEDA are combined into the new algorithm to improve the convergence and diversity. Experiments showed the superior performance of the new algorithm on several continuous problems compared with UMDA <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">c</inf> , IDEA-G and sur-shr-HEDA.

In this paper we introduce a general probabilistic graphical model for human everyday activity recognition. The proposed model is a discriminative graphical model with hidden variables for modeling body pose and sequential order of them. We use a unified framework for prediction task that is faster and more efficient than structured support vector machine and hidden conditional random fields. We have trained and tested the model on RGB-D videos and the result was comparable to the state of the art.

Probabilistic graphical modeling (PGM) provides a framework for formulating an interpretable generative process of data and expressing uncertainty about unknowns. This makes PGM very useful for understanding phenomena underlying data and for decision making. PGM has seen great success in domains where interpretable inferences are key, e.g. marketing, medicine, neuroscience, and social science. However, PGM tends to lack flexibility, which has hindered its use when it comes to modeling large scale high-dimensional complex data and performing tasks that require flexibility (e.g. in vision and language applications.)\n\nDeep learning (DL) is another framework for modeling and learning from data that has seen great empirical success in recent years. DL is very powerful and offers great flexibility, but it lacks the interpretability and calibration of PGM. \n\nThis thesis develops deep probabilistic graphical modeling (DPGM). DPGM consists in leveraging DL to make PGM more flexible. DPGM brings about new methods for learning from data that exhibit the advantages of both PGM and DL. \n\nWe use DL within PGM to build flexible models endowed with an interpretable latent structure. One family of models we develop extends exponential family principal component analysis (EF-PCA) using neural networks to improve predictive performance while enforcing the interpretability of the latent factors. Another model class we introduce enables accounting for long-term dependencies when modeling sequential data, which is a challenge when using purely DL or PGM approaches. This model class for sequential data was successfully applied to language modeling, unsupervised document representation learning for sentiment analysis, conversation modeling, and patient representation learning for hospital readmission prediction. Finally, DPGM successfully solves several outstanding problems of probabilistic topic models. \n\nLeveraging DL within PGM also brings about new algorithms for learning with complex data. For example, we develop entropy-regularized adversarial learning, a learning paradigm that deviates from the traditional maximum likelihood approach used in PGM. From the DL perspective, entropy-regularized adversarial learning provides a solution to the long-standing mode collapse problem of generative adversarial networks.

Fingerprint matching is the most important part in the field of fingerprint recognition. In this paper, a novel fingerprint matching algorithm based on the probabilistic graphical model and 3-tree model is proposed. First, minutiae matching problems are considered as a special point-set matching. Fingerprint minutiae are viewed as random variables. Each minutia pairs have some probability to be matched. Second, an algorithm is proposed to generate the graphical model and choose "signal points", which dynamically have corresponding points in other point set. We choose three base minutiae pairs as signal pairs. Third, the model is converted into a Junction Tree. A 3-tree model is built and the potentials of other minutiae pairs are calculated through Junction Tree (J.T.) algorithm. Then we translate the matching problem into the best matching problem of a weighted bipartite graph. Finally, the number of common matching pairs can be got through maximum flow algorithm. The similarity of two fingerprints is evaluated using the number of common matching pairs and the maximal posteriori probability. In order to deal with part-matching problems, we use the smallest convex hull which contains all the matched minutiae. Experiments evaluated on FVC 2004 show both effectiveness and efficiency of our methods.

The evaluation and reducing of uncertainty is central to the task of hydrological frequency analysis. In this paper a Bayesian Markov Chain Monte Carlo (MCMC) method is employed to infer the parameter values of the probabilistic distribution model and evalue the uncertainties of design flood. Comparison to the estimated results of three-parameter log-normal distribution (LN3) and the three-parameter generalized extreme value distribution (GEV), the Pearson Type 3 distribution (PIII) provides a good approximation to flood-flow data. The choice of the appropriate probabilistic model can reduce uncertainty of design flood estimation. Historical flood events might be greatly reduced uncertainty when incorporating past extreme historical data into the flood frequency analysis.

Proteins, the basic structural and functional building blocks of cellular machinery, evolve under significant constraint. Evolutionary pressures limit the acceptable variation and covariation of amino acids in a protein. As such, these evolutionary pressures yield constraints manifested in the sequence record of a family of evolutionarily related proteins. Some of these sequence constraints arise from within the proteins themselves (intramolecular constraints), while others are imposed externally (intermolecular constraints). This thesis addresses how to identify both intramolecular and intermolecular sequence constraints, represent these constraints with probabilistic graphical models, and use the models predictively and generatively. Our probabilistic graphical models support a general and powerful mechanism for representing and utilizing sequence constraints, providing a sound semantics, an efficient and transparent means of evaluating consistency with the underlying constraints, and an effective framework for exploring the space of satisfying sequences. For intramolecular constraints, our graphical models of residue coupling (GMRCs) identify and encode both residue conservation (individual residue variation) and residue coupling (pairwise residue covariation). Our GMRCs identify biologically relevant constraints and can be used to predict with high accuracy the functional class of a protein. Algorithms we have developed for sampling from GMRCs generate novel sequences that meet the underlying constraints. For intermolecular constraints, our graphical models of residue cross-coupling (GMRCCs) identify and encode residue cross-coupling (covariation between residues in two different proteins). Our GMRCCs identify cross-coupled residues that are known to confer specificity in protein-protein interactions. Using these constraints, our models are able to predict whether or not two proteins will interact. Algorithms we have developed are able to generate the sequences most likely to interact (or not) with particular design targets.

Inference is a central problem in probabilistic graphical models, and is often the main sub-step in probabilistic learning procedures. Thus, accurate inference algorithms are essential to both answer queries on a learned model, as well as to learn a robust model. Gibbs sampling is arguably one of the most popular approximate inference methods that has been widely used for probabilistic inference in several different domains including natural language processing, computer vision. etc. Here, we develop an approach that improves the performance of blocked Gibbs sampling, an advanced variant of the Gibbs sampling algorithm. Specifically, we utilize correlation among variables in the probabilistic graphical model to develop an adaptive blocked Gibbs sampler that automatically tunes its proposal distribution based on statistics derived from previous samples. Specifically, we adapt the proposal such that we sample blocks containing highly correlated variables more often than the others. This in turn helps improve probability estimates given by the sampler, by selecting hard-to-sample variables more often during the sampling procedure. Further, since adaptation breaks the Markovian property of the sampler, we develop a method to guarantee that our sampler converges to the correct stationary distribution despite being non-Markovian, by diminishing the adaptation of the selection probabilities over time. We evaluate our method with several discrete probabilistic graphical models taken from UAI challenge problems corresponding to different domains, and show that our approach is superior in terms of accuracy as compared to methods that ignore correlation information in the proposal distribution of the sampler.

Parallel probabilistic graphical model approach for nonparametric Bayesian inference

Dependence is a universal phenomenon which can be observed everywhere. In machine learning, probabilistic graphical models (PGMs) represent dependence relations with graphs. PGMs find wide applications in natural language processing (NLP), speech processing, computer vision, biomedicine, information retrieval, etc. Many traditional models, such as hidden Markov models (HMMs), Kalman filters, can be put under the umbrella of PGMs. The central idea of PGMs is to decompose (factorize) a joint probability into a product of local factors. Learning, inference and storage can be conducted efficiently over the factorization representation. In this thesis, we propose a novel framework motivated by the Minimum Shared Information Principle (MSIP): We try to find a factorization in which the information shared between factors is minimum. In other words, we try to make factors as independent as possible. The benefit by doing this is that we can train factors separately without paying a lot of efforts to guarantee consistency between them. To achieve this goal, we develop a theoretical framework called co-occurrence rate networks (CRNs) to obtain such a factorization. Experimental results on three important natural language processing tasks show that our separate training method is two orders of magnitude faster than conditional random fields, while achieving competitive quality (often better on the overall quality metric F1). The second contribution of this thesis is applying PGMs to a real-world NLP application: open relation extraction (ORE). In open relation extraction, two entities in a sentence are given, and the goal is to automatically extract their relation expression. ORE is a core technique, especially in the age of big data, for transforming unstructured information into structured data. We propose our model SimpleIE for this task. The basic idea is to decompose an extraction pattern into a sequence of simplification operations (components). The benefit by doing this is that these components can be re-combined in a new way to generate new extraction patterns. Experimental results on three benchmark data sets show that SimpleIE boosts recall and F1 by at least 15% comparing with seven ORE systems.

This chapter introduces graphical models as a powerful tool to derive efficient algorithms for inference problems. When dealing with complex interdependent variables, inference problems may become of huge complexity. In this context, the structure of the variables is of great interest. In this chapter, directed and undirected graphical models are first defined, before some crucial results are stated, such as the Hammersley–Clifford theorem of Markov random fields and the junction tree property aimed at finding groupings under which a graphical model becomes a tree. Taking advantage of the structure of the variables, belief propagation is then described, including two particular instances: the sum–product and max–sum algorithms. In the final section, the learning problem is addressed in three different contexts: parameter learning, graphical model learning, and latent graphical model learning.