Learning Bayesian Network by a Mesh of Points

In recent years there has been a growing interest in Bayesian Network learning from uncertain data. While many researchers focus on Bayesian Network learning from data with tuple uncertainty, Bayesian Network structure learning from data with attribute uncertainty gets little attention. In this paper we make a clear definition of attribute uncertain data and Bayesian Network Learning problem from such data. We propose a structure learning method named DTAU based on information theory. The algorithm assumes that the structure of a Bayesian network is a tree. It avoids enumerating all possible worlds. The dependency tree is computed with polynomial time complexity. We conduct experiments to demonstrate the effectiveness and efficiency of our method. The experiments show the clustering results on uncertain dataset by our dependency tree are acceptable.KeywordsuncertaintyBayesian Network structuredependency

Several hardness results are presented for the parent assignment problem: Given m observations of n attributes x 1, ..., x n , find the best parents for x n , that is, a subset of the preceding attributes so as to minimize a fixed cost function. This attribute or feature selection task plays an important role, e.g., in structure learning in Bayesian networks, yet little is known about its computational complexity. In this paper we prove that, under the commonly adopted full-multinomial likelihood model, the MDL, BIC, or AIC cost cannot be approximated in polynomial time to a ratio less than 2 unless there exists a polynomial-time algorithm for determining whether a directed graph with n nodes has a dominating set of size logn, a LOGSNP-complete problem for which no polynomial-time algorithm is known; as we also show, it is unlikely that these penalized maximum likelihood costs can be approximated to within any constant ratio. For the NML (normalized maximum likelihood) cost we prove an NP-completeness result. These results both justify the application of existing methods and motivate research on heuristic and super-polynomial-time algorithms.KeywordsPolynomial TimeBayesian NetworkDirected GraphPositive InstanceParent AssignmentThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Estimation of conditional dependencies over a joint multivariate probability distribution is a difficult task for big data, e.g. –omics datasets, and it becomes quickly intractable when the number of variables involved grows large. For instance, structure learning in Bayesian networks has super-exponential complexity. Dimension reduction techniques such as principal component analysis can be useful but often transform the original space and can still pose problems with scalability. This substantially limits characterization of joint probability: in general, only pairwise or k-level correlations can be analyzed efficiently. We introduce the Multivariate Independence Set Search via Progressive Addition for Conditional Markov Acyclic Networks (MISS-PACMAN), which operates a greedy selection of jointly independent feature sets from a larger set of covariates, given a variable ordering. The method is non-parametric and can be used with any kernel function. MISS-PACMAN is therefore suitable for heterogeneous big data, as it combines flexibility and scalability. In our tests on both simulated and real-world data, using random forests as kernel, MISS-PACMAN was able to select independence feature sets linearly with the number of features. Further, by combining multiple independence sets, MISS-PACMAN well-approximates the underlying conditional structure of data variables (according to the generating Bayesian network), and compares favorably with other network structure discovery algorithms, such as the Peter-Clark and the fast incremental association Markov blanket.

A new structure learning approach for Bayesian networks based on asexual reproduction optimization (ARO) is proposed in this paper. ARO can be considered an evolutionary-based algorithm that mathematically models the budding mechanism of asexual reproduction. In ARO, a parent produces a bud through a reproduction operator; thereafter, the parent and its bud compete to survive according to a performance index obtained from the underlying objective function of the optimization problem: This leads to the fitter individual. The convergence measure of ARO is analyzed. The proposed method is applied to real-world and benchmark applications, while its effectiveness is demonstrated through computer simulations. Results of simulations show that ARO outperforms genetic algorithm (GA) because ARO results in a good structure and fast convergence rate in comparison with GA.

Bayesian networks are efficient classification techniques, and widely applied in many fields, however, their structure learning is NP-hard. In this paper, a Bayesian network structure learning method called Tree-like Bayesian network (BN-TL) was proposed, which constructs the network by estimating the correlation between the features and the correlation between the class label and the features. Two metabolomics datasets about liver disease and five public datasets from the University of California at Irvine repository (UCI) were used to demonstrate the performance of BN-TL. The result shows that BN-TL outperforms the other three classifiers, including Naïve Bayesian classifier (NB), Bayesian network classifier whose structure is learned by using K2 greedy search strategy (BN-K2) and a method proposed by Kuschner in 2010 (BN-BMC) in most cases.

Bayesian networks are useful analytical models for designing the structure of knowledge in machine learning. Bayesian networks can represent probabilistic dependency relationships among the variables. One strategy of Bayesian Networks structure learning is the score and search technique. The authors present the Elephant Swarm Water Search Algorithm (ESWSA) as a novel approach to Bayesian network structure learning. In the algorithm; Deleting, Reversing, Inserting, and Moving are used to make the ESWSA for reaching the optimal structure solution. Mainly, water search strategy of elephants during drought periods is used in the ESWSA algorithm. The proposed method is compared with simulated annealing and greedy search using BDe score function. The authors have also investigated the confusion matrix performances of these techniques utilizing various benchmark data sets. As presented by the results of the evaluations, the proposed algorithm has better performance than the other algorithms and produces better scores and accuracy values.

Bayesian network model selection techniques may be used to learn and elucidate conditional relationships between features in pattern recognition tasks. The learned Bayesian network may then be used to infer unknown node-states, which may correspond to semantic tasks. One such application of this framework is scene categorization. In this paper, we employ low-level classification based on color and texture, semantic features, such as sky and grass detection, along with indoor vs. outdoor ground truth information, to create a feature set for Bayesian network structure learning. Indoor vs. outdoor inference may then be performed on a set of features derived from a testing set where node states are unknown. Experimental results show that this technique provides classification rates of 97% correct, which is a significant improvement over previous work, where a Bayesian network was constructed based on expert opinion.

Estamos en la era del aprendizaje automático y el descubrimiento automático de conocimientos a partir de datos se utiliza cada vez más para resolver problemas en nuestra vida diaria. Una clave para diseñar con éxito algoritmos inteligentes útiles es poder modelar la incertidumbre que está presente en el mundo real. Las redes bayesianas son una herramienta poderosa que modela la incertidumbre de acuerdo con la teoría de la probabilidad. Aunque la literatura contiene métodos que aprenden redes bayesianas a partir de conjuntos de datos con alta dimensionalidad, los métodos tradicionales no limitan la complejidad de inferencia de los modelos aprendidos, y a menudo producen modelos en los que la inferencia exacta es intratable. Esta tesis se centra en el aprendizaje de redes bayesianas tratables a partir de datos y contiene la siguientes cinco contribuciones: Primero, proponemos estrategias para aprender redes bayesianas en el espacio de órdenes de eliminación (i). De esta manera, podemos acotar de manera eficiente la complejidad de inferencia de las redes durante el proceso de aprendizaje. Esto es especialmente útil en problemas donde los datos están incompletos. En estos casos, el enfoque más común para aprender redes bayesianas es aplicar el algoritmo de esperanza-maximización estructural, que requiere realizar inferencia en cada iteración del proceso de aprendizaje. Aprovechando la reducida complejidad de inferencia de los modelos, proponemos un nuevo método con el propósito de garantizar la eficiencia del proceso de aprendizaje y a la vez mejorar el rendimiento del algoritmo original (ii). Además, estudiamos la complejidad de la clasificación multidimensional, una generalización de la clasificación multi-etiqueta, en redes bayesianas. Obtenemos cotas superiores para la complejidad del cómputo de las explicaciones más probables y de probabilidades marginales de las variables clase dado el valor de todas las variables predictoras (iii). Utilizamos estos límites para proponer estrategias eficientes para acotar la complejidad de los clasificadores multidimensionales basados en redes bayesianas durante el proceso de aprendizaje. Con el objetivo de mejorar el rendimiento de estos modelos, proponemos métodos para el aprendizaje discriminativo de clasificadores multidimensionales basados en redes bayesianas (iv). Finalmente, abordamos el problema de predecir la ausencia de ataques en pacientes con epilepsia en el lóbulo temporal tras someterse a cirugía (v). Para ello, utilizamos un clasificador multidimensional basados en redes bayesianas, que están especialmente diseñados para modelar las relaciones entre las variables clase, es decir, ausencia de ataques epilépticos uno, dos y cinco años después de la cirugía. ----------ABSTRACT---------- We are in the era of machine learning and the automatic discovery of knowledge from data is increasingly used to solve problems in our daily life. A key to successfully design useful intelligent algorithms is to be able to model the uncertainty that is present in the real world. Bayesian networks are a powerful tool that models uncertainty according to probability theory. Although the literature contains approaches that learn Bayesian networks from high dimensional datasets, traditional methods do not bound the inference complexity of the learned models, often producing models where exact inference is intractable. This thesis focuses on learning tractable Bayesian networks from data, and contains the following five contributions. First, we propose strategies for learning Bayesian networks in the space of elimination orders (i). In this manner, we can efficiently bound the inference complexity of the networks during the learning process. This is specially useful in problems where data is incomplete. In these cases, the most common approach to learn Bayesian networks is to apply the structural expectation-maximization algorithm, which requires performing inference at each iteration of the learning process. Taking advantage of the reduced inference complexity of the models, we propose a new method with the purpose of guaranteeing the efficiency of the learning process while improving the performance of the original algorithm (ii). Additionally, we study the complexity of multidimensional classification, a generalization of multilabel classification, in Bayesian networks. We provide upper bounds for the complexity of the most probable explanations and marginals of class variables conditioned to an instantiation of all predictor variables (iii). We use these bounds to propose efficient strategies for limiting the complexity of multidimensional Bayesian network classifiers during the learning process. With the objective of improving the performance of these models, we also propose methods for the discriminative learning of multidimensional Bayesian network classifiers (iv). Finally, we address the problem of predicting seizure freedom in patients that have undergone temporal lobe epilepsy surgery (v). For that, we use a multidimensional Bayesian network classifier, which is specially well fitted to model the relationships among the class variables, i.e., seizure freedom at one, two and five years after surgery.

In order to solve the problems that the intelligence algorithm falls into the local optimum easily and has a slow convergence in Bayesian networks (BN) structure learning, an algorithm based on adaptive selection operator with simulated annealing is proposed. This chapter conducts the adaptive selection rule in combination with conditional independence tests of BN nodes to guide the generation of neighbor. In order to better compare the adaptive effect, an algorithm based on selection operator with simulated annealing (SOSA) is proposed; at the same time 15 data sets in the three typical networks are accessed as learning samples. The results of the Bayesian Dirichlet (BD) score, Hamming distance (HD), and evolution time of the network after learning show that it has the quicker convergence and it searches the optimal solution more easily compared with simulated annealing (SA) and SOSA.

This paper addresses exact learning of Bayesian network structure from data and expert's knowledge based on score functions that are decomposable. First, it describes useful properties that strongly reduce the time and memory costs of many known methods such as hill-climbing, dynamic programming and sampling variable orderings. Secondly, a branch and bound algorithm is presented that integrates parameter and structural constraints with data in a way to guarantee global optimality with respect to the score function. It is an any-time procedure because, if stopped, it provides the best current solution and an estimation about how far it is from the global solution. We show empirically the advantages of the properties and the constraints, and the applicability of the algorithm to large data sets (up to one hundred variables) that cannot be handled by other current methods (limited to around 30 variables).

It's very significant to correctly identify the hierarchical relation among attributes correctly when constructing a diagnosis test.As we know,there are various methods to identify the hierarchical relation among attributes in different cognitive diagnosis models(CDMs).Rule Space Model(RSM) is a kind of great influence CDM which was developed by Tatsuoka and her associates.In RSM,the task of attribute identification is performed after the test items have already been developed.And then an incidence Q matrix can be determined which reflects hierarchical relation.However,in Leighton,Gierl and Hunka's(2004) Attribute Hierarchy Method(AHM) the organization of attributes(attributes,number of attributes and attribute hierarchical relation) should be determined before developing the test items.In RSM and AHM the importance of correctly identifying the attributes and their hierarchical relation cannot be overstated,and the attributes and their hierarchical relations serve as the most important input variables to the models because they provide the basis for interpreting the results.The hierarchical relation among attributes describes the domain knowledge structures.Understanding the domain knowledge structures in highly specific detail provides a rational basis for proposing and evaluating potential improvements in the measurement of general proficiency.Otherwise,improvement remains largely a trial-and-error process.It has been proved that is not reliable to get the hierarchical relation by analyzing the test items,even the resulting hierarchical relations are not unique.Now,structure learning of Bayesian Networks would be introduced to study the hierarchical relation from the examinees' attribute mastery patterns.On the one hand,this paper conducted a simulation study in which Cui,Leighton and Zheng(2006)'s attribute hierarchical relation was employed,and the sample size of 5000 was used.In the simulated study,different sliping rates was considered for the purpose of testing the robustness of Bayesian structure learning.On the other hand,this paper conducted an empirical study to research the attributes which were commonly used in different numeration representation system converting,and the sample size was 189.Also the hierarchical relation generated by Bayesian structure learning and the hierarchical relation extracted by domain expert was compared.The result of simulation study and empirical study all showed the Bayesian Networks was a very useful tool in determining the hierarchical relation among attributes in the realm of education measurement and psychometrics.It is proved that the result for reference is valuable through structure learning of Bayesian Networks in simulated experiments and practical application.Meanwhile,the Bayesian Networks also can be used to test the hierarchical relation in RSM.

The type of constraint-based methods is one of the most important approaches to learn Bayesian network (BN) structures from observational data with conditional independence (CI) tests. In this paper, we find that existing constraint-based methods often perform many redundant CI tests, which significantly reduces the learning efficiency of those algorithms. To tackle this issue, we propose a novel framework to accelerate BN structure learning by reducing redundant CI tests without sacrificing accuracy. Specifically, we first design a CI test cache table to store CI tests. If a CI test has been computed before, the result of the CI test is obtained from the table instead of computing the CI test again. If not, the CI test is computed and stored in the table. Then based on the table, we propose two CI test cache table based PC (CTPC) learning frameworks for reducing redundant CI tests for BN structure learning. Finally, we instantiate the proposed frameworks with existing well-established local and global BN structure learning algorithms. Using twelve benchmark BNs, the extensive experiments have demonstrated that the proposed frameworks can significantly accelerate existing BN structure learning algorithms without sacrificing accuracy.

Learning Bayesian network structure with immune algorithm

The problem of structures learning in Bayesian networks is to discover a directed acyclic graph that in some sense is the best representation of the given database. Score-based learning algorithm is one of the important structure learning methods used to construct the Bayesian networks. These algorithms are implemented by using some heuristic search strategies to maximize the score of each candidate Bayesian network. In this paper, a bi-velocity discrete particle swarm optimization with mutation operator algorithm is proposed to learn Bayesian networks. The mutation strategy in proposed algorithm can efficiently prevent premature convergence and enhance the exploration capability of the population. We test the proposed algorithm on databases sampled from three well-known benchmark networks, and compare with other algorithms. The experimental results demonstrate the superiority of the proposed algorithm in learning Bayesian networks.

Bayesian Network is a knowledge representation formalism that has been proven to be valuable in gene regulatory network reconstruction. However, it is showed that the structure learning of Bayesian networks is an NP-hard problem. Several heuristic searching techniques have been used to find better network structures. Among these algorithms, K2 algorithm has been found to be the most successful. However, the performance of K2 algorithm is greatly affected by a prior ordering of input nodes. If using inappropriate orderings, the accuracy of the learned structures will be relatively low. This paper represents a new structure learning method to reconstruct gene networks from the observational gene expression data. The proposed method is based on Binary Particle Swarm Optimization and K2 algorithm. It firstly identifies nodes ordering from the experimental data by using Binary Particle Swarm Optimization algorithm. Then, the nodes ordering information is used as an input of K2 algorithm so as to improve the performance and efficiency of the K2 algorithm in determining the network structure. Our method is evaluated using two benchmark network datasets with known structures. Experimental studies show that the proposed method can identify networks more efficiently and accurately than comparable methods reported in the literature.