Constraint-based and hybrid structure learning of multidimensional continuous-time Bayesian network classifiers

One of the challenges practitioners face when applying structure learning algorithms to their data involves determining a set of hyperparameters; otherwise, a set of hyperparameter defaults is assumed. The optimal hyperparameter configuration often depends on multiple factors, including the size and density of the usually unknown underlying true graph, the sample size of the input data, and the structure learning algorithm. We propose a novel hyperparameter tuning method, called the Out-of-sample Tuning for Structure Learning (OTSL), that employs out-of-sample and resampling strategies to estimate the optimal hyperparameter configuration for structure learning, given the input dataset and structure learning algorithm. Synthetic experiments show that employing OTSL to tune the hyperparameters of hybrid and score-based structure learning algorithms leads to improvements in graphical accuracy compared to the state-of-the-art. We also illustrate the applicability of this approach to real datasets from different disciplines.

We consider the problem of learning structures and parameters of continuous-time Bayesian networks (CTBNs) from time-course data under minimal experimental resources. In practice, the cost of generating experimental data poses a bottleneck, especially in the natural and social sciences. A popular approach to overcome this is Bayesian optimal experimental design (BOED). However, BOED becomes infeasible in high-dimensional settings, as it involves integration over all possible experimental outcomes. We propose a novel criterion for experimental design based on a variational approximation of the expected information gain. We show that for CTBNs, a semi-analytical expression for this criterion can be calculated for structure and parameter learning. By doing so, we can replace sampling over experimental outcomes by solving the CTBNs master-equation, for which scalable approximations exist. This alleviates the computational burden of integrating over possible experimental outcomes in high-dimensions. We employ this framework in order to recommend interventional sequences. In this context, we extend the CTBN model to conditional CTBNs in order to incorporate interventions. We demonstrate the performance of our criterion on synthetic and real-world data.

A hybrid algorithm for Bayesian network structure learning with application to multi-label learning

Bayesian networks are efficient classification techniques, and widely applied in many fields, however, their structure learning is NP-hard. In this paper, a Bayesian network structure learning method called Tree-like Bayesian network (BN-TL) was proposed, which constructs the network by estimating the correlation between the features and the correlation between the class label and the features. Two metabolomics datasets about liver disease and five public datasets from the University of California at Irvine repository (UCI) were used to demonstrate the performance of BN-TL. The result shows that BN-TL outperforms the other three classifiers, including Naïve Bayesian classifier (NB), Bayesian network classifier whose structure is learned by using K2 greedy search strategy (BN-K2) and a method proposed by Kuschner in 2010 (BN-BMC) in most cases.

A fully automatic feedforward neural network structural and weight learning algorithm is described. The Augmentation by Training with Residuals, ATR, requires neither guess of initial weight values nor the number of neurons in the hidden layer from users. The algorithm takes an incremental approach in which a hidden neuron is trained to model the mapping between the input and output of current exemplars, and is augmented to the existing network. The exemplars are then made orthogonal to the newly identified hidden neuron and used for the training of next hidden neuron. The improvement continues until a desired accuracy is reached. This new structural and weight learning algorithm is applied to the identification of a two-degree-of-freedom planar robot, a Van der Pol oscillator and a Mackay-Glass equation. The algorithm is shown to be effective in modeling all three systems and is far superior to a linear modeling scheme in the case of the robot.

In order to solve the problems that the intelligence algorithm falls into the local optimum easily and has a slow convergence in Bayesian networks (BN) structure learning, an algorithm based on adaptive selection operator with simulated annealing is proposed. This chapter conducts the adaptive selection rule in combination with conditional independence tests of BN nodes to guide the generation of neighbor. In order to better compare the adaptive effect, an algorithm based on selection operator with simulated annealing (SOSA) is proposed; at the same time 15 data sets in the three typical networks are accessed as learning samples. The results of the Bayesian Dirichlet (BD) score, Hamming distance (HD), and evolution time of the network after learning show that it has the quicker convergence and it searches the optimal solution more easily compared with simulated annealing (SA) and SOSA.

Constructing the structure of protein signaling networks by Bayesian network technology is a key issue in the field of bioinformatics. The primitive structure learning algorithms of the Bayesian network take no account of the causal relationships between variables, which is unfortunately important in the application of protein signaling networks. In addition, as a combinatorial optimization problem with a large searching space, the computational complexities of the structure learning algorithms are unsurprisingly high. Therefore, in this paper, the causal directions between any two variables are calculated first and stored in a graph matrix as one of the constraints of structure learning. A continuous optimization problem is constructed next by using the fitting losses of the corresponding structure equations as the target, and the directed acyclic prior is used as another constraint at the same time. Finally, a pruning procedure is developed to keep the result of the continuous optimization problem sparse. Experiments show that the proposed method improves the structure of the Bayesian network compared with the existing methods on both the artificial data and the real data, meanwhile, the computational burdens are also reduced significantly.

Ensuring fairness when training Machine Learning (ML) models remains a critical challenge, particularly when biases are embedded in the underlying data. This paper presents a fairness-aware graph structure learning framework demonstrating how learning fair graphs leads to fairer data for ML training and, consequently, fairer Artificial Intelligence (AI) decisioning based on such models. Our method incorporates a fairness regularization term into score-based structure learning algorithms, guiding the search towards graph structures that minimize discriminatory pathways while preserving statistical relationships. The learned fair graph structures enable the generation of synthetic datasets with mitigated biases, which can be used to train diverse ML models. This modification is non-trivial, as structure learning algorithms rely on local search strategies, while fairness is a global property that depends on the entire graph structure. Our framework is highly adaptable, compatible with various structure learning algorithms, and seamlessly incorporates different fairness metrics to meet specific contextual needs. Extensive experiments on both real-world and synthetic datasets demonstrate that our approach significantly improves fairness while maintaining competitive predictive performance, offering an interpretable and versatile solution for mitigating bias in AI systems..

Prediction models are built with various machine learning algorithms to identify defects prior to release to facilitate software testing, and save testing costs. Naive Bayes classifier is one of the best performing classification techniques in defect prediction. It assumes conditional independence of features and for defect prediction problem some of the features are not actually conditionally independent. The interesting problem is to relax these conditional independence assumptions and to check whether there is any improvement in performance of classifiers. We have built Bayesian Network structures using different classes of algorithms namely score-based, constraint-based and hybrid algorithms. We propose an approach to augment these Bayesian Network structures with class node. Bayesian Network classifiers along with Random Forests, Logistic Regression and Naive Bayes classifiers are then evaluated using measures like AUC and H-measure. We observe that RSMAX2 and Grow-Shrink classifiers (after augmentation) perform consistently better in defect prediction.

We present a hybrid algorithm called parallel simulated annealing with a greedy algorithm (PSAGA) to learn Bayesian network structures. This work focuses on simulated annealing and its parallelization with memoization to accelerate the search process. At each step of the local search, a hybrid search method combining simulated annealing with a greedy algorithm was adopted. The proposed PSAGA aims to achieve both the efficiency of parallel search and the effectiveness of a more exhaustive search. The Bayesian Dirichlet equivalence metric was used to determine an optimal structure for PSAGA. The proposed PSAGA was evaluated on seven well-known Bayesian network benchmarks generated at random. We first conducted experiments to evaluate the computational time performance of the proposed parallel search. We then compared PSAGA with existing variants of simulated annealing-based algorithms to evaluate the quality of the learned structure. Overall, the experimental results demonstrate that the proposed PSAGA shows better performance than the alternatives in terms of computational time and accuracy.

The type of constraint-based methods is one of the most important approaches to learn Bayesian network (BN) structures from observational data with conditional independence (CI) tests. In this paper, we find that existing constraint-based methods often perform many redundant CI tests, which significantly reduces the learning efficiency of those algorithms. To tackle this issue, we propose a novel framework to accelerate BN structure learning by reducing redundant CI tests without sacrificing accuracy. Specifically, we first design a CI test cache table to store CI tests. If a CI test has been computed before, the result of the CI test is obtained from the table instead of computing the CI test again. If not, the CI test is computed and stored in the table. Then based on the table, we propose two CI test cache table based PC (CTPC) learning frameworks for reducing redundant CI tests for BN structure learning. Finally, we instantiate the proposed frameworks with existing well-established local and global BN structure learning algorithms. Using twelve benchmark BNs, the extensive experiments have demonstrated that the proposed frameworks can significantly accelerate existing BN structure learning algorithms without sacrificing accuracy.

Discovering ncRNA genes is a challenging problem, which has attracted much attention recently. The accuracy of computational ncRNA prediction methods still needs to be improved, however, due to the diversity and the lack of consensus patterns of ncRNA genes. In this paper, we propose an effective computational approach based on fuzzy neural networks with structure learning (FNNSL) for novel ncRNA gene prediction. It has advantages such as explicit physical meanings of nodes and parameters in the network, and effective incorporation of prior knowledge by the fuzzy sets theory. Specifically, a structure learning algorithm is presented to decrease parameter dimensions, enhance the computational efficiency, and avoid the over-learning. In addition, a fuzzy c-means clustering method is adopted for fuzzy partitioning of input feature variables, and the corresponding implementations are compared to the other ncRNA gene prediction tools. The improved prediction accuracy demonstrates the effectiveness of the proposed approach.

Background/Aim: Identifying confounders based on causal directed acyclic graphs (DAGs) is usually based solely on expert knowledge, although epidemiological studies provide observational data. We applied data-based structure learning algorithms for the specific purpose of identifying confounders, investigating the research question whether particulate matter with an aerodynamic diameter ≤2.5 µg/m3 (PM2.5) affects progression of atherosclerosis, measured as yearly change in carotid intima media thickness (ΔcIMT). Methods: We used 5-year follow-up data of the population-based Heinz Nixdorf Recall Study. Due to computational limitations, potentially relevant variables were restricted to continuously measured data and hence included age, BMI, smoking, blood pressure, HbA1c, LDL/HDL, education, income, neighborhood unemployment (nSES) rate, physical activity and alcohol consumption. Prior knowledge was based on causal chain assumptions regarding major causes of ischaemic heart disease1, and translated to prohibited arcs, i.e. given conditions (e.g. age) are unaffected by lifestyle variables (e.g. smoking) and lifestyle variables are unaffected by subclinical medical determinants (e.g. laboratory measures). We applied a revised structure learning algorithm to learn a causal DAG, including the score-based hill-climbing algorithm, a bootstrap-check to validate the conditional independence structure between variables, and a clustering of minimal sufficient adjustment set (MSAS)-equivalent subsets within the Markov equivalence-class. Results: Based on 2279 participants, the revised structure learning resulted in one MSAS-equivalent cluster, including one DAG of 59 unambiguous arcs. The only biasing path from PM2.5 to ΔcIMT emanated from nSES. Suggested adjustment consisted of four confounder-equivalent MSAS ({nSES}; {age, physical activity, blood pressure}; {age, alcohol consumption, BMI, physical activity, smoking}; {age, BMI, education, income, physical activity, smoking}), yielding an identical exposure effect estimate of 0.002 mm (95%-CI: -0.002, 0.007) increase in cIMT per year per 5 µg/m3 increase in PM2.5. Conclusions: Data-based structure learning returned one unambiguous DAG widely matching general believes. Identical exposure effect estimates of confounder-equivalent MSAS underline a stable (causal) structure.

We have had to wait over 30 years since the naive Bayes model was first introduced in 1960 for the so-called Bayesian network classifiers to resurge. Based on Bayesian networks, these classifiers have many strengths, like model interpretability, accommodation to complex data and classification problem settings, existence of efficient algorithms for learning and classification tasks, and successful applicability in real-world problems. In this article, we survey the whole set of discrete Bayesian network classifiers devised to date, organized in increasing order of structure complexity: naive Bayes, selective naive Bayes, seminaive Bayes, one-dependence Bayesian classifiers, k -dependence Bayesian classifiers, Bayesian network-augmented naive Bayes, Markov blanket-based Bayesian classifier, unrestricted Bayesian classifiers, and Bayesian multinets. Issues of feature subset selection and generative and discriminative structure and parameter learning are also covered.

In authors' previous works, a novel self-organizing neuro-fuzzy multilayered classifier (SONeFMUC) was proposed. SONeFMUC is composed of small-scale interconnected fuzzy neuron classifiers (FNCs) arranged in layers. The structure of the classifier is revealed by means of the well known GMDH algorithm. In addition, the GMDH algorithm inherently implements feature selection, considering the most informative attributes as model inputs. However, previous simulation results indicate that the GMDH algorithm calculates a large number of FNCs with slightly higher or even the same classification capabilities than its parents. Hence, the computational cost of the GMDH is large without a direct impact to the classification accuracy. In this paper, a modified version of GMDH is proposed for an effective identification of the structure of SONeFMUC with reduced computational cost. To this end, a statistical measure of agreement of the generic FNCs in classifying the patterns of the problem is used. This measure is known as Proportion of Specific Agreement (Ps). Hence, only complementary FNCs are combined to construct a descendant FNC at the next layer and the total number of constructed FNCs is reduced. The proposed structure learning algorithm is tested on a well known classification problem of the literature, the forensic glass. Simulation results indicate the efficiency of the proposed algorithm.