Tuning structure learning algorithms with out-of-sample and resampling strategies

Learning the structure of continuous-time Bayesian networks directly from data has traditionally been performed using score-based structure learning algorithms. Only recently has a constraint-based method been proposed, proving to be more suitable under specific settings, as in modelling systems with variables having more than two states. As a result, studying diverse structure learning algorithms is essential to learn the most appropriate models according to data characteristics and task-related priorities, such as learning speed or accuracy. This article proposes alternative algorithms for learning multidimensional continuous-time Bayesian network classifiers, introducing, for the first time, constraint-based and hybrid algorithms for these models. Nevertheless, these contributions also apply to the simpler one-dimensional classification problem for which only score-based solutions exist in the literature. More specifically, the aforementioned constraint-based structure learning algorithm is first adapted to the supervised classification setting. Then, a novel algorithm of this kind, specifically tailored for the multidimensional classification problem, is presented to improve the learning times for the induction of multidimensional classifiers. Finally, a hybrid algorithm is introduced, attempting to combine the strengths of the score- and constraint-based approaches. Experiments with synthetic and real-world data are performed not only to validate the capabilities of the proposed algorithms but also to conduct a comparative study of the available competitors.

A fully automatic feedforward neural network structural and weight learning algorithm is described. The Augmentation by Training with Residuals, ATR, requires neither guess of initial weight values nor the number of neurons in the hidden layer from users. The algorithm takes an incremental approach in which a hidden neuron is trained to model the mapping between the input and output of current exemplars, and is augmented to the existing network. The exemplars are then made orthogonal to the newly identified hidden neuron and used for the training of next hidden neuron. The improvement continues until a desired accuracy is reached. This new structural and weight learning algorithm is applied to the identification of a two-degree-of-freedom planar robot, a Van der Pol oscillator and a Mackay-Glass equation. The algorithm is shown to be effective in modeling all three systems and is far superior to a linear modeling scheme in the case of the robot.

A hybrid algorithm for Bayesian network structure learning with application to multi-label learning

Ensuring fairness when training Machine Learning (ML) models remains a critical challenge, particularly when biases are embedded in the underlying data. This paper presents a fairness-aware graph structure learning framework demonstrating how learning fair graphs leads to fairer data for ML training and, consequently, fairer Artificial Intelligence (AI) decisioning based on such models. Our method incorporates a fairness regularization term into score-based structure learning algorithms, guiding the search towards graph structures that minimize discriminatory pathways while preserving statistical relationships. The learned fair graph structures enable the generation of synthetic datasets with mitigated biases, which can be used to train diverse ML models. This modification is non-trivial, as structure learning algorithms rely on local search strategies, while fairness is a global property that depends on the entire graph structure. Our framework is highly adaptable, compatible with various structure learning algorithms, and seamlessly incorporates different fairness metrics to meet specific contextual needs. Extensive experiments on both real-world and synthetic datasets demonstrate that our approach significantly improves fairness while maintaining competitive predictive performance, offering an interpretable and versatile solution for mitigating bias in AI systems..

Discovering ncRNA genes is a challenging problem, which has attracted much attention recently. The accuracy of computational ncRNA prediction methods still needs to be improved, however, due to the diversity and the lack of consensus patterns of ncRNA genes. In this paper, we propose an effective computational approach based on fuzzy neural networks with structure learning (FNNSL) for novel ncRNA gene prediction. It has advantages such as explicit physical meanings of nodes and parameters in the network, and effective incorporation of prior knowledge by the fuzzy sets theory. Specifically, a structure learning algorithm is presented to decrease parameter dimensions, enhance the computational efficiency, and avoid the over-learning. In addition, a fuzzy c-means clustering method is adopted for fuzzy partitioning of input feature variables, and the corresponding implementations are compared to the other ncRNA gene prediction tools. The improved prediction accuracy demonstrates the effectiveness of the proposed approach.

Constructing the structure of protein signaling networks by Bayesian network technology is a key issue in the field of bioinformatics. The primitive structure learning algorithms of the Bayesian network take no account of the causal relationships between variables, which is unfortunately important in the application of protein signaling networks. In addition, as a combinatorial optimization problem with a large searching space, the computational complexities of the structure learning algorithms are unsurprisingly high. Therefore, in this paper, the causal directions between any two variables are calculated first and stored in a graph matrix as one of the constraints of structure learning. A continuous optimization problem is constructed next by using the fitting losses of the corresponding structure equations as the target, and the directed acyclic prior is used as another constraint at the same time. Finally, a pruning procedure is developed to keep the result of the continuous optimization problem sparse. Experiments show that the proposed method improves the structure of the Bayesian network compared with the existing methods on both the artificial data and the real data, meanwhile, the computational burdens are also reduced significantly.

Background/Aim: Identifying confounders based on causal directed acyclic graphs (DAGs) is usually based solely on expert knowledge, although epidemiological studies provide observational data. We applied data-based structure learning algorithms for the specific purpose of identifying confounders, investigating the research question whether particulate matter with an aerodynamic diameter ≤2.5 µg/m3 (PM2.5) affects progression of atherosclerosis, measured as yearly change in carotid intima media thickness (ΔcIMT). Methods: We used 5-year follow-up data of the population-based Heinz Nixdorf Recall Study. Due to computational limitations, potentially relevant variables were restricted to continuously measured data and hence included age, BMI, smoking, blood pressure, HbA1c, LDL/HDL, education, income, neighborhood unemployment (nSES) rate, physical activity and alcohol consumption. Prior knowledge was based on causal chain assumptions regarding major causes of ischaemic heart disease1, and translated to prohibited arcs, i.e. given conditions (e.g. age) are unaffected by lifestyle variables (e.g. smoking) and lifestyle variables are unaffected by subclinical medical determinants (e.g. laboratory measures). We applied a revised structure learning algorithm to learn a causal DAG, including the score-based hill-climbing algorithm, a bootstrap-check to validate the conditional independence structure between variables, and a clustering of minimal sufficient adjustment set (MSAS)-equivalent subsets within the Markov equivalence-class. Results: Based on 2279 participants, the revised structure learning resulted in one MSAS-equivalent cluster, including one DAG of 59 unambiguous arcs. The only biasing path from PM2.5 to ΔcIMT emanated from nSES. Suggested adjustment consisted of four confounder-equivalent MSAS ({nSES}; {age, physical activity, blood pressure}; {age, alcohol consumption, BMI, physical activity, smoking}; {age, BMI, education, income, physical activity, smoking}), yielding an identical exposure effect estimate of 0.002 mm (95%-CI: -0.002, 0.007) increase in cIMT per year per 5 µg/m3 increase in PM2.5. Conclusions: Data-based structure learning returned one unambiguous DAG widely matching general believes. Identical exposure effect estimates of confounder-equivalent MSAS underline a stable (causal) structure.

Traditional power distribution networks suffer from a lack of real-time observability. This complicates development and implementation of new smart-grid technologies, such as those related to demand response, outage detection and management, and improved load monitoring. In this paper, inspired by proliferation of metering technology, we discuss topology estimation problems in structurally loopy but operationally radial distribution grids from measurements, for example, voltage data, which are either already available or can be made available with a relatively minor investment. The primary objective of this paper is to learn the operational layout of the grid. Further, the structure learning algorithm is extended to cases with missing data, where available observations are limited to a fraction of the grid nodes. The algorithms are computationally efficient—polynomial in time—which is proven theoretically and illustrated in numerical experiments on a number of test cases. The techniques developed can be applied to detect line failures in real time as well as to understand the scope of possible adversarial attacks on the grid.

Learning large network (with hundreds of variables) is gaining interest of many researchers with the emergence of high-throughput biological data sources such as micro-array data. In this paper, we investigated the two popular large scale network structure learning algorithms, sparse candidate hill climbing (SCHC) and Grow-Shrinkage(GS) algorithm. The experiments show that in fact both of them have serious effectiveness problems when the number of variables(genes) is large compared to the number of instances(experimental conditions), which is a common case in micro-array data. We further propose a new large scale structure learning algorithm based on Lasso regression. Theoretical analysis in [10] suggested that the L1-norm in lasso regression could make our algorithm especially suitable in the cases that the number of variables and instances is unbalance. Our algorithm achieves much better results than SCHC and GS on the synthetic data. We also show the effectiveness of our algorithm by learning genetic regulatory network modules from a real micro-array data (with more than 6000 genes), combined with the genome-wide location analysis data. The learned results are consistent well with biological knowledge.

The Kullback–Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and Rössler systems.

An empirical comparison of popular structure learning algorithms with a view to gene network inference

: The main objective of this project was to design, implement and evaluate new methods for performing pattern learning on structured data represented as graphs and evaluate their application to structural, relational databases relevant to the Evidence Assessment, Grouping, Linking and Evaluation (EAGLE) program. This work builds on existing methods for graph-based knowledge discovery and concept learning implemented in the SUBDUE structural pattern learning system. The graph-based structural pattern learning algorithm was extended to perform structural concept learning and structural, hierarchical conceptual clustering. The resulting system was evaluated using several structural databases, including those with known structural patterns, those of relevance to the target domains of the EAGLE program, and those developed as challenge problems within the EAGLE program.

This paper focuses on how to perform unsupervised learning of tree structures in an information theoretic setting. The approach is a purely structural one and is designed to work with representations where the correspondences between nodes are not given, but must be inferred from the structure. This is in contrast with other structural learning algorithms where the node-correspondences are assumed to be known. The learning process fits a mixture of structural models to a set of samples using a minimum descriptor length formulation. The method extracts both a structural archetype that describes the observed structural variation, and the node-correspondences that map nodes from trees in the sample set to nodes in the structural model. We use the algorithm to classify a set of shapes based on their shock graphs.

The success of deep learning in skin lesion classification mainly depends on the ultra-deep neural network and the significantly large training data set. Deep learning training is usually time-consuming, and large datasets with labels are hard to obtain, especially skin lesion images. Although pre-training and data augmentation can alleviate these issues, there are still some problems: (1) the data domain is not consistent, resulting in the slow convergence; and (2) low robustness to confusing skin lesions. To solve these problems, we propose an efficient structural pseudoinverse learning-based hierarchical representation learning method. Preliminary feature extraction, shallow network feature extraction and deep learning feature extraction are carried out respectively before the classification of skin lesion images. Gabor filter and pre-trained deep convolutional neural network are used for preliminary feature extraction. The structural pseudoinverse learning (S-PIL) algorithm is used to extract the shallow features. Then, S-PIL preliminarily identifies the skin lesion images that are difficult to be classified to form a new training set for deep learning feature extraction. Through the hierarchical representation learning, we analyze the features of skin lesion images layer by layer to improve the final classification. Our method not only avoid the slow convergence caused by inconsistency of data domain but also enhances the training of confusing examples. Without using additional data, our approach outperforms existing methods in the ISIC 2017 and ISIC 2018 datasets.

In order to solve the problems that the intelligence algorithm falls into the local optimum easily and has a slow convergence in Bayesian networks (BN) structure learning, an algorithm based on adaptive selection operator with simulated annealing is proposed. This chapter conducts the adaptive selection rule in combination with conditional independence tests of BN nodes to guide the generation of neighbor. In order to better compare the adaptive effect, an algorithm based on selection operator with simulated annealing (SOSA) is proposed; at the same time 15 data sets in the three typical networks are accessed as learning samples. The results of the Bayesian Dirichlet (BD) score, Hamming distance (HD), and evolution time of the network after learning show that it has the quicker convergence and it searches the optimal solution more easily compared with simulated annealing (SA) and SOSA.