Symbolic Regression on Network Properties

<p>Symbolic regression (SR) is a function identification process, the task of which is to identify and express the relationship between the input and output variables in mathematical models. SR is named to emphasise its ability to find the structure and coefficients of the model simultaneously. Genetic Programming (GP) is an attractive and powerful technique for SR, since it does not require any predefined model and has a flexible representation. However, GP based SR generally has a poor generalisation ability which degrades its reliability and hampers its applications to science and real-world modeling. Therefore, this thesis aims to develop new GP approaches to SR that evolve/learn models exhibiting good generalisation ability. This thesis develops a novel feature selection method in GP for high-dimensional SR. Feature selection can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalisation ability. However, feature selection is seldom considered in GP for high-dimensional SR. The proposed new feature selection method utilises GP’s built-in feature selection ability and relies on permutation to detect the truly relevant features and discard irrelevant/noisy features. The results confirm the superiority of the proposed method over the other examined feature selection methods including random forests and decision trees on identifying the truly relevant features. Further analysis indicates that the models evolved by GP with the proposed feature selection method are more likely to contain only the truly relevant features and have better interpretability. To address the overfitting issue of GP when learning from a relatively small number of instances, this thesis proposes a new GP approach by incorporating structural risk minimisation (SRM), which is a framework to estimate the generalisation performance of models, into GP. The effectiveness of SRM highly depends on the accuracy of the Vapnik-Chervonenkis (VC) dimension measuring model complexity. This thesis significantly extends an experimental method (instead of theoretical estimation) to measure the VC-dimension of a mixture of linear and nonlinear regression models in GP for the first time. The experimental method has been conducted using uniform and non-uniform settings and provides reliable VC-dimension values. The results show that our methods have an impressively better generalisation gain and evolve more compact model, which have a much smaller behavioural difference from the target models than standard GP and GP with bootstrap, The proposed method using the optimised non-uniform setting further improves the one using the uniform setting. This thesis employs geometric semantic GP (GSGP) to tackle the unsatisfied generalisation performance of GP for SR when no overfitting occurs. It proposes three new angle-awareness driven geometric semantic operators (GSO) including selection, crossover and mutation to further explore the geometry of the semantic space to gain a greater generalisation improvement in GP for SR. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, to be resistant to overfitting. The results show that compared with two kinds of state-of-the-art GSOs, the proposed new GSOs not only drive the evolutionary process fitting the target semantics more efficiently but also significantly improve the generalisation performance. A further comparison on the evolved models shows that the new method generally produces simpler models with a much smaller size and containing important building blocks of the target models.</p>

Gradient-based local optimization has been shown to improve results of genetic programming (GP) for symbolic regression. Several state-of-the-art GP implementations use iterative nonlinear least squares (NLS) algorithms such as the Levenberg-Marquardt algorithm for local optimization. The effectiveness of NLS algorithms depends on appropriate scaling and conditioning of the optimization problem. This has so far been ignored in symbolic regression and GP literature. In this study we use a singular value decomposition of NLS Jacobian matrices to determine the numeric rank and the condition number. We perform experiments with a GP implementation and six different benchmark datasets. Our results show that rank-deficient and ill-conditioned Jacobian matrices occur frequently and for all datasets. The issue is less extreme when restricting GP tree size and when using many non-linear functions in the function set.

For network graphs, numerous graph features are intimately linked to eigenvalues of the Laplacian matrix, such as connectivity and diameter. Thus, it is very important to solve eigenvalues of the Laplacian matrix for graphs. Similarly, for higher-order networks, eigenvalues of combinatorial Laplacian matrices are also important for invariants of graphs. However, for large-scale networks, it is difficult to calculate eigenvalues of the Laplacian matrix directly because it is either very difficult to obtain the whole network structure or requires a lot of computing resources. Therefore, this article makes the following contributions. Firstly, this paper proposes a random walk approach for estimating the bounds of the greatest eigenvalues of Laplacian matrices for large-scale networks. Considering the relationship between the spectral moments of the adjacency matrix and the closed paths in the network, we utilize the relationship between the adjacency matrix and the Laplacian matrix to establish the relationship between the Laplacian matrix and the closed paths. Then, we employ equiprobable random walks to sample the large graph to obtain the small graph. Through algebraic topology knowledge, we obtain the bounds of the largest eigenvalue of the Laplacian matrix of the large graph by using Laplacian spectral moments of the small graph. Secondly, for high-order networks, this paper proposes a method based on random walks and graph transformations. The graph transformation we propose mainly converts graphs with second-order simplices into ordinary weighted graphs, thereby transforming the problem of solving the spectral moments of the second-order combined Laplacian matrix into solving the spectral moments of the adjacency matrix. Then, we use the aforementioned random walk method to solve bounds of the greatest eigenvalue of the second-order combinatorial Laplacian matrix. Finally, by comparing the proposed method with existing algorithms in synthetic and real networks, its accuracy and superiority are demonstrated.

<p><strong>The Cyber-Marine project aims to maximise the utility of commercial fish by using Artificial Intelligence (AI) supported tools and non-invasive chemical methods to assess their nutrient content. Vibrational Spectroscopy has proven to be a reliable method to determine molecular characteristics of a product. Particularly, it has gained prominence in evaluating the nutrient level of produce through statistical modelling techniques, such as Partial Least Squared regression (PLSR). This thesis investigates the use of Machine Learning (ML) to relate spectral data of fish with their nutrient levels. Within ML algorithms, Genetic Programming for symbolic regression (GPSR) emerges as a notable approach, producing models that exhibit generalisability and interpretability. As an extension of GPSR, multi-tree GP (MTGP), provides a platform that can simultaneously train models for several nutrients or bioactive components. In this thesis, significant enhancements and modifications are made to key aspects of MTGP, including its evolutionary process, by the introduction of new genetic operator, and the evaluation function, through the use of a distance metric. These are designed not only to leverage the relationship between the spectral information and bioactive components, but also to optimise the interplay between different target variables thereby maximising the capabilities of the algorithm.</strong></p>

When performing symbolic regression using genetic programming, overfitting and bloat can negatively impact generalizability and interpretability of the resulting equations as well as increase computation times. A Bayesian fitness metric is introduced and its impact on bloat and overfitting during population evolution is studied and compared to common alternatives in the literature. The proposed approach was found to be more robust to noise and data sparsity in numerical experiments, guiding evolution to a level of complexity appropriate to the dataset. Further evolution of the population resulted not in overfitting or bloat, but rather in slight simplifications in model form. The ability to identify an equation of complexity appropriate to the scale of noise in the training data was also demonstrated. In general, the Bayesian model selection algorithm was shown to be an effective means of regularization which resulted in less bloat and overfitting when any amount of noise was present in the training data.

Transfer learning has attracted more attention in the machine-learning community recently. It aims to improve the learning performance on the domain of interest with the help of the knowledge acquired from a similar domain(s). However, there is only a limited number of research on tackling transfer learning in genetic programming for symbolic regression. This article attempts to fill this gap by proposing a new instance weighting framework for transfer learning in genetic programming-based symbolic regression. In the new framework, differential evolution is employed to search for optimal weights for source-domain instances, which helps genetic programming to identify more useful source-domain instances and learn from them. Meanwhile, a density estimation method is used to provide good starting points to help the search for the optimal weights while discarding some irrelevant or less important source-domain instances before learning regression models. The experimental results show that compared with genetic programming and support vector regression that learn only from the target instances, and learning from a mixture of instances from the source and target domains without any transfer learning component, the proposed method can evolve regression models which not only achieve notably better cross-domain generalization performance in stability but also reduce the trend of overfitting effectively. Meanwhile, these models are generally much simpler than those generated by the other GP methods.

Most often, climate change signals are slow moving (in human terms), low amplitude changes embedded in high amplitude, noisy data. There are many techniques for extracting such signals. This paper introduces a technique which in contrast with empirical methods produces a decomposition of a time series into a set of equations plus a This is referred to as stepwise symbolic secomposition (SSD). The extraction of symbolic equations in lieu of empirical functions assists with characterization of time series. This paper takes two examples of climate data, and applies two different techniques for characterising the low amplitude, slow change embedded therein. The record of CO2 levels at Mauna-Loa since March 1958, is used to demonstrate and contrast empirical mode decomposition (EMD), and SSD. The mean monthly tidal gauge records from the small number of gauges which have more than 120 years of data are analysed in more detail by SSD, then three techniques are used to characterize the residual. The techniques are (a) LOESS smoothing, (b) EMD, and (c) high order polynomial regression. SSD uses a genetic programming system called Eureqa from Cornell Creative Machines Lab guided by an information metric, to extract the most compact informative function it can at each step; the process is repeated on the residuals until no sufficiently informative function is found. EMD and SSD are in stark contrast in the order in which signals are decomposed. EMD extracts high frequency components first. SSD extracts components based on a mixture of parsimonious representation and variance explained. EMD leaves a low frequency filtrate of the signal in its residue, SSD tends to operate as a broad band filter, leaving high frequency noise plus a possible low frequency signal. EMD, LOESS and polynomial fitting all serve to extract low frequency components of the signal from this SSD residual. For the set of tidal gauge data, in the absence of a change in global sea level rise, the SSD procedure should randomize all segments of residual signals equally - the residual should be whitened with respect to the initial signal. It is shown however that late 20th century portions of the residuals show behaviour that is consistent with accelerating sea level rise in keeping with the bulk of the literature.

Interpretable policies for reinforcement learning by genetic programming

Robustness and Optimization of Complex Networks: Reconstructability, Algorithms and Modeling

Social networks describe social interactions between people, which are often modeled by intersection graphs. In this paper, we propose an intersection graph model that is induced by adding a sparse random bipartite graph to a given bipartite graph. Under some mild conditions, we show that the vertex–isoperimetric number and the edge–isoperimetric number of the randomly perturbed intersection graph on n vertices are Ω ( 1 / ln n ) asymptomatically almost surely. Numerical simulations for small graphs extracted from two real-world social networks, namely, the board interlocking network and the scientific collaboration network, were performed. It was revealed that the effect of increasing isoperimetric numbers (i.e., expansion properties) on randomly perturbed intersection graphs is presumably independent of the order of the network.

PageRank is a popular measure of centrality that is often applied to rank nodes in real-world networks. However, in many cases, the notion of teleportation is counterintuitive because it implies that whatever is moving around the network will jump or ‘teleport’ directly from one node to any other, without considering how far apart the nodes are. To overcome this issue, we propose here a general measure of PageRank centrality whereby the teleportation probabilities depend, in some way, on the distance separating the nodes. We accomplish this by drawing upon recent advances in non-local random walks, which allow the proposed measure to be tailored for various real-world networks and applications. To illustrate the flexibility of the proposed measure and to demonstrate how it differs from PageRank centrality, we present and discuss experimental results for a selection of real-world spatial and social networks, including an air transportation network, a collaboration network and an urban street network.

Let Gr,p be a graph obtained from a path by adjoining a cycle Cr of length r to one end and the central vertex of a star Sp on p vertices to the other end.In this paper, it is proven that unicyclic graph Gr,p with r even is determined by its Laplacian spectrum except for n = p + 4.

PurposeThe purpose of this paper is to introduce a general equation for eigensolution. Eigenvalues and eigenvectors of graphs have many applications in combinatorial optimization and structural mechanics. Some important applications of graph products consist of nodal ordering and graph partitioning for structuring the structural matrices and finite element subdomaining, respectively.Design/methodology/approachIn the existing methods for the eigensolution of Laplacian matrices, members have been added to the model of a graph product such that for its Laplacian matrix an algebraic relation between blocks become possible. These methods are categorized as topological approaches. Here, using concepts of linear algebra a general algebraic method is developed.FindingsA new algebraic method is introduced for calculating the eigenvalues of Laplacian matrices in graph products.Originality/valueThe present method provides a simple tool for calculating the eigenvalues of the Laplacian matrices without using the configurational model and merely by using the Laplacian matrices. The developed formula for calculating the eigenvalues contains approximate terms which can be managed by the analyst.

The high feature dimensionality is a challenge in music emotion recognition. There is no common consensus on a relation between audio features and emotion. The MER system uses all available features to recognize emotion; however, this is not an optimal solution since it contains irrelevant data acting as noise. In this paper, we introduce a feature selection approach to eliminate redundant features for MER. We created a Selected Feature Set (SFS) based on the feature selection algorithm (FSA) and benchmarked it by training with two models, Support Vector Regression (SVR) and Random Forest (RF) and comparing them against with using the Complete Feature Set (CFS). The result indicates that the performance of MER has improved for both Random Forest (RF) and Support Vector Regression (SVR) models by using SFS. We found using FSA can improve performance in all scenarios, and it has potential benefits for model efficiency and stability for MER task.

The paper addresses the research question of automatic emotional speech recognition for Serbian. It integrates two research issues: (i) selection of an appropriate feature set, and (ii) investigation of different classification techniques. The paper reports a set of experiments with three feature sets: (i) the prosodic feature set, (ii) the spectral feature set, and (iii) the set of both spectral and prosodic features. The linear Bayes, the perceptron rule and the kNN classifier were considered in all three experiments. The experimental results show that the highest recognition accuracy of 91.5 % was obtained with the third feature set using the linear Bayes classifier.DOI: http://dx.doi.org/10.5755/j01.eee.18.9.2806