Local Optimization Often is Ill-conditioned in Genetic Programming for Symbolic Regression

<p>Symbolic regression (SR) is a function identification process, the task of which is to identify and express the relationship between the input and output variables in mathematical models. SR is named to emphasise its ability to find the structure and coefficients of the model simultaneously. Genetic Programming (GP) is an attractive and powerful technique for SR, since it does not require any predefined model and has a flexible representation. However, GP based SR generally has a poor generalisation ability which degrades its reliability and hampers its applications to science and real-world modeling. Therefore, this thesis aims to develop new GP approaches to SR that evolve/learn models exhibiting good generalisation ability. This thesis develops a novel feature selection method in GP for high-dimensional SR. Feature selection can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalisation ability. However, feature selection is seldom considered in GP for high-dimensional SR. The proposed new feature selection method utilises GP’s built-in feature selection ability and relies on permutation to detect the truly relevant features and discard irrelevant/noisy features. The results confirm the superiority of the proposed method over the other examined feature selection methods including random forests and decision trees on identifying the truly relevant features. Further analysis indicates that the models evolved by GP with the proposed feature selection method are more likely to contain only the truly relevant features and have better interpretability. To address the overfitting issue of GP when learning from a relatively small number of instances, this thesis proposes a new GP approach by incorporating structural risk minimisation (SRM), which is a framework to estimate the generalisation performance of models, into GP. The effectiveness of SRM highly depends on the accuracy of the Vapnik-Chervonenkis (VC) dimension measuring model complexity. This thesis significantly extends an experimental method (instead of theoretical estimation) to measure the VC-dimension of a mixture of linear and nonlinear regression models in GP for the first time. The experimental method has been conducted using uniform and non-uniform settings and provides reliable VC-dimension values. The results show that our methods have an impressively better generalisation gain and evolve more compact model, which have a much smaller behavioural difference from the target models than standard GP and GP with bootstrap, The proposed method using the optimised non-uniform setting further improves the one using the uniform setting. This thesis employs geometric semantic GP (GSGP) to tackle the unsatisfied generalisation performance of GP for SR when no overfitting occurs. It proposes three new angle-awareness driven geometric semantic operators (GSO) including selection, crossover and mutation to further explore the geometry of the semantic space to gain a greater generalisation improvement in GP for SR. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, to be resistant to overfitting. The results show that compared with two kinds of state-of-the-art GSOs, the proposed new GSOs not only drive the evolutionary process fitting the target semantics more efficiently but also significantly improve the generalisation performance. A further comparison on the evolved models shows that the new method generally produces simpler models with a much smaller size and containing important building blocks of the target models.</p>

Transfer learning has attracted more attention in the machine-learning community recently. It aims to improve the learning performance on the domain of interest with the help of the knowledge acquired from a similar domain(s). However, there is only a limited number of research on tackling transfer learning in genetic programming for symbolic regression. This article attempts to fill this gap by proposing a new instance weighting framework for transfer learning in genetic programming-based symbolic regression. In the new framework, differential evolution is employed to search for optimal weights for source-domain instances, which helps genetic programming to identify more useful source-domain instances and learn from them. Meanwhile, a density estimation method is used to provide good starting points to help the search for the optimal weights while discarding some irrelevant or less important source-domain instances before learning regression models. The experimental results show that compared with genetic programming and support vector regression that learn only from the target instances, and learning from a mixture of instances from the source and target domains without any transfer learning component, the proposed method can evolve regression models which not only achieve notably better cross-domain generalization performance in stability but also reduce the trend of overfitting effectively. Meanwhile, these models are generally much simpler than those generated by the other GP methods.

Interpretable policies for reinforcement learning by genetic programming

Most often, climate change signals are slow moving (in human terms), low amplitude changes embedded in high amplitude, noisy data. There are many techniques for extracting such signals. This paper introduces a technique which in contrast with empirical methods produces a decomposition of a time series into a set of equations plus a This is referred to as stepwise symbolic secomposition (SSD). The extraction of symbolic equations in lieu of empirical functions assists with characterization of time series. This paper takes two examples of climate data, and applies two different techniques for characterising the low amplitude, slow change embedded therein. The record of CO2 levels at Mauna-Loa since March 1958, is used to demonstrate and contrast empirical mode decomposition (EMD), and SSD. The mean monthly tidal gauge records from the small number of gauges which have more than 120 years of data are analysed in more detail by SSD, then three techniques are used to characterize the residual. The techniques are (a) LOESS smoothing, (b) EMD, and (c) high order polynomial regression. SSD uses a genetic programming system called Eureqa from Cornell Creative Machines Lab guided by an information metric, to extract the most compact informative function it can at each step; the process is repeated on the residuals until no sufficiently informative function is found. EMD and SSD are in stark contrast in the order in which signals are decomposed. EMD extracts high frequency components first. SSD extracts components based on a mixture of parsimonious representation and variance explained. EMD leaves a low frequency filtrate of the signal in its residue, SSD tends to operate as a broad band filter, leaving high frequency noise plus a possible low frequency signal. EMD, LOESS and polynomial fitting all serve to extract low frequency components of the signal from this SSD residual. For the set of tidal gauge data, in the absence of a change in global sea level rise, the SSD procedure should randomize all segments of residual signals equally - the residual should be whitened with respect to the initial signal. It is shown however that late 20th century portions of the residuals show behaviour that is consistent with accelerating sea level rise in keeping with the bulk of the literature.

<p><strong>The Cyber-Marine project aims to maximise the utility of commercial fish by using Artificial Intelligence (AI) supported tools and non-invasive chemical methods to assess their nutrient content. Vibrational Spectroscopy has proven to be a reliable method to determine molecular characteristics of a product. Particularly, it has gained prominence in evaluating the nutrient level of produce through statistical modelling techniques, such as Partial Least Squared regression (PLSR). This thesis investigates the use of Machine Learning (ML) to relate spectral data of fish with their nutrient levels. Within ML algorithms, Genetic Programming for symbolic regression (GPSR) emerges as a notable approach, producing models that exhibit generalisability and interpretability. As an extension of GPSR, multi-tree GP (MTGP), provides a platform that can simultaneously train models for several nutrients or bioactive components. In this thesis, significant enhancements and modifications are made to key aspects of MTGP, including its evolutionary process, by the introduction of new genetic operator, and the evaluation function, through the use of a distance metric. These are designed not only to leverage the relationship between the spectral information and bioactive components, but also to optimise the interplay between different target variables thereby maximising the capabilities of the algorithm.</strong></p>

When performing symbolic regression using genetic programming, overfitting and bloat can negatively impact generalizability and interpretability of the resulting equations as well as increase computation times. A Bayesian fitness metric is introduced and its impact on bloat and overfitting during population evolution is studied and compared to common alternatives in the literature. The proposed approach was found to be more robust to noise and data sparsity in numerical experiments, guiding evolution to a level of complexity appropriate to the dataset. Further evolution of the population resulted not in overfitting or bloat, but rather in slight simplifications in model form. The ability to identify an equation of complexity appropriate to the scale of noise in the training data was also demonstrated. In general, the Bayesian model selection algorithm was shown to be an effective means of regularization which resulted in less bloat and overfitting when any amount of noise was present in the training data.

Networks are continuously growing in complexity, which creates challenges for determining their most important characteristics. While analytical bounds are often too conservative, the computational effort of algorithmic approaches does not scale well with network size. This work uses Cartesian Genetic Programming for symbolic regression to evolve mathematical equations that relate network properties directly to the eigenvalues of network adjacency and Laplacian matrices. In particular, we show that these eigenvalues are powerful features to evolve approximate equations for the network diameter and the isoperimetric number, which are hard to compute algorithmically. Our experiments indicate a good performance of the evolved equations for several real-world networks and we demonstrate how the generalization power can be influenced by the selection of training networks and feature sets.

This paper addresses the challenging problem of designing an adaptive feedback strategy for the stabilization of a class of nonlinearly parameterized uncertain systems in discrete time. The nonlinear least squares (NLS) algorithm is applied to estimate the unknown parameters, and a sufficient condition is proposed to ensure the strong consistency of the estimator. Based on the NLS algorithm, the global stability of the system is achieved under the output feedback design.

Genetic Programming (GP) for symbolic regression is often prone to overfitting the training data, causing poor performance on unseen data. A number of recent works in the field have been devoted to regulating this problem by investigating both the structural and functional complexity of GP individuals during the evolutionary process. This work uses the Rademacher complexity and incorporates it into the fitness function of GP, utilising it as a means of controlling the functional complexity of GP individuals. The experiment results confirm that the new GP method has a notable generalization gain compared to the standard GP and Support Vector Regression (SVR) in most of the considered problems. Further investigations also show that the new GP method generates symbolic regression models that could not only release the overfitting trend in standard GP but also are significantly smaller in size compared to their counterparts in standard GP.

Customized prediction of attendance to soccer matches based on symbolic regression and genetic programming

The paper addresses a new implementation of genetic programming by using molecular approach. Our method is based on dataflow techniques in DNA computing. After description of fundamental operations on DNA molecules and construction of logical functions the genetic programming method is introduced. We propose a way to handle these graph encoding molecules and which can be considered a genetic programming algorithm; a short discussion about experiments in implementing parts of this procedure is added.

Symbolic regression is a process to find a mathematical expression that represents the relationship between a set of explanatory variables and a measured variable. It has become a best-known problem for GP (genetic programming), as GP can use the tree representation to represent solutions as expression trees. Since the success of memetic algorithms (MAs (Memetic algorithms (MAs) can be regarded as a class of methods that combine population-based global search and local search [6, 30])) has proved the importance of local search in augmenting the global search ability of GP, GP with local search is investigated to solve symbolic regression tasks in this work. An important design issue of MAs is the balance between the global exploration of GP and the local exploitation, which has a great influence on the performance and efficiency of MAs. This work proposes a GP-based memetic algorithm for symbolic regression, termed as aMeGP (a daptive Me metic GP), which can balance global exploration and local exploitation adaptively. Compared with GP, two improvements are made in aMeGP to invoke and stop local search adaptively during evolution. The proposed aMeGP is compared with GP-based and nonGP-based symbolic regression methods on both benchmark test functions and real-world applications. The results show that aMeGP is generally better than both GP-based and nonGP-based reference methods with its evolved solutions achieving lower root mean square error (RMSE) for most test cases. Moreover, aMeGP outperforms the reference GP-based methods in the convergence ability, which can converge to lower RMSE values with faster or similar speeds.

This paper investigates the application of genetic programming (GP) for dynamic symbolic regression (SR), addressing the challenge of adapting machine learning models to evolving data in practical applications. Benchmark instances with changing underlying functions over time are defined to assess the performance of a genetic algorithm (GA) as a traditional evolutionary algorithm and an age-layered population structure (ALPS) as an open-ended evolutionary algorithm for dynamic symbolic regression. This study analyzes population dynamics by examining variable frequencies and impact changes over time in response to dynamic shifts in the training data. The results demonstrate the effectiveness of both the GA and ALPS in handling changing data, showcasing their ability to recover and evolve improved solutions after an initial drop in population quality following data changes. Population dynamics reveal that variable impacts respond rapidly to data changes, while variable frequencies shift gradually across generations, aligning with the indirect measure of fitness represented by variable impacts. Notably, the GA shows a strong dependence on mutation to avoid variables becoming permanently extinct, contrasting with the ALPS’s unexpected insensitivity to mutation rates owing to its reseeding mechanism for effective variable reintroduction.

The paper gives an overview of non-linear least square method, and presents GPS satellite pseudorange observation equations, which are usually over-determined because there are only 4 unknowns but normally more than 4 satellites being tracked, resulting in more than 4 pseudorange observation equations. Non-linear least squares(NLS) algorithm to solve pseudorange observation equations in order to find position and clock bias solutions are described in detail. There are three tests designed, one vehicle field test and two simulation tests. There are three NovAtel made GPS receivers being used in vehicle field test, while GSS6560 GPS satellite signal generator being used in simulation tests. By employing a set of differential GPS receivers, it is possible to establish rover station receiver to give a relatively high accurate reference to be compared by NLS and another GPS receiver in the vehicle field test. Results from NLS demonstrate that NLS can achieve high accuracy. Experimental study also shows that NLS algorithm possesses similar accuracy level to that of NovAtel Receiver, which represents world leading level.

This article presents a new formulation for attitude-independent calibration of three-axis magnetometers. The proposed solution employs a nonlinear least squares (NLLS) estimator that uses the nonlinear sensor model proposed by Foster and Elkaim in the well-known extended two-step (ETS) algorithm. Our solution has a few advantages over ETS and is more straightforward to implement than many subsequent algorithms. It enables the calculation of the calibration parameters’ uncertainties and does not require the assumption that all magnetometer measurements have a constant magnetic field magnitude. The former is essential for evaluating calibration quality, while the latter may be required for on-orbit magnetometer calibration. However, the NLLS has the drawback of requiring an initial parameter estimation to ensure convergence. Therefore, our work presents a new technique to calculate the scale factors and offsets for initializing the NLLS. Simulations and practical implementations demonstrate that our method performs similar to a recently computed analytical solution of the ETS (ETS-A). Furthermore, a Monte Carlo simulation shows that the NLLS algorithm provides a slightly better estimation of the scale factors than the ETS, in addition to the abovementioned advantages.