Genetic Programming with Rademacher Complexity for Symbolic Regression

<p>Symbolic regression (SR) is a function identification process, the task of which is to identify and express the relationship between the input and output variables in mathematical models. SR is named to emphasise its ability to find the structure and coefficients of the model simultaneously. Genetic Programming (GP) is an attractive and powerful technique for SR, since it does not require any predefined model and has a flexible representation. However, GP based SR generally has a poor generalisation ability which degrades its reliability and hampers its applications to science and real-world modeling. Therefore, this thesis aims to develop new GP approaches to SR that evolve/learn models exhibiting good generalisation ability. This thesis develops a novel feature selection method in GP for high-dimensional SR. Feature selection can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalisation ability. However, feature selection is seldom considered in GP for high-dimensional SR. The proposed new feature selection method utilises GP’s built-in feature selection ability and relies on permutation to detect the truly relevant features and discard irrelevant/noisy features. The results confirm the superiority of the proposed method over the other examined feature selection methods including random forests and decision trees on identifying the truly relevant features. Further analysis indicates that the models evolved by GP with the proposed feature selection method are more likely to contain only the truly relevant features and have better interpretability. To address the overfitting issue of GP when learning from a relatively small number of instances, this thesis proposes a new GP approach by incorporating structural risk minimisation (SRM), which is a framework to estimate the generalisation performance of models, into GP. The effectiveness of SRM highly depends on the accuracy of the Vapnik-Chervonenkis (VC) dimension measuring model complexity. This thesis significantly extends an experimental method (instead of theoretical estimation) to measure the VC-dimension of a mixture of linear and nonlinear regression models in GP for the first time. The experimental method has been conducted using uniform and non-uniform settings and provides reliable VC-dimension values. The results show that our methods have an impressively better generalisation gain and evolve more compact model, which have a much smaller behavioural difference from the target models than standard GP and GP with bootstrap, The proposed method using the optimised non-uniform setting further improves the one using the uniform setting. This thesis employs geometric semantic GP (GSGP) to tackle the unsatisfied generalisation performance of GP for SR when no overfitting occurs. It proposes three new angle-awareness driven geometric semantic operators (GSO) including selection, crossover and mutation to further explore the geometry of the semantic space to gain a greater generalisation improvement in GP for SR. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, to be resistant to overfitting. The results show that compared with two kinds of state-of-the-art GSOs, the proposed new GSOs not only drive the evolutionary process fitting the target semantics more efficiently but also significantly improve the generalisation performance. A further comparison on the evolved models shows that the new method generally produces simpler models with a much smaller size and containing important building blocks of the target models.</p>

Generalization ability, which reflects the prediction ability of a learned model, is an important property in genetic programming (GP) for symbolic regression. Structural risk minimization (SRM) is a framework providing a reliable estimation of the generalization performance of prediction models. Introducing the framework into GP has the potential to drive the evolutionary process toward models with good generalization performance. However, this is tough due to the difficulty in obtaining the Vapnik–Chervonenkis (VC) dimension of nonlinear models. To address this difficulty, this paper proposes an SRM-driven GP approach, which uses an experimental method (instead of theoretical estimation) to measure the VC dimension of a mixture of linear and nonlinear regression models for the first time. The experimental method has been conducted using uniform and nonuniform settings. The results show that our method has impressive generalization gains over standard GP and GP with the 0.632 bootstrap, and that the proposed method using the nonuniform setting has further improvement than its counterpart using the uniform setting. Further analyzes reveal that the proposed method can evolve more compact models, and that the behavioral difference between these compact models and the target models is much smaller than their counterparts evolved by the other GP methods.

Population diversity plays an important role in the evolutionary dynamics of genetic programming (GP). In this paper we use structural and semantic similarity measures to investigate the evolution of diversity in three GP algorithmic flavors: standard GP, offspring selection GP (OS-GP), and age-layered population structure GP (ALPS-GP). Empirical measurements on two symbolic regression benchmark problems reveal important differences between the dynamics of the tested configurations. In standard GP, after an initial decrease, population diversity remains almost constant until the end of the run. The higher variance of the phenotypic similarity values suggests that small changes on individual genotypes have significant effects on their corresponding phenotypes. By contrast, strict offspring selection within the OS-GP algorithm causes a significantly more pronounced diversity loss at both genotypic and, in particular, phenotypic levels. The pressure for adaptive change increases phenotypic robustness in the face of genotypic perturbations, leading to less genotypic variability on the one hand, and very low phenotypic diversity on the other hand. Finally, the evolution of similarities in ALPS-GP follows a periodic pattern marked by the time interval when the bottom layer is reinitialized with new individuals. This pattern is easily noticed in the lower layers characterized by shorter migration intervals, and becomes less and less noticeable on the upper layers.

Accurate estimation of river flow can have a significant importance in water resources management. In this study, Genetic programming (GP) and Support Vector Machine (SVM) methods were used to forecast daily discharge of Barandoozchay River. The daily discharge data of Barandoozchay River measured at the Dizaj hydrometric station during 2007 to 2011 was used for modeling, which 80% of the data used for training and remaining 20% used for testing of models. The results showed that in the both of considered methods, the models including discharges of one, two and three days ago had higher accuracy in verification step and the accuracy of models decreased with increasing discharge values. Comparing the performance of GP and SVM methods indicated that, however the accuracy of the GP method with the R=0.978 and RMSE=1.66 (m3/s) was slightly more than SVM method with R=0.976 and RMSE=1.80 (m3/s), but the SVM is easier than GP method. Thus, the SVM method can be used as an alternative method in forecasting daily river discharge.

<p>One of the greater issues in Genetic Programming (GP) is the computational effort required to run the evolution and discover a good solution. Phenomena such as program bloating (where genetic programs rapidly grow in size) can quickly exhaust available memory resources and slow down the evolutionary process, while the heavy cost of performing fitness evaluation can make problems which have a lot of available data very slow to solve. These issues may limit GP in some tasks it can appropriately be applied to, as well as inhibit its applications in time/space sensitive environments. In this thesis, we look at developing solutions to some of these issues in GP computational cost. First, we develop an algebraic program simplification method based on simple rules and hashing techniques, and use this method in conjunction with the standard GP on a variety of tasks. Our results suggest that program simplification can lead to a significant reduction in program size, while not significantly changing the effectiveness of the systems in finding solution programs. Secondly, we analyse the effects of program simplification on the internal GP "building blocks" to investigate whether simplification is a destructive or constructive force. Using two models for building blocks (numerical-nodes and the more complex fixed-depth subtree), we track building blocks through GP runs on a symbolic regression problem, both with and without using simplification. We find that the program simplification process can both disrupt and construct building blocks in the GP populations. However, GP systems using simplification appear to retain important building blocks, and the simplification process appears to lead to an increase in genetic diversity. These may help explain why using simplification does not reduce the effectiveness of GP systems in solving tasks. Lastly, we develop two methods of reducing the cost of fitness evaluation by reducing the number of node evaluations performed. The first method is elitism avoidance, which avoids re-evaluating programs which have been placed in the population using elitismreproduction. Thismethod reduces the CPU time for evolving solutions for six different GP tasks. The second method is a subtree caching mechanism which store fitness evaluations for subtrees in a cache so that they may be fetched when these subtrees are encountered in future fitness evaluations. Results suggest that using this mechanism can significantly reduce both the number of node evaluations and the overall CPU time used in evolving solutions, without reducing the fitness of the solutions produced.</p>

There are two important limitations of standard tree-based genetic programming (GP). First, GP tends to evolve unnecessarily large programs, what is referred to as bloat. Second, it uses inefficient search operators that operate at the syntax level. The first problem has been the subject of a fair amount of research over the years. Regarding the second problem, one approach is to use alternative search operators, for instance geometric semantic operators. However, another approach is to introduce greedy local search strategies, combining the syntactic search performed by standard GP with local search strategies for solution tuning, which is a simple strategy that has comparatively received much less attention. This work combines a recently proposed bloat-free GP called neat-GP with a local search strategy. One benefit of using a bloat-free GP is that it reduces the size of the parameter space confronted by the local searcher, offsetting some of the added computational cost. The algorithm is validated on a real-world problem with promising results.

This chapter evaluates the performance of various methods to constant creation in Grammatical Evolution (GE), and validates the results by comparing against those from a reasonably standard Genetic Programming (GP) setup. Specifically, the chapter compares a standard GE method to constant creation termed digit concatenation with what this chapter calls compact methods to constant creation. Constant creation in GE is an important issue due to the disruptive nature of ripple crossover, which can radically remap multiple terminals in an individual, and we investigate if more compact methods, which are more similar to the GP style of constant creation (Ephemeral Random Constants (ERCs), perform better. The results are surprising. Against common wisdom, a standard GE approach of digit concatenation does not produce individuals that are any larger than those from methods which are designed to use less genetic material. In fact, while GP characteristically evolves increasingly larger individuals, GE—after an initial growth or drop in sizes—tends to keep individual sizes stable despite no explicit mechanisms to control size growth. Furthermore, various GE setups perform acceptably well on unseen test data and typically outperform GP. Overall, these results encourage a belief that standard GE methods to symbolic regression are relatively resistant to pathogenic evolutionary tendencies of code bloat and overfitting.

Transfer learning has attracted more attention in the machine-learning community recently. It aims to improve the learning performance on the domain of interest with the help of the knowledge acquired from a similar domain(s). However, there is only a limited number of research on tackling transfer learning in genetic programming for symbolic regression. This article attempts to fill this gap by proposing a new instance weighting framework for transfer learning in genetic programming-based symbolic regression. In the new framework, differential evolution is employed to search for optimal weights for source-domain instances, which helps genetic programming to identify more useful source-domain instances and learn from them. Meanwhile, a density estimation method is used to provide good starting points to help the search for the optimal weights while discarding some irrelevant or less important source-domain instances before learning regression models. The experimental results show that compared with genetic programming and support vector regression that learn only from the target instances, and learning from a mixture of instances from the source and target domains without any transfer learning component, the proposed method can evolve regression models which not only achieve notably better cross-domain generalization performance in stability but also reduce the trend of overfitting effectively. Meanwhile, these models are generally much simpler than those generated by the other GP methods.

The present study quantitative structure-property relationship (QSPR) model developed for the adsorption efficiency (AE) of 70 pesticides in water sample on MWCNT/PbO2 solid phase extraction cartridge. Stepwise-multiple linear regression (SW-MLR) method employed for selection of descriptors. The selected descriptors are MATS7v, MATS6c, GATS3s, ATSC6i, C040, SpMin8_Bhi, E2v, JGI1 and Mor08u. Curiosity at the effective descriptors indicates electronic, topological and geometrical characteristics of studied pesticides are the most effective parameters on their AE on MWCNT/PbO2 nanocomposite adsorbent. Symbolic regression via genetic programming (SR-GP) utilized to offer the symbolic regression QSPR model. The accuracy and predictive power of the SR-GP model compared with traditional linear and nonlinear regression models contain multiple linear regression (MLR) and support vector regression (SVR). Inspection the fitness parameters confirmed the superiority of SR-GP model over MLR, and SVR models. In SR-GP model, the correlation coefficient (R) was 0.930 and 0.890, and the root mean square errors (RMSE) were 0.04 and 0.05 for the training and test sets, respectively. These results can be used to predict the AE for other pesticides by MWCNT/PbO2 adsorbent and designing a more efficient nano cartridge for SPE.

One of the major challenges in proteomics is peptide identification from mass spectra containing high noise ratio and small number of signal (b-/y-ions) peaks. However, the accuracy and reliability of peptide identification in such highly imbalanced MS/MS data can be improved by applying a preprocessing step prior to peptide identification aiming at discriminating b-/y-ions from noise peaks in the spectra. In this study, we report a genetic programming (GP)-based preprocessing method for de-noising highly imbalanced and noisy CID MS/MS spectra. GP now becomes a popular machine learning method via automatic programming. GP preprocesses the highly noisy MS/MS spectra by classifying peaks as noise peaks or signal peaks in a binary classification manner. Meanwhile, a set of spectral fragment features based on the MS/MS fragmentation rules is extracted from the dataset to investigate their discriminating abilities by GP. A MS/MS spectral dataset containing thousands of spectra are used to train the GP model. As the GP tree-based representation has the capability for implicit feature selection during the evolutionary process, the evolved GP model with the selected features is compared with the best threshold-based method. The results show that the GP method improved the reliability of peptide identification and increased the identification rate of a de novo sequencing tool, PEAKS, to 99.4% from 80.1% achieved by the best threshold-based method. Moreover, the result of peptide identification by a database search tool, SEQUEST, using the data preprocessed by the GP method was statistically significant compared to the other methods.

Many real-world problems have an uneven distribution of data over different classes. The imbalanced nature of data impacts the performance of classifiers. The higher counts of majority class samples influence the learning abilities of well-known classifiers. Genetic programming (GP) algorithm based on natural evolution also impacts if the data’s nature is imbalanced. The fitness function plays a pivotal role and impacts almost each building block of the GP framework. GP with the standard fitness function produces under-fitted and biased classifiers. Therefore, this paper has proposed a new fitness function in GP to classify the imbalanced data. The proposed method is used to classify nine imbalanced problems: ABL-18, ABL-9-18, BAL, YEAST2, YEAST1, ABL-9, ION, WDBC, and SPECT. The imbalanced factor of benchmark problems varies from 99:1 to 59:41. The proposed method’s performance is compared with K-Nearest-Neighbourhood (KNN) and the standard fitness function-based GP methods. The GP with newly proposed fitness function gives average AUC values for ABL-18(99:1), ABL-9-18(94:6), BAL(92:8), YEAST2(89:11), YEAST1(84:16), ABL-9(83:17), ION(64:36), WDBC(63:37), and SPECT(59:41) as 0.714, 0.812, 0.975, 0.916, 0.768, 0.654, 0.872, 0.939, and 0.704, respectively, which are higher than KNN and the standard fitness function-based GP methods. The result outcomes prove the superiority of the proposed method.

A context‐free genetic programming (GP) method is presented that simulated local scale daily extreme (maximum and minimum) temperatures based on large scale atmospheric variables. The method evolves simple and optimal models for downscaling daily temperature at a station. The advantage of the context‐free GP method is that both the variables and constants of the candidate models are optimized and consequently the selection of the optimal model. The method is applied to the Chute‐du‐Diable weather station in Northeastern Canada along with the National Center for Environmental Prediction (NCEP) reanalysis datasets. The performance of the GP based downscaling models is compared to benchmarks from a commonly used statistical downscaling model. The experiment results show that the models evolved by the GP are simpler and more efficient for downscaling daily extreme temperature than the common statistical method. The different model test results indicate that the GP approach significantly outperforms the statistical method for the downscaling of daily minimum temperature, while for the maximum temperature the two methods are almost equivalent. However, the GP method remains slightly more effective for maximum temperature downscaling than the statistical method.

The genetic programming (GP) method is proposed as a new approach to perform texture classification based directly on raw pixel data. Two alternative genetic programming representations are used to perform classification. These are dynamic range selection (DRS) and static range selection (SRS). This preliminary study uses four brodatz textures to investigate the applicability of the genetic programming method for binary texture classifications and multi-texture classifications.Results indicate that the genetic programming method, based directly on raw pixel data, is able to accurately classify different textures. The results show that the DRS method is well suited to the task of texture classification. The classifiers generated in our experiments by DRS have good performance over a variety of texture data and offer GP as a promising alternative approach for the difficult problem of texture classification.KeywordsGenetic ProgrammingClass LabelTexture ClassificationTraining AccuracyDivision PointThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Genetic Programming (GP) has been widely employed to create dispatching rules intelligently for production scheduling. The success of GP depends on a suitable terminal set of selected features. Specifically, techniques that consider feature selection in GP to enhance rule understandability for dynamic job shop scheduling (DJSS) have been successful. However, existing feature selection algorithms in GP focus more emphasis on obtaining more compact rules with fewer features than on improving effectiveness. This paper is an attempt at combining a novel GP method, GP via dynamic diversity management, with feature selection to design effective and interpretable dispatching rules for DJSS. The idea of the novel GP method is to achieve a progressive transition from exploration to exploitation by relating the level of population diversity to the stopping criteria and elapsed duration. We hypothesize that diverse and promising individuals obtained from the novel GP method can guide the feature selection to design competitive rules. The proposed approach is compared with three GP-based algorithms and 20 benchmark rules in the different job shop conditions and scheduling objectives. Experiments show that the proposed approach greatly outperforms the compared methods in generating more interpretable and effective rules for the three objective functions. Overall, the average improvement over the best-evolved rules by the other three GP-based algorithms is 13.28%, 12.57%, and 15.62% in the mean tardiness (MT), mean flow time (MFT), and mean weighted tardiness (MWT) objective, respectively.

Evolving dynamic fitness measures for genetic programming