On the Generalization Ability of Geometric Semantic Genetic Programming

<p>Symbolic regression (SR) is a function identification process, the task of which is to identify and express the relationship between the input and output variables in mathematical models. SR is named to emphasise its ability to find the structure and coefficients of the model simultaneously. Genetic Programming (GP) is an attractive and powerful technique for SR, since it does not require any predefined model and has a flexible representation. However, GP based SR generally has a poor generalisation ability which degrades its reliability and hampers its applications to science and real-world modeling. Therefore, this thesis aims to develop new GP approaches to SR that evolve/learn models exhibiting good generalisation ability. This thesis develops a novel feature selection method in GP for high-dimensional SR. Feature selection can potentially contribute not only to improving the efficiency of learning algorithms but also to enhancing the generalisation ability. However, feature selection is seldom considered in GP for high-dimensional SR. The proposed new feature selection method utilises GP’s built-in feature selection ability and relies on permutation to detect the truly relevant features and discard irrelevant/noisy features. The results confirm the superiority of the proposed method over the other examined feature selection methods including random forests and decision trees on identifying the truly relevant features. Further analysis indicates that the models evolved by GP with the proposed feature selection method are more likely to contain only the truly relevant features and have better interpretability. To address the overfitting issue of GP when learning from a relatively small number of instances, this thesis proposes a new GP approach by incorporating structural risk minimisation (SRM), which is a framework to estimate the generalisation performance of models, into GP. The effectiveness of SRM highly depends on the accuracy of the Vapnik-Chervonenkis (VC) dimension measuring model complexity. This thesis significantly extends an experimental method (instead of theoretical estimation) to measure the VC-dimension of a mixture of linear and nonlinear regression models in GP for the first time. The experimental method has been conducted using uniform and non-uniform settings and provides reliable VC-dimension values. The results show that our methods have an impressively better generalisation gain and evolve more compact model, which have a much smaller behavioural difference from the target models than standard GP and GP with bootstrap, The proposed method using the optimised non-uniform setting further improves the one using the uniform setting. This thesis employs geometric semantic GP (GSGP) to tackle the unsatisfied generalisation performance of GP for SR when no overfitting occurs. It proposes three new angle-awareness driven geometric semantic operators (GSO) including selection, crossover and mutation to further explore the geometry of the semantic space to gain a greater generalisation improvement in GP for SR. The angle-awareness brings new geometric properties to these geometric operators, which are expected to provide a greater leverage for approximating the target semantics in each operation, and more importantly, to be resistant to overfitting. The results show that compared with two kinds of state-of-the-art GSOs, the proposed new GSOs not only drive the evolutionary process fitting the target semantics more efficiently but also significantly improve the generalisation performance. A further comparison on the evolved models shows that the new method generally produces simpler models with a much smaller size and containing important building blocks of the target models.</p>

Genetic Programming (GP) has been around for over two decades and has been used in a wide range of practical applications producing human competitive results in several domains. In this paper we present a discussion and a proposal of a GP algorithm that could be conveniently implemented on an embedded system, as part of a broader research project that pursues the implementation of a complete GP system in a Field Programmable Gate Array (FPGA). Motivated by the significant time savings associated with such a platform, as well as low power consumption, low maintenance requirements, small size of the system and the possibility of performing several parallel processes. The proposal is focused on the Geometric Semantic Genetic Programming (GSGP) approach that has been recently introduced with promising results. GSGP induces a unimodal fitness landscape, simplifying the search process. The experimental work considers five variants of GSGP, that incorporate local search strategies, optimal mutations and alignment in error space. Best results were obtained by a simple variant that uses both the optimal mutation step and the standard geometric semantic mutation, using three difficult real-world problems to evaluate the methods, outperforming the original GSGP formulation in terms of fitness and empirical convergence.

High-performance concrete, besides aggregate, cement, and water, incorporates supplementary cementitious materials, such as fly ash and blast furnace slag, and chemical admixture, such as superplasticizer. Hence, it is a highly complex material and modeling its behavior represents a difficult task. This paper presents an evolutionary system for the prediction of high performance concrete strength. The proposed framework blends a recently developed version of genetic programming with a local search method. The resulting system enables us to build a model that produces an accurate estimation of the considered parameter. Experimental results show the suitability of the proposed system for the prediction of concrete strength. The proposed method produces a lower error with respect to the state-of-the art technique. The paper provides two contributions: from the point of view of the high performance concrete strength prediction, a system able to outperform existing state-of-the-art techniques is defined; from the machine learning perspective, this case study shows that including a local searcher in the geometric semantic genetic programming system can speed up the convergence of the search process.

This paper introduces Multi-Region Symbolic Regression (MR-SR), a general framework that divides the original input data space of symbolic regression problems into subspaces (regions), generates different solutions to fit these regions and then combines them. MR-SR has three main components: (1) a strategy for finding the different regions of the data input space; (2) a method for generating the functions for each region; and (3) a strategy for combining the models found by (2). The main contribution of this paper is on how we generate the functions for each region. We model the function generation problem following a multi-objective approach, where each objective corresponds to the quality of the evolved function in a region, and the number of objectives is equal to the number of regions of the data input space. We test MR-SR in two scenarios with different objectives. In the first, we used the new approach to solve the symbolic regression problem with standard GP, with the main objective of reducing error rate. In the second, we took advantage of this method for a different purpose: to reduce the dimensionality of the semantic space of a variation of GP, namely Geometric Semantic Genetic Programming (GSGP). Results in 10 datasets showed that the method using clustering k-means and a model switching strategy—which makes predictions using the best evolved function for the region of interest—obtained better results in 5 out of 10 datasets for GP with 2 regions. For GSGP the framework was less effective due to the lack of diversity of the solutions evolved.

In this article, an evolutionary algorithm known as Genetic Programming (GP) was used to design a parametric speech quality estimation model. Nowadays, GP is one of the machine learning techniques employed in a quality estimation process. In principle, the set of quality-affecting parameters was used as an input to the designed estimation model based on GP approach in order to estimate a quality of synthesized speech transmitted over IP channel (VoIP environment). The performance results obtained by the designed estimation model have confirmed the good properties of genetic programming, namely good accuracy and generalization ability; this makes it to be perspective approach to a quality estimation of this type of speech in the corresponding environment. The developed model can be helpful for network operators and service providers implementing it in planning phase or early-development stage of telecommunication services based on synthesized speech.

A novel measure of attribute significance with complexity weight

Support vector machine is a machine learning method. It takes on the good generalization ability and prediction accuracy. But the parameters of SVM model seriously affect the generalization ability and prediction accuracy of SVM model on the great extent. So an improved particle swarm optimization (PSO) algorithm based on chaotic search is introduced into the SVM model in propose a novel data classification (AMPSVM) method for processing the complex data. The first, the ergodicity, stochastic property, and regularity of chaos is used to chaotically search the current best individual, which randomly replaces the selected individual in the population in order to speed up evolution, improve the searching ability, convergence speed and accuracy. Then the improved PSO algorithm is used to select and optimize the parameters of the SVM (AMPSVM) model in order to improve the learning performance and generalization ability of the SVM model. In order to verify the effectiveness of the AMPSVM method, UCI data is selected in here. The experiment results show that the proposed AMPSVM method takes on the strong generalization ability, best sensitivity and higher classification accuracy.

Kernel selection is a key issue of support vector machine (SVM), however, for the existing criterions of kernel selection in the literature, there are some limitations in generalization ability , computational cost or fair comparison. This paper proposes a novel criterion of kernel selection for SVM named maximum robustness criterion. The properties of maximum robustness criterion guarantee that it has an excellent generalization ability. Moreover, the maximum robustness criterion is fair for different kernel functions because it is calculated in input space and has low computational cost. The experiment results on some benchmark data sets show that it not only achieves almost as good generalization ability as minimum misclassifying rate of k-fold cross validation, but also has less computational cost.

BackgroundLong-term continuous systolic blood pressure (SBP) and heart rate (HR) monitors are of tremendous value to medical (cardiovascular, circulatory and cerebrovascular management), wellness (emotional and stress tracking) and fitness (performance monitoring) applications, but face several major impediments, such as poor wearability, lack of widely accepted robust SBP models and insufficient proofing of the generalization ability of calibrated models.MethodsThis paper proposes a wearable cuff-less electrocardiography (ECG) and photoplethysmogram (PPG)-based SBP and HR monitoring system and many efforts are made focusing on above challenges. Firstly, both ECG/PPG sensors are integrated into a single-arm band to provide a super wearability. A highly convenient but challenging single-lead configuration is proposed for weak single-arm-ECG acquisition, instead of placing the electrodes on the chest, or two wrists. Secondly, to identify heartbeats and estimate HR from the motion artifacts-sensitive weak arm-ECG, a machine learning-enabled framework is applied. Then ECG-PPG heartbeat pairs are determined for pulse transit time (PTT) measurement. Thirdly, a PTT&HR-SBP model is applied for SBP estimation, which is also compared with many PTT-SBP models to demonstrate the necessity to introduce HR information in model establishment. Fourthly, the fitted SBP models are further evaluated on the unseen data to illustrate the generalization ability. A customized hardware prototype was established and a dataset collected from ten volunteers was acquired to evaluate the proof-of-concept system.ResultsThe semi-customized prototype successfully acquired from the left upper arm the PPG signal, and the weak ECG signal, the amplitude of which is only around 10% of that of the chest-ECG. The HR estimation has a mean absolute error (MAE) and a root mean square error (RMSE) of only 0.21 and 1.20 beats per min, respectively. Through the comparative analysis, the PTT&HR-SBP models significantly outperform the PTT-SBP models. The testing performance is 1.63 ± 4.44, 3.68, 4.71 mmHg in terms of mean error ± standard deviation, MAE and RMSE, respectively, indicating a good generalization ability on the unseen fresh data.ConclusionsThe proposed proof-of-concept system is highly wearable, and its robustness is thoroughly evaluated on different modeling strategies and also the unseen data, which are expected to contribute to long-term pervasive hypertension, heart health and fitness management.

BackgroundMost existing algorithms have been focused on the segmentation from several public Liver CT datasets scanned regularly (no pneumoperitoneum and horizontal supine position). This study primarily segmented datasets with unconventional liver shapes and intensities deduced by contrast phases, irregular scanning conditions, different scanning objects of pigs and patients with large pathological tumors, which formed the multiple heterogeneity of datasets used in this study.MethodsThe multiple heterogeneous datasets used in this paper includes: (1) One public contrast-enhanced CT dataset and one public non-contrast CT dataset; (2) A contrast-enhanced dataset that has abnormal liver shape with very long left liver lobes and large-sized liver tumors with abnormal presets deduced by microvascular invasion; (3) One artificial pneumoperitoneum dataset under the pneumoperitoneum and three scanning profiles (horizontal/left/right recumbent position); (4) Two porcine datasets of Bama type and domestic type that contains pneumoperitoneum cases but with large anatomy discrepancy with humans. The study aimed to investigate the segmentation performances of 3D U-Net in: (1) generalization ability between multiple heterogeneous datasets by cross-testing experiments; (2) the compatibility when hybrid training all datasets in different sampling and encoder layer sharing schema. We further investigated the compatibility of encoder level by setting separate level for each dataset (i.e., dataset-wise convolutions) while sharing the decoder.ResultsModel trained on different datasets has different segmentation performance. The prediction accuracy between LiTS dataset and Zhujiang dataset was about 0.955 and 0.958 which shows their good generalization ability due to that they were all contrast-enhanced clinical patient datasets scanned regularly. For the datasets scanned under pneumoperitoneum, their corresponding datasets scanned without pneumoperitoneum showed good generalization ability. Dataset-wise convolution module in high-level can improve the dataset unbalance problem. The experimental results will facilitate researchers making solutions when segmenting those special datasets.Conclusions(1) Regularly scanned datasets is well generalized to irregularly ones. (2) The hybrid training is beneficial but the dataset imbalance problem always exits due to the multi-domain homogeneity. The higher levels encoded more domain specific information than lower levels and thus were less compatible in terms of our datasets.

In 2012, Moraglio and coauthors introduced new genetic operators for Genetic Programming, called geometric semantic genetic operators. They have the very interesting advantage of inducing a unimodal error surface for any supervised learning problem. At the same time, they have the important drawback of generating very large data models that are usually very hard to understand and interpret. The objective of this work is to alleviate this drawback, still maintaining the advantage. More in particular, we propose an elitist version of geometric semantic operators, in which offspring are accepted in the new population only if they have better fitness than their parents. We present experimental evidence, on five complex real-life test problems, that this simple idea allows us to obtain results of a comparable quality (in terms of fitness), but with much smaller data models, compared to the standard geometric semantic operators. In the final part of the paper, we also explain the reason why we consider this a significant improvement, showing that the proposed elitist operators generate manageable models, while the models generated by the standard operators are so large in size that they can be considered unmanageable.

State forecast of machine using support vector machines has good generalization ability in situation of rare samples. Appropriate parameter selection is very crucial to the learning results and generalization ability of support vector machines. In addition, embedding dimension influences the phase space reconstitution of nonlinear systems, as well as the precision of machine state forecasting. In this paper, an approach to optimize the parameters of SVM and the embedding dimension based on genetic algorithms was proposed. The proposed model is applied to the tendency forecasting of the vibration of shovel electric drive system. The results show that it can avoid blindness of manually selection of parameters and meanwhile improves the prediction performance greatly.

With the increasing popularity of online social networks (OSNs), a huge number of social bots have emerged. Social bots are involved in various cybercrimes like cyberbullying and rumor dissemination, which have seriously affected the normal order of OSNs. Nowadays, existing studies in this field almost focus on English OSNs like Twitter and Facebook. However, it is difficult to directly apply these detection technologies to Sina Weibo, which is one of the largest Chinese microblogging services in the world. In addition, social bots are evolving rapidly and time-consuming feature engineering may not perform well in detecting newly emerging social bots. In this paper, we propose a new joint approach with Temporal and Profile information for social bot detection (TPBot). The approach includes data collection module, feature extraction module, and detection module. To begin with, data collection module uses a web crawler to obtain user data from Sina Weibo. Next, the feature extraction module regards the user posts as temporal data to extract temporal-semantic and temporal-metadata features. Furthermore, this module extracts features based on users’ profile. Finally, a detection model based on BiGRU and attention mechanism is designed in the detection module. The results show that TPBot performs better than baselines with the F1-score of 0.9837 on the Sina Weibo dataset. Moreover, we have also conducted an experiment on the two datasets collected from Twitter to evaluate the generalization ability of TPBot. It is found that TPBot outperforms baselines on the new datasets and has good generalization ability.

As an important branch of artificial intelligence, computer vision plays a huge role in the rapid development of artificial intelligence. From a biological point of view, in the acquisition and processing of information, vision is much more important than hearing, touch, etc., because 70% of the human cerebral cortex is processing visual information. Therefore, advances in computer vision technology are critical to the development of artificial intelligence that is designed to allow machines to think and handle things like humans. The acquisition and processing of visual information has always been the focus of computer vision research, and it is also difficult. The main problem of traditional computer vision technology in the processing of visual information is that the extracted image features are less discriminative, the generalization ability of image features in complex background scenes is insufficient, and the recognition ability on object recognition is poor. In response to these problems, based on the visual neural mechanism, this paper establishes an appropriate computer model for the neuronal cells in the human primary visual cortex, models the recognition response mechanism of the visual ventral system, and performs image feature extraction on the training samples. And object recognition. The results show that compared with the traditional methods, the proposed method effectively improves the discrimination of image features, and the image features extracted under complex background scenes have good generalization ability. On this basis, the training samples can be effectively recognized. The results show that the model based on the visual neural mechanism, the recognition of the edge, orientation and contour of the training sample show the advantages of the biological vision mechanism in object recognition.

Aiming at the obstacle avoidance control problem of small quadrotor, a method of quadrotor obstacle avoidance based on reinforcement learning is proposed. The proposed method can make training converge quickly and has good environmental robustness. The proposed methods include: (1) a framework adopts perception module and decision module to improve the generalization ability of the obstacle avoidance model; (2) An Actor-Critic framework-based Proximal Policy Optimization (PPO) algorithm to provide quadrotor with policy-based decision-making capabilities; The experimental simulation results show that the strategy-based framework converges quickly and has a high success rate, the training time is much lower than that of the value-based framework. The monocular vision observation ability is limited, which leads to deviations between local observation and global state, So LSTM layer is usually added to increase model performance. Policy -based decision can have a good obstacle avoidance effect without adding the LSTM layer, and have good generalization ability after short relearning after changing.