AutoML Loss Landscapes

Machine learning pipeline potentially consists of several stages of operations like data preprocessing, feature engineering and machine learning model training. Each operation has a set of hyper-parameters, which can become irrelevant for the pipeline when the operation is not selected. This gives rise to a hierarchical conditional hyper-parameter space. To optimize this mixed continuous and discrete conditional hierarchical hyper-parameter space, we propose an efficient pipeline search and configuration algorithm which combines the power of Reinforcement Learning and Bayesian Optimization. Empirical results show that our method performs favorably compared to state of the art methods like Auto-sklearn, TPOT, Tree Parzen Window, and Random Search.

Given the increasing importance of machine learning (ML) in our lives, several algorithmic fairness techniques have been proposed to mitigate biases in the outcomes of the ML models. However, most of these techniques are specialized to cater to a single family of ML models and a specific definition of fairness, limiting their adaptibility in practice. We introduce a general constrained Bayesian optimization (BO) framework to optimize the performance of any ML model while enforcing one or multiple fairness constraints. BO is a model-agnostic optimization method that has been successfully applied to automatically tune the hyperparameters of ML models. We apply BO with fairness constraints to a range of popular models, including random forests, gradient boosting, and neural networks, showing that we can obtain accurate and fair solutions by acting solely on the hyperparameters. We also show empirically that our approach is competitive with specialized techniques that enforce model-specific fairness constraints, and outperforms preprocessing methods that learn fair representations of the input data. Moreover, our method can be used in synergy with such specialized fairness techniques to tune their hyperparameters. Finally, we study the relationship between fairness and the hyperparameters selected by BO. We observe a correlation between regularization and unbiased models, explaining why acting on the hyperparameters leads to ML models that generalize well and are fair.

Automated Machine Learning (AutoML) is a research area that aims to help humans solve Machine Learning (ML) problems by automatically discovering good model pipelines (algorithms and their hyperparameters for every stage of a machine learning process) for a given dataset. Since we have a combinatorial optimization problem for which it is impossible to evaluate all possible pipelines, most AutoML systems use Evolutionary Algorithm (EA) or Bayesian Optimization (BO) to find a good solution. As these systems usually evaluate the pipelines' performance using the k-fold cross-validation method, the chance of finding an overfitted solution increases with the number of pipelines evaluated. Therefore, to avoid the aforementioned issue, we propose an Auto-ML system, named Auto-ML System for Text Classification (ASTeC), that uses the Bootstrap Bias Corrected CV (BBC-CV) to evaluate the pipelines' performance. More specifically, the proposed system combines EA, BO, and BBC-CV to find a good model pipeline for the text classification task. We evaluate our proposal by comparing it against two state-of-the-art systems, the Tree-based Pipeline Optimization Tool (TPOT) and Google Cloud AutoML service. To do so, we use seven public datasets composed of written Brazilian Portuguese texts from the sentiment analysis domain. Statistical tests show that our system is equivalent to or better than both of them in all evaluated datasets.

In recent decades, various software effort estimation (SEE) algorithms have been suggested. Unfortunately, generating high-precision accuracy is still a major challenge in the context of SEE. The use of traditional techniques and parametric approaches is largely inaccurate because they produce biased and subjective accuracy. Meanwhile, none of the machine learning methods performed well. This study applies the AdaBoost ensemble learning method and random forest (RF), on the other hand the Bayesian optimization method is applied to determine the hyperparameters of this model. The PROMISE repository and the ISBSG dataset were used to build the SEE model. The developed model was comprehensively compared with four machine learning methods (classification and regression tree, k-nearest neighbor, multilayer perceptron, and support vector regression) under 3-fold cross validation (CV). It can be seen that the RF method based on AdaBoost ensemble learning and bayesian optimization outperforms this approach. In addition, the AdaBoost-based model assigns a feature importance rating, which makes it a promising tool in software effort prediction.

Neuromorphic computing is emerging as a promising Beyond Moore computing paradigm that employs event-triggered computation and non-von Neumann hardware. Spike Timing Dependent Plasticity (STDP) is a well-known bio-inspired learning rule that relies on activities of locally connected neurons to adjust the weights of their respective synapses. In this work, we analyze a basic STDP rule and its sensitivity on the different hyperparameters for training spiking neural networks (SNNs) with supervision, customized for a neuromorphic hardware implementation with integer weights. We compare the classification performance on four UCI datasets (iris, wine, breast cancer and digits) that depict varying levels of complexity. We perform a search for optimal set of hyperparameters using both grid search and Bayesian optimization. Through the use of Bayesian optimization, we show the general trends in hyperparameter sensitivity in SNN classification problem. With the best sets of hyperparameters, we achieve accuracies comparable to some of the best performing SNNs on these four datasets. With a highly optimized supervised STDP rule we show that these accuracies can be achieved with just 20 epochs of training.

With the sustainable development of the construction industry, recycled aggregate (RA) has been widely used in concrete preparation to reduce the environmental impact of construction waste. Compressive strength is an essential measure of the performance of recycled aggregate concrete (RAC). In order to understand the correspondence between relevant factors and the compressive strength of recycled concrete and accurately predict the compressive strength of RAC, this paper establishes a model for predicting the compressive strength of RAC using machine learning and hyperparameter optimization techniques. RAC experimental data from published literature as the dataset, extreme gradient boosting (XGBoost), random forest (RF), K-nearest neighbour (KNN), support vector machine regression Support Vector Regression (SVR), and gradient boosted decision tree (GBDT) RAC compressive strength prediction models were developed. The models were validated and compared using correlation coefficients (R2), Root Mean Square Error (RMSE), mean absolute error (MAE), and the gap between the experimental results of the predicted outcomes. In particular, The effects of different hyperparameter optimization techniques (Grid search, Random search, Bayesian optimization-Tree-structured Parzen Estimator, Bayesian optimization- Gaussian Process Regression) on model prediction efficiency and prediction accuracy were investigated. The results show that the optimal combination of hyperparameters can be searched in the shortest time using the Bayesian optimization algorithm based on TPE (Tree-structured Parzen Estimator); the BO-TPE-GBDT RAC compressive strength prediction model has higher prediction accuracy and generalisation ability. This high-performance compressive strength prediction model provides a basis for RAC’s research and practice and a new way to predict the performance of RAC.

Load forecasting impacts directly financial returns and information in electrical systems planning. A promising approach to load forecasting is the Echo State Network (ESN), a recurrent neural network for the processing of temporal dependencies. The low computational cost and powerful performance of ESN make it widely used in a range of applications including forecasting tasks and nonlinear modeling. This paper presents a Bayesian optimization algorithm (BOA) of ESN hyperparameters in load forecasting with its main contributions including helping the selection of optimization algorithms for tuning ESN to solve real-world forecasting problems, as well as the evaluation of the performance of Bayesian optimization with different acquisition function settings. For this purpose, the ESN hyperparameters were set as variables to be optimized. Then, the adopted BOA employs a probabilist model using Gaussian process to find the best set of ESN hyperparameters using three different options of acquisition function and a surrogate utility function. Finally, the optimized hyperparameters are used by the ESN for predictions. Two datasets have been used to test the effectiveness of the proposed forecasting ESN model using BOA approaches, one from Poland and another from Brazil. The results of optimization statistics, convergence curves, execution time profile, and the hyperparameters’ best solution frequencies indicate that each problem requires a different setting for the BOA. Simulation results are promising in terms of short-term load forecasting quality and low error predictions may be achieved, given the correct options settings are used. Furthermore, since there is not an optimal global optimization solution known for real-world problems, correlations among certain values of hyperparameters are useful to guide the selection of such a solution.

This study presents a data-driven approach for optimizing the operating altitude of Airborne Wind Energy (AWE) systems to maximize net energy production. Determining the optimal operating altitude of an AWE system is challenging, as the wind speed constantly varies with both time and altitude. Furthermore, without expensive auxiliary equipment, the wind speed is only measurable at the AWE system's operating altitude. The work presented in this paper shows how tools from machine learning can be blended with real-time control to optimize the AWE system's operating altitude efficiently, without the use of auxiliary wind profiling equipment. Specifically, Bayesian Optimization, which is a data-driven technique for finding the optimum of an unknown and expensive-to-evaluate objective function, is applied to the real-time control of an AWE system. The underlying objective function is modeled by a Gaussian Process (GP); then, Bayesian Optimization utilizes the predictive uncertainty information from the GP to decide the best subsequent operating altitude. In the AWE application, conventional Bayesian Optimization is extended to handle the time-varying nature of the wind shear profile (wind speed vs. time). Using real wind data, our method is validated against three baseline approaches. Our simulation results show that the Bayesian Optimization method is successful in dramatically increasing power production over these baselines.

In recent years, the rapid growth in data availability and advances in computational methodologies have greatly expanded the potential to analyze complex systems and extract actionable knowledge. This thesis demonstrates the use of machine learning (ML) and other data-driven techniques in environmental sciences; with a specific focus on enhanced weathering (EW), a technique the IPCC is counting upon to achieve the climate warming target of 2C, and on global crop fertilization, whose optimization is crucial for maximizing food security while avoiding environmental pollution. Part I (Chapters 2–5) addresses two EW experiments. Chapter 2 investigates how a wide range of biotic and abiotic factors influence EW in a large-scale column experiment. Through ML, we predict four EW indicators from the experimental inputs. Then, explainable artificial intelligence (xAI) methods are used to identify the most influential drivers, providing new insights into the mechanisms that govern EW. Building on this, chapter 3 explores how the same experimental inputs affect the earthworms within the system. By combining ML and xAI, livable conditions for earthworms are found to be primarily determined by water drainage-related features. Chapter 4 introduces two novel Bayesian optimization (BO) algorithms designed for experimental design in batch settings. The most effective algorithm is applied to the EW experiment to identify new experimental combinations that simultaneously reduce uncertainty and enhance predicted CDR outcomes. In Chapter 5, a second EW experiment is analyzed, this time using a reactor with variable operational settings. Geochemical modeling of such time series data relies on unmeasured inputs and does not account for operational dynamics. Therefore, the unknown inputs are optimized using BO and the optimized geochemical model is combined with a correction model, improving the predictive performance. Through xAI, we investigate the impact of the operational conditions. Part II (Chapters 6–7) focuses on global, crop-specific inorganic fertilization rates. In Chapter 6, a new global dataset is constructed with annual fertilizer application rates from 1961–2019 at the crop and country level. Specifically, ML is used to predict the historical rates from environmental, agrological and socioeconomic features, and the dominant drivers are investigated using xAI. Finally, Chapter 7 analyzes the temporal trajectories of fertilizer intensification to allow comparison across countries, regions, nutrients and crops. A logistic–lognormal function is defined to model crop-specific fertilizer use over time and Bayesian inference is applied to estimate its parameters. From these, critical points of the trajectories are derived and compared.

While it is inarguable that the advent of the silicon transistor was revolutionary for humankind, there is an ever-growing need for electronics that can operate at limits beyond the scope of silicon. Just as Feynman predicted in his infamous speech "Plenty of Room at the Bottom", quantum materials are this next phase of electronic materials.1 Over the past twenty years the condensed matter physics community has made strides in the development of these materials but we are still a ways off of large scale manufacturing of said materials. Because of this, the research pursued in this dissertation presents a novel approach to advancing the experimental synthesis of 2D materials using Chemical Vapor Deposition via Density Functional Theory, thermodynamics, and machine learning. Presented in this dissertation are the theory, simulations, experimentation, and machine learning developed for the purpose of determining experimental parameters for the synthesis of 2D materials. This idea is probed by analyzing a material that is already known. In Chapter 1 of this dissertation, we provide an overview of 2D materials and the current techniques for 2D material synthesis. We explore the current methods for calculating synthesis and why this area of research needs to be pursued. In Chapter 2 of this dissertation, we discuss the theoretical and experimental frameworks utilized in the research presented. This chapter starts by describing the current theoretical techniques for quantum simulations via their mathematical structure to develop and understanding for why each successive approach is chosen. We then move on to discuss how these quantum simulation practices are utilized to estimate the thermodynamic properties of molecules and crystal structures. These thermodynamic properties are then used to estimate the energy differences in a proposed chemical reaction to form our desired 2D material. While this approach is straight-forward, it can be computationally intensive. Because of this, the next section moves on to the derivation of a simple approach to overcoming the limitations of current formation energy calculations, modeling all of the temperature and pressure space with one input pressure value, reducing computational cost further. Chapter 2 continues by also discussing the computational techniques utilized for computing expected experimental measurement results-including Raman response and x-ray diffraction. The chapter then continues to describe the methods used for building a machine learning code to track experimental results, within which the computed expected experimental results are utilized for testing the trained models. Finally, this chapter ends with the computational method used deciding an initial set of exploratory experimental parameters and why it is necessary to do so. In Chapter 3 of this dissertation, we discuss the experimental techniques used in the process of synthesizing and characterizing our samples. The chapter starts by discussing the methods of synthesis via Chemical Vapor Deposition and a newly devised "Hot Plate Method". The chapter then moves on to describe sample transferring techniques utilized for sample characterization purposes. Finally, the chapter covers the characterization processes used for our data collection. In Chapter 4, the methods from Chapter 2 and Chapter 3 and utilized to discover a novel way to synthesize graphene at atmospheric pressure and lower relative temperatures than standard methods by utilizing a non-explosive solid source. This chapter discusses the specific computations utilized and their results as well as characterization of the new material. In Chapter 5, we review the use of neural networks for converging experimental convergence. While techniques like Bayesian Optimization (BO) already exist for this purpose, BO is computationally intensive while neural networks retain a low computational cost-even at high samples. For this purpose, we explore the use of Density Functional Theory in conjunction with experimental results to converge experiments. Finally, in Chapter 6 of this dissertation, the future direction of each technique is analyzed and discussed to illustrate a potential path towards more advanced techniques.--Author's abstract

Stroke may cause death for anyone, including youngsters. One of the early stroke detection techniques is a Computerized Tomography (CT) scan. This research aimed to optimize hyperparameter in Deep Learning, Random Search and Bayesian Optimization for determining the right hyperparameter. The CT scan images were processed by scaling, grayscale, smoothing, thresholding, and morphological operation. Then, the images feature was extracted by the Gray Level Co-occurrence Matrix (GLCM). This research was performed a feature selection to select relevant features for reducing computing expenses, while deep learning based on hyperparameter setting was used to the data classification process. The experiment results showed that the Random Search had the best accuracy, while Bayesian Optimization excelled in optimization time.

Predicting the adhesive force between steel reinforcement and concrete is crucial as it influences stress distribution and the overall mechanical behavior of reinforced concrete. This study proposes a novel approach to enhance bond strength prediction using machine learning (ML) models optimized through Bayesian optimization (BO). A dataset comprising 401 beam tests with six key factors was used to train three distinct ML algorithms—Support Vector Regression (SVR), Random Forest (RF), and Extreme Gradient Boosting (XGBoost). The prediction models were first trained on the full dataset, with BO applied to fine-tune hyperparameters and improve accuracy. Among these models, the BO-XGBoost achieved the best performance, with an R2 of 0.74, MAE of 1.412 MPa, and RMSE of 1.516 MPa on the test set, and R2 = 0.80, MAE = 0.950 MPa, RMSE = 1.200 MPa on the training set. In addition, a simplified model was developed, incorporating only three critical variables—rebar thickness, reinforcement tensile strength, and concrete compressive capacity—to make the model more applicable in real-world engineering scenarios. To further interpret the model’s predictions, Shapley additive explanations (SHAP) were employed, revealing the specific influence of each variable on bond strength. This study demonstrates that the integration of ML with Bayesian optimization can significantly improve the accuracy of bond strength predictions, offering valuable insights for structural design optimization.

Suspended sediment load (SSL) is essential to river and dam engineering. Due to the complexity and stochastic nature of sedimentation, SSL prediction is a challenging task and conventional methods often fail to generate accurate results. Aiming to provide an improved estimation, this paper contributes to a new forecasting framework by integrating the seasonal adjustment (SA) and Bayesian optimization (BOP) into a machine learning (ML) model (denoted as BMS). The SA is used for de-seasonalisation and trend extraction; the BOP is to optimize the ML architecture. The BMS is evaluated using the daily SSL records from the Yangtze River. Its performance is appraised by statistical criteria of Nash-Sutcliffe efficiency (NSE), correlation coefficient (CC), root mean squared error (RMSE) and mean absolute error (MAE). With the de-noising and hyper-parameter tuning modules, the BMS effectively heightens the accuracy of the standard ML models. The most significant improvement occurs in the Boosting model, with its augments in NSE and CC by 4.3% and 1.6%, and reductions in RMSE and MAE by 24.9% and 24.2%. The BMS gains by comparison even under flood conditions, where it remarkably reduces the errors of the constituent models by up to 47.9% for RMSE and 48.3% for MAE.

Background Accurate prediction of outcomes following a heart transplant is critical to explaining risks and benefits to patients and decision-making when considering potential organ offers. Given the large number of potential variables to be considered, this task may be most efficiently performed using machine learning (ML). Purpose We trained and tested different ML algorithms to accurately predict outcomes following a cardiac transplant using the United Network of Organ Sharing (UNOS) database. Methods We included 67 939 adult and pediatric patients enrolled in the UNOS database between January 1994 and December 2016 who underwent cardiac transplantation (median age 53 [IQR 38 – 60], 72.7% males). In our models, as an input, we included 114 features that have been collected from recipients and donors prior to transplant. The primary outcome was all-cause mortality at one-year post-transplant. We evaluated three different ML methods: XGBoost, Random Forest (RF) and L2 regularized logistic regression. Algorithms were trained and tested using shuffled 10-fold cross-validation (CV) as well as rolling CV. In the rolling CV, to mimic prospective procedure, ML models were trained by incrementally adding patients according to transplant year and testing models on the data in the following year. The hyperparameters, controlling the learning process, were tuned using Bayesian optimization. Prognostic accuracy for one-year all-cause mortality was characterized using the area under the receiver-operating characteristic curve (AUC). Results In total, 8,394 patients died within 1 year of transplant. We observed a substantial difference in prognostic accuracy between the shuffled 10-fold CV and the rolling CV. In the 10-fold CV, XGBoost and RF achieved high predictive performance with AUC of 0.848 (95% CI: 0.842–0.854) and 0.891 (95% CI: 0.886–0.896), respectively. In the rolling CV, which is a more realistic setting, AUC dropped to 0.673 (95% CI: 0.661–0.684) for XGBoost and 0.670 (0.657–0.683) for RF. Predictive performance of L2 regularized logistic regression remained stable across the two CV procedures, achieving AUC 0.669 (95% CI: 0.662–0.676) in the 10-fold shuffled CV and 0.665 (95% CI: 0.649–0.680) in the rolling CV procedure (Figure). Conclusions Our study suggests that ML models could be used to predict mortality in the first year post-transplant. We also show that the choice of CV procedure is crucial for evaluating ML models, particularly in data collected over a long period of time. The difference between the shuffled and rolling CV in the predictive performance of the tree-based ML models might indicate temporal dataset shift. In the rolling CV, all three methods achieved similar predictive performance. Funding Acknowledgement Type of funding sources: Public grant(s) – National budget only. Main funding source(s): Research Foundation Flanders (FWO)

Background and objectiveThe increasing global prevalence of diabetes has led to a surge in complications, significantly burdening healthcare systems and affecting patient quality of life. Early prediction of these complications is critical for timely intervention, yet existing models often rely heavily on clinical indicators while underutilizing fundamental laboratory test parameters. This study aims to bridge this gap by leveraging the 12 most frequently tested laboratory indicators in diabetic patients to develop an optimized predictive model for diabetes complications.MethodsA comprehensive dataset was established through meticulous data collection from a high-volume tertiary hospital, followed by extensive data cleaning and classification. Various machine learning classifiers, including Random Forest, XGBoost, Support Vector Machine (SVM), and Multilayer Perceptron (MLP), were trained on this dataset to evaluate their predictive performance. We further introduced an ensemble learning model with Bayesian optimization to enhance accuracy and cost-efficiency. Additionally, feature importance analysis was conducted to refine the model by reducing testing costs while maintaining high predictive accuracy.ResultsOur ensemble model with Bayesian optimization demonstrated superior performance, achieving over 90% accuracy in predicting various diabetic complications, with an outstanding 98.50% accuracy and 99.76% AUC for diabetic nephropathy. Feature correlation analysis enabled a refined model that not only improved predictive accuracy but also reduced overall medical costs by 2.5% through strategic feature elimination.ConclusionsThis study makes three key contributions: (1) Development of a high-quality dataset based on fundamental laboratory indicators, (2) Creation of a highly accurate predictive model using ensemble learning and Bayesian optimization, particularly excelling in diabetic nephropathy prediction, and (3) Implementation of a cost-efficient diagnostic approach that reduces testing expenses without compromising accuracy. The proposed model provides a strong foundation for future research and practical clinical applications, demonstrating the potential of integrating machine learning with cost-conscious medical testing.