Hyperparameters optimization for the machine learning

The development of hyperparameter optimization (HPO) algorithms is an important topic within both the machine learning and data management domains. While numerous strategies employing early stopping mechanisms have been proposed to bolster HPO efficiency, there remains a notable deficiency in understanding how the selection of early stopping metrics influences the reliability of early stopping decisions and, by extension, the broader HPO outcomes. This paper undertakes a systematic exploration of the impact of metric selection on the effectiveness of early stopping-based HPO. Specifically, we introduce a set of metrics that incorporate uncertainty and highlight their practical significance in enhancing the reliability of early stopping decisions. Our empirical experiments on HPO and NAS benchmarks show that using training loss as an early stopping metric in the early training stages improves HPO outcomes by up to 24.76% compared to the more widely accepted validation loss. Furthermore, integrating uncertainty into the metric yields an additional improvement of up to 4% under budget constraints, translating into meaningful resource savings and scalability benefits in large-scale HPO scenarios. These findings demonstrate the critical role of metric selection while shedding light on the potential implications of integrating uncertainty as a metric. This research provides empirical insights that serve as a compass for the selection and formulation of metrics, thereby contributing to a more profound comprehension of mechanisms underpinning early stopping-based HPO.

Reinforcement-based Method for Simultaneous Clustering Algorithm Selection and its Hyperparameters Optimization

Support Vector Regression (SVR) is one of the frequently used methods for regression estimation. Kernel selection in SVR is of great importance for the training and estimation of the model. Determining the parameters of the selected kernel is at least as important as the selected kernel. The two parameters of the Radial Basis Function (RBF) kernel are determined stochastically with 5 different current and popular meta-heuristic optimization algorithms and their performances are evaluated on 7 different data sets. The performances of Particle Swarm Optimization (PSO), Salp Swarm Algorithm (SSA), Kepler Optimization Algorithm (KOA), Weighted Average Algorithm (WAA) and Hippopotamus Optimization (HO) algorithms in hyper-parameter optimization are compared. Data sets are separated into training and test with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{k}$</tex>-10 cross-validation. WAA was the model with the highest <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{R}^{\mathbf{2}}$</tex> value among the models, with values of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$0.8521,0.9493$</tex>, and 0.8392 in 3 of the 7 data sets, respectively. HO was the model with the highest <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{R}^{\mathbf{2}}$</tex> value among the models, with values of 0.9374, and 0.9375 in 2 of the 7 datasets, respectively. The SVR model, which did not undergo hyper-parameter optimization with the optimization algorithm, was the model with the highest <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{R}^{\mathbf{2}}$</tex> value among the models, with values of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 8 6 6 9}$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{0. 5 9 2 8}$</tex> in 2 of the 7 datasets. Among the algorithms performing hyperparameter optimization, KOA was found to be the fastest algorithm in terms of run time, while HO was found to be the slowest.

Finding the optimal hyper-parameters values for a given problem is essential for most machine learning algorithms. In this paper, we propose a novel hyper-parameter optimization algorithm that is very simple to implement and still competitive with the state-of-the-art L-SHADE variant of Differential Evolution. While the most common method for hyper-parameter optimization is a combination of grid and manual search, random search has recently shown itself to be more effective and has been proposed as a baseline against which to measure other methods. In this paper, we compare three optimization algorithms, namely, the state-of-the-art L-SHADE algorithm, the random search algorithm, and our novel and simple adaptive random search algorithm. We find that our simple adaptive random search strategy is capable of finding parameters that achieve results comparable to the state-of-the-art L-SHADE algorithm, both of which achieve significantly better performance than random search when optimizing the hyper-parameters of the state-of-the-art XGBoost algorithm for 11 datasets. Because of the significant performance increase of our simple algorithm when compared to random search, we propose this as the new go-to method for tuning the hyper-parameters of machine learning algorithms when desiring a simple-to-implement algorithm, and also propose to use this algorithm as a new baseline against which other strategies should be measured.

Background: A grid search, at the cost of training and testing a large number of models, is an effective way to optimize the prediction performance of deep learning models. A challenging task concerning grid search is time management. Without a good time management scheme, a grid search can easily be set off as a "mission" that will not finish in our lifetime. In this study, we introduce a heuristic three-stage mechanism for managing the running time of low-budget grid searches with deep learning, sweet-spot grid search (SSGS) and randomized grid search (RGS) strategies for improving model prediction performance, in an application of predicting the 5-year, 10-year, and 15-year risk of breast cancer metastasis. Methods: We develop deep feedforward neural network (DFNN) models and optimize the prediction performance of these models through grid searches. We conduct eight cycles of grid searches in three stages, focusing on learning a reasonable range of values for each of the adjustable hyperparameters in Stage 1, learning the sweet-spot values of the set of hyperparameters and estimating the unit grid search time in Stage 2, and conducting multiple cycles of timed grid searches to refine model prediction performance with SSGS and RGS in Stage 3. We conduct various SHAP analyses to explain the prediction, including a unique type of SHAP analyses to interpret the contributions of the DFNN-model hyperparameters. Results: The grid searches we conducted improved the risk prediction of 5-year, 10-year, and 15-year breast cancer metastasis by 18.6%, 16.3%, and 17.3%, respectively, over the average performance of all corresponding models we trained using the RGS strategy. Conclusions: Grid search can greatly improve model prediction. Our result analyses not only demonstrate best model performance but also characterize grid searches from various aspects such as their capabilities of discovering decent models and the unit grid search time. The three-stage mechanism worked effectively. It not only made our low-budget grid searches feasible and manageable but also helped improve the model prediction performance of the DFNN models. Our SHAP analyses not only identified clinical risk factors important for the prediction of future risk of breast cancer metastasis, but also DFNN-model hyperparameters important to the prediction of performance scores.

Ensemble forecasts are important due to their ability to characterize forecast uncertainty, which is fundamental when forecasting extreme weather. Ensemble forecasts are however often biased and underdispersed and thus need to be post-processed.A common approach for this is the use of ensemble model output statistics (EMOS), where a parametric distribution is fitted with a limited number of predictors. With recent advances in computer science and increased amounts of data available, machine learning techniques, like random forests, are becoming more popular for high dimensional regression problems. In this research, we explore the use of the quantile regression forest (QRF), a random forest adapted for conditional quantile estimation, applied to medium range gridded probabilistic precipitation forecasts. QRFs are non-parametric and allow for a larger number of predictors, which means they can possibly consider more dependencies that might otherwise not be captured with a simple EMOS.A QRF takes several hyperparameters that influence the way the decision trees in the forest are constructed. We explore the minimum number of samples needed in a leaf to split it (minimum node size) and the number of predictors explored in each split (mtry). A hyperparameter space is constructed by setting ranges for both minimum node size and mtry, and the optimal hyperparameter set is determined by performing a cross validated grid search. Here, each model is assessed based on the continuous ranked probability skill score (CRPSS). For comparison, EMOS is applied with a zero-adjusted gamma (ZAGA) distribution, using a limited number of predictors that are physically correlated to precipitation. Both methods are verified on a separate testing data set and evaluated using several scores, including CRPSS and Brier skills score (BSS).We consider 4 years (November 2018 &amp;#8211; October 2022) of archived operational ECMWF-IFS ensemble forecasts for the Netherlands. The data is split into November 2018 &amp;#8211; October 2021 for training and cross-validation, and October 2021 &amp;#8211; October 2022 for testing, separating data for season, initialization time and lead-time. Forecasts are post-processed up to +10 days. Ensemble statistics on 60+ forecast variables are used as predictors. Spatially and temporally aggregated, gauge-adjusted radar observations are used as predictand. The raw ensemble is considered as the benchmark.The results of this research will determine what method will be used to post-process the ensemble precipitation forecasts in the context of the early warning center (EWC) of the Royal Netherlands Meteorological Institute. The most suitable method could differ between shorter and longer lead times.

Accurate internet traffic prediction is a key challenge in managing next-generation networks such as 6G. This paper presents a novel approach based on Graph Neural Networks (GNNs) for predicting internet traffic in 6G networks. The proposed model integrates Graph Attention Networks (GAT) and Transformer architectures to learn spatial and temporal dependencies in traffic data. A K-Nearest Neighbors (KNN)-based graph construction method is utilized to represent spatial relationships between network cells. The model’s performance is enhanced by leveraging a Genetic Algorithm (GA) for hyperparameter optimization. Experimental results demonstrate the effectiveness of the proposed model in achieving superior prediction accuracy, as evidenced by improvements in RMSE, and MAE compared to baseline models. This work offers a scalable solution for traffic prediction in 6G networks.

As mobile internet technology rapidly advances, the importance of smartphones outfitted with diverse sensors has grown, particularly for transportation mode detection (TMD). Information regarding transportation modes is essential for improving urban development, traffic control, and route planning. Despite extensive research in TMD utilizing conventional machine learning techniques, the results often fall below expectations, largely due to their dependence on handcrafted features. Moreover, while long short‐term memory (LSTM) methods are employed for TMD, they generally fail to capture specific local nuances in complex time‐series data and encounter difficulties optimizing feature selection. To address these issues, we propose a refined TMD model incorporating a sophisticated spatial attention‐based transductive LSTM (TLSTM) network, further improved by reinforcement learning (RL) with an innovative scope loss function for efficient feature selection. The TLSTM framework is adept at identifying minor temporal variations in data, advancing beyond traditional LSTM models by applying transductive learning (TL) to concentrate on data points proximate to the test instance. The scope loss function balances the utilization of known data with the exploration of new data vistas. Additionally, our model integrates an upgraded differential evolution (DE) algorithm for intricate hyperparameter optimization, which includes a novel mutation strategy coupled with k‐means clustering to pinpoint crucial clusters. Extensive evaluations on the SHL, HTC, and US‐TMD datasets demonstrate that our advanced TMD algorithm outperforms existing competitive models. The F‐measure recorded by our system on these datasets stands at 90.140%, 88.315%, and 89.110%, respectively. This state‐of‐the‐art approach highlights the efficacy of TLSTM amalgamated with spatial attention and RL techniques in boosting the precision and dependability of TMD.

Soil organic carbon (SOC) plays a critical role in global carbon cycling, influencing climate regulation, soil fertility, and sustainable land management. However, accurate SOC prediction remains a challenging task due to the complex, high-dimensional, and nonlinear nature of soil data. Recent advances in machine learning (ML) have improved SOC estimation, yet these models often suffer from overfitting and computational inefficiency when effective feature selection and hyperparameter tuning are not applied. To address these challenges, we propose a novel integration of the Ninja Optimization Algorithm (NiOA) for simultaneous feature selection and hyperparameter optimization, aimed at enhancing both predictive accuracy and computational efficiency. In our experimental setup, 80% of the dataset was allocated for training and 20% for testing. The baseline Support Vector Machine (SVR) model achieved a mean squared error (MSE) of 0.00513, which was reduced to 0.00011 after applying binary NiOA (bNiOA) for feature selection. After full NiOA-based hyperparameter tuning, the MSE improved further to <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="m1"><mml:mrow><mml:mn>7.52</mml:mn><mml:mo>×</mml:mo><mml:mn>1</mml:mn><mml:msup><mml:mrow><mml:mn>0</mml:mn></mml:mrow><mml:mrow><mml:mo>−</mml:mo><mml:mn>7</mml:mn></mml:mrow></mml:msup></mml:mrow></mml:math> , corresponding to a 99.98% reduction in prediction error. Thus, the proposed NiOA-enhanced framework demonstrates considerable potential in advancing SOC modeling. It offers a scalable, interpretable, and high-precision solution that can be effectively applied in data-scarce environments, particularly in support of sustainable land management and climate change adaptation strategies.

The generalization of Physics-Informed Neural Networks (PINNs) used to solve the inhomogeneous Helmholtz equation in a simplified three-dimensional room is investigated. PINNs are appealing since they can efficiently integrate a partial differential equation and experimental data by minimizing a loss function. However, a previous study experienced limitations in acoustics regarding the source term. A challenging but realistic excitation case is a confined (e.g., single-point) excitation area, yielding a smooth spatial wave field periodically with the wavelength. Compared to studies using smooth (unrealistic) sound excitation, the network’s generalization capabilities regarding a realistic sound excitation are addressed. Different methods like hyperparameter optimization, adaptive refinement, Fourier feature engineering, and locally adaptive activation functions with slope recovery are tested to tailor the PINN’s accuracy to an experimentally validated finite element analysis reference solution computed with openCFS. The hyperparameter study and optimization are conducted regarding the network depth and width, the learning rate, the used activation functions, and the deep learning backends (PyTorch 2.5.1, TensorFlow 2.18.0 1, TensorFlow 2.18.0 2, JAX 0.4.39). A modified (feature-engineered) PINN architecture was designed using input feature engineering to include the dispersion relation of the wave in the neural network. For smoothly (unrealistic) distributed sources, it was shown that the standard PINNs and the feature-engineered PINN converge to the analytic solution, with a relative error of 0.28% and 2×10−4%, respectively. The locally adaptive activation functions with the slope lead to a relative error of 0.086% with a source sharpness of s=1 m. Similar relative errors were obtained for the case s=0.2 m using adaptive refinement. The feature-engineered PINN significantly outperformed the results of previous studies regarding accuracy. Furthermore, the trainable parameters were reduced to a fraction by Bayesian hyperparameter optimization (around 5%), and likewise, the training time (around 3%) was reduced compared to the standard PINN formulation. By narrowing this excitation towards a single point, the convergence rate and minimum errors obtained of all presented network architectures increased. The feature-engineered architecture yielded a one order of magnitude lower accuracy of 0.20% compared to 0.019% of the standard PINN formulation with a source sharpness of s=1 m. It outperformed the finite element analysis and the standard PINN in terms time needed to obtain the solution, needing 15 min and 30 s on an AMD Ryzen 7 Pro 8840HS CPU (AMD, Santa Clara, CA, USA) for the FEM, compared to about 20 min (standard PINN) and just under a minute of the feature-engineered PINN, both trained on a Tesla T4 GPU (NVIDIA, Santa Clara, CA, USA).

With the advances in computational power and numerical methods, analysts can now rely on first-principle simulations to predict ultra-fine details in a variety of applications. Advances in machine learning (ML) have produced algorithms that can now learn high-level abstractions via hierarchical models. This project aims to leverage advances in ML techniques and the available high-resolution simulation data to develop a novel modeling and simulation (M\&S) methodology for reactor safety analysis. While application-agnostic ML techniques are available, complex physics constraints need to be incorporated into ML techniques to build ML-based closures for computationally efficient predictive simulations. This project intends to develop a physics-guided data-driven multi-scale methodology for M\&S of advanced reactors. The project focuses on thermal fluid (T/F) phenomena, which play major roles in advanced reactor safety. Specifically, we propose a data-driven coarse-mesh turbulence model based on local flow features for the transient analysis of thermal mixing and stratification in a sodium-cooled fast reactor (SFR). The model has a coarse-mesh setup to ensure computational efficiency, while it is trained by fine-mesh computational fluid dynamics (CFD) data with Reynolds-averaged Navier-Stokes (RANS) turbulence model to ensure accuracy. Three different neural networks are developed and tested for loss-of-flow transients in the hot pool of SFR, i.e. the densely connected convolutional neural network (DCNN), long-short-term-memory network based on proper orthogonal decomposition (POD-LSTM), and the DCNN informed by LSTM (DCNN-LSTM). The performances of these three neural networks are evaluated based on baseline models. The DCNN-LSTM model has been chosen for further hyperparameter optimization. Furthermore, based on a simplified two-dimensional case, uncertainty quantification (UQ) of the developed ML-based closure are investigated with three methods, i.e. Monte Carlo dropout, deep ensemble, and Bayesian neural network. The developed ML-based turbulent viscosity closure relation based on deep ensemble is then integrated into the system analysis module SAM and serves as a term in the conservation equations. Such a SAM-ML based procedure guarantees that the obtained results are consistent with the physical constraints of the thermal-fluid system. The SAM-ML simulation on the same loss-of-flow transient showed comparable accuracy with the CFD simulation but with a much coarser mesh setup. Last but not least, the ML-based closure improvement with the support of higher-fidelity data from large eddy simulation (LES) is discussed. As a first step towards this direction, a baseline LES simulation is performed to obtain comparable data with RANS results. Based on the early results, future investigation on further improving the ML-based closure is discussed. We believe the developed approach that combines scientific machine learning with nuclear system analysis code can benefit the advanced reactor community as more accurate safety analyses will better characterize reactor safety margins and reduce licensing efforts.

Machine learning techniques have achieved remarkable development in recent years. However, the performance of many machine learning models usually involves careful tuning of hyperparameters. The hyperparameter optimization (HPO) is usually a high-dimensional black box optimization problem and often faces expensive function evaluations. Besides, the task of HPO has gained great attention in academy and industry. In this paper, a novel method for hyperparameter optimization is proposed, referred to as surrogate model assisted quantum-inspired evolutionary algorithm (SA-QEA), which incorporates the principles of quantum-inspired evolutionary algorithm (QEA) and an efficient search framework based on a surrogate model. In the proposed algorithm, we adopt a single individual QEA with neighborhood exploration as the evolution scheme to generate the candidate solutions, and multivariate adaptive regression splines (MARS) is used as a surrogate to approximate the objective function around the individuals. Through conducting comprehensive experimental evaluations on two benchmark problems and three machine learning models, we test our proposed algorithm and compare it with other widely used methods. The results achieve competitive performance and demonstrate the effectiveness of SA-QEA for hyperparameter optimization.

AdaBoost algorithm is a typical Boosting algorithm, which belongs to a successful representative in the Boosting family. This algorithm can upgrade a weak classifier with a better classification effect than random classification to a strong classifier with high classification accuracy, where n_estimators represents the number of iterations of the base classifier. If the value is too large, it will easily cause the model to overfit, if it is too small, it is easy. The model is under-fitting, and the parameter setting is not set randomly, but according to the current status of the data set. Aiming at the problem that the number of iterations in the AdaBoost algorithm is uncertain, this paper introduces a Bayesian optimization algorithm for hyperparameter tuning, which makes the value of hyper parameter in AdaBoost algorithm suitable for the current data set, and finally obtains a hyperparameter optimization AdaBoost algorithm. The experiment result shows the method that adopt Bayesian optimization algorithm for hyperparameter optimization and apply the optimized hyperparameter value to the AdaBoost algorithm does not only improves the classification accuracy of the AdaBoost algorithm, but also avoids overfitting and underfitting of the model.

Contact: Oliver.Serang@Childrens.Harvard.edu

Predicting hospital readmissions among diabetic patients has been of substantial interest to many researchers and health decision makers to provide quality and cost effective health care. In this paper, we optimize an efficient classifier for hospital readmission prediction within 30 days of discharge, the hybrid fuzzy weighted k-nearest neighbor (HFWkNN) method. The optimization is performed through the hyperparameter γ, ε and εmin introduced in the membership function of HFWkNN. These hyperparameters are important since they directly control the behavior of the training phase and significantly affect the performance of the method. A relationship is established between the performance of the proposed HFWkNN and the hyperparameters using two powerful optimization algorithms; grid search and random search. Experimental results show improved performance using the optimized hyperparameters in the resulting hospital readmission prediction model. To show that HFWkNN model can be generalized, the results are compared with those of several kNN-based algorithms using two additional classification datasets in addition to the hospital readmission dataset. They are IRIS dataset and breast cancer dataset. These are common benchmark sets with real-world data. The model so far achieved higher classification accuracy than FkNN model. The best hyperparameter values for HFWkNN with grid search are γ=0.2249, ε=1.112 and ε <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">min</sub> =0.01. Also, HFWkNN shows a performance of 80.00%, meaning that it has generalized well on the different data sets.