Bayesian Optimization for Hyperparameter Tuning in Healthcare for Diabetes Prediction

Optimizing hyperparameters is crucial for enhancing the performance of deep learning (DL) models. The process of configuring optimal hyperparameters, known as hyperparameter tuning, can be performed using various methods. Traditional approaches like grid search and random search have significant limitations. In contrast, Bayesian optimization (BO) utilizes a surrogate model and an acquisition function to intelligently navigate the hyperparameter space, aiming to provide deeper insights into performance disparities between naïve and advanced methods. This study evaluates BO's efficacy compared to baseline methods such as random search, manual search, and grid search across multiple DL architectures, including multi-layer perceptron (MLP), convolutional neural network (CNN), and LeNet, applied to the Modified National Institute of Standards and Technology (MNIST) and CIFAR-10 datasets. The findings indicate that BO, employing the tree-structured parzen estimator (TPE) search method and expected improvement (EI) acquisition function, surpasses alternative methods in intricate DL architectures such as LeNet and CNN. However, grid search shows superior performance in smaller DL architectures like MLP. This study also adopts a multi-objective (MO) perspective, balancing conflicting performance objectives such as accuracy, F1 score, and model size (parameter count). This MO assessment offers a comprehensive understanding of how these performance metrics interact and influence each other, leading to more informed hyperparameter tuning decisions.

With the sustainable development of the construction industry, recycled aggregate (RA) has been widely used in concrete preparation to reduce the environmental impact of construction waste. Compressive strength is an essential measure of the performance of recycled aggregate concrete (RAC). In order to understand the correspondence between relevant factors and the compressive strength of recycled concrete and accurately predict the compressive strength of RAC, this paper establishes a model for predicting the compressive strength of RAC using machine learning and hyperparameter optimization techniques. RAC experimental data from published literature as the dataset, extreme gradient boosting (XGBoost), random forest (RF), K-nearest neighbour (KNN), support vector machine regression Support Vector Regression (SVR), and gradient boosted decision tree (GBDT) RAC compressive strength prediction models were developed. The models were validated and compared using correlation coefficients (R2), Root Mean Square Error (RMSE), mean absolute error (MAE), and the gap between the experimental results of the predicted outcomes. In particular, The effects of different hyperparameter optimization techniques (Grid search, Random search, Bayesian optimization-Tree-structured Parzen Estimator, Bayesian optimization- Gaussian Process Regression) on model prediction efficiency and prediction accuracy were investigated. The results show that the optimal combination of hyperparameters can be searched in the shortest time using the Bayesian optimization algorithm based on TPE (Tree-structured Parzen Estimator); the BO-TPE-GBDT RAC compressive strength prediction model has higher prediction accuracy and generalisation ability. This high-performance compressive strength prediction model provides a basis for RAC’s research and practice and a new way to predict the performance of RAC.

Brain tumor is one of the most aggressive diseases nowadays, resulting in a very short life span if it is diagnosed at an advanced stage. The treatment planning phase is thus essential for enhancing the quality of life for patients. The use of Magnetic Resonance Imaging (MRI) in the diagnosis of brain tumors is extremely widespread, but the manual interpretation of large amounts of images requires considerable effort and is prone to human errors. Hence, an automated method is necessary to identify the most common brain tumors. Convolutional Neural Network (CNN) architectures are successful in image classification due to their high layer count, which enables them to conceive the features effectively on their own. The tuning of CNN hyperparameters is critical in every dataset since it has a significant impact on the efficiency of the training model. Given the high dimensionality and complexity of the data, manual hyperparameter tuning would take an inordinate amount of time, with the possibility of failing to identify the optimal hyperparameters. In this paper, we proposed a Bayesian Optimization-based efficient hyperparameter optimization technique for CNN. This method was evaluated by classifying 3064 T-1-weighted CE-MRI images into three types of brain tumors (Glioma, Meningioma, and Pituitary). Based on Transfer Learning, the performance of five well-recognized deep pre-trained models is compared with that of the optimized CNN. After using Bayesian Optimization, our CNN was able to attain 98.70% validation accuracy at best without data augmentation or cropping lesion techniques, while VGG16, VGG19, ResNet50, InceptionV3, and DenseNet201 achieved 97.08%, 96.43%, 89.29%, 92.86%, and 94.81% validation accuracy, respectively. Moreover, the proposed model outperforms state-of-the-art methods on the CE-MRI dataset, demonstrating the feasibility of automating hyperparameter optimization.

Given the increasing importance of machine learning (ML) in our lives, several algorithmic fairness techniques have been proposed to mitigate biases in the outcomes of the ML models. However, most of these techniques are specialized to cater to a single family of ML models and a specific definition of fairness, limiting their adaptibility in practice. We introduce a general constrained Bayesian optimization (BO) framework to optimize the performance of any ML model while enforcing one or multiple fairness constraints. BO is a model-agnostic optimization method that has been successfully applied to automatically tune the hyperparameters of ML models. We apply BO with fairness constraints to a range of popular models, including random forests, gradient boosting, and neural networks, showing that we can obtain accurate and fair solutions by acting solely on the hyperparameters. We also show empirically that our approach is competitive with specialized techniques that enforce model-specific fairness constraints, and outperforms preprocessing methods that learn fair representations of the input data. Moreover, our method can be used in synergy with such specialized fairness techniques to tune their hyperparameters. Finally, we study the relationship between fairness and the hyperparameters selected by BO. We observe a correlation between regularization and unbiased models, explaining why acting on the hyperparameters leads to ML models that generalize well and are fair.

This paper incorporates Bayesian estimation and optimization into a portfolio selection framework, particularly for high-dimensional portfolios in which the number of assets is larger than the number of observations. We leverage a constrained minimization approach, called the linear programming optimal (LPO) portfolio, to directly estimate effective parameters appearing in the optimal portfolio. We propose two refinements for the LPO strategy. First, we explore improved Bayesian estimates, instead of sample estimates, of the covariance matrix of asset returns. Second, we introduce Bayesian optimization (BO) to replace traditional grid-search cross-validation (CV) in tuning hyperparameters of the LPO strategy. We further propose modifications in the BO algorithm by (1) taking into account the time-dependent nature of financial problems and (2) extending the commonly used expected improvement acquisition function to include a tunable trade-off with the improvement’s variance. Allowing a general case of noisy observations, we theoretically derive the sublinear convergence rate of BO under the newly proposed EIVar and thus our algorithm has no regret. Our empirical studies confirm that the adjusted BO results in portfolios with higher out-of-sample Sharpe ratio, certainty equivalent, and lower turnover compared to those tuned with CV. This superior performance is achieved with a significant reduction in time elapsed, thus also addressing time-consuming issues of CV. Furthermore, LPO with Bayesian estimates outperforms the original proposal of LPO, as well as the benchmark equally weighted and plugin strategies.

In the telecommunications industry, predicting customer churn is crucial for maintaining business sustainability. High churn rates can negatively impact profitability, necessitating effective retention strategies. This research aims to enhance the accuracy of telecommunications customer churn prediction by optimizing the C4.5 classification algorithm through feature selection and hyperparameter tuning. The methods used include Information Gain for feature selection and hyperparameter tuning with Random Search and Grid Search. This study utilizes the Telco Customer Churn dataset from Kaggle, split into an 80:20 ratio for training and testing data. Six approaches are applied: (1) the basic C4.5 algorithm, (2) C4.5 with Information Gain, (3) C4.5 with Random Search, (4) C4.5 with Grid Search, (5) C4.5 with a combination of Information Gain and Random Search, and (6) C4.5 with a combination of Information Gain and Grid Search. The results indicate that the C4.5 algorithm alone achieves an accuracy of 74.09%, while applying Information Gain increases accuracy to 78.42%. Hyperparameter tuning with Random Search achieves the highest accuracy of 80.05%, whereas Grid Search reaches 77.71%. Combining Information Gain with Random Search results in an accuracy of 78.99%, while combining Information Gain with Grid Search yields an accuracy of 78.85%. These findings suggest that hyperparameter tuning using Random Search significantly improves accuracy compared to other methods, while Information Gain feature selection does not have a significant impact on performance in this context.

BackgroundNeoadjuvant immunotherapy has been shown to improve survival in patients with gastric cancer. This study sought to develop and validate a radiomics-based machine learning (ML) model for patients with locally advanced gastric cancer (LAGC), specifically to predict whether patients will achieve a major pathological response (MPR) following neoadjuvant immunotherapy. With its predictive capabilities, this tool shows promise for enhancing clinical decision-making processes in the future.MethodsThis study utilized a multicenter cohort design, retrospectively gathering clinical data and computed tomography (CT) images from 268 patients diagnosed with advanced gastric cancer who underwent neoadjuvant immunotherapy between January 2019 and December 2023 from two medical centers. Radiomic features were extracted from CT images, and a multi-step feature selection procedure was applied to identify the top 20 representative features. Nine ML algorithms were implemented to build prediction models, with the optimal algorithm selected for the final prediction model. The hyperparameters of the chosen model were fine-tuned using Bayesian optimization and grid search. The performance of the model was evaluated using several metrics, including the area under the curve (AUC), accuracy, and Cohen’s kappa coefficient.ResultsThree cohorts were included in this study: the development cohort (DC, n = 86), the internal validation cohort (IVC, n = 59), and the external validation cohort (EVC, n = 52). Nine ML models were developed using DC cases. Among these, an optimized Bayesian-LightGBM model, demonstrated robust predictive performance for MPR following neoadjuvant immunotherapy in LAGC patients across all cohorts. Specifically, within DC, the LightGBM model attained an AUC of 0.828, an overall accuracy of 0.791, a Cohen’s kappa coefficient of 0.552, a sensitivity of 0.742, a specificity of 0.818, a positive predictive value (PPV) of 0.586, a negative predictive value (NPV) of 0.867, a Matthews correlation coefficient (MCC) of 0.473, and a balanced accuracy of 0.780. Comparable performance metrics were validated in both the IVC and the EVC, with AUC values of 0.777 and 0.714, and overall accuracies of 0.729 and 0.654, respectively. These results suggested good fitness and generalization of the Bayesian-LightGBM model. Shapley Additive Explanations (SHAP) analysis identified significant radiomic features contributing to the model’s predictive capability. The SHAP values of the features wavelet.LLH_gldm_SmallDependenceLowGrayLevelEmphasis, wavelet.HHL_glrlm_RunVariance, and wavelet.LLH_glszm_LargeAreaHighGrayLevelEmphasis were ranked among the top three, highlighting their significant contribution to the model’s predictive performance. In contrast to existing radiomic models that exclusively focus on neoadjuvant chemotherapy, our model integrates both neoadjuvant immunotherapy and chemotherapy, thereby offering more precise predictive capabilities.ConclusionThe radiomics-based ML model demonstrated significant efficacy in predicting the pathological response to neoadjuvant immunotherapy in LAGC patients, thereby providing a foundation for personalized treatment strategies.

The endeavors toward sustainable transportation systems are a key concern for planners and decision-makers where increasing public transport attractiveness is essential. In this paper, a machine-learning-based predictive modeling approach is proposed for metro ridership prediction, considering the built environment around the stations; it is in the best interest of sustainable transport planning to ultimately contribute to the achievement of Sustainable Development Goals (UN-SDGs). A total of twelve parameters are considered as input features including time of day, day of the week, station, and nine types of land use density. Hence, a time-series database is used for model development and testing. Several machine learning (ML) models were evaluated for their predictive performance: ridge regression, lasso regression, elastic net, k-nearest neighbor, support vector regression, decision tree, random forest, extremely randomized trees, adaptive boosting, gradient boosting, extreme gradient boosting, and stacking ensemble learner. Bayesian optimization and grid search are combined with 10-fold cross-validation to tune the hyperparameters of each model. The performance of the developed models was validated based on the test dataset using five quantitative performance measures. The results demonstrated that, among the base learners, the decision tree showed the highest performance with an R2 of 87.4% on the test dataset. KNN and SVR were the second and third-best models among the base learners. Furthermore, the feature importance investigation explains the relative contribution of each type of land use density to the prediction of the metro ridership. The results showed that governmental land use density, educational facilities land use density, and mixed-use density are the three factors that play the most critical role in determining total ridership. The outcomes of this research could be of great help to the decision-making process for the best achievement of sustainable development goals in relation to sustainable transport and land use.

Efficient irrigation scheduling is crucial for optimizing crop yields and conserving water resources. In this paper, we present robust models for predicting soil moisture trends in cotton based on the integration of IoT, machine learning (ML), and deep learning (DL) frameworks. We access soil moisture and weather forecast data from IoT field sensors using web APIs, then process and incorporate them into ML and DL models to predict soil moisture trends. Specifically, in DL we used the Long Short-Term Memory (LSTM) with two LSTM architectures (Single Stream LSTM and Dual LSTM), while in ML we used the Gradient Boosting (GB), XGBoost, Random Forest (RF), and AdaBoost methods. Experiment results show that, Single Stream LSTM achieved MSE (0.0710) and (0.9684) R 2 outperforming its Dual counterpart which recorded MSE (0.1200) and (0.9466) R 2 , while under ML, Gradient Boosting (MSE = 0.2822, R² = 0.9909) outperforms other ML models. In general, LSTM methods achieved the lowest error while ML methods recorded the highest R 2 -score. Predictive methods presented in this paper, have shown great potential in estimating soil moisture patterns for optimal irrigation schedules in cotton, thereby enhancing water use efficiency and increasing crops yield.

The evolution of Electronic Health Records (EHR) has facilitated comprehensive patient record-keeping, enhancing healthcare delivery and decision-making processes. Despite these advancements, analyzing EHR data using ensemble machine learning methods poses unique challenges. These challenges include data dimensionality, imbalanced class distributions, and the need for effective hyperparameter tuning to optimize model performance. The study conducted a thorough comparative analysis of various ensemble machine learning (EML) models using Electronic Health Record (EHR) datasets. After addressing data imbalance and reducing dimensionality, the accuracy of the EML models showed significant improvement. Notably, the Gradient Boosting Machine (GBM) and CatBoost models exhibited superior performance with an accuracy of 73%, achieved through experiments involving dimensionality reduction and handling of imbalanced data. Furthermore, optimization techniques such as Grid Search and Random Search were employed to enhance the EML models. The results of model optimization revealed that the GBM + Random Search model performed the best, achieving an accuracy of 74%, followed by the XGBoost + Grid Search model with an accuracy of 73%. The GBM model also excelled in distinguishing between positive and negative classes, boasting the highest Area under Curve (AUC) value of 0.78, indicative of its superior classification capabilities compared to other models. This study emphasizes the significance of incorporating cutting-edge EML techniques into clinical workflows and emphasizes the revolutionary potential of GBM in classification modeling for patient severity conditions. Future research should focus on deep learning (DL) applications and the integration of these models.

This study investigates the application of machine learning models for real-time dynamic pricing strategies in the retail and e-commerce sectors. We employed three prominent supervised machine learning models—Linear Regression, Random Forest, and Gradient Boosting Machines (GBM)—to predict optimal prices using a dataset sourced from Kaggle. The models were trained and evaluated with a 70:30 train-test split, while hyperparameter tuning was performed using grid search and cross-validation. The results indicate that the Gradient Boosting Machines (GBM) model consistently outperformed the other models, achieving the lowest Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), and demonstrating a higher R-squared (R²) value. The comparative analysis highlights GBM's ability to capture complex interactions in dynamic pricing data, making it a robust choice for accurate price forecasting. The Random Forest model also delivered satisfactory results, balancing accuracy and computational efficiency, whereas the Linear Regression model showed higher prediction errors due to its limitations in modeling non-linear relationships. Real-time testing in a simulated environment confirmed the models' adaptability and responsiveness in a dynamic marketplace. These findings provide actionable insights for retail and e-commerce businesses, emphasizing the importance of model selection, hyperparameter optimization, and system integration to implement efficient dynamic pricing strategies. Future work should explore more extensive datasets and real-world applications to address seasonal variations, regional preferences, and consumer behavior, ensuring a more comprehensive and practical deployment of machine learning-driven dynamic pricing models.

This study explores the potential of the GPBoost approach for groundwater quality assessment in comparison to three other gradient boosting-based algorithms. Three methods, random search, grid search, and Bayesian optimization were used to find the optimal values of various hyperparameters with all four-gradient boosting-based algorithms. One hundred and two samples of Entropy weighted water quality index with 14 input parameters are used for assessing groundwater quality. The calculated EWQI values for drinking range between 80.4 and 394.96 in pre-monsoon and 39.6 to 338.79 during the post-monsoon period. Moreover, spatial distribution maps displayed that the central portions of the study area fall under medium water quality. The performances of models were compared based on multiple statistical criteria, including Correlation Coefficient (CC), root mean square error (RMSE), and mean absolute error (MAE). The results reveal that the CC value by all modeling approaches is more than 0.93, suggesting a comparable performance by all methods. Results in terms of RMSE values in predicting the EWQI values suggest GPBoost (random search) model performed better than the other three models, thus suggesting a competitive performance by GPBoost in comparison to other gradient boosting-based approaches. Relative importance analysis provided by random and grid search methods highlights the significance of NO3 −, Mg2+, TDS, EC, and TH as important input parameters for predicting EWQI.

Machine learning (ML) methods have shown great potential in predicting colorectal cancer (CRC) survival. However, the ML models introduced thus far have mainly focused on binary outcomes and have not considered the time-to-event nature of this type of modeling. This study aims to evaluate the performance of ML approaches for modeling time-to-event survival data and develop transparent models for predicting CRC-specific survival. The data set used in this retrospective cohort study contains information on patients who were newly diagnosed with CRC between December 28, 2012, and December 27, 2019, at West China Hospital, Sichuan University. We assessed the performance of 6 representative ML models, including random survival forest (RSF), gradient boosting machine (GBM), DeepSurv, DeepHit, neural net-extended time-dependent Cox (or Cox-Time), and neural multitask logistic regression (N-MTLR) in predicting CRC-specific survival. Multiple imputation by chained equations method was applied to handle missing values in variables. Multivariable analysis and clinical experience were used to select significant features associated with CRC survival. Model performance was evaluated in stratified 5-fold cross-validation repeated 5 times by using the time-dependent concordance index, integrated Brier score, calibration curves, and decision curves. The SHapley Additive exPlanations method was applied to calculate feature importance. A total of 2157 patients with CRC were included in this study. Among the 6 time-to-event ML models, the DeepHit model exhibited the best discriminative ability (time-dependent concordance index 0.789, 95% CI 0.779-0.799) and the RSF model produced better-calibrated survival estimates (integrated Brier score 0.096, 95% CI 0.094-0.099), but these are not statistically significant. Additionally, the RSF, GBM, DeepSurv, Cox-Time, and N-MTLR models have comparable predictive accuracy to the Cox Proportional Hazards model in terms of discrimination and calibration. The calibration curves showed that all the ML models exhibited good 5-year survival calibration. The decision curves for CRC-specific survival at 5 years showed that all the ML models, especially RSF, had higher net benefits than default strategies of treating all or no patients at a range of clinically reasonable risk thresholds. The SHapley Additive exPlanations method revealed that R0 resection, tumor-node-metastasis staging, and the number of positive lymph nodes were important factors for 5-year CRC-specific survival. This study showed the potential of applying time-to-event ML predictive algorithms to help predict CRC-specific survival. The RSF, GBM, Cox-Time, and N-MTLR algorithms could provide nonparametric alternatives to the Cox Proportional Hazards model in estimating the survival probability of patients with CRC. The transparent time-to-event ML models help clinicians to more accurately predict the survival rate for these patients and improve patient outcomes by enabling personalized treatment plans that are informed by explainable ML models.

Oil viscosity is an important parameter for estimating the flow behavior of crude oil under varying pressure and temperature regimes encountered in the subsurface and surface components of the petroleum production system. Although experimental data provides the most accurate characterization of this property, obtaining such data in the laboratory is often time-consuming and expensive. Alternatively, one of several empirical correlations can be used, albeit with often poor generalization to samples from differing thermodynamic systems or crude oil types. In this paper, we use a data-driven approach based on machine learning (ML) algorithms to create high-performing models for predicting the dead oil and bubblepoint oil viscosity of Nigerian crudes. Random Forest (RF) and Gradient Boosting (GB) ensembles, as well as Multi-Layer Perceptron (MLP) and Radial Basis Function (RBF) networks, are compared to several widely used correlations, and their performance is evaluated on a large dataset of crude oil samples collected from various Niger Delta oil fields. The ML models are built with minimal and easily accessible input parameters shared by the correlations, allowing for an objective assessment of their superiority to these correlations given the same input framework. These are reservoir temperature and oil API gravity for the dead oil viscosity modeling, and the solution gas-oil ratio and dead oil viscosity for the bubblepoint oil viscosity modeling. Furthermore, the model's reliability is tested by partitioning the dataset into training, validating, and testing sets, and optimal model configurations are achieved through Bayesian Optimization. On the dead oil viscosity modeling task, we achieve a modest improvement over the best-performing correlation developed by Al-Khafaji (1987). The RF model's R2 and mean absolute error (MAE) scores on the testing data are 0.831 and 3.423, respectively, versus 0.809 and 4.439 for the correlation method. However, for the bubblepoint oil viscosity modeling, the improvement recorded using the RF model over the best-performing correlation developed by Beggs and Robinson (1975) is remarkable. In this case, the R2 and MAE scores for the RF model are 0.924 and 0.441, respectively, compared to 0.574 and 0.813 for the correlation method. As a result, our ML models have outperformed conventional correlations for modeling oil viscosity. Ultimately, oil viscosity estimates from the developed ML models can provide a much more accurate characterization of the fluid behavior of Niger Delta crudes than the conventional correlations can. Furthermore, the ML models allow for continuous improvement of their generalization ability by introducing new data from various sources to the models.

The availability of data is an important thing in the decision-making process. With the trend of using Machine Learning and Data Mining, humans are facilitated in learning and finding hidden knowledge in the data. One technique used in the classification of data is the Deep Learning method. Deep Learning is widely used because of its ability to solve problems with satisfying results. In other words, Deep Learning has shown to be the one of the most powerful and widely used methods. However, the key of success depends on the setting the right parameters. With this reason, we need a way to optimize the use of hyperparameter in Deep Learning. The method commonly used in this case is Grid Search. However, this method requires a long time and high computation. While other methods such as Random Search and Bayesian Optimization, both are very reliable and efficient compared to Grid Search. The results of the experiments conducted in this study indicate that Bayesian Optimization is better than Random Search in performing hyperparameter optimization processes in Deep Learning.