A monitoring framework for deployed machine learning models with supply chain examples

The ongoing Russia–Ukraine conflict has exacerbated the global crisis of natural gas supply, particularly in Europe. During the winter season, major importers of liquefied natural gas (LNG), such as South Korea and Japan, were directly affected by fluctuating spot LNG prices. This study aimed to use machine learning (ML) to predict the Japan Korea Marker (JKM), a spot LNG price index, to reduce price fluctuation risks for LNG importers such as the Korean Gas Corporation (KOGAS). Hence, price prediction models were developed based on long short-term memory (LSTM), artificial neural network (ANN), and support vector machine (SVM) algorithms, which were used for time series data prediction. Eighty-seven variables were collected for JKM prediction, of which eight were selected for modeling. Four scenarios (scenarios A, B, C, and D) were devised and tested to analyze the effect of each variable on the performance of the models. Among the eight variables, JKM, national balancing point (NBP), and Brent price indexes demonstrated the largest effects on the performance of the ML models. In contrast, the variable of LNG import volume in China had the least effect. The LSTM model showed a mean absolute error (MAE) of 0.195, making it the best-performing algorithm. However, the LSTM model demonstrated a decreased in performance of at least 57% during the COVID-19 period, which raises concerns regarding the reliability of the test results obtained during that time. The study compared the ML models’ prediction performances with those of the traditional statistical model, autoregressive integrated moving averages (ARIMA), to verify their effectiveness. The comparison results showed that the LSTM model’s performance deviated by an MAE of 15–22%, which can be attributed to the constraints of the small dataset size and conceptual structural differences between the ML and ARIMA models. However, if a sufficiently large dataset can be secured for training, the ML model is expected to perform better than the ARIMA. Additionally, separate tests were conducted to predict the trends of JKM fluctuations and comprehensively validate the practicality of the ML models. Based on the test results, LSTM model, identified as the optimal ML algorithm, achieved a performance of 53% during the regular period and 57% d during the abnormal period (i.e., COVID-19). Subject matter experts agreed that the performance of the ML models could be improved through additional studies, ultimately reducing the risk of price fluctuations when purchasing spot LNG.

The rising machine learning (ML) models have become the preferred way for landslide detection based on remote sensing images, but the performance of these models in a sample-free area are rarely concerned in many studies. In this study, we used a cross-validation method (training model in one area and validation in another) to compare the model portability of trained ML models applied in an “off-site” area, as a consideration of the landslide detection ability of these models in sample-free areas. We integrate nighttime light imagery, multi-seasonal optical Landsat time-series and digital elevation data, and we employed support vector machines (SVM), artificial neural networks (ANN) and random forest (RF) models to classify the satellite imagery and identify landslides. Samples of two scenarios generated from two subareas of the Jiuzhaigou disaster-stricken region are used for the cross-application and accuracy evaluation of three ML models. The results revealed that when the trained models are applied in areas outside those in which they were developed, the landslide identification accuracy of these three models has declined. Especially for the SVM and ANN models, the accuracy is greatly reduced and there appears a seriously imbalanced user’s and producer’s accuracy. However, although the performance of the RF model is lower than that of SVM and ANN models in their local area, the RF model exhibits stable portability, and retains the original performance and achieves a satisfactory balance between overestimation and underestimation in “off-site” areas. An additional validation from a new area proved that the landslide detection performance of the RF model with stable portability is higher than that of the SVM and ANN models in “off-site” areas. The results suggest that evaluating the model portability through cross-application can be a useful way to determine the most suitable model for landslide detection in “off-site” areas with a similar geographic environment to model development areas, so as to maximize the accuracy of landslide detection based on limited samples.

Perinatal depression (PND) is one of the most common medical complications during pregnancy and postpartum period, affecting 10–20% of pregnant individuals. Black and Latina women have higher rates of PND, yet they are less likely to be diagnosed and receive treatment. Machine learning (ML) models based on Electronic Medical Records (EMRs) have been effective in predicting postpartum depression in middle-class White women but have rarely included sufficient proportions of racial and ethnic minorities, which contributed to biases in ML models for minority women. Our goal is to determine whether ML models could serve to predict depression in early pregnancy in racial/ethnic minority women by leveraging EMR data. We extracted EMRs from a hospital in a large urban city that mostly served low-income Black and Hispanic women (N=5,875) in the U.S. Depressive symptom severity was assessed from a self-reported questionnaire, PHQ-9. We investigated multiple ML classifiers, used Shapley Additive Explanations (SHAP) for model interpretation, and determined model prediction bias with two metrics, Disparate Impact, and Equal Opportunity Difference. While ML model (Elastic Net) performance was low (ROCAUC=0.67), we identified well-known factors associated with PND, such as unplanned pregnancy and being single, as well as underexplored factors, such as self-report pain levels, lower levels of prenatal vitamin supplement intake, asthma, carrying a male fetus, and lower platelet levels blood. Our findings showed that despite being based on a sample mostly composed of 75% low-income minority women (54% Black and 27% Latina), the model performance was lower for these communities. In conclusion, ML models based on EMRs could moderately predict depression in early pregnancy, but their performance is biased against low-income minority women.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

Breast cancer continues to be a leading cause of death among women in the world. The prediction of survival outcomes based on treatment modalities, i.e., chemotherapy, hormone therapy, surgery, and radiation therapy is an essential step towards personalization in treatment planning. However, Machine Learning (ML) models may improve these predictions by investigating intricate relationships between clinical variables and survival. This study investigates the performance of several ML models to predict survival rate in patients undergoing diverse breast cancer treatments i.e., chemotherapy, hormone therapy, surgery and radiation using multiple clinical parameters. The dataset consisted of 5000 samples and turned into downloaded from Kaggle. The models assessed blanketed Support Vector Machines (SVM), K-Nearest Neighbor (KNN), AdaBoost, Gradient Boosting, Random Forest, Gaussian Naive Bayes, Logistic Regression, Extreme Gradient Boosting (XG boost), and Decision tree. Performance of the models was assessed using parameters such as Accuracy, Precision, Recall, F1-Score and Area under the Receiver Operating Characteristic Curve (AUC-ROC). SHAP (SHapley Additive exPlanations) values analysis was done to provide an explanation for the impact of a feature on model predictions using Waterfall and Beeswarm plots. Anticipated baseline (E(f(x))) were in comparison to the predictions (f(x)) for each therapy group. Performance of Gradient Boosting was shown to be the best with an Accuracy: 0.972, Precision: 0.973, Recall: 0.972, F1-Score: 0.973, and AUC-ROC Score: 0.997. Chemotherapy had a notably bad impact on survival, with an f(x) of -0.274 and an E(f(x)) of -0.025. Hormone therapy showed the maximum detrimental effect on survival, with an f(x) of -0.408. Surgical operation had an extraordinarily impartial impact (f(x) = -0.041), even as radiation therapy positively impacted survival results with an f(x) of 0.22. Gradient Boosting was the most predictive algorithm for breast cancer survival outcomes. This SHAP-primarily based analysis provides a complete knowledge of ways one-of-a-kind treatments have an effect on survival predictions in breast cancer patients. Radiation therapy indicates the most tremendous effect on survival, whilst hormone therapy reveals the maximum poor effect. Future studies need to explore personalized treatment strategies that comprise these insights to enhance patient effects.

Accurate wind speed forecasting is essential for daily activities and socio‐economic production. This study employs an explainable machine learning (ML) model to correct wind speed forecasts from a numerical weather prediction (NWP) model over weather stations across lead times of 1–24 hr in Zhejiang Province, China. Eleven meteorological variables derived from the NWP model are used as predictors. The results demonstrate that ML model substantially improves the original NWP forecasts, reducing the root mean square error of approximately 20%–40% at most stations and about 10% at certain coastal stations. The corrected forecasts also consistently achieve higher Kling‐Gupta Efficiency scores and Forecast Accuracy across all lead times. To enhance model interpretability, we apply Split Count and SHapley Additive exPlanation analyses. The results indicate that the friction velocity (UST) is the most influential predictor and reveal a threshold beyond which the ML model's correction performance improves markedly. The ML model performance slightly declines in coastal regions, likely due to the inconsistent representation of UST between the land and water surface in the NWP model. Monthly analysis further shows superior model performance in November and December, when UST values are higher. This study underscores the value of explainable ML models in improving NWP wind forecasts and provides insights for correcting meteorological variables in NWP models.

Examining the condition of groundwater resources and the impact of droughts is valuable for effective water resources management. Today, machine learning (ML) models are recognized as one of the useful tools in time series predictions. In this study, the groundwater condition of one of the most important aquifers in northwest Iran was investigated using MODFLOW, followed by estimating the groundwater resource index (GRI) utilizing the multivariate adaptive regression spline (MARS) and least squares support vector regression (LSSVR) for a period between 2001 and 2019. Meteorological and hydrological drought indicators along with precipitation and flow rate were used as input variables for prediction. The simulation results revealed a groundwater level decrease since the aquifer withdrawal amount is more than the recharge amount. Besides, results showed that there is a limited interaction between surface water and groundwater resources, mainly caused by the decrease in the river flow and aquifer groundwater level drop. Both ML models performed well in GRI estimation, using groundwater flow, streamflow drought index, standardized precipitation index, and runoff as input variables. The performance of the MARS model with RMSE, MAE, and NSE error evaluation criteria of 0.37, − 0.19, and 0.83, respectively, exerted slightly better results than LSSVR with RMSE, MAE, and NSE of 0.48, − 0.06, and 0.80, respectively. The findings reveal the appropriate performance of both models in forecasting drought indicators, highlighting the necessity of using ML models in hydrology and drought prediction problems.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

Machine Learning Can Accurately Predict Overnight Stay, Readmission, and 30-Day Complications Following Anterior Cruciate Ligament Reconstruction

Enhancing estimation of water quality index using stacking machine learning techniques: The case of Southern Bug River.

Machine Learning (ML) models have been extensively applied in various fields to enhance prediction. For instance, in cybersecurity, they examine large amounts of data, establish trends in the data, and draw insights from previous events, to enhance detection and respond to cyber threats. Random Forest, Logistic Regression, K-Nearest Neighbor and LSTM are some of the popular ML models vastly used for anomaly detection. The accuracy of these models is therefore the cornerstone of the organization’s information systems security since wrong predictions result to false positives and negatives which significantly reduce employees’ output and may result into workers’ frustrations when interacting with the information systems. Among the many factors that affect ML model performance, data pre-processing has been underscored. Using the various publicly available datasets, this paper examines the impact of data preprocessing techniques on selected ML model architectures’ performance. Training time, Accuracy, Precision, Recall and F1 scores are used for evaluating the ML models’ performance.

The increasing application of machine learning (ML) in critical areas such as healthcare and finance highlights the importance of fairness in ML models, challenged by biases in training data that can lead to discrimination. We introduce ‘FairUS’, a novel pre-processing method for reducing bias in ML models utilizing the Conditional Generative Adversarial Network (CTGAN) to synthesize upsampled data. Unlike traditional approaches that focus solely on balancing subgroup sample sizes, FairUS strategically optimizes the quantity of synthesized data. This optimization aims to achieve an ideal balance between enhancing fairness and maintaining the overall performance of the model. Extensive evaluations of our method over several canonical datasets show that the proposed method enhances fairness by 2.7 times more than the related work and 4 times more than the baseline without mitigation, while preserving the performance of the ML model. Moreover, less than a third of the amount of synthetic data was needed on average. Uniquely, the proposed method enables decision-makers to choose the working point between improved fairness and model’s performance according to their preferences.

Background/Objective: Subarachnoid hemorrhage (SAH) is associated with high morbidity and mortality rates, necessitating prognostic algorithms to guide decisions. Our study evaluates the use of machine learning (ML) models for predicting 1-month and 1-year mortality among SAH patients using national electronic health records (EHR) system. Methods: Retrospective cohort of 29,274 SAH patients, identified through national EHR system from January 2017 to December 2022, was analyzed, with mortality data obtained from central civil registration system in Türkiye. Variables included (n = 102) pre- (n = 65) and post-admission (n = 37) data, such as patient demographics, clinical presentation, comorbidities, laboratory results, and complications. We employed logistic regression (LR), decision trees (DTs), random forests (RFs), and artificial neural networks (ANN). Model performance was evaluated using area under the curve (AUC), average precision, and accuracy. Feature significance analysis was conducted using LR. Results: The average age was 56.23 ± 16.45 years (47.8% female). The overall mortality rate was 22.8% at 1 month and 33.3% at 1 year. One-month mortality increased from 20.9% to 24.57% (p < 0.001), and 1-year mortality rose from 30.85% to 35.55% (p < 0.001) in the post-COVID period compared to the pre-COVID period. For 1-month mortality prediction, the ANN, LR, RF, and DT models achieved AUCs of 0.946, 0.942, 0.931, and 0.916, with accuracies of 0.905, 0.901, 0.893, and 0.885, respectively. For 1-year mortality, the AUCs were 0.941, 0.927, 0.926, and 0.907, with accuracies of 0.884, 0.875, 0.861, and 0.851, respectively. Key predictors of mortality included age, cardiopulmonary arrest, abnormal laboratory results (such as abnormal glucose and lactate levels) at presentation, and pre-existing comorbidities. Incorporating post-admission features (n = 37) alongside pre-admission features (n = 65) improved model performance for both 1-month and 1-year mortality predictions, with average AUC improvements of 0.093 ± 0.011 and 0.089 ± 0.012, respectively. Conclusions: Our study demonstrates the effectiveness of ML models in predicting mortality in SAH patients using big data. LR models’ robustness, interpretability, and feature significance analysis validate its importance. Including post-admission data significantly improved all models’ performances. Our results demonstrate the utility of big data analytics in population-level health outcomes studies.

BackgroundThe large amount of heterogeneous data collected in surgical/endoscopic practice calls for data-driven approaches as machine learning (ML) models. The aim of this study was to develop ML models to predict endoscopic sleeve gastroplasty (ESG) efficacy at 12 months defined by total weight loss (TWL) % and excess weight loss (EWL) % achievement. Multicentre data were used to enhance generalizability: evaluate consistency among different center of ESG practice and assess reproducibility of the models and possible clinical application. Models were designed to be dynamic and integrate follow-up clinical data into more accurate predictions, possibly assisting management and decision-making.MethodsML models were developed using data of 404 ESG procedures performed at 12 centers across Europe. Collected data included clinical and demographic variables at the time of ESG and at follow-up. Multicentre/external and single center/internal and temporal validation were performed. Training and evaluation of the models were performed on Python’s scikit-learn library. Performance of models was quantified as receiver operator curve (ROC-AUC), sensitivity, specificity, and calibration plots.ResultsMulticenter external validation: ML models using preoperative data show poor performance. Best performances were reached by linear regression (LR) and support vector machine models for TWL% and EWL%, respectively, (ROC-AUC: TWL% 0.87, EWL% 0.86) with the addition of 6-month follow-up data.Single-center internal validation: Preoperative data only ML models show suboptimal performance. Early, i.e., 3-month follow-up data addition lead to ROC-AUC of 0.79 (random forest classifiers model) and 0.81 (LR models) for TWL% and EWL% achievement prediction, respectively. Single-center temporal validation shows similar results.ConclusionsAlthough preoperative data only may not be sufficient for accurate postoperative predictions, the ability of ML models to adapt and evolve with the patients changes could assist in providing an effective and personalized postoperative care. ML models predictive capacity improvement with follow-up data is encouraging and may become a valuable support in patient management and decision-making.

Transfer learning enables prediction of steel corrosion in concrete under natural environments