Hybrid modeling for streamflow prediction in ungauged rivers with uncertainty quantification: a comparative analysis

When establishing a safety case for a high-level nuclear waste repository, coupled thermal, hydraulic, mechanical and chemical (THMC) processes are modelled to assess radionuclide transport and the integrity of barriers. Modelling these processes requires numerous parameters, which are all subject to uncertainty. However, complex and extensive uncertainty quantification (UQ) methods can come at great computational cost, especially when performed on multi-parametric problems. Therefore, UQ methods are sought with manageable computational effort while still capturing the relevant information to allow a meaningful interpretation of the results. The aim of this study is the comparison of three different UQ methods with different levels of complexity and computational cost in terms of their results. Radionuclide transport and THM-calculations are performed by employing these methods to the French Callovo-Oxfordian claystone as a reference material. The three methods include: (1) a UQ method based on the complete sampling of input parameter distributions by a Monte Carlo approach, (2) a UQ method based on a minimal number of data points, by sampling quantiles of the input parameter distributions, as well as the bounds of the distributions’ intervals, and (3) a UQ method that involves a first-order second-moment reliability approach. The results depict the different limitations and benefits of the UQ methods analysed while highlighting that a comprehensive understanding of parameter sensitivity and modelling approach are key to choosing the appropriate UQ method. • Applicability of UQ methods depends on complexity of problem. • FOSM and quantile method can be appropriate for linear problems and non-coupled processes. • MC approach remains method of choice for complex non-linear analyses.

Soil organic carbon (SOC) contents and stocks provide valuable insights into soil health, nutrient cycling, greenhouse gas emissions, and overall ecosystem productivity. Given this, remote sensing data coupled with advanced machine learning (ML) techniques have eased SOC level estimation while revealing its patterns across different ecosystems. However, despite these advances, the intricacies of training reliable and yet certain SOC models for specific end-users remain a great challenge. To address this, we need robust SOC uncertainty quantification techniques. Here, we introduce a methodology that leverages conformal prediction to address the uncertainty in estimating SOC contents while using remote sensing data. Conformal prediction generates statistically reliable uncertainty intervals for predictions made by ML models. Our analysis, performed on the LUCAS dataset in Europe and incorporating a suite of relevant environmental covariates, underscores the efficacy of integrating conformal prediction with another ML model, specifically random forest. In addition, we conducted a comparative assessment of our results against prevalent uncertainty quantification methods for SOC prediction, employing different evaluation metrics to assess both model uncertainty and accuracy. Our methodology showcases the utility of the generated prediction sets as informative indicators of uncertainty. These sets accurately identify samples that pose prediction challenges, providing valuable insights for end-users seeking reliable predictions in the complexities of SOC estimation.

This paper proposes an uncertainty quantification (UQ)-incorporated design optimization technique that integrates UQ considering the statistical characteristics of sensing margin with machine learning (ML) for the design of bitline sense amplifiers (BLSAs) in dynamic random access memory (DRAM), including offset compensation operation. The increased offset due to process variation exacerbated by device scaling degrades the reliability of DRAM read operations, making quantitative analysis of process variation essential. Conventional Monte Carlo (MC)-based UQ analysis provides high accuracy but requires extensive simulation costs. Accordingly, this study systematically defines key input and output variables for conventional BLSA and two representative offset compensation BLSA structures, introduces Sobol sequence-based quasi-Monte Carlo (QMC) sampling to improve simulation efficiency, and generalizes the Standard UQ methodology to predict statistical distributions of circuit performance parameters under process variation. Furthermore, a 3-Stage UQ algorithm that sequentially learns nominal, mean, and standard deviation values is proposed to enhance the efficiency of ML model training. HSPICE simulation results based on the TSMC 65 nm process demonstrated that the proposed 3-Stage UQ method reduced the simulation time required for ML training data generation by an average of 43% compared to the Standard UQ method and achieved excellent prediction performance even in data-limited environments. Additionally, design optimization combined with Bayesian Optimization (BO) confirmed performance improvements across all circuit structures. The methodology presented in this study enables effective process variation-aware optimization for DRAM designs requiring high reliability and yield, and is expected to contribute to next-generation memory circuit design automation.

Uncertainty quantification (UQ) is important to machine learning (ML) force fields to assess the level of confidence during prediction, as ML models are not inherently physical and can therefore yield catastrophically incorrect predictions. Established a-posteriori UQ methods, including ensemble methods, the dropout method, the delta method, and various heuristic distance metrics, have limitations such as being computationally challenging for large models due to model re-training. In addition, the uncertainty estimates are often not rigorously calibrated. In this work, we propose combining the distribution-free UQ method, known as conformal prediction (CP), with the distances in the neural network’s latent space to estimate the uncertainty of energies predicted by neural network force fields. We evaluate this method (CP+latent) along with other UQ methods on two essential aspects, calibration, and sharpness, and find this method to be both calibrated and sharp under the assumption of independent and identically-distributed (i.i.d.) data. We show that the method is relatively insensitive to hyperparameters selected, and test the limitations of the method when the i.i.d. assumption is violated. Finally, we demonstrate that this method can be readily applied to trained neural network force fields with traditional and graph neural network architectures to obtain estimates of uncertainty with low computational costs on a training dataset of 1 million images to showcase its scalability and portability. Incorporating the CP method with latent distances offers a calibrated, sharp and efficient strategy to estimate the uncertainty of neural network force fields. In addition, the CP approach can also function as a promising strategy for calibrating uncertainty estimated by other approaches.

This study proposes a novel approach to improve forecast accuracy and uncertainty quantification in time series forecasting through the synthesis of Machine Learning (ML) models with Dynamic Bayesian Predictive Synthesis (DBPS). We synthesize predictive distributions from three ML models—LightGBM, LSTM, and Prophet—through DBPS, notably employing Generalized Autoregressive Conditional Heteroskedasticity (GARCH) models to estimate uncertainties for ML forecasts. Empirical analysis using meteorological data from Delhi, India, demonstrates that our proposed method outperforms individual ML models and conventional ensemble methods, achieving improvements of up to 25.2% in RMSE and 25.4% in MAE. Furthermore, evaluating predictive distributions using Log Predictive Density Ratio (LPDR) reveals superior performance to other models, indicating a more accurate capture of observational uncertainty. Analysis of DBPS weights reveals that the offsetting effect of prediction biases is one of the factors contributing to improved forecast accuracy. These findings suggest that the synthesis of ML models and DBPS enables more reliable forecasting and improved uncertainty quantification.

This paper proposes an efficient and accurate non-intrusive uncertainty quantification (UQ) method in computational fluid dynamics (CFD). Emphasis is placed on developing an UQ method that can accurately predict stochastic behaviors of output solution with small number of sampling simulations, and is also accurate for non-smooth output uncertainty responses. The proposed method is based on Kriging surrogate model, and the Kriging function values are used to evaluate output uncertainties robustly even with non-smooth responses, while using both the fit uncertainty and the gradient information of the Kriging predictors for dynamic adaptive sampling. The proposed Kriging-model-based UQ method shows a superior performance in estimating the non-smooth responses of output solution in terms of accuracy and robustness compared to the existing polynomial chaos expansion and the adaptive sampling method based on only the Kriging predictor fit uncertainty. The proposed method is first tested on analytical non-smooth functions under uniform uncertainties, and then applied to the transonic RAE 2822 airfoil flow under normal uncertainties in freestream Mach number by coupling the proposed UQ method with CFD.

This paper provides an overview of the following three modeling approaches – Physics-based Modeling, Machine Learning, and Hybrid Modeling (Physics-based Modeling & Machine Learning combined), and addresses their applicability for multiphase flow simulations in specific use-cases. Physics-based modeling builds on well understood concepts in, e.g., thermodynamics, fluid dynamics, fluid modeling and optimization techniques. It requires deep domain knowledge as well as accurate fluid data and may incur significant computational cost. Machine Learning systems are based on learning algorithms, which find relationships between sensor data and output variables in a training dataset. The approach requires a good understanding of the learning algorithms and statistics. Small datasets or changes in operational conditions limit the suitability of this approach. Hybrid Models combine Physics with Machine Learning, and these models are on a sliding scale between pure Physics-based Models and pure Machine Learning Models. The individual use-case defines how the Hybrid Model is configured and where it sits on the sliding scale. Here, we will investigate three different use-cases: Physics supporting Machine Learning: Use physics to generate data to support training of machine learning models when data is sparse. Machine Learning supporting Physics: Use machine learning to provide additional input to physics modeling when available data does not suffice to achieve accurate numerical solutions. True Hybrid: Close the circle and use physics for feature engineering to improve the machine learning models, which then provide synthetic data as input to physics-based simulations. We observe that there is not one universal solution. On the one hand, flow simulations may address situations which are stochastic in nature and not deterministic. Here, a machine learning model based on physics-based simulations fits the purpose well. On the other hand, there are situations when machine learning can deduce relations between parameters such that you can provide additional input into physics-based models. One such example is to address the oil-water split at very high water cuts. By adopting the appropriate combination, the hybrid approach results in superior accuracy and offers the ability to address a broader range of applications than physics or machine learning alone.

In recent years, Machine Learning (ML) models have led to a substantial improvement in hydrological predictions. It appears these models can distill information from catchment properties that is relevant for the relationship between meteorological drivers and streamflow, which has so far eluded hydrologists.In the first part of this talk, I shall demonstrate some of our attempts towards understanding these improvements. Utilising Autoencoders and intrinsic dimension estimators, we have shown that the wealth of available catchment properties can effectively be summarised into merely three features, insofar as they are relevant for streamflow prediction. Hybrid models, which combine the flexibility of ML models with mechanistic mass-balance models, are equally adept at predicting as pure ML models but come with only a few interpretable interior states. Combining these findings will, hopefully, bring us closer to understanding what these ML models seem to have 'grasped'.In the second part of the talk, I will address the issue of uncertainty quantification. I contend that error modelling should not be attempted on the residuals. Rather, we should model the errors where they originate, i.e., on the inputs, model states, and/or parameters. Such stochastic models are more adept at expressing the intricate distributions exhibited by real data. However, they come at the cost of a very large number of unobserved latent variables and thus pose a high-dimensional inference problem. This is particularly pertinent when our models include ML components. Fortunately, advances in inference algorithms and parallel computing infrastructure continue to extend the limits on the number of variables that can be inferred within a reasonable timeframe. I will present a straightforward example of a stochastic hydrological model with input uncertainty, where Hamiltonian Monte Carlo enables a comprehensive Bayesian inference of model parameters and the actual rain time-series simultaneously.

• ML models for predicting SCM concretes compressive strength developed • Multiple datasets devised from a global dataset were used for training • Multiple ML models used in conjunction with multiple datasets for best results • Accurate prediction results obtained for all SCM concrete types studied • Prediction uncertainty bounds established by conformal prediction analysis Supplementary cementitious materials (SCM) used as partial replacements of cement can reduce carbon emissions and construction costs. The amount and types of SCM used directly influence the compressive strength of concrete, which is a crucial indicator of concrete performance. However, predicting compressive strength is challenging for SCM-based concretes due to their complex compositions, especially when multiple types of SCM are used. This study used five machine learning (ML) models, namely artificial neural network (ANN), random forest (RF), extra trees (ET), extreme gradient boosting (XGB), and adaptive boosting (AdaBoost) to predict the 28-day compressive strength of SCM concretes incorporating fly ash (FA), silica fume (SF), and ground granulated blast furnace slag (GGBFS). Firstly, a global dataset comprising 1456 data points was organized into seven individual and six combined datasets for training of the five ML models. After this, the performances of the ML models’ predictions using individual and combined datasets were compared using root mean squared error (RMSE) and coefficient of determination (R 2 ) to identify the optimal ML model-dataset configurations for each type of SCM-based concrete. It is found that in general, XGB models developed using the combined datasets demonstrated superior prediction accuracy, achieving an R² value exceeding 0.8. Both Shapley Additive Explanation (SHAP) and partial dependence plots (PDPs) were employed to illustrate the effects of input variables on compressive strength. Finally, uncertainty quantification using the Jackknife method combined with a normalized nonconformity measure was employed to provide accurate uncertainty bounds for compressive strength predictions.

Machine learning (ML) models provide more choices to patients with diabetes mellitus (DM) to more properly manage blood glucose (BG) levels. However, because of numerous types of ML algorithms, choosing an appropriate model is vitally important. In a systematic review and network meta-analysis, this study aimed to comprehensively assess the performance of ML models in predicting BG levels. In addition, we assessed ML models used to detect and predict adverse BG (hypoglycemia) events by calculating pooled estimates of sensitivity and specificity. PubMed, Embase, Web of Science, and Institute of Electrical and Electronics Engineers Explore databases were systematically searched for studies on predicting BG levels and predicting or detecting adverse BG events using ML models, from inception to November 2022. Studies that assessed the performance of different ML models in predicting or detecting BG levels or adverse BG events of patients with DM were included. Studies with no derivation or performance metrics of ML models were excluded. The Quality Assessment of Diagnostic Accuracy Studies tool was applied to assess the quality of included studies. Primary outcomes were the relative ranking of ML models for predicting BG levels in different prediction horizons (PHs) and pooled estimates of the sensitivity and specificity of ML models in detecting or predicting adverse BG events. In total, 46 eligible studies were included for meta-analysis. Regarding ML models for predicting BG levels, the means of the absolute root mean square error (RMSE) in a PH of 15, 30, 45, and 60 minutes were 18.88 (SD 19.71), 21.40 (SD 12.56), 21.27 (SD 5.17), and 30.01 (SD 7.23) mg/dL, respectively. The neural network model (NNM) showed the highest relative performance in different PHs. Furthermore, the pooled estimates of the positive likelihood ratio and the negative likelihood ratio of ML models were 8.3 (95% CI 5.7-12.0) and 0.31 (95% CI 0.22-0.44), respectively, for predicting hypoglycemia and 2.4 (95% CI 1.6-3.7) and 0.37 (95% CI 0.29-0.46), respectively, for detecting hypoglycemia. Statistically significant high heterogeneity was detected in all subgroups, with different sources of heterogeneity. For predicting precise BG levels, the RMSE increases with a rise in the PH, and the NNM shows the highest relative performance among all the ML models. Meanwhile, current ML models have sufficient ability to predict adverse BG events, while their ability to detect adverse BG events needs to be enhanced. PROSPERO CRD42022375250; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=375250.

Uncertainty estimation in healthcare involves quantifying and understanding the inherent uncertainty or variability associated with medical predictions, diagnoses, and treatment outcomes. In this era of Artificial Intelligence (AI) models, uncertainty estimation becomes vital to ensure safe decision-making in the medical field. Therefore, this review focuses on the application of uncertainty techniques to machine and deep learning models in healthcare.A systematic literature review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Our analysis revealed that Bayesian methods were the predominant technique for uncertainty quantification in machine learning models, with Fuzzy systems being the second most used approach. Regarding deep learning models, Bayesian methods emerged as the most prevalent approach, finding application in nearly all aspects of medical imaging.Most of the studies reported in this paper focused on medical images, highlighting the prevalent application of uncertainty quantification techniques using deep learning models compared to machine learning models. Interestingly, we observed a scarcity of studies applying uncertainty quantification to physiological signals. Thus, future research on uncertainty quantification should prioritize investigating the application of these techniques to physiological signals.Overall, our review highlights the significance of integrating uncertainty techniques in healthcare applications of machine learning and deep learning models. This can provide valuable insights and practical solutions to manage uncertainty in real-world medical data, ultimately improving the accuracy and reliability of medical diagnoses and treatment recommendations.

Shear viscosity, though being a fundamental property of all fluids, is computationally expensive to calculate from equilibrium molecular dynamics simulations. Recently, machine learning (ML) methods have been used to augment molecular simulations in many contexts, thus showing promise to estimate viscosity too in a relatively inexpensive manner. However, ML methods face significant challenges—such as overfitting, when the size of the data set is small, as is the case with viscosity. In this work, we train seven ML models to predict the shear viscosity of a Lennard–Jones fluid, with particular emphasis on addressing issues arising from a small data set. Specifically, the issues related to model selection, performance estimation and uncertainty quantification were investigated. First, we show that the widely used performance estimation procedure of using a single unseen data set shows a wide variability—in estimating the errors on—small data sets. In this context, the common practice of using cross validation (CV) to select the hyperparameters (model selection) can be adapted to estimate the generalization error (performance estimation) as well. We compare two simple CV procedures for their ability to do both model selection and performance estimation, and find that k-fold CV based procedure shows a lower variance of error estimates. Also, these CV procedures naturally lead to an ensemble of trained ML models. We discuss the role of performance metrics in training and evaluation and propose a method to rank the ML models based on multiple metrics. Finally, two methods for uncertainty quantification—Gaussian process regression (GPR) and ensemble method—were used to estimate the uncertainty on individual predictions. The uncertainty estimates from GPR were also used to construct an applicability domain using which the ML models provided even more reliable predictions on an independent viscosity data set generated in this work. Overall, the procedures prescribed in this work, together, lead to robust ML models for small data sets.

This study focuses on using various machine learning (ML) models to evaluate the shear behaviors of ultra-high-performance concrete (UHPC) beams reinforced with glass fiber-reinforced polymer (GFRP) bars. The main objective of the study is to predict the shear strength of UHPC beams reinforced with GFRP bars using ML models. We use four different ML models: support vector machine (SVM), artificial neural network (ANN), random forest (R.F.), and extreme gradient boosting (XGBoost). The experimental database used in the study is acquired from various literature sources and comprises 54 test observations with 11 input features. These input features are likely parameters related to the composition, geometry, and properties of the UHPC beams and GFRP bars. To ensure the ML models' generalizability and scalability, random search methods are utilized to tune the hyperparameters of the algorithms. This tuning process helps improve the performance of the models when predicting the shear strength. The study uses the ACI318M-14 and Eurocode 2 standard building codes to predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams. The ML models' predictions are compared to the results obtained from these building code standards. According to the findings, the XGBoost model demonstrates the highest predictive test performance among the investigated ML models. The study employs the SHAP (SHapley Additive exPlanations) analysis to assess the significance of each input parameter in the ML models' predictive capabilities. A Taylor diagram is used to statistically compare the accuracy of the ML models. This study concludes that ML models, particularly XGBoost, can effectively predict the shear capacity behavior of GFRP bars-reinforced UHPC I-shaped beams.

Systematic literature review of machine learning based software development effort estimation models

Accurate estimation of reference evapotranspiration (ETo) plays a vital role in irrigation and water resource planning. The Penman–Monteith method recommended by the Food and Agriculture Organization (FAO PM56) is widely used and considered a standard to calculate ETo. However, FAO PM56 cannot be used with limited meteorological variables, so it is compulsory to choose an alternative model for ETo estimation, which requires fewer variables. This study built ten machine learning (ML) models based on multi-function, neural network, and tree-based structure against the FAO PM56 method. For this purpose, time series temperature data on a monthly scale are only used to train ML models. The developed ML models were applied to estimate ETo at different test stations and the obtained results were compared with the FAO PM56 method to verify and validate their performance in ETo estimation for the selected stations. In addition, multiple statistical indicators, including root-mean-square error (RMSE), coefficient of determination (R2), mean absolute error (MAE), Nash–Sutcliffe efficiency (NSE), and correlation coefficient (r) were calculated to compare the performance of each ML model on ETo estimation. Among the applied ML models, the ETo tree boost (TB) ML model outperformed the other ML models in estimating ETo in diverse climatic conditions based on statistical indicators (R2, NSE, r, RMSE, and MAE). Moreover, the observed R2, NSE, and r were the highest for the TB ML model, while RMSE and MAE were found to be the lowest at the study sites compared to other applied ML models. Lastly, ETo point data yielded from the TB ML model was used in an interpolation process to create monthly and annual ETo maps. Based on the ETo maps, this study suggests mainly a focus on areas with high ETo values and proper irrigation scheduling of crops to ensure water sustainability.