Rigorous calibration and validation of high sigma defect prediction models

Prediction models: stepwise development and simultaneous validation is a step back

A number of live birth predictive model during assisted reproductive technology treatment have been available in recent years, but few targeted evaluating the chances of live birth in poor ovarian response(POR) patients. The aim of this study was to develop a nomogram based on POSEIDON criteria to predict live birth in patients with expected POR. This retrospective cohort study using clinical data from 657 patients in POSEIDON Groups 3 and 4 (antral follicle count [AFC] ≤5 and AMH <1.2 ng/ml) in the CenterforReproductiveMedicine, First Affiliated Hospital of Xinjiang Medical University, and Construction a nomogram model t. Among 657 expected POR patients, 111 (16.89%) had live births, and 546 (83.11%) did not have live births. These were divided into a training set(n=438) and a validation set (n=219). Multivariate logistic regression analysis showed that the age (OR = 0.91, 95% CI: 0.86-0.97), BMI (OR = 1.98, 95% CI: 1.09-3.67), AMH (OR = 3.48, 95% CI: 1.45-8.51), normal fertilized oocytes (OR = 1.40, 95% CI: 1.21-1.63), and the basal FSH (OR = 0.89, 95% CI: 0.80-0.98) of the female were independent factors predicting live birth in patients with expected POR. Then, an individualized nomogram prediction model was built from these five factors. The area under the ROC curve of the live birth prediction model was 0.820 in the training set and 0.879 in the validation set. We have developed a nomogram combining clinical and laboratory factors to predict the probability of live birth in patients with an expected POR during IVF/ICSI, which can helpful for clinician in decision-making. However, the data comes from the same center, needs a prospective multicenter study for further in-depth evaluation and validation of this prediction model.

Background: Adding genetic and other biomarkers to breast cancer risk prediction models could markedly improve model discrimination; however, these expanded models have not been validated in a range of populations. In particular, the calibration of these new models how well the predicted absolute risks match observed risks has not been established. Good calibration is essential to confirm the utility of these risk models in precision prevention and treatment programs. Large cohort studies provide an ideal setting to validate risk models, as they can be used to validate both relative and absolute risks. However, in practice, genetic and biomarker data are often not available in the full cohort, but only on a sub sample of cases and controls. When the rules for sampling cases and controls into the sub sample are known, inverse-probability-of-sampling (IPW) weights can be used to estimate empirical absolute risks. When the sampling rules are unknown or complicated, the IPW weights can be estimated by regressing selection into the sub sample on matching and other inclusion criteria. Methods: We evaluated the performance of recently published breast cancer risk prediction models [Maas et al. JAMA Oncol 2016] in the Nurses Health Study (NHS) and Nurses Health Study II (NHSII). We first assess a prediction model that only includes questionnaire data (BMI, hormone replacement therapy (HRT), alcohol consumption, smoking status, height, parity, age at menarche and menopause, age at first birth, and family history of breast cancer). These data are available on all subjects in the NHS and NHSII blood subcohorts: 32,826 women in NHS (with disease follow-up from 1990-2012) and 29,611 women in NHS II (1999-2013). We will then validate a model that includes both questionnaire data and a polygenic risk score based on 92 established risk SNPs. Genetic data are available on case-control samples nested within the blood subcohorts: 2308 breast cancer cases and 3344 controls from NHS and 612 breast cancer cases and 933 controls from NHSII. We estimated IPW weights among controls using logistic regression in the blood subcohorts, with sampling as control being the outcome and the following predictors: age at baseline, menopausal status, HRT, length of HRT use for premenopausal women at baseline, and length of follow up time. We used the iCARE software package (Maas P, Chatterjee N, Wheeler W et al. 2015) to calculate predicted 5 and 10-year absolute risks of breast cancer based on the published models, empirical 5 and 10-year incidence across deciles of predicted risk, and Hosmer-Lemeshow goodness of fit and AUC statistics. Results: For the risk model without genetic information, predicted risks in the blood subcohorts ranged from 6.5/1,000 (1st decile) to 20.1/1,000 (10th decile) for NHS. Although empirical risks increased across deciles at approximately the same rate as predicted rates, empirical risks were higher than predicted (Hosmer-Lemeshow p&amp;lt;10-10). Validation analyses in the NHS II blood subcohort are ongoing. Due to matching and selection on control status, the baseline distribution of questionnaire risk factors differed between the blood subcohorts and the controls from the nested case-control samples. The IPW-weighted distribution in controls closely matched the distribution in the full subcohorts, suggesting a well-calculated weight. We will present IPW-based validation of the risk model in the nested case-control samples (work in progress). Conclusions: These results confirm that breast cancer risk prediction models can discriminate between high-risk and low-risk women, but they also highlight that the accuracy of absolute risk estimates can vary across populations. Findings from this study can add insights into model improvement and model application. Moreover, the method of using IPW weights to approximate a full cohort analysis provides a potential solution for utilizing nested case-control studies in future validation analyses. This abstract is also being presented as Poster A05. Citation Format: Chi Gao, Parichoy Pal Choudhury, Paige Maas, Rulla Tamimi, Heather Eliassen, Nilanjan Chatterjee, Montserrat Garcia-Closas, Peter Kraft. Validation of breast cancer risk prediction model using Nurses Health and Nurse Health II Studies. [abstract]. In: Proceedings of the AACR Special Conference: Improving Cancer Risk Prediction for Prevention and Early Detection; Nov 16-19, 2016; Orlando, FL. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2017;26(5 Suppl):Abstract nr PR02.

Objectives To perform a temporal validation of a predictive model for death in children with visceral leishmaniasis (VL). Methods A temporal validation of a children-exclusive predictive model of death due to VL (Sampaio et al. 2010 model), using a retrospective cohort, hereby called validation cohort. The validation cohort convenience sample was made of 156 patients less than 15 years old hospitalized between 2008 and 2018 with VL. Patients included in the Sampaio et al. 2010 study are here denominated derivation cohort, which was composed of 546 patients hospitalized in the same hospital setting in the period from 1996 to 2006. The calibration and discriminative capacity of the model to predict death by VL in the validation cohort were then assessed through the procedure of logistic recalibration that readjusted its coefficients. The calibration of the updated model was tested using Hosmer–Lemeshow test and Spiegelhalter test. A ROC curve was built and the value of the area under this curve represented the model's discrimination. Results The validation cohort found a lethality of 6.4%. The Sampaio et al. 2010 model demonstrated inadequate calibration in the validation cohort (Spiegelhalter test: p=0.007). It also presented unsatisfactory discriminative capacity, evaluated by the area under the ROC curve = 0.618. After the coefficient readjustment, the model showed adequate calibration (Spiegelhalter test, p=0.988) and better discrimination, becoming satisfactory (AUROC = 0.762). The score developed by Sampaio et al. 2010 attributed 1 point to the variables dyspnea, associated infections, and neutrophil count <500/mm3; 2 points to jaundice and mucosal bleeding; and 3 points to platelet count <50,000/mm3. In the recalibrated model, each one of the variables had a scoring of 1 point for each. Conclusion The temporally validated model, after coefficient readjustment, presented adequate calibration and discrimination to predict death in children hospitalized with VL.

A fundamental part of medical research is the development and validation of diagnostic and prognostic prediction models [1,2]. These prediction models aim to predict the absolute probability that a certain disease or condition is currently present (diagnostic models) or that an outcome will occur within a specific follow-up period (prognostic models) for an individual subject. Prediction models typically rely on multiple predictors, which can include demographic characteristics, medical history and physical examination items, or more complex measurements from, for example, medical imaging, electrophysiology, pathology, and biomarkers. Also for diagnostic models, estimates of probabilities are rarely based on a single test, and doctors naturally integrate several patient characteristics and symptoms [3]. A broad range of prediction modeling techniques exist, like regression approaches, neural network models, decision tree models, genetic programming models, and support vector machine learning models, although prediction models developed by a multivariable regression approach are by far prevailing. It is widely recommended that a developed prediction model should not be used in practice before being externally validated—at least once—in other individuals than those used for model development [4–7]. Unfortunately, most prediction models are poorly or not at all validated, rendering interpretation of their generalizability difficult. In addition, many systematic reviews showed that for the same outcome or same target population, numerous competing models exist [8–10]. Generally speaking, researchers often ignore existing prediction models and develop yet another prediction model from their own data [2]. This practice sustains a cycle of underpowered prediction model development studies and poor knowledge about the generalizability and applicability of developed prediction models. Evidence synthesis and meta-analysis of individual participant data (IPD) from multiple studies seems to be a unique opportunity to address these problems, as it allows researchers to develop and directly validate models on large datasets and across a wide range of populations and settings, to directly test a model’s generalizability (Fig 1) [11–13]. Fig 1 Trends in publications of IPD-MA studies focusing on the development and/or validation of diagnostic or prognostic prediction models. There is currently little guidance on how to conduct an IPD meta-analysis (IPD-MA) for developing and/or validating diagnostic or prognostic prediction models [15]. To date, most IPD-MA articles focus on estimating relative quantities, like a risk ratio, hazard ratio, or odds ratio for a specific treatment or a specific etiologic factor. In contrast, prediction modeling research is focused on developing and validating multivariable models aimed at calculating an absolute risk estimate of the combined variables, rather than estimating the relative effect of a specific treatment or etiologic factor. Furthermore, prediction modeling studies focus entirely on the role and joint contribution of multiple covariates, whereas intervention studies in principle rely on randomization to reduce the role of covariates (Table 1). Hence, IPD-MAs of randomized intervention and etiological studies, which are beyond the scope of this paper and are instead addressed in the accompanying paper [16], differ from IPD-MAs of multivariable prediction models, which are the focus of this paper. Table 1 The main differences between IPD-MA of treatment intervention studies and of multivariable prediction modeling studies. We provide an overview of the advantages and limitations of IPD-MAs aiming to develop a novel prediction model or to validate one or more existing models across multiple datasets. This overview is based on published guidelines and existing recommendations for the conduct of prediction modeling studies and of IPD-MA research. We illustrate this overview with examples of recently published IPD-MAs of prediction models across various medical domains. Our aim is to help researchers, readers, reviewers, and editors to identify and understand the key issues involved with such IPD-MA projects.

Macroscopic traffic flow modelling of large-scale freeway networks with field data verification: State-of-the-art review, benchmarking framework, and case studies using METANET

Abstract. Accurate flood damage data are essential for developing reliable flood risk assessments and designing effective risk management strategies. However, empirical flood damage data remain limited, particularly at the object level, hindering the calibration and validation of predictive models. Existing datasets are often highly aggregated and lack the granularity required for detailed analysis. This paper presents two comprehensive, micro-scale datasets documenting flood damage to 256 buildings, comprising both residential buildings and business premises, surveyed in the aftermath of the 2022 flood event in the Marche region of Italy. The georeferenced datasets include information on hazard characteristics, buildings' vulnerability features, physical damage description across structural and non-structural components, indirect damage, and implemented mitigation measures. In addition, original survey forms are provided to support future data collections in different contexts. Datasets and survey forms are available at the link: https://doi.org/10.5281/zenodo.15591850 (Rrokaj et al., 2025). The quality and richness of these datasets make them a valuable resource for improving flood risk modelling and supporting local stakeholders in identifying intervention priorities.

The need for a large number of costly experimental tests is a major challenge in the certification of composite parts in the aerospace industry. While computational simulation, specifically the finite element method (FEM), can accelerate this process through analysis supported by testing, the time-consuming calibration of composite failure models remains a drawback. This often requires an engineer with years of experience to perform trial-and-error simulations. To address this, an automated machine learning framework has been proposed to simulate and calibrate the failure models of composites in an accelerated and low-cost manner. In this framework, FEM parameters are calibrated using probabilistic machine learning in a step-by-step and targeted approach. The single-edge notch tension (SENT) test was chosen as a case study and simulated explicitly with predefined initial parameters. The result was compared with experiments to calculate the calibration error. Then, Gaussian process regression (GPR) was used in an in-house developed program to automatically select new failure parameters based on initial values and the calibration error in the previous step. The program continues this process until the error becomes negligible. For validation, the calibrated model was used to predict the behavior of a center-notch tension (CNT) test. The presented automated framework provides a robust and reliable calibrated model in a significantly faster manner. It also reduces a substantial amount of computational costs, eliminates the need for high expertise, and saves valuable engineering time.

ObjectivesTo enable risk stratification of patients with various types of arterial disease by the development and validation of models for prediction of recurrent vascular event risk based on vascular risk...

As a result of the highly heterogeneous tumor microenvironment and a series of complex phenomena that occur at different scales, several modeling approaches have emerged. The tumor growth models can be categorized by the behaviors they aim to capture, the hallmarks of cancer that are being considered, and the temporal and biological scales represented. Moreover, these models can also be differentiated according to the mathematical framework that is being employed, being either discrete models, continuum models, or hybrid models. Therefore, with this large set of possible models, and with each model choice having many unknown parameters, it is important to develop an adaptive model selection and validation framework in order to choose the best model for the desired application. In this work, we employ the Occam Plausibility Algorithm (OPAL) framework [1] for model selection and validation of a large body of tumor growth models. We select several continuum tumor growth models, half of them using the phase-field modeling approach, derived from the models developed in [2], and the other half being reaction-diffusion type models derived from [3]. The OPAL framework brings together model calibration, determination of sensitivities of outputs to parameter variances, and calculation of model plausibilities for model selection. Firstly, we identify the model classes and compute the sensitivity of the quantity of interest (QoI), here the tumor volume as a function of time. The models with parameters that do not influence the QoI are eliminated. The next step consists in dividing the models into Occam categories according to their number of parameters, where the models with fewer parameters are assembled to category 1. Following the principle of Occam's Razor, one of main goals of OPAL is to select the simplest model among the several models that leads to the same prediction. The models from the first category are calibrated using Bayes' rule for a specific calibration scenario. The next step is to compute the plausibilities for the models in this category and choose the most plausible model. Finally, with the most plausible model known, we solve the problem in the validation scenario. The model is valid depending on the accuracy with which the model predictions of the QoI agree with experimental data. If the model is not valid, we move to the next category; otherwise, we select this model as being the simplest and the most plausible valid model. We demonstrate that the OPAL is able to assist in the selection of the best tumor growth model from a defined set of models. The model selected is valid for particular scenarios. In this work, we consider avascular tumor growth models to reproduce the data obtained in a murine model of glioma growth. The brain data is generated with diffusion weighted magnetic resonance imaging over a period of ten days. The methodologies and results confirm that the OPAL strategy provides a powerful framework for model selection, parameters estimation, and model validation in the presence of uncertainties.

Multisite calibration &amp; validation of model reduces the uncertainty present in the model parameters and improve the SWAT model output. In the present study, an effort was made to set up SWAT model along with multisite calibration and validation for Kishna upper basin of India. A total of six gauging stations maintained by Central water commission across the area were selected for SWAT model calibration and validation. Sequential Uncertainty Fitting (SUFI-2) programme of SWAT-CUP (Calibration and Uncertainty Programme) was employed to calibrate the SWAT model. During sensitivity analysis of SWAT model, parameters which are highly sensitive that influences flow characteristics of different gauging station were identified. These parameters along with parameter range were used to calibration and validation of SWAT model each of the sub basins. Observed flow and predicted stream flows were compared with certain model performance indices. These indices indicated that for all six gauging stations, the SWAT model found to be very good. The performance indices like R2, NSE, RSR and PBIAS were ranged from 0.70 to 0.84, 0.67 to 0.76, 0.49 to 0.57 and -11.7 to 24.0% respectively during model calibration period and 0.70 to 0.93, 0.67 to 0.83, 0.41 to 0.58 and 8.5 to 22.9% respectively during model validation period. A good agreement between observed and simulated flows at all the gauging stations were observed. It can be concluded that SWAT model output was improved with multisite calibration and validation of the model.

Prediction models can identify fall-prone individuals. Prediction models can be based on either data from research cohorts (cohort-based) or routinely collected data (RCD-based). We review and compare cohort-based and RCD-based studies describing the development and/or validation of fall prediction models for community-dwelling older adults. Medline and Embase were searched via Ovid until January 2023. We included studies describing the development or validation of multivariable prediction models of falls in older adults (60+). Both risk of bias and reporting quality were assessed using the PROBAST and TRIPOD, respectively. We included and reviewed 28 relevant studies, describing 30 prediction models (23 cohort-based and 7 RCD-based), and external validation of two existing models (one cohort-based and one RCD-based). The median sample sizes for cohort-based and RCD-based studies were 1365 [interquartile range (IQR) 426-2766] versus 90441 (IQR 56442-128157), and the ranges of fall rates were 5.4% to 60.4% versus 1.6% to 13.1%, respectively. Discrimination performance was comparable between cohort-based and RCD-based models, with the respective area under the receiver operating characteristic curves ranging from 0.65 to 0.88 versus 0.71 to 0.81. The median number of predictors in cohort-based final models was 6 (IQR 5-11); for RCD-based models, it was 16 (IQR 11-26). All but one cohort-based model had high bias risks, primarily due to deficiencies in statistical analysis and outcome determination. Cohort-based models to predict falls in older adults in the community are plentiful. RCD-based models are yet in their infancy but provide comparable predictive performance with no additional data collection efforts. Future studies should focus on methodological and reporting quality.

Temporal validation of 30-day mortality prediction models for transcatheter aortic valve implantation using statistical process control – An observational study in a national population

To construct and validate a risk prediction model for patients with meropenem-induced liver injury (MiLI). A retrospective case-control study was conducted to collect data on inpatients treated with meropenem at Shiyan People's Hospital, Hubei, China from January 2018 to December 2022; this study served as the model construction dataset. Univariate analysis and multiple logistic regression analysis were employed to identify the related factors for MiLI, and a nomogram risk prediction model for MiLI was constructed. The recognition ability and prediction accuracy of the model were evaluated using the receiver operating characteristic (ROC) and calibration curves. The clinical efficacy was assessed via the decision curve analysis (DCA). The internal validation was performed using the bootstrap method, and external validation was conducted based on an external dataset from Shiyan Taihe Hospital between October 2021 and December 2023. A total of 1,625 individuals were included in the model construction dataset, of which 62 occurred MiLI. The external validation dataset included 1,032 cases, with 74 patients developing liver injury. Six variables were independent factors for MiLI and included in the final prediction model: being male (OR = 2.080, 95% CI: 1.050-4.123, P = 0.036), ICU admission (OR = 8.207, 95% CI: 4.094-16.453, P < 0.001), gallbladder disease (OR = 8.240, 95% CI: 3.605-18.832, P < 0.001), baseline ALP (OR = 1.012, 95% CI: 1.004-1.019, P = 0.004), GGT (OR = 1.010, 95% CI: 1.005-1.015, P < 0.001), and PLT (OR = 0.997, 95% CI: 0.994-0.999, P = 0.020). The c-statistic value for internal validation of the prediction model was 0.821; the sensitivity and specificity were 0.997 and 0.924, respectively. The c-statistic value of the prediction model in the model construction dataset was 0.837 (95% CI, 0.789-0.885), while in the external validation dataset was 0.851 (95% CI, 0.802-0.901). The P-values of the calibration curve in the two datasets were 0.935 and 0.084, respectively. Being male, ICU admission, gallbladder disease, higher levels of baseline ALP and GGT, and lower levels of baseline PLT were the risk factors for MiLI. The nomogram model built based on these factors demonstrated favorable performance in discrimination, calibration, clinical applicability, and internal-external validation. The nomogram model can assist clinicians in early identification of high-risk patients receiving meropenem, predicting the risk of MiLI, and ensuring safe medication practices.

BackgroundThe incidence rate of hypertriglyceridaemic acute pancreatitis (HTG-AP) has been steadily increasing due to changes in lifestyle and dietary patterns. Moreover, HTG-AP tends to be more severe than pancreatitis caused by other aetiologies, which may be related to pancreatic steatosis (PS). However, currently, no universally accepted multifactorial clinical scoring system specifically for predicting the severity of HTG-AP exists. This study aimed to identify predictors of hypertriglyceridaemic severe acute pancreatitis (HTG-SAP) and specifically incorporated PS into a visual model for predicting HTG-SAP early.MethodsA total of 346 patients with HTG-AP were included. These patients were classified into HTG-SAP (n = 94) and hypertriglyceridaemic non-severe acute pancreatitis (HTG-NSAP, n = 252) groups. An additional 51 patients were included for prospective internal validation of the predictive model. SPSS 29.0 and R version 4.4 software programs were used for statistical data analysis and for establishing and validating the predictive model, employing various methods, including univariate analysis, binary logistic regression, calibration curve analysis, and decision curve analysis (DCA).ResultsEight variables, namely, respiratory rate (RR), D-dimer (D-D), blood urea nitrogen (BUN), serum calcium (Ca2+), potential of hydrogen (pH), and the presence of pancreatic necrosis (PN), pleural effusion (PE) and PS, were identified as independent predictors for HTG-SAP via multivariate binary logistic analysis. The AUC of the new HTG-SAP model was 0.937 (95% CI [0.908–0.966]), which was greater than those of the modified CT severity index (MCTSI), the Bedside Index for Severity in Acute Pancreatitis (BISAP) score, and the Sequential Organ Failure Assessment (SOFA) score (AUC: 0.832, 0.784, and 0.782, respectively) (P < 0.001). The calibration curve strongly aligned the predicted outcomes and the actual observations. DCA indicated that clinical intervention would be beneficial for patients who are predicted to be at risk of developing HTG-SAP.ConclusionRR; D-D, BUN, and Ca2+ levels; pH, and the presence of PN, PE, and PS are independent predictors of HTG-SAP. The prediction model developed based on these predictors highly consistent and practical for predicting HTG-SAP.