Can the Predictive Analytics Toolkit (PAT) handle a genomic data set?

IntroductionPrediction models for gestational hypertension and preeclampsia have been developed with data and assumptions from developed countries. Their suitability and application for low resource settings have not been tested. This review aimed to identify and assess the methodological quality of prediction models for gestational hypertension and pre-eclampsia with reference to their application in low resource settings.MethodsUsing combinations of keywords for gestational hypertension, preeclampsia and prediction models seven databases were searched to identify prediction models developed with maternal data obtained before 20 weeks of pregnancy and including at least three predictors (Prospero registration CRD 42017078786). Prediction model characteristics and performance measures were extracted using the CHARMS, STROBE and TRIPOD checklists. The National Institute of Health quality assessment tools for observational cohort and cross-sectional studies were used for study quality appraisal.ResultsWe retrieved 8,309 articles out of which 40 articles were eligible for review. Seventy-seven percent of all the prediction models combined biomarkers with maternal clinical characteristics. Biomarkers used as predictors in most models were pregnancy associated plasma protein-A (PAPP-A) and placental growth factor (PlGF). Only five studies were conducted in a low-and middle income country.ConclusionsMost of the studies evaluated did not completely follow the CHARMS, TRIPOD and STROBE guidelines in prediction model development and reporting. Adherence to these guidelines will improve prediction modelling studies and subsequent application of prediction models in clinical practice. Prediction models using maternal characteristics, with good discrimination and calibration, should be externally validated for use in low and middle income countries where biomarker assays are not routinely available.

A fundamental part of medical research is the development and validation of diagnostic and prognostic prediction models [1,2]. These prediction models aim to predict the absolute probability that a certain disease or condition is currently present (diagnostic models) or that an outcome will occur within a specific follow-up period (prognostic models) for an individual subject. Prediction models typically rely on multiple predictors, which can include demographic characteristics, medical history and physical examination items, or more complex measurements from, for example, medical imaging, electrophysiology, pathology, and biomarkers. Also for diagnostic models, estimates of probabilities are rarely based on a single test, and doctors naturally integrate several patient characteristics and symptoms [3]. A broad range of prediction modeling techniques exist, like regression approaches, neural network models, decision tree models, genetic programming models, and support vector machine learning models, although prediction models developed by a multivariable regression approach are by far prevailing. It is widely recommended that a developed prediction model should not be used in practice before being externally validated—at least once—in other individuals than those used for model development [4–7]. Unfortunately, most prediction models are poorly or not at all validated, rendering interpretation of their generalizability difficult. In addition, many systematic reviews showed that for the same outcome or same target population, numerous competing models exist [8–10]. Generally speaking, researchers often ignore existing prediction models and develop yet another prediction model from their own data [2]. This practice sustains a cycle of underpowered prediction model development studies and poor knowledge about the generalizability and applicability of developed prediction models. Evidence synthesis and meta-analysis of individual participant data (IPD) from multiple studies seems to be a unique opportunity to address these problems, as it allows researchers to develop and directly validate models on large datasets and across a wide range of populations and settings, to directly test a model’s generalizability (Fig 1) [11–13]. Fig 1 Trends in publications of IPD-MA studies focusing on the development and/or validation of diagnostic or prognostic prediction models. There is currently little guidance on how to conduct an IPD meta-analysis (IPD-MA) for developing and/or validating diagnostic or prognostic prediction models [15]. To date, most IPD-MA articles focus on estimating relative quantities, like a risk ratio, hazard ratio, or odds ratio for a specific treatment or a specific etiologic factor. In contrast, prediction modeling research is focused on developing and validating multivariable models aimed at calculating an absolute risk estimate of the combined variables, rather than estimating the relative effect of a specific treatment or etiologic factor. Furthermore, prediction modeling studies focus entirely on the role and joint contribution of multiple covariates, whereas intervention studies in principle rely on randomization to reduce the role of covariates (Table 1). Hence, IPD-MAs of randomized intervention and etiological studies, which are beyond the scope of this paper and are instead addressed in the accompanying paper [16], differ from IPD-MAs of multivariable prediction models, which are the focus of this paper. Table 1 The main differences between IPD-MA of treatment intervention studies and of multivariable prediction modeling studies. We provide an overview of the advantages and limitations of IPD-MAs aiming to develop a novel prediction model or to validate one or more existing models across multiple datasets. This overview is based on published guidelines and existing recommendations for the conduct of prediction modeling studies and of IPD-MA research. We illustrate this overview with examples of recently published IPD-MAs of prediction models across various medical domains. Our aim is to help researchers, readers, reviewers, and editors to identify and understand the key issues involved with such IPD-MA projects.

Risk prediction models for postoperative delirium in elderly patients with fragility hip fracture: A systematic review and critical appraisal

In patients with coronary heart disease (CHD), loneliness is associated with increased risk of morbidity and mortality. No predictive tool is available to detect patients who are influenced by loneliness to a degree that impacts mortality. To: (i) propose a prediction model that detects patients influenced by loneliness to a degree that increases one-year all-cause mortality, (ii) evaluate model classification performance of the prediction model, and (iii) investigate potential questionnaire response errors. A cohort of patients with CHD (n=7169) responded to a national cross-sectional survey, including two questions on loneliness. Information on cohabitation and follow-up information on one-year all-cause mortality were obtained from national registers. Prediction model development was based on the prognostic values of item responses in the questionnaire on loneliness and of cohabitation, evaluated with Cox-proportional Hazards Ratio (HR). Item responses which significantly predicted one-year mortality were included in the high-risk loneliness (HiRL) prediction model. Sensitivity, specificity and likelihood ratio were calculated to evaluate model classification performance. Sources of response errors were evaluated using verbal probing technique in an additional cohort (n=7). The TRIPOD checklist has been used to ensure transparent reporting. Two item responses significantly predicted one-year mortality HR=2.24 (95%CI=1.24-4.03) and HR=2.65 (95%CI=1.32-5.32) and were thus included in the model. Model classification performance showed a likelihood ratio of 1.89. Response error was evaluated as low. Based on the prognostic value in a loneliness questionnaire, a prediction model suitable to screen patients with CHD for high-risk loneliness was suggested. The HiRL prediction model is a short and easy-to-use screening tool that offers clinical staff to identify patients with CHD who are influenced by loneliness to a degree that impacts mortality. However, further evaluation of model performance and questionnaire validation is recommended before integrating the model into clinical practice.

The ability to accurately estimate the risk of peri-operative mortality after lung resection is important. There are concerns about the performance and validity of existing models developed for this purpose, especially when predicting mortality within 90 days of surgery. The aim of this study was therefore to develop a clinical prediction model for mortality within 90 days of undergoing lung resection. A retrospective database of patients undergoing lung resection in two UK centres between 2012 and 2018 was used to develop a multivariable logistic risk prediction model, with bootstrap sampling used for internal validation. Apparent and adjusted measures of discrimination (area under receiving operator characteristic curve) and calibration (calibration-in-the-large and calibration slope) were assessed as measures of model performance. Data were available for 6600 lung resections for model development. Predictors included in the final model were age, sex, performance status, percentage predicted diffusion capacity of the lung for carbon monoxide, anaemia, serum creatinine, pre-operative arrhythmia, right-sided resection, number of resected bronchopulmonary segments, open approach and malignant diagnosis. Good model performance was demonstrated, with adjusted area under receiving operator characteristic curve, calibration-in-the-large and calibration slope values (95% confidence intervals) of 0.741 (0.700, 0.782), 0.006 (-0.143, 0.156) and 0.870 (0.679, 1.060), respectively. The RESECT-90 model demonstrates good statistical performance for the prediction of 90-day mortality after lung resection. A project to facilitate large-scale external validation of the model to ensure that the model retains accuracy and is transferable to other centres in different geographical locations is currently underway.

BackgroundCatastrophic antiphospholipid syndrome (CAPS) is a life-threatening form of antiphospholipid syndrome (APS), with rarely occurrence but high mortality. Current classification diagnosis criteria require more than 3 organs involvement in 1-week...

Abstract. Numerical air quality models (AQMs) have been applied more frequently over the past decade to address diverse scientific and regulatory issues associated with deteriorated air quality in China. Thorough evaluation of a model's ability to replicate monitored conditions (i.e., a model performance evaluation or MPE) helps to illuminate the robustness and reliability of the baseline modeling results and subsequent analyses. However, with numerous input data requirements, diverse model configurations, and the scientific evolution of the models themselves, no two AQM applications are the same and their performance results should be expected to differ. MPE procedures have been developed for Europe and North America, but there is currently no uniform set of MPE procedures and associated benchmarks for China. Here we present an extensive review of model performance for fine particulate matter (PM2.5) AQM applications to China and, from this context, propose a set of statistical benchmarks that can be used to objectively evaluate model performance for PM2.5 AQM applications in China. We compiled MPE results from 307 peer-reviewed articles published between 2006 and 2019, which applied five of the most frequently used AQMs in China. We analyze influences on the range of reported statistics from different model configurations, including modeling regions and seasons, spatial resolution of modeling grids, temporal resolution of the MPE, etc. Analysis using a random forest method shows that the choices of emission inventory, grid resolution, and aerosol- and gas-phase chemistry are the top three factors affecting model performance for PM2.5. We propose benchmarks for six frequently used evaluation metrics for AQM applications in China, including two tiers – “goals” and “criteria” – where goals represent the best model performance that a model is currently expected to achieve and criteria represent the model performance that the majority of studies can meet. Our results formed a benchmark framework for the modeling performance of PM2.5 and its chemical species in China. For instance, in order to meet the goal and criteria, the normalized mean bias (NMB) for total PM2.5 should be within 10 % and 20 %, while the normalized mean error (NME) should be within 35 % and 45 %, respectively. The goal and criteria values of correlation coefficients for evaluating hourly and daily PM2.5 are 0.70 and 0.60, respectively; corresponding values are higher when the index of agreement (IOA) is used (0.80 for goal and 0.70 for criteria). Results from this study will support the ever-growing modeling community in China by providing a more objective assessment and context for how well their results compare with previous studies and to better demonstrate the credibility and robustness of their AQM applications prior to subsequent regulatory assessments.

In the evolving landscape of biomedical biometrics, where multimodal approaches are increasingly crucial for reliable user authentication, this research presents a comprehensive study. The primary focus is on the construction and performance evaluation of a robust big data prediction model within a cloud computing environment. The advent of big data and cloud computing has revolutionized the field of biomedical biometrics, offering immense potential for advanced data analysis and prediction. This research presents the development and evaluation of a robust prediction model for multimodal biometric data in biomedical applications. The proposed model incorporation of Reliable Discrete Variable Topology (RDVT) into the prediction model. RDVT introduces a topological data structure that enhances data reliability and ensures the integrity of multimodal biometric information. The construction and training of the prediction model are meticulously detailed, encompassing data preprocessing, feature extraction, clustering, classification, and model evaluation. Additionally, the integration of a fuzzy clustering algorithm enhances the model's ability to handle uncertainty and imprecision in biometric data. The advancement of multimodal biometrics in the biomedical field by introducing the Reliable Discrete Variable Topology (RDVT) and a big data prediction model based on a fuzzy clustering algorithm in a cloud computing environment. The model's performance is rigorously assessed through extensive experimentation, including accuracy, precision, recall, and F1-score measurements.

Prediction models are used in reproductive medicine to calculate the probability of pregnancy without treatment, as well as the probability of pregnancy after ovulation induction, intrauterine insemination or in vitro fertilization. The performance of such prediction models is often evaluated with a receiver operating characteristic (ROC) curve. The area under the ROC curve, also known as c-statistic, is then used as a measure of model performance. The value of this c-statistic is low for most prediction models in reproductive medicine. Here, we demonstrate that low values of the c-statistic are to be expected in these prediction models, but we also show that this does not imply that these models are of limited use in clinical practice. The calibration of the model (the correspondence between model-based probabilities and observed pregnancy rates) as well as the availability of a clinically useful distribution of probabilities and the ability to correctly identify the appropriate form of management are more meaningful concepts for model evaluation.

Clinical prediction modeling plays a pivotal part in modern clinical care, particularly in predicting the risk of in-hospital mortality. Recent modeling efforts have focused on leveraging intraoperative data sources to improve model performance. However, the individual and collective benefit of pre- and intraoperative data for clinical decision-making remains unknown. We hypothesized that pre- and intraoperative predictors contribute equally to the net benefit in a decision curve analysis (DCA) of in-hospital mortality prediction models that include pre- and intraoperative predictors. Data from the VitalDB database featuring a subcohort of 6043 patients were used. A total of 141 predictors for in-hospital mortality were grouped into preoperative (demographics, intervention characteristics, and laboratory measurements) and intraoperative (laboratory and monitor data, drugs, and fluids) data. Prediction models using either preoperative, intraoperative, or all data were developed with multiple methods (logistic regression, neural network, random forest, gradient boosting machine, and a stacked learner). Predictive performance was evaluated by the area under the receiver-operating characteristic curve (AUROC) and under the precision-recall curve (AUPRC). Clinical utility was examined with a DCA in the predefined risk preference range (denoted by so-called treatment threshold probabilities) between 0% and 20%. AUROC performance of the prediction models ranged from 0.53 to 0.78. AUPRC values ranged from 0.02 to 0.25 (compared to the incidence of 0.09 in our dataset) and high AUPRC values resulted from prediction models based on preoperative laboratory values. A DCA of pre- and intraoperative prediction models highlighted that preoperative data provide the largest overall benefit for decision-making, whereas intraoperative values provide only limited benefit for decision-making compared to preoperative data. While preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for low treatment thresholds up to 5% to 10%, preoperative laboratory measurements become the dominant source for decision support for higher thresholds. When it comes to predicting in-hospital mortality and subsequent decision-making, preoperative demographics, comorbidities, and surgery-related data provide the largest benefit for clinicians with risk-averse preferences, whereas preoperative laboratory values provide the largest benefit for decision-makers with more moderate risk preferences. Our decision-analytic investigation of different predictor categories moves beyond the question of whether certain predictors provide a benefit in traditional performance metrics (eg, AUROC). It offers a nuanced perspective on for whom these predictors might be beneficial in clinical decision-making. Follow-up studies requiring larger datasets and dedicated deep-learning models to handle continuous intraoperative data are essential to examine the robustness of our results.

BackgroundCesarean scar pregnancy (CSP) may lead to uterine rupture and heavy bleeding as the pregnancy progresses. Removal of the lesion by transvaginal hysterotomy is considered an effective first-line treatment for CSP, but there are few comprehensive prognostic models to determine which patients are candidates for transvaginal hysterotomy. This study aims to formulate a risk scoring system using a large cohort of CSP patients to evaluate the risk of adverse events when treated with transvaginal hysterotomy.MethodsA total of 273 CSP patients who underwent transvaginal hysterotomy at the First Affiliated Hospital of Sun Yat-sen University in China from January 2009 to February 2023 were studied, retrospectively. Patients’ clinical characteristics, surgical outcomes and detailed ultrasound characteristics were recorded according to a newly developed consensus. Multivariable logistic regression was employed to identify the risk factors associated with adverse events, and subsequently, a predictive model was constructed. A scoring system was developed by stratifying these risk factors and assigning scores to them using a mathematical model. Calibration was assessed with the Hosmer-Lemeshow test, and internal validation was performed using stratified 5-fold cross-validation. The diagnostic accuracy, sensitivity, specificity, and the area under the receiver operating characteristics curve were calculated to evaluate the performance of both the developed prediction model and the scoring system.ResultsOf 273 patients, 244 (89.4%) were successfully treated without experiencing any adverse events, while 29 (10.6%) had encountered at least one adverse event. The study identified gestational age [odds ratio (OR) =1.244, 95% confidence interval (CI): 1.052–1.422], the exact extent of the protrusion (OR =2.568, 95% CI: 1.627–4.047), and the vascularity of the lesion (OR =3.739, 95% CI: 1.551–9.016) as independent predictors for adverse events. Based on these findings, a prediction model and an 8-point scoring system were created. Both the prediction model and the scoring system demonstrated excellent performance. The average area under the receiver operating characteristics curve (AUC) of the prediction model was 0.828 (95% CI: 0.769–0.887), and the average accuracy was 0.901 (95% CI: 0.864–0.938). For the scoring system, the average area under the receiver operating characteristics curve was 0.829 (95% CI: 0.775–0.883), and the average accuracy of 0.898 (95% CI: 0.835–0.961). The model and scoring system had an acceptable goodness of fit according to the Hosmer-Lemeshow test (P=0.223 and P=0.707, respectively).ConclusionsThe prognostic model and a scoring system derived from a large cohort of CSP patients who underwent transvaginal hysterotomy can effectively predict adverse events. The scoring system is easy-to-use and can intuitively help clinicians identify candidates for transvaginal hysterotomy. It facilitates the implementation of more individualized and effective management strategies for CSP, and potentially enhances patient prognosis.

<p>Soil spectroscopy in the mid-infrared (MIR) allows the fast and cost-effective derivation of multiple physical and chemical soil properties, e.g., soil organic carbon (SOC) and soil texture, from a single reflectance spectrum. The recent development of extensive soil spectral libraries and field-portable handheld FTIR spectrometers have opened up new opportunities for the widespread application of soil reflectance spectroscopy in the geo- and environmental sciences. Compared to laboratory measurements on pre-treated soil material, field recordings of MIR spectra are impacted by in situ environmental conditions that modify and degrade the measured reflectance signal, most prominently variations in soil moisture and particle size across samples. These conditions prevent leveraging available MIR soil spectral libraries to build predictive models of soil properties directly.</p><p>We evaluated the capacity of the External Parameter Orthogonalization (EPO) algorithm to compensate for moisture and particle size-induced effects on MIR reflectance spectra recorded in the field to generate laboratory-equivalent spectra from the in-situ data, which would allow calibrations of predictive soil property models from soil spectral library data to be transferred to field-recorded spectra. An archive of 230 soils collected across five soil regions in Germany covering a broad range of parent materials, soil texture classes and organic carbon contents was used to evaluate the approach. For each soil sample, MIR reflectance spectra had been acquired both in the field, i.e., measured in situ on the soil surface, and in the laboratory on pre-treated (sieved and ground) soil material. Field spectra were corrected for environmental effects by EPO and used to predict SOC and soil texture with predictive models developed on the laboratory spectra.</p><p>Analysis of the EPO-transformed spectra showed that the algorithm could compensate for some of the significant environmental effects present in the field data, e.g., non-linear baseline shifts and large-scale water absorption features, effectively reducing variation across the soil samples that is not linked to the physical and chemical soil properties of interest. EPO-transformation of the spectra further allowed a robust transfer of calibrations developed on laboratory spectra of pre-treated soils to the field spectra. Predictive accuracies for SOC and soil texture were lower than for pure laboratory applications but generally in line with models developed with an extensive regional calibration sample directly on the field MIR spectra.</p><p>The correction of field MIR spectra with the EPO algorithm thus represents a promising approach to integrating existing soil spectral libraries into the development of predictive soil property models for in-situ MIR reflectance spectra as it would allow the development of predictive models without requiring a large number of additional regional calibration samples for field application of MIR soil spectroscopy.</p>

Prediction models are essential in clinical decision-making for estimating the probability of current (diagnosis, screening) or future (prognosis) outcomes. Network meta-analysis (NMA) serves as a powerful tool to compare the performance of multiple prediction models simultaneously. However, there is hardly any guidance on methods and reporting for studies employing NMA to evaluate prediction models. To provide an overview of NMAs assessing prediction model (external validation) performance, regardless of whether they use aggregate data (AD) or individual participant data (IPD). In addition, we offer recommendations for improving the reporting and conduct of NMAs in prediction model research. We searched PubMed and Embase up to September 1, 2025, to identify studies that addressed the evaluation of diagnostic or prognostic prediction model performance using NMA. We included articles that employed NMA to compare and assess at least three prediction models. We summarized the identified studies based on, eg, their application (diagnostic vs prognostic), data use (AD vs IPD), medical contexts in which the models were assessed, and evaluation metrics applied (eg, discrimination, calibration, and (re)classification). In addition, we examined the statistical approaches employed, the NMA assumptions (such as consistency, transitivity, and exchangeability), and the ranking methods used for model comparison. After screening 2436 articles, 28 were included. Twenty-six studies (92.9%) used AD, while two (7.1%) used IPD. Hospital care was the most common setting (n = 22; 78.6%), with respirology (n = 7; 25.0%) and cardiology (n = 5; 17.9%) as the most frequently studied clinical domains. Key NMA assumptions were addressed differently across the 28 NMAs: 14.3% (n = 4) discussed transitivity, similarity, or exchangeability, and 53.6% (n = 15) tested for consistency. The statistical approach also varied, with 60.7% of studies (n = 17) reporting Bayesian methods and 17.9% (n = 5) reporting frequentist approaches. Surface under the cumulative ranking was the predominant ranking method (n = 18). Most NMAs included 5 to 10 models in the network, with 5 NMAs analyzing more than 20 models. Performance metrics varied, with 39.3% of studies (n = 11) reporting discrimination measures, such as C statistics, while none reported calibration metrics. Sensitivity or specificity was provided in 64.3% of studies (n = 18), and no articles reported advanced decision-analytic metrics like the decision curve analysis. This scoping review highlights the limited and diverse use of NMA methods in evaluating prediction models, with a predominant reliance on aggregate rather than IPD, and inconsistent consideration of key NMA assumptions and model performance metrics. We provide recommendations for the reporting and conduct of an NMA of prediction model validation performance. Prediction models are tools used in medicine to estimate a patient's risk of developing a disease or experiencing a health outcome. Many different prediction models exist for the same condition, and it can be difficult for doctors and researchers to know which model performs best. One way to compare multiple models is through a statistical method called network meta-analysis (NMA), which is commonly used to compare treatments but has rarely been applied to prediction models. In our study, we reviewed all published NMAs that evaluated prediction models to see how they were conducted and reported. We looked at whether the studies reported important performance measures, such as how well the models could distinguish between patients with and without the outcome (discrimination), and how well the predictions matched actual outcomes (calibration). We also checked if key NMA assumptions were considered and how analyses were conducted. We found that most studies used summary data instead of patient-level data. Many did not report crucial performance measures and rarely checked NMA assumptions. There was also limited transparency in how the models were analyzed, making it difficult for others to reproduce the results. Our findings show that while NMA has great potential to help compare prediction models and identify the most reliable ones, current practice often lacks the detailed reporting needed to make these comparisons fully trustworthy. We recommend better reporting, sharing of data and analysis code, and careful checking of assumptions to help researchers and doctors choose the most reliable models, ultimately improving patient care.

Developing and Evaluating Prediction Models in Rehabilitation Populations

Abstract. The application of regional-scale air quality models is an important tool in air quality assessment and management. For this reason, the understanding of model abilities and performances is mandatory. The main objective of this research was to investigate the spatial and temporal variability of background particulate matter (PM) concentrations, to evaluate the regional air quality modelling performance in simulating PM concentrations during statically stable conditions and to investigate processes that contribute to regionally increased PM concentrations with a focus on eastern and central Europe. The temporal and spatial variability of observed PM was analysed at 310 rural background stations in Europe during 2011. Two different regional air quality modelling systems (offline coupled European Monitoring and Evaluation Programme, EMEP, and online coupled Weather Research and Forecasting with Chemistry) were applied to simulate the transport of pollutants and to further investigate the processes that contributed to increased concentrations during high-pollution episodes. Background PM measurements from rural background stations, wind speed, surface pressure and ambient temperature data from 920 meteorological stations across Europe, classified according to the elevation, were used for the evaluation of individual model performance. Among the sea-level stations (up to 200 m), the best modelling performance, in terms of meteorology and chemistry, was found for both models. The underestimated modelled PM concentrations in some cases indicated the importance of the accurate assessment of regional air pollution transport under statically stable atmospheric conditions and the necessity of further model improvements.