EVALUATING RISK-PREDICTION MODELS USING DATA FROM ELECTRONIC HEALTH RECORDS.

It is well known that in logistic regression, where the outcome is measured with error, a biased estimate of the association between the outcome and a risk factor may result if no proper adjustment is made. Hence, it seems tempting to always adjust for possible misclassification of the outcome. Here we show that it is not always beneficial to do so because, though the adjustment reduces the bias, it also inflates the variance, leading to a possibly larger mean squared error of the estimate. In the context of a data set on agricultural injuries, numerical evidence is provided through simulation studies.

IntroductionPostoperative pulmonary complications (PPCs) following major abdominal surgery result in substantial morbidity and mortality, yet stratifying patients for risk-modifying interventions remains challenging. This study aimed to identify and externally validate PPC risk prediction models in an international, prospective cohort.MethodA systematic review was conducted to identify risk prediction models for PPC following abdominal surgery. External validation was performed using data from a prospective dataset of adult patients undergoing major abdominal surgery from January to April 2019 in the UK, Ireland, and Australia. The primary outcome was identification of PPC within 30-days (StEP-COMPAC criteria definition). Model discrimination and diagnostic accuracy were compared.ResultsSix unique risk prediction models were eligible from 2819 records (112 full texts). These were validated across 11,591 patients, with an overall PPC rate of 7.8% (n = 903). The Assess Respiratory Risk in Surgical Patients in Catalonia (ARISCAT) score provided the best discrimination (AUROC: 0.709 (95% CI: 0.692-0.727), yet no risk prediction model demonstrated good discrimination (AUROC >0.7).ConclusionsThe risk of PPC for patients following major abdominal surgery in the pre-covid era is not well described by existing prediction tools. New prediction tools are required to account for additional variation introduced for patients affected by SARS-CoV-2 infection.

Accurate risk prediction models are needed to identify different risk groups for individualized prevention and treatment strategies. In the Nurses' Health Study, to examine the effects of several biomarkers and genetic markers on the risk of rheumatoid arthritis (RA), a three-phase nested case-control (NCC) design was conducted, in which two sequential NCC subcohorts were formed with one nested within the other, and one set of new markers measured on each of the subcohorts. One objective of the study is to evaluate clinical values of novel biomarkers in improving upon existing risk models because of potential cost associated with assaying biomarkers. In this paper, we develop robust statistical procedures for constructing risk prediction models for RA and estimating the incremental value (IncV) of new markers based on three-phase NCC studies. Our method also takes into account possible time-varying effects of biomarkers in risk modeling, which allows us to more robustly assess the biomarker utility and address the question of whether a marker is better suited for short-term or long-term risk prediction. The proposed procedures are shown to perform well in finite samples via simulation studies.

In machine learning problems, we usually assume that the validation accuracy is a good estimation of prediction accuracy for datasets without ground truth. In reality, this assumption may not hold. Therefore, we propose an approach to estimate the prediction accuracy of a target model on unlabeled datasets. The proposed approach uses multiple target homogeneous models to assign each unlabeled sample a confidence value, based on the number of models agreeing on the predicted label. With the confidence values, the prediction accuracy of the target model on the datasets can be estimated. In the experiments, the target model is a convolutional neural network (CNN) model, and the homogeneous models only differ in initial weights. The experiments are conducted with datasets from a wide variety of music genres. The estimation performance of the proposed approach is compared with the reversed testing qualities (RTQ) and the ensemble average qualities (EAQ) approaches. The RTQ approach was proposed to estimate the prediction accuracy of trained models, and the EAQ approach was originally designed for estimating the predictive uncertainty of individual samples. We apply all three compared models to estimate prediction accuracy of datasets by using a linear model. The parameters of the linear model are either computed by using multiple labeled datasets or one labeled dataset. The experimental results show that when compared with the RTQ approach, the proposed approach has much lower estimation errors for some datasets. When compared with the EAQ approach, the proposed approach is more robust for datasets with large distribution shifts. Finally, we show an additional benefit of the proposed approach. In case that the estimated accuracy is unsatisfactory, we may re-train the target model with a new training set, which contains the original training samples plus new training samples with manual labeling from the unlabeled dataset. The experimental results confirm that it is more effective to select (and label) new samples from those with low confidence values than those randomly selected. Overall, the proposed approach is a promising approach for estimating prediction accuracy on unlabeled datasets.

Solar-based power generation presents challenges for system and grid operators due to the intermittent nature of power supply. Predicting the performance of photovoltaic (PV) power plants and rooftop systems can often be challenging due to difficulties in data collection and incoherencies in interconnected systems. Following the hierarchical aggregation structure from geographical and temporal similarities between PV systems, we suggest a simplified approach to predicting the performance of individual installations and evaluating the impact of these hypothetical installations on the overall grid. We use the hierarchical nature of power generation and ascertain weather datasets to predict the performance of new or existing systems for locations with unmeasured input data. We demonstrate an approach that could improve grid stability by using a hierarchical model on publicly available datasets on utility and rooftop installations. Ensemble machine learning algorithms are trained with 16 weeks of known hourly input training features to form a baseline model for known locations. The prediction accuracy is then directly compared for locations with known and unknown input features, both on a granular and subregion level. We observe a reduction in prediction accuracy by 6–8% using the hierarchical approach. The accuracy of the hierarchical model can be further enhanced beyond our work by increasing the training dataset temporally, as well as by augmenting nested layers of the hierarchy.

BackgroundBluetongue (BT) is a disease of concern to animal breeders, so the question on their minds is whether they can predict the risk of the disease before it occurs. The main objective of this study is to enhance the accuracy of BT risk prediction by relying on machine learning (ML) approaches to help in fulfilling this inquiry. Several risk factors of BT that affect the occurrence and magnitude of animal infection with the virus have been reported globally. Additionally, risk factors, such as sex, age, species, and season, unevenly affect animal health and welfare. Therefore, the seroprevalence study data of 233 apparently healthy animals (125 sheep and 108 goats) from five different provinces in Egypt were used to analyze and compare the performance of the algorithms in predicting BT risk.ResultsLogistic regression (LR), decision tree (DT), random forest (RF), and a feedforward artificial neural network (ANN) were used to develop predictive BT risk models and compare their performance to the base model (LR). Model performance was assessed by the area under the receiver operating characteristics curve (AUC), accuracy, true positive rate (TPR), false positive rate (FPR), false negative rate (FNR), precision, and F1 score. The results indicated that RF performed better than other models, with an AUC score of 81%, ANN of 79.6%, and DT of 72.85%. In terms of performance and prediction, LR showed a much lower value (AUC = 69%). Upon further observation of the results, it was discovered that age and season were the most important predictor variables reported in classification and prediction.ConclusionThe findings of this study can be utilized to predict and control BT risk factors in sheep and goats, with better diagnostic discrimination in terms of accuracy, TPR, FNR, FPR, and precision of ML models over traditional and commonly used LR models. Our findings advocate that the implementation of ML algorithms, mainly RF, in farm decision making and prediction is a promising technique for analyzing cross-section studies, providing adequate predictive power and significant competence in identifying and ranking predictors representing potential risk factors for BT.

Polypharmacy Increases Heart Failure (HF) Readmission Rate

Bias from nondifferential but dependent misclassification of exposure and outcome.

Machine learning models developed from electronic health records data have been increasingly used to predict risk of mortality for general oncology patients. But these models may have suboptimal performance because of patient heterogeneity. The objective of this work is to develop a new modeling approach to predicting short-term mortality that accounts for heterogeneity across multiple subgroups in the presence of a large number of electronic health record predictors. We proposed a two-stage approach to addressing heterogeneity among oncology patients of different cancer types for predicting their risk of mortality. Structured data were extracted from the University of Pennsylvania Health System for 20,723 patients of 11 cancer types, where 1,340 (6.5%) patients were deceased. We first modeled the overall risk for all patients without differentiating cancer types, as is done in the current practice. We then developed cancer type-specific models using the overall risk score as a predictor along with preselected type-specific predictors. The overall and type-specific models were compared with respect to discrimination using the area under the precision-recall curve (AUPRC) and calibration using the calibration slope. We also proposed metrics that characterize the degree of risk heterogeneity by comparing risk predictors in the overall and type-specific models. The two-stage modeling resulted in improved calibration and discrimination across all 11 cancer types. The improvement in AUPRC was significant for hematologic malignancies including leukemia, lymphoma, and myeloma. For instance, the AUPRC increased from 0.358 to 0.519 (∆ = 0.161; 95% CI, 0.102 to 0.224) and from 0.299 to 0.354 (∆ = 0.055; 95% CI, 0.009 to 0.107) for leukemia and lymphoma, respectively. For all 11 cancer types, the two-stage approach generated well-calibrated risks. A high degree of heterogeneity between type-specific and overall risk predictors was observed for most cancer types. Our two-stage modeling approach that accounts for cancer type-specific risk heterogeneity has improved calibration and discrimination than a model agnostic to cancer types.

Cheese-making traits in dairy cattle are important to the dairy industry but are difficult to measure at the individual level because there are limitations on collecting phenotypic information. Mid-infrared spectroscopy has its advantages, but it can only be used during monthly milk recordings. Recently, in-line devices for real-time analysis of milk quality have been developed. The AfiLab recording system (Afimilk) offers significant benefits as phenotypes can be collected from each cow at each milking session. The objective of this study was to assess the potential of integrating AfiLab real-time milk analyzer measures with the stacking ensemble learning technique using heterogeneous base learners for the in-line daily monitoring of cheese-making traits in Holstein cattle with a view to developing a precision livestock farming system for monitoring the technological quality of milk. Data and samples for wet-laboratory analyses were collected from 499 Holstein cows belonging to 2 farms where the AfiLab system was installed. The traits of concern were 9 milk coagulation traits [3 milk coagulation properties (MCP), and 6 curd firming traits (CFt)], and 7 cheese-making traits [3 cheese yield (CY) traits, and 4 milk nutrient recovery in the curd (REC) traits]. The near-infrared AfiLab spectral data and on-farm information (days in milk and parity) were used to assess the predictive ability of different statistical methods [elastic net (EN), gradient boosting machine (GBM), extreme gradient boosting (XGBoost), and artificial neural network (ANN)] across different cross-validation scenarios. These statistical methods were considered the base learners, which were then combined in a stacking ensemble learning. Results indicate that including information on the cows (days in milk and parity) in the AfiLab infrared prediction increased its accuracy by 10.3% for traditional MCP, 13.8% for curd firming, 9.8% for CY, and 11.2% for REC traits compared with those obtained from near-infrared AfiLab alone. The statistical approaches exhibited high prediction accuracies (R2) averaged across the cross-validation scenarios for traditional MCP (0.58 for ANN, 0.55 for EN and GBM, 0.52 for XGBoost, and 0.62 for stacking ensemble), CFt (0.55 for ANN, 0.54 for EN and GBM, 0.53 for XGBoost, and 0.61 for stacking ensemble), and similar R2 averages for CY and REC (0.55 for ANN, 0.54 for EN and GBM, 0.53 for XGBoost, and 0.61 for stacking ensemble). The ANN approach was more accurate than the other base learners (EN, GBM, and XGBoost) and improved accuracy across cross-validation scenarios on average by 7% for traditional MCP, 5% for CFt, 8% for CY, and 7% for REC. The stacking ensemble method improved prediction accuracy by 3% to 31% for traditional MCP, 2% to 26% for CFt, 1% to 38% for CY traits, and 2% to 27% for REC traits compared with the base learners. The prediction accuracies of the different approaches evaluated tended to decrease from the 10-fold cross-validation to the independent validation scenario, although there was a smaller reduction in prediction accuracy with the stacking ensemble learning technique across all the cross-validation scenarios. Our results show that combining in-line on-farm information with stacking ensemble machine learning represents an effective alternative for obtaining robust daily predictions of milk cheese-making traits.

Despite advances in medical therapy, reperfusion, and mechanical support, cardiogenic shock remains associated with excess morbidity and mortality. Accurate risk stratification may improve patient management. We compared the accuracy of established risk scores for cardiogenic shock. Patients admitted to tertiary care center cardiac care units in the province of Alberta in 2015 were assessed for cardiogenic shock. The Acute Physiology and Chronic Health Evaluation-II (APACHE-II), CardShock, intra-aortic balloon pump (IABP) Shock II, and sepsis-related organ failure assessment (SOFA) risk scores were compared. Receiver operating characteristic curves were used to assess discrimination of in-hospital mortality and compared using DeLong's method. Calibration was assessed using the Hosmer-Lemeshow goodness-of-fit test. The study included 3021 patients, among whom 510 (16.9%) had cardiogenic shock. Patients with cardiogenic shock had longer median hospital stays (median 11.0 vs 4.1 days, P < .001) and were more likely to die (29.0% vs 2.5%, P < .001). All risk scores were adequately calibrated for predicting hospital morality except for the APACHE-II score (Hosmer-Lemeshow P < .001). Discrimination of in-hospital mortality with the APACHE-II (area under the curve [AUC]: 0.72, 95% confidence interval [CI]: 0.66-0.76) and IABP-Shock II (AUC: 0.73, 95% CI: 0.68-0.77) scores were similar, while the CardShock (AUC: 0.76, 95% CI: 0.72-0.81) and SOFA (AUC: 0.76, 95%CI: 0.72-0.81) scores had better discrimination for predicting in-hospital mortality. In a real-world population of patients with cardiogenic shock, existing risk scores had modest prognostic accuracy, with no clear superior score. Further investigation is required to improve the discriminative abilities of existing models or establish novel methods.

Diagnostic accuracy of risk prediction models in postmenopausal patients with suspected ovarian cancer

Although markers identified by genome-wide association studies have individually strong statistical significance, their performance in prediction remains limited. Our goal was to use animal breeding genomic prediction models to predict additive genetic contributions for systolic blood pressure (SBP) using whole genome sequencing data with different validation designs.The additive genetic contributions of SBP were estimated via linear mixed model. Rare variants (MAF<0.05) were collapsed through the k-means method to create a "collapsed single-nucleotide polymorphisms." Prediction of the additive genomic contributions of SBP was conducted using genomic Best Linear Unbiased Predictor (GBLUP) and BayesCπ. Estimates of predictive accuracy were compared using common single-nucleotide polymorphisms (SNPs) versus common and collapsed SNPs, and for prediction within and across families.The additive genetic variance of SBP contributed to 18% of the phenotypic variance (h2 = 0.18). BayesCπ had slightly better prediction accuracies than GBLUP. In both models, within-family predictions had higher accuracies both in the training and testing set than didacross-family design. Collapsing rare variants via the k-means method and adding to the common SNPs did not improve prediction accuracies. The prediction model, including both pedigree and genomic information, achieved a slightly higher accuracy than using either source of information alone.Prediction of genetic contributions to complex traits is feasible using whole genome sequencing and statistical methods borrowed from animal breeding. The relatedness of individuals between the training and testing set strongly affected the performance of prediction models. Methods for inclusion of rare variants in these models need more development.

Robust multivariate time series forecasting is crucial in many cyberphysical and Internet of Things applications. Existing state-of-the-art robust forecasting models decompose time series into independent functions covering trends and periodicities. However, these independent functions fail to capture correlations among multiple time series, thereby reducing prediction accuracy. Moreover, existing robust forecasting models treat certain abrupt but normal changes, e.g., caused by holidays, as outliers because they occur infrequently and have data distributions that resemble those of outliers. This exacerbates model bias and reduces prediction accuracy. This paper aims to capture correlations across multiple time series and abrupt but normal changes, thereby improving prediction accuracy. We employ weak labels to partition the dataset into source and target domains. Then, we propose the Domain Adversarial Robust Forecaster (DARF). This forecasting model is based on adversarial domain adaptation and includes two novel modules: Correlated Robust Forecaster (CORF) and Domain Critic. Specifically, CORF constitutes an encoder-decoder framework proficient at robust multivariate time series forecasting, and Domain Critic works to reduce data bias. Extensive experiments and discussions show that DARF is capable of state-of-the-art forecasting accuracy.

Natural or artificial selection could shape genetic architecture, e.g., the relationship between minor allele frequency (MAF) and the effect sizes of causal variants (CVs). This study aimed to investigate the impact of the MAF-effect size relationship (as a selection signature, S) on genomic prediction and heritability estimation in livestock, using both simulated data (Holstein) and real datasets (Holstein and pigs). We evaluated the performance of two models: (1) selection-adjusted genomic best linear unbiased prediction (GBLUP-S), and (2) MAF-stratified selection-adjusted genomic best linear unbiased prediction (GBLUP-SMS). Simulation results demonstrated that for traits under strong negative selection (S < -1), both GBLUP-S and GBLUP-SMS outperformed classic GBLUP. The prediction accuracy of GBLUP-S improved by 0.011-0.031, while GBLUP-SMS achieved a gain of 0.005-0.025. Furthermore, GBLUP-SMS exhibited lower sensitivity to variations in S-values, whereas GBLUP-S heavily relied on accurate S specification. When the true S was matched, GBLUP-SMS generated more unbiased (or comparable) heritability estimates and higher prediction accuracy relative to GBLUP-S. Critically, mismatched S in GBLUP-S led to increased bias in heritability estimates and reduced prediction accuracy. Cross-validation with real phenotypic data from Holsteins and pigs demonstrated that implementing selection-adjust methods improved prediction accuracy by 0.015 for FP in Holsteins and 0.01 for T1 in pigs, while enhancing the unbiasedness of heritability estimates across all traits. Negative selection signatures were identified for cattle (S = -0.5) and pig T1, T2, and T3 (S = -1.5, -1, and -2, respectively). These findings advance the theoretical framework of GBLUP-based genomic prediction and heritability estimation.