Model performance metrics for multivariate image analysis

Using spatial information for evaluating the quality of prediction maps from hyperspectral images: A geostatistical approach

Objective: Thalassemia is a hereditary hemoglobinopathy and remains a significant public health problem, particularly in Mediterranean regions. Although genetic testing represents the gold standard for subtype classification, access to such testing is limited in many clinical settings. This pilot study aimed to explore the feasibility of using machine learning models based on routinely available clinical and laboratory parameters to support the differentiation of thalassemia subtypes in the absence of genetic testing.Methods: This retrospective cross-sectional study included 83 individuals (57 thalassemia major, 11 thalassemia intermedia, and 15 healthy controls). Demographic, clinical, and laboratory variables were analyzed using the R programming language. A supervised Random Forest algorithm was applied for multiclass classification. Model performance was assessed using accuracy, class-specific sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). To further evaluate the distinction between thalassemia major and intermedia, a simplified logistic regression model was constructed, and Firth logistic regression was applied to address the small sample size and class imbalance.Results: The Random Forest model demonstrated an overall test-set accuracy of 85.7%. Sensitivity was 80% for thalassemia major and 100% for both thalassemia intermedia and healthy controls. Variable importance analysis identified red cell distribution width (RDW), hematocrit, ferritin, and hemoglobin as the most influential predictors. In the simplified logistic regression model distinguishing thalassemia major from intermedia, RDW was the only variable reaching statistical significance (p = 0.0476). Model performance metrics, including high AUC values, should be interpreted cautiously given the limited sample size.Conclusion: The Random Forest model demonstrated an overall test-set accuracy of 85.7%. Sensitivity was 80% for thalassemia major and 100% for both thalassemia intermedia and healthy controls. Variable importance analysis identified red cell distribution width (RDW), hematocrit, ferritin, and hemoglobin as the most influential predictors. In the simplified logistic regression model distinguishing thalassemia major from intermedia, RDW was the only variable reaching statistical significance (p = 0.0476). Model performance metrics, including high AUC values, should be interpreted cautiously given the limited sample size.

Prediction models are essential in clinical decision-making for estimating the probability of current (diagnosis, screening) or future (prognosis) outcomes. Network meta-analysis (NMA) serves as a powerful tool to compare the performance of multiple prediction models simultaneously. However, there is hardly any guidance on methods and reporting for studies employing NMA to evaluate prediction models. To provide an overview of NMAs assessing prediction model (external validation) performance, regardless of whether they use aggregate data (AD) or individual participant data (IPD). In addition, we offer recommendations for improving the reporting and conduct of NMAs in prediction model research. We searched PubMed and Embase up to September 1, 2025, to identify studies that addressed the evaluation of diagnostic or prognostic prediction model performance using NMA. We included articles that employed NMA to compare and assess at least three prediction models. We summarized the identified studies based on, eg, their application (diagnostic vs prognostic), data use (AD vs IPD), medical contexts in which the models were assessed, and evaluation metrics applied (eg, discrimination, calibration, and (re)classification). In addition, we examined the statistical approaches employed, the NMA assumptions (such as consistency, transitivity, and exchangeability), and the ranking methods used for model comparison. After screening 2436 articles, 28 were included. Twenty-six studies (92.9%) used AD, while two (7.1%) used IPD. Hospital care was the most common setting (n = 22; 78.6%), with respirology (n = 7; 25.0%) and cardiology (n = 5; 17.9%) as the most frequently studied clinical domains. Key NMA assumptions were addressed differently across the 28 NMAs: 14.3% (n = 4) discussed transitivity, similarity, or exchangeability, and 53.6% (n = 15) tested for consistency. The statistical approach also varied, with 60.7% of studies (n = 17) reporting Bayesian methods and 17.9% (n = 5) reporting frequentist approaches. Surface under the cumulative ranking was the predominant ranking method (n = 18). Most NMAs included 5 to 10 models in the network, with 5 NMAs analyzing more than 20 models. Performance metrics varied, with 39.3% of studies (n = 11) reporting discrimination measures, such as C statistics, while none reported calibration metrics. Sensitivity or specificity was provided in 64.3% of studies (n = 18), and no articles reported advanced decision-analytic metrics like the decision curve analysis. This scoping review highlights the limited and diverse use of NMA methods in evaluating prediction models, with a predominant reliance on aggregate rather than IPD, and inconsistent consideration of key NMA assumptions and model performance metrics. We provide recommendations for the reporting and conduct of an NMA of prediction model validation performance. Prediction models are tools used in medicine to estimate a patient's risk of developing a disease or experiencing a health outcome. Many different prediction models exist for the same condition, and it can be difficult for doctors and researchers to know which model performs best. One way to compare multiple models is through a statistical method called network meta-analysis (NMA), which is commonly used to compare treatments but has rarely been applied to prediction models. In our study, we reviewed all published NMAs that evaluated prediction models to see how they were conducted and reported. We looked at whether the studies reported important performance measures, such as how well the models could distinguish between patients with and without the outcome (discrimination), and how well the predictions matched actual outcomes (calibration). We also checked if key NMA assumptions were considered and how analyses were conducted. We found that most studies used summary data instead of patient-level data. Many did not report crucial performance measures and rarely checked NMA assumptions. There was also limited transparency in how the models were analyzed, making it difficult for others to reproduce the results. Our findings show that while NMA has great potential to help compare prediction models and identify the most reliable ones, current practice often lacks the detailed reporting needed to make these comparisons fully trustworthy. We recommend better reporting, sharing of data and analysis code, and careful checking of assumptions to help researchers and doctors choose the most reliable models, ultimately improving patient care.

Some special topics in multivariate image analysis

Application of partial least squares and radial basis function neural networks in multivariate imaging analysis-quantitative structure activity relationship: Study of cyclin dependent kinase 4 inhibitors

Purpose: Numerous normal tissue complication probability (NTCP) models for radiation pneumonitis (RP) have been published in the literature, but minimal consideration is given to their predictive performance. Using several published model parameters, we quantified the range of model performance metrics to differentiate between and assess clinical utility of predictive RP models. Methods: Clinical, demographic, and dosimetric data was collected for 453 NSCLC patients treated with definitive photon radiotherapy. A series of logistic NTCP models were developed to assess overall performance. A baseline mean lung dose (MLD) model was considered in addition to our institutions best model, which incorporates MLD and smoking status. The model parameters from several publications were identified and incorporated in the analysis. The considered endpoint was CTCAEv3.0 grade>=3. Metrics of model performance including area under the curve (AUC) and Spearmans rank correlation (rs) were calculated for each of the models to facilitate comparison. Results: The MLD plus smoking status model performed best in terms of observed AUC (0.685) and rs (0.337); however, model performance was similar to that achieved by all of the other tested models (AUC range=[0.663‐0.685] and rs range=[0.263‐0.337]). In all considered models, predictive performance was poor (i.e. AUC<0.7 and rs<0.4). Additionally, the narrow range of AUC/rs values indicates that there is marginal difference in the performance of any one model over another. Conclusion: The range of quantitative performance metrics indicates that there is minimal ability to differentiate between the selected models. The achieved values for AUC and rs indicate poor overall model accuracy, which limits the practical utility of NTCP models for RP prediction. Furthermore, model parameters from the literature cannot by applied to our dataset to improve RP prediction. Other model formulations and still unidentified patient factors should be considered to develop more applicable and clinically useful NTCP models for treatment personalization.

Data science techniques such as machine learning are rapidly becoming available to engineers building models from system data, such as aircraft operations data. These techniques require validation for use in fielded systems providing recommendations to operators or maintainers. The methods for validating and testing machine learned algorithms generally focus on model performance metrics such as accuracy or F1-score. Many aviation datasets are highly imbalanced, which can invalidate some underlying assumptions of machine learning models. Two simulations are performed to show how some common performance metrics respond to imbalanced populations. The results show that each performance metric responds differently to a sample depending on the imbalance ratio between two classes. The results indicate that traditional methods for repairing underlying imbalance in the sample may not provide the rigorous validation necessary in safety critical applications. The two simulations indicate that authorities must be cautious when mandating metrics for model acceptance criteria because they can significantly influence the model parameters.

A simple imaging system has been developed for acquiring multivariate images in order to characterise the heterogeneity of food materials. The objective of the present work is, first, to demonstrate the capability of this acquisition system to discriminate food products of different natures. Secondly, our goal is to apply Partial Least Squares regression on these multivariate images and to evaluate the interest of various strategies of classification. A data set containing 24 images (702 × 524) acquired at different wavelengths for four food products is analysed. After the establishment of the PLS2 models employed for predicting the indicator variables, four strategies of classification of observations are tested. The first classification is done by selecting the largest component of the indicator variables. The others are based on the measurement of distances to the barycentres of the qualitative groups. Distances calculated can be either Euclidian distances or Mahalonobis distances. Except the strategy based on the Euclidian distance on scores, the strategies are rather equivalent, with a slight advantage to the Euclidian distance on predicted indicators. Another possibility addressed by the use of linear discriminant analysis (LDA) on multivariate images is to represent the qualitative groups as artificial images. The largest confusion appears between both cereal products while others are well classified. Copyright © 2006 John Wiley & Sons, Ltd.

Hydrological models are widely used to support water resources planning, flood and drought assessment, watershed management, and climate change impact analysis. The credibility of such applications depends strongly on how model performance is evaluated against observed data. Numerous statistical performance metrics have been proposed for hydrological model evaluation; however, their mathematical formulations, sensitivity to flow regimes, and interpretability differ substantially. As a result, the choice of evaluation criteria can strongly influence conclusions regarding model adequacy and comparative performance. This review synthesizes commonly used hydrological model performance metrics, including the Nash-Sutcliffe Efficiency (NSE), coefficient of determination (R²), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Percent Bias (PBIAS), Kling-Gupta Efficiency (KGE), RMSE-observations standard deviation ratio (RSR), and Willmott’s index of agreement. For each metric, definitions, formulations, interpretation guidelines, strengths, and limitations are discussed based on classical and recent literature. Particular attention is given to the sensitivity of metrics to high-flow and low-flow conditions, systematic bias, variability, and error magnitude, as well as their suitability for different water resources applications. Comparative analysis highlights that no single metric can adequately capture all aspects of hydrological model performance, and metric-dependent ranking of models is common across studies. The review study emphasizes the importance of multi-criteria evaluation frameworks and application-specific metric selection, especially under non-stationary climatic conditions. Practical recommendations are provided to support transparent, consistent, and meaningful performance evaluation in hydrological modeling and water resources decision-making.

Purpose of ReviewThis review aims to better understand the utility of machine learning algorithms for predicting spatial patterns of contaminants in the United States (U.S.) drinking water.Recent FindingsWe found 27 U.S. drinking water studies in the past ten years that used machine learning algorithms to predict water quality. Most studies (42%) developed random forest classification models for groundwater. Continuous models show low predictive power, suggesting that larger datasets and additional predictors are needed. Categorical/classification models for arsenic and nitrate that predict exceedances of pollution thresholds are most common in the literature because of good national scale data coverage and priority as environmental health concerns. Most groundwater data used to develop models were obtained from the United States Geological Survey (USGS) National Water Information System (NWIS). Predictors were similar across contaminants but challenges are posed by the lack of a standard methodology for imputation, pre-processing, and differing availability of data across regions.SummaryWe reviewed 27 articles that focused on seven drinking water contaminants. Good performance metrics were reported for binary models that classified chemical concentrations above a threshold value by finding significant predictors. Classification models are especially useful for assisting in the design of sampling efforts by identifying high-risk areas. Only a few studies have developed continuous models and obtaining good predictive performance for such models is still challenging. Improving continuous models is important for potential future use in epidemiological studies to supplement data gaps in exposure assessments for drinking water contaminants. While significant progress has been made over the past decade, methodological advances are still needed for selecting appropriate model performance metrics and accounting for spatial autocorrelations in data. Finally, improved infrastructure for code and data sharing would spearhead more rapid advances in machine-learning models for drinking water quality.

Hepatitis C is a viral infectious disease caused that affects 58 million people globally making it a public health crisis, and it is reported that 20-30% of cases develop to severe liver disease. This creates the need for data-driven based method that such as Machine learning that depends on data inputs in making accurate outputs with high precision. This study explores the use of Machine learning algorithms in optimizing the detection of hepatitis C status of patients. This research utilizes a publicly available dataset that contains 12 explanatory features and one target feature that addresses the patient’s diagnosis status. Data preprocessing techniques such as checking of missing values, assessment for possible outliers and Exploratory data analysis was carried on the dataset to improve the quality of the dataset for predictive analysis. The EDA results reviews that majority of the features had significant relationship (p-value < 0.05) with the target variable. Also, we observed the distribution of the disease diagnosis by gender reviews that females showed higher prevalence, and when we consider age, older people are more likely to contract the HCV. This result aligns with research findings from recent studies. The dataset was scaled and separated using a ratio split of 80:20 (80% of data for training). We applied SMOTE algorithm to the training data to balance the diagnosis distribution of the target variable. The model was trained using a wide parameter space using a cross-validation fold of 10 to select the best parameters. The ML models evaluated in this research include Logistic Regression, Support Vector Machine, Random Forest, XGBOOST, Decision Tree, and Extra Trees. The model performance metrics shows that XGBOOST was superior with an accuracy score of 98%, followed by Random Forest (97%), Decision Tree (94%), Extra Trees (93%), SVM (92%), and Logistic Regression (87%). The Kappa statistics for XGBOOST which was estimated as 0.8827 shows that difference between the predicted instances and the actual instances was minimal. The performance metrics of the XGBOOST was also contrast with ML algorithms trained by recent researchers in detecting HCV and it shows our model was more precise underscoring the significance of datapreprocessing, feature-scaling, SMOTE and hyperparameter tuning in predictive analysis in healthcare.

Summary: The potential of deep learning (DL) in genomic selection (GS) is constrained by the significant technical expertise required to design and implement neural networks. While DL has revolutionized fields like language processing and structural biology, its application in GS has not yet consistently outperformed traditional models like mixed linear models. The key to unlocking DL's power in GS lies in the exploration of network architectures tailored to genomic data, a process that demands intensive programming and poses a barrier for many researchers. To overcome this challenge, we developed Artificial Intelligence for Efficient and Versatile Evaluation and Representation (AI4EVER), a freely available graphical software platform that enables users to explore and apply machine learning (ML) models without any coding. AI4EVER integrates a graphical user interface (GUI) with a Python-based ML backend. The platform currently supports five models: Ridge Regression, Random Forest, Gradient Boosted Decision Trees, Multi-Layer Perceptron, and a customizable Keras-based neural network that can simultaneously predict multiple traits in a single model. A key feature of AI4EVER is optional incorporation of genome-wide association study (GWAS) results (p-values) as feature weights during model training, enabling biologically informed DL workflows. The platform further provides real-time visualization of model performance metrics and automated feature-importance outputs to enhance interpretability. AI4EVER also separates model training and prediction workflows, allowing trained models to be reused for independent prediction datasets. Using a representative maize dataset, we demonstrate that AI4EVER enables access to advanced AI, empowers genomic researchers to accelerate data-driven decision-making in breeding programs, ultimately lowering the barrier to artificial intelligence-enabled genetic improvement in crops and animals and human health management.

Recreational water quality is currently monitored at Sandpoint Beach on Lake St. Clair using culture-based enumeration of Escherichia coli. Using water quality and weather data collected over 4 yr, several multiple linear regression (MLR)-based models were developed for near real-time prediction of E. coli concentration and were tested using independent data from the fifth year. Model performance was assessed by the determination of metrics such as RMSE, accuracy, specificity, sensitivity, and area under the receiver operating characteristic curve (AUROC). Each of the developed MLR models described herein resulted in increased correct responses for both exceedance and non-exceedance of the applicable standard as compared to predictions based on E. coli measurements (persistence models, using the previous day's E. coli concentration), which is the method currently being used. The AUROC values for persistence models are between 0.5 and 0.6, as compared to >0.7 for all the MLR models described herein. Among the MLR models, model performance improved when qualitative sky weather condition, which is commonly reported but was not previously used in similar models, was included. To select the best model, a principal coordinate analysis was used to combine multiple model performance metrics and provide a more sensitive tool for model comparison. Although models developed using 2, 3, and 4 yr of monitoring data provided reasonable performance, the model developed using the most recent 2-yr data was marginally better. Thus, data from the most recent 2 yr are likely sufficient as a training dataset for updating the MLR model for Sandpoint Beach in the future.

There have been several attempts to develop machine learning (ML) models to identify human infecting viruses from their genomic sequences, with varying degrees of success. Direct comparison between models is problematic, because these models are typically trained and evaluated on different datasets with alternative data splitting schemes, features, and model performance metrics. In this paper we present a standardized dataset of mammal infecting and non-infecting viral pathogens, refined from the previous work of Mollentze et al. to include the latest literature evidence, roughly doubling the number of curated host-virus records available to the community, and new host target labels, primate and mammal. The new host labels were included for several reasons, including previous reports that classification performance is better at broader taxonomic ranks and the idea that there may be more data for primate infection that might serve as a suitable proxy for zoonotic potential and avoidance of false positives for human infection due to absence of evidence. On this dataset, we report the performance of eight machine learning models for predicting mammal-infecting viruses from their genomic sequences. We find that randomly assigning cases in our improved dataset to training/testing sets, when compared to the original assignments into training/testing in Mollentze et al., increases the overall average ROC AUC of prediction of human infection from 0.663 ± 0.070 to 0.784 ± 0.013, consistent with the reduction in phylogenetic distance between train and test sets (relative entropy change from 3.00 to 0.08). The broadest host category of mammal infection can be predicted most reliably at 0.850 ± 0.020. We share our improved dataset and code to enable standardized comparisons of machine learning methods to predict human host infections. Overall, we have presented preliminary evidence that classification of virus host infection is more tractable at higher taxonomic ranks, that unsurprisingly reducing the phylogenetic distance between training and test sets can improve predictive performance, that peptide kmer features appear to be harmful to out of sample model performance, and we are left with the question of whether models for virus host prediction can reasonably be expected to perform well in out of sample scenarios given the likelihood that viruses do not share a common ancestor. Consistent with this concern, when the data is resampled such that there is no overlap between viral families in training and test sets (relative entropy > 24), models perform no better than random chance at prediction of human infection regardless of whether kmers are included (ROC AUC 0.50 ± 0.08) or not (ROC AUC 0.50 ± 0.04).

Accurately determining sex using features like facial bone profiles and teeth is crucial for identifying unknown victims. Lateral cephalometric radiographs effectively depict the lateral cranial structure, aiding the development of computational identification models. This study develops and evaluates a sex prediction model using cephalometric radiographs with several convolutional neural network (CNN) architectures. The primary goal is to evaluate the model's performance on standardized radiographic data and real-world cranial photographs to simulate forensic applications. Six CNN architectures-VGG16, VGG19, MobileNetV2, ResNet50V2, InceptionV3, and InceptionResNetV2-were employed to train and validate 340 cephalometric images of Indonesian individuals aged 18 to 40 years. The data were divided into training (70%), validation (15%), and testing (15%) subsets. Data augmentation was implemented to mitigate class imbalance. Additionally, a set of 40 cranial images from anatomical specimens was employed to evaluate the model's generalizability. Model performance metrics included accuracy, precision, recall, and F1-score. CNN models were trained and evaluated on 340 cephalometric images (255 females and 85 males). VGG19 and ResNet50V2 achieved high F1-scores of 95% (females) and 83% (males), respectively, using cephalometric data, highlighting their strong class-specific performance. Although the overall accuracy exceeded 90%, the F1-score better reflected model performance in this imbalanced dataset. In contrast, performance notably decreased with cranial photographs, particularly when classifying female samples. That is, while InceptionResNetV2 achieved the highest F1-score for cranial photographs (62%), misclassification of females remained significant. Confusion matrices and per-class metrics further revealed persistent issues related to data imbalance and generalization across imaging modalities. Basic CNN models perform well on standardized cephalometric images but less effectively on photographic cranial images, indicating a domain shift between image types that limits generalizability. Improving real-world forensic performance will require further optimization and more diverse training data. Not applicable.