An exploratory data analysis: the performance differences of a medical code prediction system on different demographic groups

Accurately determining sex using features like facial bone profiles and teeth is crucial for identifying unknown victims. Lateral cephalometric radiographs effectively depict the lateral cranial structure, aiding the development of computational identification models. This study develops and evaluates a sex prediction model using cephalometric radiographs with several convolutional neural network (CNN) architectures. The primary goal is to evaluate the model's performance on standardized radiographic data and real-world cranial photographs to simulate forensic applications. Six CNN architectures-VGG16, VGG19, MobileNetV2, ResNet50V2, InceptionV3, and InceptionResNetV2-were employed to train and validate 340 cephalometric images of Indonesian individuals aged 18 to 40 years. The data were divided into training (70%), validation (15%), and testing (15%) subsets. Data augmentation was implemented to mitigate class imbalance. Additionally, a set of 40 cranial images from anatomical specimens was employed to evaluate the model's generalizability. Model performance metrics included accuracy, precision, recall, and F1-score. CNN models were trained and evaluated on 340 cephalometric images (255 females and 85 males). VGG19 and ResNet50V2 achieved high F1-scores of 95% (females) and 83% (males), respectively, using cephalometric data, highlighting their strong class-specific performance. Although the overall accuracy exceeded 90%, the F1-score better reflected model performance in this imbalanced dataset. In contrast, performance notably decreased with cranial photographs, particularly when classifying female samples. That is, while InceptionResNetV2 achieved the highest F1-score for cranial photographs (62%), misclassification of females remained significant. Confusion matrices and per-class metrics further revealed persistent issues related to data imbalance and generalization across imaging modalities. Basic CNN models perform well on standardized cephalometric images but less effectively on photographic cranial images, indicating a domain shift between image types that limits generalizability. Improving real-world forensic performance will require further optimization and more diverse training data. Not applicable.

SummaryAccurate prediction of the solubility of chemical substances in solvents remains a challenge. The sparsity of high-quality solubility data is recognized as the biggest hurdle in the development of robust data-driven methods for practical use. Nonetheless, the effects of the quality and quantity of data on aqueous solubility predictions have not yet been scrutinized. In this study, the roles of the size and the quality of data sets on the performances of the solubility prediction models are unraveled, and the concepts of actual and observed performances are introduced. In an effort to curtail the gap between actual and observed performances, a quality-oriented data selection method, which evaluates the quality of data and extracts the most accurate part of it through statistical validation, is designed. Applying this method on the largest publicly available solubility database and using a consensus machine learning approach, a top-performing solubility prediction model is achieved.

Disparities in mental health care exist between regional and demographic groups. While screening is recommended as part of a correctional mental health strategy, little work has been done to explore whether it can narrow regional and demographic disparities in access to care. We compared treatment access rates by sex, race, age, and region in relation to screening results. We conducted a retrospective cohort study using administrative data. All 7965 admissions to the prison system were followed for a median of 14 months. Males and non-Indigenous minority racial groups had lower rates of treatment regardless of screening results; they were less likely both to self-report needs and to receive treatment if these needs were reported. Regional differences revealed higher treatment rates in Atlantic Canada and Ontario, as well as higher rates of inmates self-reporting needs on screening who did not receive treatment in the Atlantic, Québec, and Pacific regions. There were minimal differences between inmates of different age groups. Findings suggest potential resource gaps and/or differences in the performance of screening to detect mental health needs across demographic and regional groups. Screening did not narrow, and may have widened, differences between groups.

Abstract. Benchmarking model performance across large samples of catchments is useful to guide model selection and future model development. Given uncertainties in the observational data we use to drive and evaluate hydrological models, and uncertainties in the structure and parameterisation of models we use to produce hydrological simulations and predictions, it is essential that model evaluation is undertaken within an uncertainty analysis framework. Here, we benchmark the capability of several lumped hydrological models across Great Britain by focusing on daily flow and peak flow simulation. Four hydrological model structures from the Framework for Understanding Structural Errors (FUSE) were applied to over 1000 catchments in England, Wales and Scotland. Model performance was then evaluated using standard performance metrics for daily flows and novel performance metrics for peak flows considering parameter uncertainty. Our results show that lumped hydrological models were able to produce adequate simulations across most of Great Britain, with each model producing simulations exceeding a 0.5 Nash–Sutcliffe efficiency for at least 80 % of catchments. All four models showed a similar spatial pattern of performance, producing better simulations in the wetter catchments to the west and poor model performance in central Scotland and south-eastern England. Poor model performance was often linked to the catchment water balance, with models unable to capture the catchment hydrology where the water balance did not close. Overall, performance was similar between model structures, but different models performed better for different catchment characteristics and metrics, as well as for assessing daily or peak flows, leading to the ensemble of model structures outperforming any single structure, thus demonstrating the value of using multi-model structures across a large sample of different catchment behaviours. This research evaluates what conceptual lumped models can achieve as a performance benchmark and provides interesting insights into where and why these simple models may fail. The large number of river catchments included in this study makes it an appropriate benchmark for any future developments of a national model of Great Britain.

Label distribution learning (LDL) can more accurately represent the degree of correlation between labels and samples than multi-label learning. However, LDL usually has limited available label data, which is not conducive to training deep learning models. Data augmentation refers to methods for solving limited data problems by introducing unobserved data or latent variables to increase the size and quality of the training dataset. In this paper, we augment the dataset by mapping the features and label distributions of the generated samples to the same subspace, and using the generator to learn the distribution of the original data in this space. First, we use the encoder and generator to extract effective information from sample features and label distributions, respectively. Second, we randomly fuse the existing label distribution to generate a new label distribution, then map it to the subspace and restore the corresponding features through the decoder. Finally, these generated samples are mixed with the original training set to train a model for predicting label distribution. Experimental results on nine real-world datasets show that our proposed algorithm can improve the performance of deep learning models to a certain extent.

There have been several attempts to develop machine learning (ML) models to identify human infecting viruses from their genomic sequences, with varying degrees of success. Direct comparison between models is problematic, because these models are typically trained and evaluated on different datasets with alternative data splitting schemes, features, and model performance metrics. In this paper we present a standardized dataset of mammal infecting and non-infecting viral pathogens, refined from the previous work of Mollentze et al. to include the latest literature evidence, roughly doubling the number of curated host-virus records available to the community, and new host target labels, primate and mammal. The new host labels were included for several reasons, including previous reports that classification performance is better at broader taxonomic ranks and the idea that there may be more data for primate infection that might serve as a suitable proxy for zoonotic potential and avoidance of false positives for human infection due to absence of evidence. On this dataset, we report the performance of eight machine learning models for predicting mammal-infecting viruses from their genomic sequences. We find that randomly assigning cases in our improved dataset to training/testing sets, when compared to the original assignments into training/testing in Mollentze et al., increases the overall average ROC AUC of prediction of human infection from 0.663 ± 0.070 to 0.784 ± 0.013, consistent with the reduction in phylogenetic distance between train and test sets (relative entropy change from 3.00 to 0.08). The broadest host category of mammal infection can be predicted most reliably at 0.850 ± 0.020. We share our improved dataset and code to enable standardized comparisons of machine learning methods to predict human host infections. Overall, we have presented preliminary evidence that classification of virus host infection is more tractable at higher taxonomic ranks, that unsurprisingly reducing the phylogenetic distance between training and test sets can improve predictive performance, that peptide kmer features appear to be harmful to out of sample model performance, and we are left with the question of whether models for virus host prediction can reasonably be expected to perform well in out of sample scenarios given the likelihood that viruses do not share a common ancestor. Consistent with this concern, when the data is resampled such that there is no overlap between viral families in training and test sets (relative entropy > 24), models perform no better than random chance at prediction of human infection regardless of whether kmers are included (ROC AUC 0.50 ± 0.08) or not (ROC AUC 0.50 ± 0.04).

There have been several attempts to develop machine learning (ML) models to identify human infecting viruses from their genomic sequences, with varying degrees of success. Direct comparison between models is problematic, because these models are typically trained and evaluated on different datasets with alternative data splitting schemes, features, and model performance metrics. In this paper we present a standardized dataset of mammal infecting and non-infecting viral pathogens, refined from the previous work of Mollentze et al. to include the latest literature evidence, roughly doubling the number of curated host-virus records available to the community, and new host target labels, primate and mammal. The new host labels were included for several reasons, including previous reports that classification performance is better at broader taxonomic ranks and the idea that there may be more data for primate infection that might serve as a suitable proxy for zoonotic potential and avoidance of false positives for human infection due to absence of evidence. On this dataset, we report the performance of eight machine learning models for predicting mammal-infecting viruses from their genomic sequences. We find that randomly assigning cases in our improved dataset to training/testing sets, when compared to the original assignments into training/testing in Mollentze et al., increases the overall average ROC AUC of prediction of human infection from 0.663 ± 0.070 to 0.784 ± 0.013, consistent with the reduction in phylogenetic distance between train and test sets (relative entropy change from 3.00 to 0.08). The broadest host category of mammal infection can be predicted most reliably at 0.850 ± 0.020. We share our improved dataset and code to enable standardized comparisons of machine learning methods to predict human host infections. Overall, we have presented preliminary evidence that classification of virus host infection is more tractable at higher taxonomic ranks, that unsurprisingly reducing the phylogenetic distance between training and test sets can improve predictive performance, that peptide kmer features appear to be harmful to out of sample model performance, and we are left with the question of whether models for virus host prediction can reasonably be expected to perform well in out of sample scenarios given the likelihood that viruses do not share a common ancestor. Consistent with this concern, when the data is resampled such that there is no overlap between viral families in training and test sets (relative entropy > 24), models perform no better than random chance at prediction of human infection regardless of whether kmers are included (ROC AUC 0.50 ± 0.08) or not (ROC AUC 0.50 ± 0.04).

Model-informed precision dosing (MIPD) optimizes drug doses based on pharmacokinetic (PK) model predictions, necessitating careful selection of models tailored to patient characteristics. This study evaluates the predictive performance of various vancomycin PK models across diverse age and BMI categories, drawing insights from a large multi-site database. Adults receiving vancomycin intravenous therapy at United States health systems between January 1, 2022, and December 31, 2023, were included. Patient demographics, vancomycin administration records, and therapeutic drug monitoring levels (TDMs) were collected from the InsightRX database. Age and body mass index (BMI)-based subgroups were formed to assess model performance, with predictions made iteratively. The optimal model for each age-BMI subgroup was chosen based on predefined criteria: models were filtered for mean percentage error (MPE) ≤ 20% and normalized root mean squared error (RMSE) < 8 mg/L, and then the most accurate among them was selected. A total of 384,876 treatment courses across 155 US health systems were analyzed, contributing 841,604 TDMs. Eleven models were compared, showing varying accuracy across age-BMI categories (41%-73%), with higher accuracy observed once TDMs were available for Bayesian estimates of individual PK parameters. Models performed more poorly in younger adults compared to older adults, and the optimal model differed depending on age-BMI categories and prediction methods. Notably, in the a priori period, the Colin model performed best in adults aged 18-64 years across most BMI categories; the Goti/Tong model performed best in the older, non-obese adults; and the Hughes model performed best in many of the obese categories. Our study identifies specific vancomycin PK models that demonstrate superior predictions across age-BMI categories in MIPD applications. Our findings underscore the importance of tailored model selection for vancomycin management, especially highlighting the need for improved models in younger adult patients. Further research into the clinical implications of model performance is warranted to enhance patient care outcomes.

Soil water retention curve (SWRC) is key hydraulic property for vadose zone hydrology. Selected 22 SWRC models with different structures were divided into four types: traditional, single‐segment, two‐segment, and three‐segment models. According to the accuracy and model selection criterion, these models were compared and evaluated using water retention data from saturation to oven dryness of 94 soil samples covering almost all soil types. Among the models, the EG model (exponential model combined with the Groenevelt–Grant model) shows the best performance, but the BC (Brooks–Corey) and VG (van Genuchten) models performed the worst. The relationship between model performance and complexity showed that traditional models with residual water content performed worse than segmental models from saturation to oven dryness, and the segmental models performed worse with an increase in the segment number. In general, models with more parameters perform better, but when the parameter number is greater than five, the model performance gradually worsens due to the equifinality of different parameters. The correlation analysis between soil properties on model performance showed that soil texture has a more significant effect on model performance than bulk density and organic matter content, because the models are primarily affected by their mathematical forms rather than the physical significance of the parameters. Each model picks its limitations and may apply to only one or particular group(s) of soil(s). Therefore, the complementarities of different models may be a great choice to improve SWRC fitting and simulation in arid soil.

Atmospheric nitrogen oxide and nitrogen dioxide (NO + NO2, together termed as NOX) estimates from annual photochemical simulations for years 2002–2016 are compared to surface network measurements of NOX and total gas-phase-oxidized reactive nitrogen (NOY) to evaluate the Community Multiscale Air Quality (CMAQ) modeling system performance by U.S. region, season, and time of day. In addition, aircraft measurements from 2011 Deriving Information on Surface Conditions from Column and Vertically Resolved Observations Relevant to Air Quality are used to evaluate how emissions, chemical mechanism, and measurement uncertainty each contribute to the overall model performance. We show distinct seasonal and time-of-day patterns in NOX performance. Summertime NOX is overpredicted with bimodal peaks in bias during early morning and evening hours and persisting overnight. The summertime morning NOX bias dropped from between 28% and 57% for earlier years (2002–2012) to between −2% and 7% for later years (2013–2016). Summer daytime NOX tends to be unbiased or underpredicted. In winter, the evening NOX overpredictions remain, but NOX is unbiased or underpredicted overnight, in the morning, and during the day. NOX overpredictions are most pronounced in the Midwestern and Southern United States with Western regions having more of a tendency toward model underpredictions of NOX. Modeled NOX performance has improved substantially over time, reflecting updates to the emission inputs and the CMAQ air quality model. Model performance improvements are largest for years simulated with CMAQv5.1 or later and for emission inventory years 2014 and later, coinciding with reduced onroad NOX emissions from vehicles with newer emission control technologies and improved treatment of chemistry, deposition, and vertical mixing in CMAQ. Our findings suggest that emissions temporalization of specific mobile source sectors have a small impact on model performance, while chemistry updates improve predictions of NOY but do not improve summertime NOX bias in the Baltimore/DC area. Sensitivity runs performed for different locations across the country suggest that the improvement in summer NOX performance can be attributed to updates in vertical mixing incorporated in CMAQv5.1.

Automatic Breast Ultrasound (ABUS) image segmentation is of great significance for breast cancer diagnosis and treatment. However, similar to most medical datasets, ABUS image datasets are often small-scale and seriously imbalanced, which makes ABUS image segmentation become a challenge. To solve this problem, we propose the Residual Swin Transformer Unet with Consistency Regularization (RSTUnet-CR) which can make full use of non-lesion and unlabeled images for high-precision tumor segmentation on ABUS images. We design a consistency-regularization decoder to reconstruct the input image, which can learn well from non-lesion and unlabeled data. The reconstruction task makes the model more suitable for the imbalanced medical image datasets. In addition, observing that the ABUS images have global semantic correlation, we establish long-distance dependence of images by the residual Swin Transformer block to improve segmentation performance. We evaluate our method on the ABUS dataset collected from 256 subjects and demonstrate the superiority of the proposed method over other state-of-the-art methods in this imbalanced dataset.

Medical diagnosis is a crucial step for patient treatment. However, diagnosis is prone to bias due to imbalanced datasets. To overcome the imbalanced dataset problem, simple minority oversampling technique (SMOTE) was proposed that can generate new synthetic samples at data level to create the balance between minority and majority classes. However, the synthetic samples are generated on a random basis which causes class mixture problem; thus, resulting in deteriorating the classification performance and biased diagnosis. In order to overcome the SMOTE shortcomings, some modified methods were proposed that try to generate synthetic samples along the line segment of selected minority samples. Most of these methods adopt one of the two policies for selecting minority samples to generate synthetic samples: borderline region sampling or safe region sampling. However, they both suffer from over-generalisation problem. We propose a modified SMOTE-based resampling method called RSMOTE to alleviate the medical imbalanced dataset problem. We provide an in-depth analysis and verify the performance of RSMOTE over imbalanced medical datasets. In this paper, the proposed RSMOTE divides the minority sample domain into four regions (normal, semi-normal, semi-critical, and critical) based on the minority sample density analysis. RSMOTE discovers the minority sample region globally and applies the resampling near a specific group of samples. Our analysis and experiments verify that if synthetic samples are generated in the regions with high minority sample density, classification performance will be improved due to low risk of class mixture. Unlike some safe region methods, RSMOTE decides the region of minority samples on a global basis, thus removing the over-generalisation problem. Classic and additional evaluation metrics are considered to measure the effectiveness of the modified method: Recall, FP Rate, Precision, F-Measure, ROC area, and Average Aggregated Metric. We carried out experiments over various imbalanced medical datasets. Based on the minority sample density analysis, we propose RSMOTE method that divides the minority sample domain into four regions. The proposed RSMOTE includes four re-sampling methods that each of them carries out resampling on a specific region. According to the experimental results, resampling on the regions with high minority sample density obtained better results while those with lower minority sample density got the inferior results. Thus, we conclude that the RSMOTE is a more flexible resampling method for the imbalanced medical datasets that is capable of generating samples with various minority sample densities.

The purpose of the present meta-analysis was to determine if biodata scale scores differ based on demographic group membership (i.e., gender, race, age) and to evaluate the contextual factors that amplify or mitigate these effects (e.g., construct domain, scoring method). Despite the popularity of biodata scales for personnel selection purposes, previous research findings do not provide clear evidence as to whether adverse impact is a serious concern. To address this gap, a meta-analysis of 43 studies (56 independent samples) was conducted, providing estimates of group differences across demographic groups (female-male, Black-White, Hispanic-White, Asian-White, and age) and relevant construct domains. The majority of biodata scale scores were found to exhibit small group differences across construct domains and demographic groups (≈75% had Cohen's d less than |.20|). Group differences were also relatively small when compared with other popular selection methods. When considered with previous research, these findings provide further support for the use of biodata scales in personnel selection scenarios. (PsycInfo Database Record (c) 2025 APA, all rights reserved).

Purpose: Numerous normal tissue complication probability (NTCP) models for radiation pneumonitis (RP) have been published in the literature, but minimal consideration is given to their predictive performance. Using several published model parameters, we quantified the range of model performance metrics to differentiate between and assess clinical utility of predictive RP models. Methods: Clinical, demographic, and dosimetric data was collected for 453 NSCLC patients treated with definitive photon radiotherapy. A series of logistic NTCP models were developed to assess overall performance. A baseline mean lung dose (MLD) model was considered in addition to our institutions best model, which incorporates MLD and smoking status. The model parameters from several publications were identified and incorporated in the analysis. The considered endpoint was CTCAEv3.0 grade&gt;=3. Metrics of model performance including area under the curve (AUC) and Spearmans rank correlation (rs) were calculated for each of the models to facilitate comparison. Results: The MLD plus smoking status model performed best in terms of observed AUC (0.685) and rs (0.337); however, model performance was similar to that achieved by all of the other tested models (AUC range=[0.663‐0.685] and rs range=[0.263‐0.337]). In all considered models, predictive performance was poor (i.e. AUC&lt;0.7 and rs&lt;0.4). Additionally, the narrow range of AUC/rs values indicates that there is marginal difference in the performance of any one model over another. Conclusion: The range of quantitative performance metrics indicates that there is minimal ability to differentiate between the selected models. The achieved values for AUC and rs indicate poor overall model accuracy, which limits the practical utility of NTCP models for RP prediction. Furthermore, model parameters from the literature cannot by applied to our dataset to improve RP prediction. Other model formulations and still unidentified patient factors should be considered to develop more applicable and clinically useful NTCP models for treatment personalization.

Mortality risk after hospitalization for heart failure (HF) is high, especially in the first 90days. This study aimed to construct a model automatically predicting 90day post-discharge mortality using electronic health record (EHR) data 48h after admission and artificial intelligence. All HF-related admissions from 2015 to 2020 in a single hospital were included in the model training. Comprehensive EHR data were collected 48h after admission. Natural language processing was applied to textual information. Deaths were identified from the French national database. After variable selection with least absolute shrinkage and selection operator, a logistic regression model was trained. Model performance [area under the receiver operating characteristic curve (AUC)] was tested in two independent cohorts of patients admitted to two hospitals between March and December 2021. The derivation cohort included 2257 admissions (248 deaths after hospitalization). The evaluation cohorts included 348 and 388 admissions (34 and 38 deaths, respectively). Forty-two independent variables were selected. The model performed well in the derivation cohort [AUC: 0.817; 95% confidence interval (CI) (0.789-0.845)] and in both evaluation cohorts [AUC: 0.750; 95% CI (0.672-0.829) and AUC: 0.723; 95% CI (0.644-0.803]), with better performance than previous models in the literature. Calibration was good: 'low-risk' (predicted mortality ≤8%), 'intermediate-risk' (8-12.5%) and 'high-risk' (>12.5%) patients had an observed 90day mortality rate of 3.8%, 8.4% and 19.4%, respectively. The study proposed a robust model for the automatic prediction of 90day mortality risk 48h after hospitalization for decompensated HF. This could be used to identify high-risk patients for intensification of therapeutic management.