Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis

ObjectiveTo systematically evaluate the methodological quality and predictive performance of risk prediction models for permanent pacemaker implantation (PPMI) following transcatheter aortic valve replacement (TAVR), identify key predictive factors, and assess the risk of bias and clinical applicability of these models.MethodsA comprehensive search was conducted across multiple databases, including PubMed, Web of Science, The Cochrane Library, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL), China National Knowledge Infrastructure (CNKI), Wanfang Database, China Science and Technology Journal Database (VIP), and SinoMed. The search included all records from database inception to January 1, 2025. Two independent researchers screened studies and extracted relevant data.ResultsA total of 11 studies were included, covering 11 risk prediction models with sample sizes ranging from 184–35,410. The incidence of PPMI after TAVR varied between 7.3% and 31.0%. Frequently identified predictors (present in at least two studies) included right bundle branch block (RBBB), self-expandable valves, PR interval, QRS interval, and atrioventricular block (AVB). All models reported the area under the receiver operating characteristic curve (AUROC), ranging from 0.660–0.916, with seven studies providing calibration metrics. Internal validation was performed in three studies, while one study included both internal and external validation. Ten studies were assessed as having a high risk of bias, primarily due to deficiencies in data analysis. The pooled AUROC for the nine validated models was 0.76 (95% confidence interval: 0.72–0.80), indicating moderate discriminatory ability.ConclusionExisting risk prediction models for PPMI after TAVR demonstrate moderate predictive performance but are limited by a high risk of bias, as assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Future research should focus on developing more robust models through larger sample sizes, rigorous methodologies, and multi-center external validation.Systematic Review RegistrationThe protocol for this study is registered with https://www.crd.york.ac.uk/PROSPERO/view/CRD42025629869, PROSPERO CRD42025629869.

ObjectiveThis study aimed to systematically evaluate published predictive models for dental caries in children and adolescents.DesignA systematic review and meta-analysis of observational studies.Data sourcesComprehensive searches were conducted in PubMed, Web...

BackgroundSeveral prediction models for cognitive frailty (CF) in older adults have been developed. However, the existing models have varied in predictors and performances, and the methodological quality still needs to be determined.ObjectivesWe aimed to summarize and critically appraise the reported multivariable prediction models in older adults with CF.MethodsPubMed, Embase, Cochrane Library, Web of Science, Scopus, PsycINFO, CINAHL, China National Knowledge Infrastructure, and Wanfang Databases were searched from the inception to March 1, 2022. Included models were descriptively summarized and critically appraised by the Prediction Model Risk of Bias Assessment Tool (PROBAST).ResultsA total of 1,535 articles were screened, of which seven were included in the review, describing the development of eight models. Most models were developed in China (n = 4, 50.0%). The most common predictors were age (n = 8, 100%) and depression (n = 4, 50.0%). Seven models reported discrimination by the C-index or area under the receiver operating curve (AUC) ranging from 0.71 to 0.97, and four models reported the calibration using the Hosmer–Lemeshow test and calibration plot. All models were rated as high risk of bias. Two models were validated externally.ConclusionThere are a few prediction models for CF. As a result of methodological shortcomings, incomplete presentation, and lack of external validation, the models’ usefulness still needs to be determined. In the future, models with better prediction performance and methodological quality should be developed and validated externally.Systematic review registrationwww.crd.york.ac.uk/prospero, identifier CRD42022323591.

Risk prediction model for chemotherapy-induced nausea and vomiting in cancer patients: a systematic review.

Risk prediction models for postoperative delirium in elderly patients with fragility hip fracture: A systematic review and critical appraisal

BackgroundRisk prediction models for depression in patients with coronary heart disease are increasingly being developed. However, the quality and applicability of these models in clinical practice remain uncertain.ObjectiveTo systematically evaluate depression risk prediction models in patients with coronary heart disease (CHD).MethodsDatabases including PubMed, Web of Science, Embase, Cochrane Library, CNKI, Wanfang, VIP, and SinoMed were searched for relevant studies from inception to September 29, 2024. Two researchers independently screened the literature, extracted data, and used the Prediction Model Risk of Bias Assessment Tool (PROBAST) to evaluate the models' risk of bias and applicability.ResultsEight studies, encompassing 13 risk prediction models and involving 8,035 CHD patients, were included, with 1,971 patients diagnosed with depression. Common predictors included age, educational level, gender, and cardiac function classification. The area under the curve (AUC) for the models ranged from 0.772 to 0.961, indicating overall good performance; however, risk of bias was high, primarily due to issues in the analysis phase, such as inadequate handling of missing values, univariate analysis for variable selection, and lack of external validation.ConclusionDepression risk prediction models for CHD patients generally perform well, but high risk of bias and limited applicability remain concerns. Future studies should focus on developing and validating more robust models to aid healthcare professionals in early identification of high-risk patients for depression.Systematic Review Registrationhttps://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42024625641, identifier (CRD42024625641).

ObjectiveThe prognosis for acute exacerbation of chronic obstructive pulmonary disease (AECOPD) is not optimistic, and severe AECOPD leads to an increased risk of mortality. Prediction models help distinguish between high‐ and low‐risk groups. At present, many prediction models have been established and validated, which need to be systematically reviewed to screen out more suitable models that can be used in the clinic and provide evidence for future research.MethodsWe searched PubMed, EMBASE, Cochrane Library and Web of Science databases for studies on risk models for AECOPD mortality from their inception to 10 April 2022. The risk of bias was assessed using the prediction model risk of bias assessment tool (PROBAST). Stata software (version 16) was used to synthesize the C‐statistics for each model.ResultsA total of 37 studies were included. The development of risk prediction models for mortality in patients with AECOPD was described in 26 articles, in which the most common predictors were age (n = 17), dyspnea grade (n = 11), altered mental status (n = 8), pneumonia (n = 6) and blood urea nitrogen (BUN, n = 6). The remaining 11 articles only externally validated existing models. All 37 studies were evaluated at a high risk of bias using PROBAST. We performed a meta‐analysis of five models included in 15 studies. DECAF (dyspnoea, eosinopenia, consolidation, acidemia and atrial fibrillation) performed well in predicting in‐hospital death [C‐statistic = 0.91, 95% confidence interval (CI): 0.83, 0.98] and 90‐day death [C‐statistic = 0.76, 95% CI: 0.69, 0.82] and CURB‐65 (confusion, urea, respiratory rate, blood pressure and age) performed well in predicting 30‐day death [C‐statistic = 0.74, 95% CI: 0.70, 0.77].ConclusionsThis study provides information on the characteristics, performance and risk of bias of a risk model for AECOPD mortality. This pooled analysis of the present study suggests that the DECAF performs well in predicting in‐hospital and 90‐day deaths. Yet, external validation in different populations is still needed to prove this performance.

BackgroundPrognostic models are used widely in the oncology domain to guide medical decision-making. Little is known about the risk of bias of prognostic models developed using machine learning and the barriers to their clinical uptake in the oncology domain.MethodsWe conducted a systematic review and searched MEDLINE and EMBASE databases for oncology-related studies developing a prognostic model using machine learning methods published between 01/01/2019 and 05/09/2019. The primary outcome was risk of bias, judged using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We described risk of bias overall and for each domain, by development and validation analyses separately.ResultsWe included 62 publications (48 development-only; 14 development with validation). 152 models were developed across all publications and 37 models were validated. 84% (95% CI: 77 to 89) of developed models and 51% (95% CI: 35 to 67) of validated models were at overall high risk of bias. Bias introduced in the analysis was the largest contributor to the overall risk of bias judgement for model development and validation. 123 (81%, 95% CI: 73.8 to 86.4) developed models and 19 (51%, 95% CI: 35.1 to 67.3) validated models were at high risk of bias due to their analysis, mostly due to shortcomings in the analysis including insufficient sample size and split-sample internal validation.ConclusionsThe quality of machine learning based prognostic models in the oncology domain is poor and most models have a high risk of bias, contraindicating their use in clinical practice. Adherence to better standards is urgently needed, with a focus on sample size estimation and analysis methods, to improve the quality of these models.

ObjectiveThis study aims to evaluate the risk of bias (ROB) and reporting quality of idiopathic pulmonary fibrosis (IPF) prediction models by assessing characteristics of these models.MethodsThe development and/or validation of IPF prognostic models were identified via an electronic search of PubMed, Embase, and Web of Science (from inception to 12 August, 2021). Two researchers independently assessed the risk of bias (ROB) and reporting quality of IPF prediction models based on the Prediction model Risk Of Bias Assessment Tool (PROBAST) and Transparent Reporting of a multivariable prognostic model for Individual Prognosis or Diagnosis (TRIPOD) checklist.ResultsTwenty prognostic model studies for IPF were included, including 7 (35%) model development and external validation studies, 8 (40%) development studies, and 5 (25%) external validation studies. According to PROBAST, all studies were appraised with high ROB, because of deficient reporting in the domains of participants (45.0%) and analysis (67.3%), and at least 55% studies were susceptible to 4 of 20 sources of bias. For the reporting quality, none of them completely adhered to the TRIPOD checklist, with the lowest mean reporting score for the methods and results domains (46.6% and 44.7%). For specific items, eight sub-items had a reporting rate ≥80% and adhered to the TRIPOD checklist, and nine sub-items had a very poor reporting rate, less than 30%.ConclusionStudies adhering to PROBAST and TRIPOD checklists are recommended in the future. The reproducibility and transparency can be improved when studies completely adhere to PROBAST and TRIPOD checklists.

Artificial intelligence (AI) prediction models can accurately identify high-risk populations by integrating multi-dimensional clinical data, providing decision support for doctors in formulating individualized discharge plans and optimizing follow-up intervention strategies, thereby reducing the risk of readmission from the source. Currently, the number of AI prediction models for readmission of critically ill patients is increasing, but the quality and applicability of these models in clinical practice and future research remain uncertain. To systematically evaluate published studies on AI prediction models for critically ill patients. This study conducted a computerized search of the CNKI, Wanfang Data, VIP, SinoMed, PubMed, Web of Science, Cochrane, and Embase databases, with the time range from 2020 to June 25, 2025. Information such as study design, data sources, outcome definitions, sample size, predictors, model development, and performance was extracted from the selected studies. The Prediction Model Risk of Bias Assessment Tool (PROBAST) checklist was used to evaluate the risk of bias and applicability. A total of 387 studies were retrieved, and after screening, 31 studies with their 31 prediction models were included in this review. All studies developed risk prediction models for readmission of critically ill patients using artificial intelligence algorithms. The readmission risk of critically ill patients ranged from 1.3% to 13.7%. The most commonly used predictors were structured data. The reported area under the curve (AUC) ranged from 0.66 to 0.98. All studies had a high risk of bias, mainly due to poor reporting quality in the analysis domain and insufficient applicability. The pooled AUC of the 24 validation models was 0.82, with a 95% confidence interval of 0.77-0.87. These study results constitute a comprehensive set of high-quality evidence, demonstrating that AI prediction models exhibit moderate-to-high predictive performance and that their predictive performance is significantly higher than that of traditional prediction models. No Patient or Public Contribution. This Meta-analysis is based on the systematic review and statistical combination of the published clinical research data. The processes of research design, data extraction, and result interpretation did not involve the participation of patients or the public. The protocol for this study has been registered in PROSPERO (registration number: CRD42025637829).

Background/Objectives: Renal cancer has a heterogeneous characteristic. Prognostic models can enable a better evaluation of the prognosis. This study aimed to analyze the clinical applicability and risk of bias of prognostic models described in the literature for predicting cancer-specific mortality in renal cancer patients who have undergone nephrectomy. Methods: A systematic review (PROSPERO [CRD42021243529]) of all scientific articles that evaluate prognostic models for cancer-specific mortality due to renal cancer was performed. Descriptive analysis and application of the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) and the Prediction Model Risk Of Bias Assessment Tool (PROBAST) were used. Results: Most of the 40 reviewed studies used retrospective cohort designs, mainly based on hospital records or the SEER database, and focused on patients undergoing nephrectomy. While 50% developed models without validation, the rest included internal or external validation methods, with nomograms being the most common format for presenting results. Cox regression was the main modeling technique, although problems such as poor treatment of missing data, inadequate reporting of events per variable, and limited assessment of model assumptions were prevalent. According to the PROBAST assessment, all studies showed a high risk of bias, particularly in the scope of analysis, and only 40% had good applicability. Conclusions: All the studies analyzed were found to have a high risk of bias, and only 40% demonstrated good applicability. Hence, it is necessary to develop cancer-specific prognostic models for renal cancer based on the CHARMS and PROBAST frameworks.

Malignant melanoma is the fifth most common cancer in the UK, with rates continuing to rise, resulting in considerable burden to patients and the NHS. The objectives were to evaluate the effectiveness and cost-effectiveness of current and alternative follow-up strategies for stage IA and IB melanoma. Three systematic reviews were conducted. (1) The effectiveness of surveillance strategies. Outcomes were detection of new primaries, recurrences, metastases and survival. Risk of bias was assessed using the Cochrane Collaboration's Risk-of-Bias 2.0 tool. (2) Prediction models to stratify by risk of recurrence, metastases and survival. Model performance was assessed by study-reported measures of discrimination (e.g. D-statistic, Harrel's c-statistic), calibration (e.g. the Hosmer-Lemeshow 'goodness-of-fit' test) or overall performance (e.g. Brier score, R2). Risk of bias was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). (3) Diagnostic test accuracy of fine-needle biopsy and ultrasonography. Outcomes were detection of new primaries, recurrences, metastases and overall survival. Risk of bias was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool. Review data and data from elsewhere were used to model the cost-effectiveness of alternative surveillance strategies and the value of further research. (1) The surveillance review included one randomised controlled trial. There was no evidence of a difference in new primary or recurrence detected (risk ratio 0.75, 95% confidence interval 0.43 to 1.31). Risk of bias was considered to be of some concern. Certainty of the evidence was low. (2) Eleven risk prediction models were identified. Discrimination measures were reported for six models, with the area under the operating curve ranging from 0.59 to 0.88. Three models reported calibration measures, with coefficients of ≥ 0.88. Overall performance was reported by two models. In one, the Brier score was slightly better than the American Joint Committee on Cancer scheme score. The other reported an R2 of 0.47 (95% confidence interval 0.45 to 0.49). All studies were judged to have a high risk of bias. (3) The diagnostic test accuracy review identified two studies. One study considered fine-needle biopsy and the other considered ultrasonography. The sensitivity and specificity for fine-needle biopsy were 0.94 (95% confidence interval 0.90 to 0.97) and 0.95 (95% confidence interval 0.90 to 0.97), respectively. For ultrasonography, sensitivity and specificity were 1.00 (95% confidence interval 0.03 to 1.00) and 0.99 (95% confidence interval 0.96 to 0.99), respectively. For the reference standards and flow and timing domains, the risk of bias was rated as being high for both studies. The cost-effectiveness results suggest that, over a lifetime, less intensive surveillance than recommended by the National Institute for Health and Care Excellence might be worthwhile. There was considerable uncertainty. Improving the diagnostic performance of cancer nurse specialists and introducing a risk prediction tool could be promising. Further research on transition probabilities between different stages of melanoma and on improving diagnostic accuracy would be of most value. Overall, few data of limited quality were available, and these related to earlier versions of the American Joint Committee on Cancer staging. Consequently, there was considerable uncertainty in the economic evaluation. Despite adoption of rigorous methods, too few data are available to justify changes to the National Institute for Health and Care Excellence recommendations on surveillance. However, alternative strategies warrant further research, specifically on improving estimates of incidence, progression of recurrent disease; diagnostic accuracy and health-related quality of life; developing and evaluating risk stratification tools; and understanding patient preferences. This study is registered as PROSPERO CRD42018086784. This project was funded by the National Institute for Health Research Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol 25, No. 64. See the NIHR Journals Library website for further project information.

BackgroundThere is an increasing prevalence of cardiovascular disease (CVD) in China, which represents the leading cause of mortality. Precise CVD risk identification is the fundamental prevention component. This study sought to systematically review the CVD risk prediction models derived and/or validated in the Chinese population to promote primary CVD prevention.MethodsReports were included if they derived or validated one or more CVD risk prediction models in the Chinese population. PubMed, Embase, CINAHL, Web of Science, Scopus, China National Knowledge Infrastructure (CNKI), VIP database, etc., were searched. The risk of bias was assessed with the Prediction Model Risk of Bias Assessment Tool (PROBAST). Meta-analysis was performed in R using the package metamisc.ResultsFrom 55,183 records, 22 studies were included. Twelve studies derived 18 CVD risk prediction models, of which seven models were derived based on a multicentre cohort including more than two provinces of mainland China, and one was a model developed based on a New Zealand cohort including Chinese individuals. The number of predictors ranged from 6 to 22. The definitions of predicted outcomes showed considerable heterogeneity. Fourteen articles described 29 validations of 8 models. The Framingham model and pooled cohort equations (PCEs) are the most frequently validated foreign tools. Discrimination was acceptable and similar for men and women among models (0.60–0.83). The calibration estimates changed substantially from one population to another. Prediction for atherosclerotic cardiovascular disease Risk in China (China-PAR) showed good calibration [observed/expected events ratio = 0.99, 95% PI (0.57,1.70)] and female sex [1.10, 95% PI (0.23,5.16)].ConclusionsSeveral models have been developed or validated in the Chinese population. The usefulness of most of the models remains unclear due to incomplete external validation and head-to-head comparison. Future research should focus on externally validating or tailoring these models to local settings.Trail registrationThis systematic review was registered at PROSPERO (International Prospective Register of Systematic Reviews, CRD42021277453).

BackgroundFailure to extubate successfully from mechanical ventilation is a critical event associated with poor prognosis in ICU patients, significantly prolonging hospital stays and increasing mortality rates. It is widely accepted in academic circles that developing prediction models for extubation failure can facilitate precise extubation decisions. Despite the rapid proliferation of relevant prediction models, their methodological quality and bedside applicability remain ambiguous.ObjectiveThis study aims to outline the predictive factors associated with the risk of extubation failure in patients undergoing mechanical ventilation in the Intensive Care Unit (ICU) and to summarize the existing predictive models.MethodsWe searched the China National Knowledge Infrastructure (CNKI), Wanfang Database, VIP Database, China Biomedical Database, PubMed, Embase, Web of Science, and Cochrane Library. We included both prospective and retrospective studies that developed or validated risk prediction models for extubation failure in patients undergoing mechanical ventilation in the ICU. The Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the bias and applicability of the models.ResultsThis analysis includes 14 studies. Frequency analysis of the predictors revealed that there are 15 predictors that appeared at least twice, among which mechanical ventilation duration, GCS score, APACHE II score, age, and hemoglobin were the most common predictors. From the perspective of the models, only 2 studies conducted both internal and external validation, 3 studies ultimately employed machine learning, while 11 studies utilized traditional modeling methods. However, we found that many studies faced issues such as insufficient sample sizes, missing crucial methodological information, and all models being rated as having a high risk of bias.ConclusionMost published predictive models lack methodological rigor, leading to a heightened risk of bias. Future research should prioritize the enhancement of methodological rigor and the external validation of risk prediction models for extubation failure in ICU patients receiving mechanical ventilation. Additionally, it is essential to emphasize adherence to scientific methods and transparent reporting to improve the accuracy and generalizability of research findings.Systematic review registrationhttps://www.crd.york.ac.uk/PROSPERO/recorddashboard, Registration number:CRD420251124371.

The number of risk prediction models for sarcopenia in patients undergoing maintenance haemodialysis (MHD) is increasing. However, the quality, applicability, and reporting adherence of these models in clinical practice and future research remain unknown. To systematically review published studies on risk prediction models for sarcopenia in patients undergoing MHD. Systematic review and meta-analysis of observational studies. This systematic review adhered to the PRISMA guidelines. Search relevant domestic and international databases, which were searched from the inception of the databases until November 2023. The Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist was used to extract data. The Prediction Model Risk of Bias Assessment Tool (PROBAST) checklist was used to assess the risk of bias and applicability. The Transparent Reporting of a Multivariate Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) was used to assess the reporting adherence. A total of 478 articles were retrieved, and 12 prediction models from 11 articles were included after the screening process. The incidence of sarcopenia in patients undergoing MHD was 16.38%-37.29%. The reported area under the curve (AUC) ranged from 0.73 to 0.955. All studies had a high risk of bias, mainly because of inappropriate data sources and poor reporting in the field of analysis. The combined AUC value of the six validation models was 0.91 (95% confidence interval: 0.87-0.94), indicating that the model had a high discrimination. Although the included studies reported to some extent the discrimination of predictive models for sarcopenia in patients undergoing MHD, all studies were assessed to have a high risk of bias according to the PROBAST checklist, following the reporting guidelines outlined in the TRIPOD statement, and adherence was incomplete in all studies. CRD42023476067.