Diagnostic accuracy of a novel endometriosis staging system: an external validation study

Comparison of the Recently Published “AAGL Endometriosis Classification” with Other Staging Systems to Predict Surgical Complexity

ObjectivesTo systematically review AKI outcome prediction models and their external validation studies, to describe the discrepancy of reported accuracy between the results of internal and external validations, and to identify variables frequently included in the prediction models.MethodsWe searched the MEDLINE and Web of Science electronic databases (until January 2016). Studies were eligible if they derived a model to predict mortality of AKI patients or externally validated at least one of the prediction models, and presented area under the receiver-operator characteristic curves (AUROC) to assess model discrimination. Studies were excluded if they described only results of logistic regression without reporting a scoring system, or if a prediction model was generated from a specific cohort.ResultsA total of 2204 potentially relevant articles were found and screened, of which 12 articles reporting original prediction models for hospital mortality in AKI patients and nine articles assessing external validation were selected. Among the 21 studies for AKI prediction models and their external validation, 12 were single-center (57%), and only three included more than 1,000 patients (14%). The definition of AKI was not uniform and none used recently published consensus criteria for AKI. Although good performance was reported in their internal validation, most of the prediction models had poor discrimination with an AUROC below 0.7 in the external validation studies. There were 10 common non-renal variables that were reported in more than three prediction models: mechanical ventilation, age, gender, hypotension, liver failure, oliguria, sepsis/septic shock, low albumin, consciousness and low platelet count.ConclusionsInformation in this systematic review should be useful for future prediction model derivation by providing potential candidate predictors, and for future external validation by listing up the published prediction models.

Bedside airway examination tests, for assessing the physical status of the airway in adults with no apparent anatomical airway abnormalities, are designed as screening tests. Screening tests are expected to have high sensitivities. We found that all investigated index tests had relatively low sensitivities with high variability. In contrast, specificities were consistently and markedly higher than sensitivities across all tests. The standard bedside airway examination tests should be interpreted with caution, as they do not appear to be good screening tests. Among the tests we examined, the upper lip bite test showed the most favourable diagnostic test accuracy properties. Given the paucity of available data, future research is needed to develop tests with high sensitivities to make them useful, and to consider their use for screening difficult face mask ventilation and failed intubation. The 27 studies in 'Studies awaiting classification' may alter the conclusions of the review, once we have assessed them.

In reply

Clinically relevant deep learning for detection and quantification of geographic atrophy from optical coherence tomography: a model development and external validation study

Primary ciliary dyskinesia (PCD) is a rare disorder causing chronic otosinopulmonary disease, generally diagnosed through evaluation of respiratory ciliary ultrastructure and/or genetic testing. Nasal nitric oxide (nNO) measurement is used as a PCD screening test because patients with PCD have low nNO levels, but its value as a diagnostic test remains unknown. To perform a systematic review to assess the utility of nNO measurement (index test) as a diagnostic tool compared with the reference standard of electron microscopy (EM) evaluation of ciliary defects and/or detection of biallelic mutations in PCD genes. Ten databases were searched for reference sources from database inception through July 29, 2016. Study inclusion was limited to publications with rigorous nNO index testing, reference standard diagnostic testing with EM and/or genetics, and calculable diagnostic accuracy information for cooperative patients (generally >5 yr old) with high suspicion of PCD. Meta-analysis provided a summary estimate for sensitivity and specificity and a hierarchical summary receiver operating characteristic curve. The Quality Assessment of Diagnostic Accuracy Studies-2 tool was used to assess study quality, and Grading of Recommendations Assessment, Development, and Evaluation was used to assess the certainty of evidence. In 12 study populations (1,344 patients comprising 514 with PCD and 830 without PCD), using a reference standard of EM alone or EM and/or genetic testing, summary sensitivity was 97.6% (92.7-99.2) and specificity was 96.0% (87.9-98.7), with a positive likelihood ratio of 24.3 (7.6-76.9), a negative likelihood ratio of 0.03 (0.01-0.08), and a diagnostic odds ratio of 956.8 (141.2-6481.5) for nNO measurements. After studies using EM alone as the reference standard were excluded, the seven studies using an extended reference standard of EM and/or genetic testing showed a summary sensitivity of nNO measurements of 96.3% (88.7-98.9) and specificity of 96.4% (85.1-99.2), with a positive likelihood ratio of 26.5 (5.9-119.1), a negative likelihood ratio of 0.04 (0.01-0.12), and a diagnostic odds ratio of 699.3 (67.4-7256.0). Certainty of the evidence was graded as moderate. nNO is a sensitive and specific test for PCD in cooperative patients (generally >5 yr old) with high clinical suspicion for this disease. With a moderate level of evidence, this meta-analysis confirms that nNO testing using velum closure maneuvers has diagnostic accuracy similar to EM and/or genetic testing for PCD when cystic fibrosis is ruled out. Thus, low nNO values accompanied by an appropriate clinical phenotype could be used as a diagnostic PCD test, though EM and/or genetics will continue to provide confirmatory information.

IntroductionPostoperative pulmonary complications (PPCs) following major abdominal surgery result in substantial morbidity and mortality, yet stratifying patients for risk-modifying interventions remains challenging. This study aimed to identify and externally validate PPC risk prediction models in an international, prospective cohort.MethodA systematic review was conducted to identify risk prediction models for PPC following abdominal surgery. External validation was performed using data from a prospective dataset of adult patients undergoing major abdominal surgery from January to April 2019 in the UK, Ireland, and Australia. The primary outcome was identification of PPC within 30-days (StEP-COMPAC criteria definition). Model discrimination and diagnostic accuracy were compared.ResultsSix unique risk prediction models were eligible from 2819 records (112 full texts). These were validated across 11,591 patients, with an overall PPC rate of 7.8% (n = 903). The Assess Respiratory Risk in Surgical Patients in Catalonia (ARISCAT) score provided the best discrimination (AUROC: 0.709 (95% CI: 0.692-0.727), yet no risk prediction model demonstrated good discrimination (AUROC >0.7).ConclusionsThe risk of PPC for patients following major abdominal surgery in the pre-covid era is not well described by existing prediction tools. New prediction tools are required to account for additional variation introduced for patients affected by SARS-CoV-2 infection.

See https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD014461.pub2/full for a more recent review that covers this topic and has superseded this review. Low-back pain (LBP) is a common condition seen in primary care. A principal aim during a clinical examination is to identify patients with a higher likelihood of underlying serious pathology, such as vertebral fracture, who may require additional investigation and specific treatment. All 'evidence-based' clinical practice guidelines recommend the use of red flags to screen for serious causes of back pain. However, it remains unclear if the diagnostic accuracy of red flags is sufficient to support this recommendation. To assess the diagnostic accuracy of red flags obtained in a clinical history or physical examination to screen for vertebral fracture in patients presenting with LBP. Electronic databases were searched for primary studies between the earliest date and 7 March 2012. Forward and backward citation searching of eligible studies was also conducted. Studies were considered if they compared the results of any aspect of the history or test conducted in the physical examination of patients presenting for LBP or examination of the lumbar spine, with a reference standard (diagnostic imaging). The selection criteria were independently applied by two review authors. Three review authors independently conducted 'Risk of bias' assessment and data extraction. Risk of bias was assessed using the 11-item QUADAS tool. Characteristics of studies, patients, index tests and reference standards were extracted. Where available, raw data were used to calculate sensitivity and specificity with 95% confidence intervals (CI). Due to the heterogeneity of studies and tests, statistical pooling was not appropriate and the analysis for the review was descriptive only. Likelihood ratios for each test were calculated and used as an indication of clinical usefulness. Eight studies set in primary (four), secondary (one) and tertiary care (accident and emergency = three) were included in the review. Overall, the risk of bias of studies was moderate with high risk of selection and verification bias the predominant flaws. Reporting of index and reference tests was poor. The prevalence of vertebral fracture in accident and emergency settings ranged from 6.5% to 11% and in primary care from 0.7% to 4.5%. There were 29 groups of index tests investigated however, only two featured in more than two studies. Descriptive analyses revealed that three red flags in primary care were potentially useful with meaningful positive likelihood ratios (LR+) but mostly imprecise estimates (significant trauma, older age, corticosteroid use; LR+ point estimate ranging 3.42 to 12.85, 3.69 to 9.39, 3.97 to 48.50 respectively). One red flag in tertiary care appeared informative (contusion/abrasion; LR+ 31.09, 95% CI 18.25 to 52.96). The results of combined tests appeared more informative than individual red flags with LR+ estimates generally greater in magnitude and precision. The available evidence does not support the use of many red flags to specifically screen for vertebral fracture in patients presenting for LBP. Based on evidence from single studies, few individual red flags appear informative as most have poor diagnostic accuracy as indicated by imprecise estimates of likelihood ratios. When combinations of red flags were used the performance appeared to improve. From the limited evidence, the findings give rise to a weak recommendation that a combination of a small subset of red flags may be useful to screen for vertebral fracture. It should also be noted that many red flags have high false positive rates; and if acted upon uncritically there would be consequences for the cost of management and outcomes of patients with LBP. Further research should focus on appropriate sets of red flags and adequate reporting of both index and reference tests.

Peripheral arterial disease is a major cause of the development of diabetic foot ulcers, lower limb amputation and mortality in patients with diabetes. Diagnosing peripheral arterial disease is of clinical importance but is difficult in this cohort of patients. Several diagnostic bedside tests exist, but there is uncertainty as to which is the most accurate. To determine the diagnostic accuracy of five index tests (audible waveform assessment, visual waveform assessment, toe-brachial pressure index, ankle-brachial pressure index and exercise ankle-brachial pressure index) for the diagnosis of peripheral arterial disease in patients with diabetes as determined by a reference test (computed tomography angiography or magnetic resonance angiography). In selected sites, to evaluate the performance of a sixth test, the Podiatry Ankle Duplex scan, a new point-of-care duplex ultrasound scan. Prospective multicentre diagnostic accuracy study. Primary (general practice and community clinics) and secondary care National Health Service hospitals (inpatient and outpatient) in the United Kingdom. Patients were eligible for inclusion if they were aged ≥ 18 years and had a known history of diabetes. Exclusion criteria included contraindications to computed tomography angiography or magnetic resonance angiography, if their peripheral arterial disease status was known on imaging, or they had a known history of peripheral arterial disease intervention. Participants underwent all bedside tests (Podiatry Ankle Duplex scan performed in three centres only), which were performed in a logical sequence to reduce the risk of bias. The reference scan was to be performed within 6 weeks of the index tests. The primary outcome measure of diagnostic performance is test sensitivity. Secondary outcomes included specificity, likelihood ratios, predictive values and diagnostic odds ratio, as well as patient acceptability of tests, technical success and health economic outcomes. Based on the 573 reference tests performed, 222 (39%) participants had evidence of peripheral arterial disease. All routinely used index tests showed relatively low sensitivities: audible waveform 36%, 99% confidence interval 27% to 45%; visual waveform 42%, confidence interval 33% to 51%; toe-brachial pressure index 55%, confidence interval 46% to 64%; ankle-brachial pressure index 41%, confidence interval 32% to 50%; and exercise ankle-brachial pressure index 41%, confidence interval 31% to 51%. The Podiatry Ankle Duplex scan had a higher sensitivity 89%, confidence interval 74% to 100%, as compared to all other index tests. A large proportion of reference scans were performed outside the 6-week time window (45%), although peripheral arterial disease is a chronic condition and is unlikely to have developed/progressed in the time frame. Additionally, the peripheral arterial disease prevalence was lower than anticipated during study design (39% vs. 50%). Lastly, despite efforts to standardise assessment and minimise bias, there is the possibility that one test may have influenced the interpretation of others. Index tests used routinely in clinical practice for diagnosing peripheral arterial disease in patients with diabetes showed poor diagnostic accuracy for peripheral arterial disease and should not be recommended for diagnosing peripheral arterial disease in this population. A review of peripheral arterial disease diagnostic testing in patients with diabetes should be undertaken. It may be necessary to perform more reliable imaging, such as full lower limb duplex ultrasound as a first-line diagnostic test. This synopsis presents independent research funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme as award number NIHR131855.

Rapid immunoassays are widely used for diagnosing heparin-induced thrombocytopenia (HIT). The STic Expert HIT kit is commonly employed, but its diagnostic accuracy in unselected, real-world populations requires robust evaluation against an appropriate reference standard. This study evaluates the diagnostic accuracy of the STic Expert HIT kit using the heparin-induced platelet aggregation test (PAT) as the comparator. This cross-sectional diagnostic accuracy study was conducted from January 2023 to April 2025. A consecutive series of 320 patients with suspected HIT underwent parallel testing with the STic Expert HIT kit (index test) and PAT (comparator). The 4T score was calculated prospectively for all patients. Borderline visual results on the STic kit were classified according to epidemiological recommendations: as false negative if PAT-positive and false positive if PAT-negative. Diagnostic accuracy measures-sensitivity, specificity, predictive values, and likelihood ratios-were calculated with 95% confidence intervals. The study is reported in accordance with the STARD 2015 checklist. Among 320 consecutive patients (median age 45.6 years; range 0.19-89), the prevalence of PAT-confirmed HIT was 16.3% (52/320). Following the STARD guidelines, the STic Expert HIT kit demonstrated a sensitivity of 57.5% (95% CI: 42.2-71.6) and specificity of 92.5% (95% CI: 88.8-95.1). The positive predictive value was 58.8% (95% CI: 46.2-70.4) and negative predictive value was 92.1% (95% CI: 89.6-94.1). The positive likelihood ratio was 7.67 (95% CI: 5.10-11.52) and negative likelihood ratio was 0.46 (95% CI: 0.32-0.66). The area under the ROC curve was 0.75 (95% CI: 0.67-0.83). Borderline results constituted 6.6% (21/320) of tests; of these, 33.3% (7/21) were true positives and 66.7% (14/21) were false positives. In this large, real-world diagnostic accuracy study, the STic Expert HIT kit demonstrated limited sensitivity for HIT diagnosis. The test is not suitable for ruling out HIT due to its suboptimal sensitivity. Borderline results are common and require integration with pre-test probability. These findings highlight the need for confirmatory functional testing in suspected HIT, particularly in high-probability settings.

Background and purpose — External validation of machine learning (ML) prediction models is an essential step before clinical application. We assessed the proportion, performance, and transparent reporting of externally validated ML prediction models in orthopedic surgery, using the Transparent Reporting for Individual Prognosis or Diagnosis (TRIPOD) guidelines. Material and methods — We performed a systematic search using synonyms for every orthopedic specialty, ML, and external validation. The proportion was determined by using 59 ML prediction models with only internal validation in orthopedic surgical outcome published up until June 18, 2020, previously identified by our group. Model performance was evaluated using discrimination, calibration, and decision-curve analysis. The TRIPOD guidelines assessed transparent reporting. Results — We included 18 studies externally validating 10 different ML prediction models of the 59 available ML models after screening 4,682 studies. All external validations identified in this review retained good discrimination. Other key performance measures were provided in only 3 studies, rendering overall performance evaluation difficult. The overall median TRIPOD completeness was 61% (IQR 43–89), with 6 items being reported in less than 4/18 of the studies. Interpretation — Most current predictive ML models are not externally validated. The 18 available external validation studies were characterized by incomplete reporting of performance measures, limiting a transparent examination of model performance. Further prospective studies are needed to validate or refute the myriad of predictive ML models in orthopedics while adhering to existing guidelines. This ensures clinicians can take full advantage of validated and clinically implementable ML decision tools.

Purpose: There have been many attempts to preoperatively evaluate the chemotherapy response of osteosarcoma patients using 99mTc-MIBI scintigraphy. However, the evaluations were lacking in consistency. We performed this systematic review and meta-analysis to systematically evaluate the ability of 99mTc-MIBI scintigraphy in preoperatively assessing the response of osteosarcoma patients to neoadjuvant chemotherapy.Methods: For this systematic review and meta-analysis, PubMed, Web of Science, OVID, the Cochrane Library, and CNKI were searched. Eligible studies were included based on the defined criteria. The index test was 99mTc-MIBI scintigraphy, the reference standard was tumor necrosis rate. Quality Assessment of Diagnostic Accuracy Studies-2 was adopted for quality assessment of included studies. The statistical pooling analysis, meta-regression analysis, subgroup analysis, sensitivity analysis, and publication bias of our research were performed using STATA 15.Results: Eight articles with 189 osteosarcoma patients were included in this systematic review and meta-analysis. Our results demonstrated that the threshold effect of our meta-analysis was significant. The uptake change ratio of 99mTc-MIBI scintigraphy had a pooled sensitivity, specificity, positive and negative likelihood ratio, diagnostic odds ratio, and the area under curve of 0.98 (0.58–1.00), 0.68 (0.47–0.84), 3.1 (1.7–5.5), 0.03 (0.00–0.90), 103 (4–3,003), and 0.91 (0.88–0.93) in preoperative assessment of response of osteosarcoma patients to neoadjuvant chemotherapy. Meta-regression analysis and subgroup analysis indicated the factors of method and cut off value may introduce the heterogeneity. The pooled sensitivity, specificity, positive and negative likelihood ratio, diagnostic odds ratio, and the area under curve of washout rate of 99mTc-MIBI were 0.87 (0.69–0.95), 0.91 (0.75–0.97), 9.3 (3.2–27.0), 0.15 (0.06–0.37), 64 (14–301), and 0.89 (0.86–0.92), respectively. Sensitivity analysis and publication bias demonstrated our meta-analysis was reliable.Conclusion: Both the ΔUR and WR derived from 99mTc-MIBI scintigraphy were valuable in preoperatively assessing the response of osteosarcoma patients to neoadjuvant chemotherapy, and ΔUR may possess a more outstanding diagnostic accuracy than WR.

International external validation of the SORG machine learning algorithms for predicting 90-day and one-year survival of patients with spine metastases using a Taiwanese cohort

Objective:To conduct a systematic review, critical appraisal, and external validation of survival prediction tools for patients undergoing intrahepatic cholangiocarcinoma (iCCA) resection.Summary background data:Despite the development of several survival prediction tools in recent years for patients undergoing iCCA resections, there is a lack of critical appraisal and external validation of these models.Methods:We conducted a systematic review and critical appraisal of survival and recurrence prediction models for patients undergoing curative-intent iCCA resections. Studies were evaluated based on their model design, risk of bias, reporting, performance, and validation results. We identified the best model and externally validated it using our institution’s data.Results:This review included a total of 31 studies, consisting of 26 studies with original prediction tools and 5 studies that only conducted external validations. Among the 26, 54% of the studies conducted internal validations, 46% conducted external validations, and only 1 study scored a low risk of bias. Harrell’s C-statistics ranged from 0.67 to 0.76 for internal validation and from 0.64 to 0.75 for external validation. Only 81% of the studies reported model calibration. Our external validation of the best model (Intrahepatic Cholangiocarcinoma [ICC]-Metroticket) estimated Harrell’s and Uno’s C-statistics of 0.67 (95% CI: 0.56–0.77) and Uno’s time-dependent area under the receiver operating characteristic curve (AUC) of 0.71 (95% CI: 0.53–0.88), with a Brier score of 0.20 (95% CI: 0.15–0.26) and good calibration plots.Conclusions:Many prediction models have been published in recent years, but their quality remains poor, and minimal methodological quality improvement has been observed. The ICC-Metroticket was selected as the best model (Uno’s time-dependent AUC of 0.71) for 5-year overall survival prediction in patients undergoing curative-intent iCCA resection.

Polycystic ovary morphology (PCOM) on ultrasonography is considered as a cardinal feature of polycystic ovarian syndrome (PCOS). Its relevance as a diagnostic criterion for PCOS was reaffirmed in the most recent International Evidence-Based Guideline for the Assessment and Management of PCOS. However, there remains a lack of clarity regarding the best practices and specific ultrasonographic markers to define PCOM. The aim of this systematic review and diagnostic meta-analysis was to assess the diagnostic accuracy of various ultrasonographic features of ovarian morphology in the diagnosis of PCOS. Relevant studies published from 1 January 1990 to 12 June 2023 were identified by a systematic search in PubMed, Web of Science, Scopus, CINAHL, and CENTRAL. Studies that generated diagnostic accuracy measures (e.g. proposed thresholds, sensitivity, specificity) for PCOS using the following ultrasonographic markers met criteria for inclusion: follicle number per ovary (FNPO) or per single cross-section (FNPS), ovarian volume (OV), and stromal features. Studies on pregnant or post-menopausal women were excluded. Risk of bias and applicability assessment for diagnostic test accuracy studies were determined using the QUADAS-2 and QUADAS-C tool for a single index test or between multiple index tests, respectively. Diagnostic meta-analysis was conducted using a bivariate model of pooled sensitivity and specificity, and visualized using forest plots and summary receiver-operating characteristic (SROC) curves. From a total of 2197 records initially identified, 31 studies were included. Data from five and two studies were excluded from the meta-analysis due to duplicate study populations or limited data for the index test, leaving 24 studies. Pooled results of 20 adult studies consisted of 3883 control participants and 3859 individuals with PCOS. FNPO was the most accurate diagnostic marker (sensitivity: 84%, CI: 81-87%; specificity: 91%, CI: 86-94%; AUC: 0.905) in adult women. OV and FNPS had similar pooled sensitivities (OV: 81%, CI: 76-86%; FNPS: 81%, CI: 70-89%) but inferior pooled specificities (OV: 81%, CI: 75-86%; FNPS: 83%, CI: 75-88%) and AUCs (OV: 0.856; FNPS: 0.870) compared to FNPO. Pooled results from four adolescent studies consisting of 210 control participants and 268 girls with PCOS suggested that OV may be a robust ultrasonographic marker for PCOS diagnosis albeit the current evidence remains limited. The majority of the studies had high risk of bias for the patient selection (e.g. lack of randomized/consecutive patient selection) and index test (e.g. lack of pre-proposed thresholds for comparison) domains across all ultrasonographic markers. As such, diagnostic meta-analysis was unable to determine the most accurate cutoff for ultrasonographic markers to diagnose PCOS. Subgroup analysis suggested that stratification based on previously proposed diagnostic thresholds, age, BMI, or technology did not account for the heterogeneity in diagnostic accuracy observed across the studies. Studies that diagnosed PCOS using the Rotterdam criteria had improved sensitivity for FNPO. Studies from North America had lower diagnostic accuracy when compared to Asian studies (FNPO: sensitivity) and European studies (OV: specificity, diagnostic odds ratio and positive likelihood ratio). Geographic differences in diagnostic accuracy may potentially be due to differences in age, BMI, and diagnostic criteria of the PCOS group across regions. This diagnostic meta-analysis supports the use of FNPO as the gold standard in the ultrasonographic diagnosis of PCOS in adult women. OV and FNPS provide alternatives if total antral follicle counts cannot be accurately obtained. Our findings support the potential for ultrasonographic evidence of PCOM in adolescents as more data becomes available. Subgroup analysis suggests the need to investigate any relative contributions of geographical differences on PCOS phenotypes. These findings may provide the basis for the development of strategies and best practices toward a standardized definition of PCOM and a more accurate ultrasonographic evaluation of PCOS.