Receiver Operating Characteristic Curve and Equal Error Rate

295: Accuracy of Various Methods of B-Line Assessment In the Diagnosis of Congestive Heart Failure Compared by Receiver Operating Characteristic Curves

Receiver Operating Characteristic (ROC) curves can be used to describe detectability of noise-limited signals. The basic concepts underlying the theory of ROC curves and the experimental determination of ROC curves are discussed. ROC curves are used to describe the detectability of the image of 2 mm Lucite beads (similar to certain kinds of small gall-stones) in a noisy background of radiographic mottle. Results are shown for four typical radiographic screen-film combinations of differing optical and noise properties. The qualitative relationship between the ROC curves and the mathematical model describing radiographic mottle is examined. The ROC curves describing the detectability of the bad images when radiographed with two screen-film combinations of equal system speed but different optical characteristics are compared to the quality of blood vessel images obtained with the same screen-film combinations. It is concluded that it is important to distinguish between the assessment of image quality based on the relative sharpness of high contrast images, versus the assessment of image quality based on the relative detectability of low contrast noise-limited images.

Objective To investigate features of fasting plasma glucose (FPG) during different trimesters and hemoglobin A1c (HbA1c) in the third trimester of pregnant women in Zhujing Region of Shanghai. Methods A total of 606 cases of pregnant women who received prenatal examination and 75 g oral glucose tolerance test (OGTT) in Jinshan Branch of Shanghai Sixth People′s Hospital from January 1 to December 31, 2014 were collected as research subjects. According to the results of 75 g OGTT, they were divided into gestational diabetes mellitus (GDM) group (n=136) and non-GDM group (n=470). The clinical data, such as age, body mass index (BMI), as well as the concentration of FPG during different pregnancy trimesters and level of HbA1c in the third trimester were collected retrospectively in both two groups by retrospective analysis method. According to the quartile method, level of HbA1c in the third trimester of all subjects were divided into four ranges: Q1 (HbA1c level<5.1%), Q2 (5.1%≤HbA1c level<5.2%), Q3 (5.2%≤HbA1c level<5.5%), and Q4 (HbA1c level≥5.5%), respectively. The age, BMI, concentrations of FPG and the level of HbA1c between two groups were compared by independent-samples t test. Chi-square test was used to compare the GDM incidence of all subjects with different ranges of HbA1c level in the third trimester, and further comparison was conducted by adjusting inspection level. Then receiver operator characteristic (ROC) curve of HbA1c level in the third trimester for predicting the incidence of GDM was drawn, and the area under ROC curve (ROC-AUC) was calculated. The optimal critical value of HbA1c level in the third trimester for predicting the incidence of GDM was obtained when the Youden index reaching the maximum value. And its sensitivity and specificity were calculated. This study met the requirements of the World Medical Association Declaration of Helsinki revised in 2013. Results ①The age and BMI of GDM group were higher than those in non-GDM group, and there were statistically significant differences (t=5.306, 5.250; P<0.001). Among the subjects in GDM group, the concentration of FPG in second trimester was the highest among 3 trimesters, which was (4.9±0.6) mmol/L, and the concentration of FPG in second trimester was higher than that in first and third trimester, respectively, and both the differences were statistically significant (t=2.087, 1.960; P=0.039, 0.041). Among the subjects in non-GDM group, the concentration of FPG in first trimester was the highest among 3 trimesters, which was (4.6±0.3) mmol/L, and the concentration of FPG in first trimester was higher than that in second and third trimester, respectively, and both the differences were statistically significant (t=15.230, 5.613; P<0.001). The concentration of FPG in first, second and third trimesters of GDM group were higher than those of non-GDM group, respectively, and there were statistically significant differences (t=5.416, 15.526, 4.471; P<0.001). Besides, HbA1c level in third trimester of GDM group was (5.6±0.4)%, which was higher than that of non-GDM group (5.4±0.4)%, and there was significant difference (t=5.845, P<0.01). ②There was statistical difference in GDM incidences in all subjects with different ranges of HbA1c levels in third trimester (χ2=22.707, P<0.001). Multiple comparison results showed that the incidences of GDM in all subjects with Q1 and Q2 ranges of HbA1c levels in third trimester were higher than that in subjects with Q4 range of HbA1c levels in third trimester, respectively, and both the differences were statistically significant (χ2=15.071, 16.785; P<0.001). ③The results of ROC curve analysis of HbA1c level in third trimester in predicting the incidence of GDM showed that the ROC-AUC was 0.647 (95%CI: 0.625-0.672, P<0.001), and the optimal cut-off value of HbA1c level in third trimester to predict GDM incidence was 5.5%, and the sensitivity of HbA1c level in predicting the incidence of GDM was 63.4%, and the sensitivity was 69.7%. Conclusions The concentrations of FPG in GDM pregnant women in Zhujing Region of Shanghai begin to increase from the first trimester and peak during the second trimester, and decrease during the third trimester after lifestyle intervention. It is advisable for GDM pregnant women to control HbA1c levels below 5.5% during the third trimester. Key words: Diabetes, gestational; Fasting plasma glucose; Hemoglobin A, glycosylated; Glucose tolerance test; ROC curve; Forecasting; Pregnant women

A receiver operating characteristic (ROC) curve visually demonstrates the tradeoff between sensitivity and specificity as a function of varying a classification threshold. It is a common practice to use ROC curves to measure the accuracy of predictions by different methods. Although this method has been used primarily in medical and engineering fields, it could be used effectively in sports as well. More precisely, an ROC plots the sensitivity versus (1 - specificity), and the area under the curve gives a measure of the prediction. So, the ideal best prediction should have one square unit of area under the ROC curve, where it achieves both 100% sensitivity and 100% specificity (which, in reality, rarely happens). Consequently, when we compare two methods, the one with the greater area under its ROC is judged best. This paper shows the effectiveness of using the ROC curves in analyzing cricket data. In particular, the quality of the decisions made by umpires is investigated. Also, the comparison of the accuracy of the methods for revising the target for matches shortening due to weather interruptions is the key interest in our investigation.

We consider the distributed detection problem, in which a set of decision makers (DMs) receive observations of the environment and transmit finite-valued messages to other DMs according to prespecified communication protocols. A designated primary DM makes the final decision on one out of two alternative hypotheses. All DMs make decisions, in order to maximize a measure of organizational performance. We discuss three different types of decision rules (deterministic, independent randomization, and dependent randomization), and their implications on the organizational performance. Each DM is described by its individual Receiver Operating Characteristic (ROC) curve, which is concave. We determine that concavity is not guaranteed for the case of the ROC curve of a team of DMs, even if the decision rules are perfectly continuous and if the individual ROC curves are strictly concave and smooth, unless dependent randomization is allowed.

In power distribution fault data, the percentage of faults with different causes could be very different and varies from region to region. This data imbalance issue seriously affects the performance evaluation of fault diagnosis algorithms. Due to the limitations of conventional accuracy (ACC) and geometric mean (G-mean) measures, this paper discusses the application of Receiver Operating Characteristic (ROC) curves in evaluating distribution fault diagnosis performance. After introducing how to obtain ROC curves, Artificial Neural Networks (ANN), Logistic Regression (LR), Support Vector Machines (SVM), Artificial Immune Recognition Systems (AIRS), and K-Nearest Neighbor (KNN) algorithm are compared using ROC curves and Area Under the Curve (AUC) on real-world fault datasets from Progress Energy Carolinas. Experimental results show that AIRS performs best most of the time and ANN is potentially a good algorithm with a proper decision threshold.

Receiver operating characteristic (ROC) analysis has been used in a variety of settings since it was first declassified by the United States government over 60 years ago. One venue in which it has received particular attention is in the field of radiology. In radiology, as in other areas of application, ROC analysis is used to assess the ability of a diagnostic test to distinguish between two opposing states. One useful descriptor in ROC analysis is the area under the ROC curve. At times, it is useful and insightful to average ROC curves in order to create a single curve that summarizes all of the data from multiple readers. In this paper, we investigate the impact of correlated readings on the average area under two readers' ROC curves using several common averaging strategies, and then apply the results to a radiologic study.

Receiver Operating Characteristic (ROC) curves are used to describe and compare the accuracy of diagnostic test or the ability of a continuous biomarker in discriminating between the subjects into healthy or diseased cases in medical field. The most familiar form of ROC curve is Bi-normal (Gaussian) ROC curve model, which assumes that the test scores or a monotone transformation of the test scores are from two normal populations (i.e. healthy and diseased). It may not be true all the time, it may violate the assumptions of normal distribution in some situations and also we cannot adopt the model as it is when the sample size is small. In this paper, we have proposed ROC curve model for Rayleigh distribution which can be used even when sample size is small. The properties of Bi-Rayleigh ROC model are studied and Area Under the ROC Curve (AUC) are derived. The proposed model is supported by real life example as well as simulation studies. The confidence interval for the population parameter is studied with simulation studies of varying sample sizes. It is found that Bi-Rayleigh ROC model provides better accuracy of classification than the conventional bi-normal ROC model. In the ROC model section, Bi-Rayleigh ROC model and its AUC are derived and its properties are discussed. In application section, two real life example viz. Head trauma data and Multiple Sclerosis data were used for the proposed model and then these are compared with the conventional bi- normal model. In the simulation section, simulation studies have been carried out for different values of parameters to support the Bi-Rayleigh ROC model. In the last section, the confidence intervals for the parameter σi, i=0,1 are discussed for real life example and simulated data sets. II. ROC MODEL AND AREA UNDER THE CURVE Let the scores (S) be a random variable in healthy (H) and diseased (D) populations respectively. The ROC curve is plotted over values of threshold value (t) and is given by y(t) = h (x(t)) where x(t) is the False positive rate which is given by x(t) = P(S>t|H) and y(t) is True positive rate which is given by y(t) = P (S>t|D) The general functional form of ROC can be represented as

To assess examiner reliability when scoring dental fluorosis in Malaysian children using clinical (Dean's Index) and photographic methods. The upper central incisors of 111 children were examined both clinically and photographically for fluorosis status using Dean's index. Twenty children were re-examined after a two-week interval for intra-examiner reliability by a single examiner. In addition, two independent examiners and the clinical examiner scored 111 photographic images of the same children in a standardized manner. Fluorosis scores were compared individually between examiners for both clinical and photographic scoring. Examiner reliability was assessed using both simple and weighted kappa statistics at tooth level. Sensitivity, specificity, positive-negative predictive values and a Receiver Operating Characteristic (ROC) curve were also calculated to determine the accuracy of the test. Across the three examiners, the prevalence of fluorosis (Dean's score ≥ 2) using photographs was lower (ranged from 23% to 26%) than the prevalence recorded by clinical examination (30%). The kappa score for intra-examiner reliability for the duplicate clinical examination was excellent (0.89). Inter-examiner reliability between the photographic method and the clinical examination (gold standard) for each examiner was substantial with weighted kappa values ranging from 0.74 to 0.77. The photographic method indicated higher specificity (99%) than sensitivity (79%) and the area under the ROC curve was also high (0.89) which suggests good accuracy of the diagnostic test. These results suggest that photographic examination of fluorosis on central incisors can be recorded with good examiner reliability. The recorded fluorosis prevalence was lower using the photographic scores.

Objective To investigate the correlation between serum endocan level and risk of diabetic peripheral neuropathy(DPN). Methods A total of 67 patients with type 2 diabetes mellitus (T2DM) were recruited from the Department of Endocrinology in Gansu Provincial Hospital From April 2015 to November 2015. All patients were divided into DPN group (n=25) and simple T2DM group (n=42). Meanwhile, gender and age matched healthy subjects were included as the normal control group(NC group, n=28). Serum endocan was measured by ELISA, and other metabolic parameters were also tested. Pearson correlation analysis, logistic regression analysis and area of receiver operator characteristic (ROC) curve analysis were used to analyze the correlation between endocan level and DPN risk. Results Compared with NC group, the serum level of endocan was significantly increased in simple T2DM group and DPN group(F=24.26, P<0.05). In addition, compared with simple T2DM group, the serum level of endocan in DPN group was markedly higher (t=5.668, P<0.05). Endocan was positively correlated with the course of disease and age(r=0.438, 0.394, all P<0.05). Endocan level, smoking and age were the independent risk factors for DPN(OR=1.692, 9.880, 1.093, 95% CI: 1.279-2.237, 2.039-47.878, 1.024-1.168, all P<0.05). The results of the ROC analysis revealed that the area under ROC curve was 0.824(95% CI: 0.727-0.921, P=0.000). Conclusion Serum level of endocan is related to DPN, and it may be used as a novel biomarker for DPN. Key words: Endocan; Diabetic peripheral neuropathy; Type 2 diabetes mellitus

Maximizing area under ROC curve for biometric scores fusion

Receiver operating characteristics (ROC) has been widely used as a performance evaluation tool to measure effectiveness of medical modalities. It is derived from a standard detection theory with false alarm and detection power interpreted as false positive (FP) and true positive (TP) respectively in terms of medical diagnosis. The ROC curve is plotted based on TP versus FP via hard decisions. This paper presents a three dimensional (3D) ROC analysis which extends the traditional two-dimensional (2D) ROC analysis by including a threshold parameter in a third dimension resulting from soft decisions, (SD). As a result, a 3D ROC curve can be plotted based on three parameters, TP, FP and SD. By virtue of such a 3D ROC curve three two-dimensional (2D) ROC curves can be derived, one of which is the traditional 2D ROC curve of TP versus FP with SD reduced to hard decision. In order to illustrate its utility in medical diagnosis, its application to magnetic resonance (MR) image classification is demonstrated.

In this paper, we address the problem of score fusion in biometric authentication. Single valued metrics related to the receiver operating characteristics (ROC) curve, such as Equal Error Rate (EER) and False Rejection Rate (FRR) when False Acceptance Rate equals zero, are extensively used for evaluating biometric authentication performances. Various requirements and preferences, for example, lower EER, or smaller FRR, may be imposed on biometric authentication systems in different application scenarios. We propose a novel method of score fusion based on quasi-convex optimization to directly improve biometric authentication metrics. Experiments based on a face recognition system demonstrate the effectiveness of the proposed method.

An Empirical Comparison of Discrete Ratings and Subjective Probability Ratings

Background: HER2+ breast cancer (BC) is a heterogeneous disease. Despite the novel targeted therapies against HER2, 20% of patients relapse. In this context, the identification of prognostic and predictor biomarkers is needed.MicroRNAs are involved in BC development and prognosis. We aim at assessing the prognostic significance of two micro-RNAs (named as A and B) in HER2+ BC tissue, to propose signature to predict relapse in this setting of patients.Methods: This study was conducted in a cohort of 46 non-consecutive HER2+ BC patients diagnosed at Hospital Clínico València-INCLIVA. All BC patients received standard treatment for localized disease. RNA was isolated from formalin-fixed paraffin-embedded primary tumor and retro-transcribed to cDNA. MicroRNA expression levels were measured by real-time qPCR and results were normalized according to the expression of housekeeping mir-16 miRNA.Non-parametric Mann-Whitney U test was used to ascertain the statistical significance of differences in miRNA expression levels between disease-free and relapse. Receiver operator characteristics (ROC) curves were constructed and AUC, accuracy, specificity and sensitivity, were calculated as a biomarker performance parameter. The best cutoff value was established based on the highest value obtained in ROC curve analysis according to the maximization of the sum of sensitivity and specificity. Kaplan-Meier curves were plotted for disease-free survival (DFS) based on the best cutoff value. As both microRNA showed high correlation (r=0.85), a genetic signature was developed using Principal Component Analysis (PCA) and extracting the first component. MicroRNA targeted genes were downloaded from miRTarBase (release 7.0). Cytoscape (v3.8.0) and ClueGo (v2.5.6) and were used to perform functional enrichment analysis with the Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway database. To select significant terms/pathways, p-values were adjusted by the Benjamni-Hochberg method (p&amp;lt;0.05). Results: The expression of both microRNA-A and B was higher in the primary tumor of those patients who relapsed after treatment compared to those free of disease (p= 0.003 and 0.0002, respectively). ROC curves demonstrated that microRNA-A and B might discriminate those HER2+ BC patients who relapse. MicroRNA-A predicts relapse with an AUC 0.740, sensitivity of 0.810, specificity of 0.583 and accuracy of 0.689, and microRNA-B presented an AUC 0.821, sensitivity of 0.810, specificity of 0.750, and accuracy of 0.778. The survival analysis showed that microRNA-A and B strongly predicted shorter DFS (p= 0.014 and 0.0012, respectively).Two-microRNA signature combining (PCA1= 99.4% absorbed variance) microRNA-A and microRNA-B expression levels had high performance in discriminating relapsed from non-relapsed HER2+ BC patients (AUC 0.748, accuracy 0.689, sensitivity 1, specificity 0.417). Signature predicted DFS (p= 0.009). Functional enrichment analysis returned 55 significant pathways. Of these, important pathways related to tumorigenesis and response to treatment were enriched. Interestingly, P53 pathway, apoptosis, and cell cycle were in the top 5 enriched pathways. Conclusions: We demonstrated the value of microRNA A and B as potential prognostic biomarkers. High expression was related to poor prognosis in over cohort of HER2+ BC patients. Both microRNAs were able to strongly discriminate patients who will relapse and predict shorter DFS. Moreover, our signature was able to predict the outcome, suggesting a potential prognostic role at diagnosis. Both microRNAs are related to important biological pathways associated to BC progression. Further investigations to optimize the micro-RNA signature are on-going. Citation Format: Anna Adam-Artigues, Juan Antonio Carbonell-Asíns, Sheila Zúñiga-Trejos, Santiago Moragón, Maria Teresa Martínez, Cristina Hernando, Eduardo Tormo, Octavio Burgués, Ana Lluch, Begoña Bermejo, Pilar Eroles, Juan Miguel Cejalvo. Identification of a novel two-microRNA signature for recurrence prediction in HER2 positive breast cancer [abstract]. In: Proceedings of the 2020 San Antonio Breast Cancer Virtual Symposium; 2020 Dec 8-11; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2021;81(4 Suppl):Abstract nr PS19-20.