Class noise detection based on software metrics and ROC curves

Receiver operating characteristic (ROC) analysis has been used in a variety of settings since it was first declassified by the United States government over 60 years ago. One venue in which it has received particular attention is in the field of radiology. In radiology, as in other areas of application, ROC analysis is used to assess the ability of a diagnostic test to distinguish between two opposing states. One useful descriptor in ROC analysis is the area under the ROC curve. At times, it is useful and insightful to average ROC curves in order to create a single curve that summarizes all of the data from multiple readers. In this paper, we investigate the impact of correlated readings on the average area under two readers' ROC curves using several common averaging strategies, and then apply the results to a radiologic study.

Receiver operating characteristics (ROC) has been widely used as a performance evaluation tool to measure effectiveness of medical modalities. It is derived from a standard detection theory with false alarm and detection power interpreted as false positive (FP) and true positive (TP) respectively in terms of medical diagnosis. The ROC curve is plotted based on TP versus FP via hard decisions. This paper presents a three dimensional (3D) ROC analysis which extends the traditional two-dimensional (2D) ROC analysis by including a threshold parameter in a third dimension resulting from soft decisions, (SD). As a result, a 3D ROC curve can be plotted based on three parameters, TP, FP and SD. By virtue of such a 3D ROC curve three two-dimensional (2D) ROC curves can be derived, one of which is the traditional 2D ROC curve of TP versus FP with SD reduced to hard decision. In order to illustrate its utility in medical diagnosis, its application to magnetic resonance (MR) image classification is demonstrated.

Objective To investigate features of fasting plasma glucose (FPG) during different trimesters and hemoglobin A1c (HbA1c) in the third trimester of pregnant women in Zhujing Region of Shanghai. Methods A total of 606 cases of pregnant women who received prenatal examination and 75 g oral glucose tolerance test (OGTT) in Jinshan Branch of Shanghai Sixth People′s Hospital from January 1 to December 31, 2014 were collected as research subjects. According to the results of 75 g OGTT, they were divided into gestational diabetes mellitus (GDM) group (n=136) and non-GDM group (n=470). The clinical data, such as age, body mass index (BMI), as well as the concentration of FPG during different pregnancy trimesters and level of HbA1c in the third trimester were collected retrospectively in both two groups by retrospective analysis method. According to the quartile method, level of HbA1c in the third trimester of all subjects were divided into four ranges: Q1 (HbA1c level<5.1%), Q2 (5.1%≤HbA1c level<5.2%), Q3 (5.2%≤HbA1c level<5.5%), and Q4 (HbA1c level≥5.5%), respectively. The age, BMI, concentrations of FPG and the level of HbA1c between two groups were compared by independent-samples t test. Chi-square test was used to compare the GDM incidence of all subjects with different ranges of HbA1c level in the third trimester, and further comparison was conducted by adjusting inspection level. Then receiver operator characteristic (ROC) curve of HbA1c level in the third trimester for predicting the incidence of GDM was drawn, and the area under ROC curve (ROC-AUC) was calculated. The optimal critical value of HbA1c level in the third trimester for predicting the incidence of GDM was obtained when the Youden index reaching the maximum value. And its sensitivity and specificity were calculated. This study met the requirements of the World Medical Association Declaration of Helsinki revised in 2013. Results ①The age and BMI of GDM group were higher than those in non-GDM group, and there were statistically significant differences (t=5.306, 5.250; P<0.001). Among the subjects in GDM group, the concentration of FPG in second trimester was the highest among 3 trimesters, which was (4.9±0.6) mmol/L, and the concentration of FPG in second trimester was higher than that in first and third trimester, respectively, and both the differences were statistically significant (t=2.087, 1.960; P=0.039, 0.041). Among the subjects in non-GDM group, the concentration of FPG in first trimester was the highest among 3 trimesters, which was (4.6±0.3) mmol/L, and the concentration of FPG in first trimester was higher than that in second and third trimester, respectively, and both the differences were statistically significant (t=15.230, 5.613; P<0.001). The concentration of FPG in first, second and third trimesters of GDM group were higher than those of non-GDM group, respectively, and there were statistically significant differences (t=5.416, 15.526, 4.471; P<0.001). Besides, HbA1c level in third trimester of GDM group was (5.6±0.4)%, which was higher than that of non-GDM group (5.4±0.4)%, and there was significant difference (t=5.845, P<0.01). ②There was statistical difference in GDM incidences in all subjects with different ranges of HbA1c levels in third trimester (χ2=22.707, P<0.001). Multiple comparison results showed that the incidences of GDM in all subjects with Q1 and Q2 ranges of HbA1c levels in third trimester were higher than that in subjects with Q4 range of HbA1c levels in third trimester, respectively, and both the differences were statistically significant (χ2=15.071, 16.785; P<0.001). ③The results of ROC curve analysis of HbA1c level in third trimester in predicting the incidence of GDM showed that the ROC-AUC was 0.647 (95%CI: 0.625-0.672, P<0.001), and the optimal cut-off value of HbA1c level in third trimester to predict GDM incidence was 5.5%, and the sensitivity of HbA1c level in predicting the incidence of GDM was 63.4%, and the sensitivity was 69.7%. Conclusions The concentrations of FPG in GDM pregnant women in Zhujing Region of Shanghai begin to increase from the first trimester and peak during the second trimester, and decrease during the third trimester after lifestyle intervention. It is advisable for GDM pregnant women to control HbA1c levels below 5.5% during the third trimester. Key words: Diabetes, gestational; Fasting plasma glucose; Hemoglobin A, glycosylated; Glucose tolerance test; ROC curve; Forecasting; Pregnant women

Receiver Operating Characteristic Analysis: A Primer 1

This paper presents a decision analysis method for evaluating computer intrusion detection systems. The method integrates and extends receiver operating characteristic (ROC) and cost analysis methods to provide an expected cost metric. We demonstrate that both the ROC analysis and cost analysis methods are incomplete. Furthermore, we demonstrate how a decision tree can combine and extend the ROC and cost analysis methods to provide an expected cost metric that reflects the intrusion detection system's ROC curve, costs, and assessments of the hostility of the environment as summarized by the prior probability of intrusion. We further demonstrate how this method can be used to decide the optimal operating point on an intrusion detector's ROC curve, choose the best intrusion detection system, compare the value of one intrusion detection system with another's, determine the value of an intrusion detector over no detector, and determine how to adjust the operation of an intrusion detector to respond to changes in its environment. General results are given and the method is illustrated in several numerical examples that involve both hypothetical and real intrusion detection systems. We demonstrate that, contrary to common advice, the value of an intrusion detection system depends not only on its ROC curve, but also on various costs (such as those associated with making incorrect decisions about detection) and the hostility of the operating environment. Conclusions are drawn about the design and evaluation of intrusion detection systems and the role for decision analysis in that design and evaluation.

Importance of construction of models for predicting software quality attributes is increasing leading to usage of artificial intelligence techniques such as Artificial Neural Network (ANN). The goal of this paper is to empirically compare traditional strategies such as Logistic Regression (LR) and ANN to assess software quality. The study used data collected from public domain NASA data set. We find the effect of software metrics on fault proneness. The fault proneness models were predicted using LR regression and ANN methods. The performance of the two methods was compared by Receiver Operating Characteristic (ROC) analysis. The areas under the ROC curves are 0.78 and 0.745 for the LR and ANN model, respectively. The predicted model shows that software metrics are related to fault proneness. The models predict faulty classes with more than 70 percent accuracy. The study showed that ANN method can also be used in constructing software quality models and more similar studies should further investigate the issue. Based on these results, it is reasonable to claim that such a model could help for planning and executing testing by focusing resources on fault-prone parts of the design and code.Keywordsempirical validationmetricssoftware qualityartificial neural network

Receiver operating characteristic (ROC) analysis is a fundamental tool used for the evaluation and comparison of diagnostic systems that provides estimates of the combinations of sensitivity and specificity that can be achieved with a given technique. Along with critical considerations of practical limitations, such as throughput and time to availability of results, ROC analyses can be applied to provide meaningful assessments and comparisons of available biodosimetry methods. Accordingly, guidance from the Food and Drug Administration to evaluate biodosimetry devices recommends using ROC analysis. However, the existing literature for the numerous biodosimetry methods that have been developed to address the needs for triage either do not contain ROC analyses or present ROC analyses where the dose distributions of the study samples are not representative of the populations to be screened. The use of non-representative sample populations can result in a significant spectrum bias, where estimated performance metrics do not accurately characterize the true performance under real-world conditions. Particularly, in scenarios where a large group of people is screened because they were potentially exposed in a large-scale radiation event, directly measured population data do not exist. However, a number of complex simulations have been performed and reported in the literature that provide estimates of the required dose distributions. Based on these simulations and reported data about the output and uncertainties of biodosimetry assays, we illustrate how ROC curves can be generated that incorporate a realistic representative sample. A technique to generate ROC curves for biodosimetry data is presented along with representative ROC curves, summary statistics and discussion based on published data for triage-ready electron paramagnetic resonance in vivo tooth dosimetry, the dicentric chromosome assay and quantitative polymerase chain reaction assay. We argue that this methodology should be adopted generally to evaluate the performance of radiation biodosimetry screening assays so that they can be compared in the context of their intended use.

Receiver Operating Characteristic (ROC) curves can be used to describe detectability of noise-limited signals. The basic concepts underlying the theory of ROC curves and the experimental determination of ROC curves are discussed. ROC curves are used to describe the detectability of the image of 2 mm Lucite beads (similar to certain kinds of small gall-stones) in a noisy background of radiographic mottle. Results are shown for four typical radiographic screen-film combinations of differing optical and noise properties. The qualitative relationship between the ROC curves and the mathematical model describing radiographic mottle is examined. The ROC curves describing the detectability of the bad images when radiographed with two screen-film combinations of equal system speed but different optical characteristics are compared to the quality of blood vessel images obtained with the same screen-film combinations. It is concluded that it is important to distinguish between the assessment of image quality based on the relative sharpness of high contrast images, versus the assessment of image quality based on the relative detectability of low contrast noise-limited images.

295: Accuracy of Various Methods of B-Line Assessment In the Diagnosis of Congestive Heart Failure Compared by Receiver Operating Characteristic Curves

Receiver operating characteristic (ROC) analysis is performed by a curve, called ROC curve, plotted based on detection probability, P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> , versus false alarm probability, P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> , and has been widely used as an evaluation tool for signal detection. Specifically, the area under an ROC curve (AUC) is calculated and used as a detection measure. Unfortunately, finding distributions of P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> and P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> to generate a continuous ROC curve is practically infeasible. This article investigates approaches to generating a discrete 2D ROC curve of ( P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> , P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> ) without appealing for probability distributions. Since P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> and P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> are determined by the same threshold τ to specify a detector, an ROC curve of ( P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> , P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> ) can only be used to evaluate the effectiveness of a detector but not target detectability (TD) and also not background suppressibility (BS). To address this issue, a 3D ROC curve is generated as a function of ( P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> , P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> , τ) by introducing a specific threshold parameter τ as a third independent variable. As a result, a 3D ROC curve along with its derived three 2D ROC curves of ( P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> , P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> ), ( P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</sub> , τ), and ( P <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> , τ) can further be used to design new quantitative measures to evaluate the effectiveness of a detector and its TD and BS. To demonstrate the full utility of 3D ROC analysis in target detection, extensive experiments are performed on two types of targets, prior target detection and anomaly detection, to conduct a comprehensive analysis on 3D ROC curves using new designed detection measures to evaluate target/anomaly detection performance.

To determine the cut-off values of body mass index (BMI) and waist circumference to predict hypertension in adults in north India. A community based cross-sectional study was conducted in 801 subjects in Kanpur, aged 20 years and above, using multistage stratified random sampling technique. A pre-tested structured questionnaire was used to elicit the required information from the study participants and the diagnostic criteria for hypertension were taken according to the Seventh Joint National Committee Report on Hypertension (JNC-7). Receiver operating characteristic (ROC) analysis was used to estimate the cut-off values of BMI and waist circumference to predict hypertension. The ROC analysis revealed that BMI is a good predictor of hypertension for both men (area under the ROC curve 0.714) and women (area under the ROC curve 0.821). The cut-off values of BMI for predicting hypertension were identified as ≥ 24.5 kg/m(2) in men and ≥ 24.9 kg/m(2) in women. Similarly, the ROC analysis for waist circumference showed that it is a good predictor of hypertension both for men (area under the ROC curve 0.784) and women (area under the ROC curve 0.815). The cut-offs for waist circumference for predicting hypertension were estimated as ≥ 83 cm for men and ≥ 78 cm for women. Adults with high BMI or high waist circumference had a higher prevalence of hypertension, respectively. Simple anthropometric measurements such as BMI and waist circumference can be used for screening people at increased risk of hypertension in order to refer them for more careful and early diagnostic evaluation. Policies and programs are required for primary and secondary prevention of hypertension.

When an investigator is developing a new image-processing technique or manipulating an image-acquisition technique they are confronted with the question of whether the new technique will improve clinical diagnosis. A first approach is to look at individual physical properties of the image such as image contrast and resolution. Although these properties might be useful, it has long been known that the noise characteristics of the image system need to be taken into consideration to appropriately evaluate the quality of an image whether it will be used to detect, classify, and/ˆ•or estimate a signal (Cunningham and Shaw, 1999). One useful measure of the noise characteristics is the noise-equivalent quanta (NEQ) that expresses the image noise in terms of the number of Poisson-distributed input photons per unit area at each spatial frequency (Wagner and Brown, 1985). The NEQ can be thought of as a measure inversely related to the amount of noise as a function of spatial frequency. However, when the diagnostic decision involves a human observer, medical image quality can be defined in terms of human performance in visual tasks that are relevant to clinical diagnosis (Barrett, 1993). In this context, receiver operating characteristics (ROC) studies are the standard method of evaluating the impact of a particular image manipulation on clinical diagnosis. In these studies, the physicians scrutinize a set of medical images (under the different image-acquisition or processing conditions) and rate their confidence about the presence of the lesion. The investigator infers from these ratings a measure of performance known as the area under the ROC curve. Often, the number of possible conditions is large and ROC studies become time consuming and costly because they require a large number of human observations. Other times, the investigator might want to optimize a parameter or a set of parameters. In such cases, the number of conditions suffers a combinatorial explosion, and therefore ROC studies become unfeasible. Thus it is desirable to develop a metric of image quality that could be used for fast evaluation and optimization of image quality but also would have the predictive power of ROC studies. Computer-model observers are algorithms that attempt to predict human visual performance in noisy images and might represent the desired metric of image quality when the diagnostic decision involves a human observer and a visual task. Development of models to predict human visual signal detection in noise goes back to work by Rose (1948) who studied the detectability of a flat-topped disk embedded in white noise (see Burgess, 1999a, for a review). In the last two decades, many studies have concentrated on finding a model observer that can predict human performance across many types of synthetic backgrounds. More recently, model observers have been applied to real medical-image backgrounds. The hope is that eventually model observers will become common metrics of task-based image quality for evaluation of medical-image quality as well as optimization of imaging systems.

Nonclinical studies provide the opportunity to anchor biochemical with morphologic findings; however, liver injury is often complex and heterogeneous, confounding the ability to relate biochemical changes with specific patterns of injury. The aim of the current study was to compare diagnostic performance of hepatobiliary markers for specific manifestations of drug-induced liver injury in rat using data collected in a recent hepatic toxicogenomics initiative in which rats (n = 3205) were given 182 different treatments for 4 or 14 days. Diagnostic accuracy of alanine aminotransferase (ALT), aspartate aminotransferase (AST), total bilirubin (Tbili), serum bile acids (SBA), alkaline phosphatase (ALP), gamma glutamyl transferase (GGT), total cholesterol (Chol), and triglycerides (Trig) was evaluated for specific types of liver histopathology by Receiver Operating Characteristic (ROC) analysis. To assess the relationship between biochemical and morphologic changes in the absence of hepatocellular necrosis, a second ROC analysis was performed on a subset of rats (n = 2504) given treatments (n = 152) that did not cause hepatocellular necrosis. In the initial analysis, ALT, AST, Tbili, and SBA had the greatest diagnostic utility for manifestations of hepatocellular necrosis and biliary injury, with comparable magnitude of area under the ROC curve and serum hepatobiliary marker changes for both. In the absence of hepatocellular necrosis, ALT increases were observed with biochemical or morphologic evidence of cholestasis. In both analyses, diagnostic utility of ALP and GGT for biliary injury was limited; however, ALP had modest diagnostic value for peroxisome proliferation, and ALT, AST, and total Chol had moderate diagnostic utility for phospholipidosis. None of the eight markers evaluated had diagnostic value for manifestations of hypertrophy, cytoplasmic rarefaction, inflammation, or lipidosis.

Morphological-Hemodynamic Characteristics of Intracranial Bifurcation Mirror Aneurysms

Objective To investigate a possible role for serum inflammatory cytokines [interleukin (IL)-6, 1β and tumor necrosis factor (TNF)-α, interferon-γ] as pre-operative diagnosis index of ovarian cyst torsion. Methods From January 2011 to December 2014, a total of forty cases who were admitted to Shangqiu First People's Hospital with suspected clinical diagnosis of ovarian cyst torsion were prospectively assigned to the study. They were divided into two groups according to surgery examination, study group (n=23, ovarian cyst torsion patients) and control group (n=17, non-ovarian cyst torsion patients). Blood samples were drawn pre-operatively and examined for serum levels of IL-6, -1β and TNF-α, interferon-γ by enzyme-linked immunosorbent assay (ELISA). The levels of serum inflammatory cytokines, receiver operating characteristic (ROC) curve between two groups were analyzed. Meanwhile, the critical value, sensitivity, specificity, positive predictive value, negative predictive value and Youden's index of serum inflammatory cytokines in the diagnosis of ovarian cyst torsion were detected. The study protocol was approved by the Ethical Review Board of Investigation in Human Being of Shangqiu First People's Hospital in Henan Province. Informed consent was obtained from each participates. Results Levels of IL-6, -1β and TNF-α, interferon-γ in study group were higher than those in group control[IL-6: (10.92±0.63) pg/mL vs (10.20±0.58) pg/mL; IL-1β: (53.28±15.29) pg/mL vs (34.59±10.27) pg/mL; TNF-α: (8.96±1.50) pg/mL vs (8.02±1.34) pg/mL; interferon-γ: (25.17±6.56) pg/mL vs (16.52±3.71) pg/mL], and all the differences were statistically significant (t=3.69, 4.36, 2.05, 4.88; P=0.000 1, 0.000 1, 0.005 3, 0.034). According to the ROC curve, the area under curve of IL-6, -1β were 0.838 and 0.831, the sensitivity of diagnosing ovarian cyst torsion were 78.3% and 91.3%, the specificity were 82.5% and 82.4%, respectively. Conclusions Elevated serum levels of IL-6 and IL-1β were associated with the occurrence of ovarian cyst torsion. In combination with vague clinical signs of ovarian cyst torsion, serum IL-6 and IL-1β could be the pre-operative diagnostic indices of ovarian cyst torsion. Key words: Cystic ovarian tumors torsion; Serum; Diagnosis; Preoperative period; ROC curve