• Cite Icon65
  • https://doi.org/10.1145/1102351.1102419Copy DOI Icon

ROC confidence bands

  • Jan 1, 2005
  • Sofus A Macskassy +2 more
Show More
  • Abstract
  • Literature Map
  • References
  • Citations
  • Similar Papers
Abstract

This paper is about constructing confidence bands around ROC curves. We first introduce to the machine learning community three band-generating methods from the medical field, and evaluate how well they perform. Such confidence bands represent the region where the "true" ROC curve is expected to reside, with the designated confidence level. To assess the containment of the bands we begin with a synthetic world where we know the true ROC curve---specifically, where the class-conditional model scores are normally distributed. The only method that attains reasonable containment out-of-the-box produces non-parametric, "fixed-width" bands (FWBs). Next we move to a context more appropriate for machine learning evaluations: bands that with a certain confidence level will bound the performance of the model on future data. We introduce a correction to account for the larger uncertainty, and the widened FWBs continue to have reasonable containment. Finally, we assess the bands on 10 relatively large benchmark data sets. We conclude by recommending these FWBs, noting that being non-parametric they are especially attractive for machine learning studies, where the score distributions (1) clearly are not normal, and (2) even for the same data set vary substantially from learning method to learning method.

Similar Papers
  • Research Article
  • Citations14

Simultaneous and exact interval estimates for the contrast of two groups based on an extremely high dimensional variable: application to mass spec data

  • Apr 25, 2007
  • Bioinformatics
  • Yuhyun Park +6
  • PDF
  • Research Article
  • Citations80

Chemist versus Machine: Traditional Knowledge versus Machine Learning Techniques

  • Nov 09, 2020
  • Trends in Chemistry
  • Janine George +1
  • Peer Review Report

Editor's evaluation: Robust and Efficient Assessment of Potency (REAP) as a quantitative tool for dose-response curve estimation

  • May 09, 2022
  • Philip Boonstra
  • PDF
  • Research Article
  • Citations95

Impact of train/test sample regimen on performance estimate stability of machine learning in cardiovascular imaging

  • Jul 14, 2021
  • Scientific Reports
  • Vikash Singh +5
  • Book Chapter
  • Citations1

Effects of smoothing on distribution approximations

  • Jan 01, 2003
  • Peter Hall +1
  • Conference Article
  • Citations1

An Innovative Machine Learning Method for Predicting Well Performance in Unconventional Reservoirs with a Relatively Small Data Set

  • Mar 07, 2023
  • Hui-Hai Liu +3
  • Research Article
  • Citations1

Demystifying machine learning in endourology - understanding models, applications, and clinical impact: a review from EAU endourology.

  • Sep 25, 2025
  • Current opinion in urology
  • Chady Ghnatios +2
  • Peer Review Report

Decision letter: VO2max prediction based on submaximal cardiorespiratory relationships and body composition in male runners and cyclists: a population study

  • Apr 04, 2023
  • Beat Knechtle +1
  • PDF
  • Research Article
  • Citations13

Machine learning guided postnatal gestational age assessment using new-born screening metabolomic data in South Asia and sub-Saharan Africa

  • Sep 07, 2021
  • BMC Pregnancy and Childbirth
  • Sunil Sazawal +25
  • Research Article

Using of data science in healthcare

  • Apr 24, 2021
  • Problems of Innovation and Investment Development
  • Ihor Ponomarenko +1
  • Book Chapter
  • Citations45

CloudSVM: Training an SVM Classifier in Cloud Computing Systems

  • Jan 01, 2013
  • F Ozgur Catak +1
  • Front Matter
  • Citations6

Commentary: Machine learning and cardiac surgery risk prediction

  • Aug 24, 2020
  • The Journal of Thoracic and Cardiovascular Surgery
  • David M Shahian +1
  • PDF
  • Research Article
  • Citations70

A review of machine learning models applied to genomic prediction in animal breeding.

  • Sep 06, 2023
  • Frontiers in Genetics
  • Narjice Chafai +3
  • Research Article

VENTRICULAR-ARTERIAL DECOUPLING IS ASSOCIATED WITH IN-HOSPITAL ADVERSE EVENTS IN NORMOTENSIVE PULMONARY EMNBOLISM

  • Oct 01, 2022
  • Canadian Journal of Cardiology
  • G Prosperi-Porta +8
  • PDF
  • Research Article
  • Citations29

Development and Validation of a Risk Prediction Model for Venous Thromboembolism in Lung Cancer Patients Using Machine Learning.

  • Mar 07, 2022
  • Frontiers in Cardiovascular Medicine
  • Haike Lei +14
Cactus Communications logo

Copyright 2026 Cactus Communications. All rights reserved.