Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-cd9895bd7-gbm5v Total loading time: 0 Render date: 2024-12-23T05:21:08.007Z Has data issue: false hasContentIssue false

16 - Multireader ROC Analysis

from Part III - Perception Metrology

Published online by Cambridge University Press:  20 December 2018

By
Stephen L. Hillis
Edited by
Ehsan Samei  and
Elizabeth A. Krupinski
Ehsan Samei
Affiliation:
Duke University Medical Center, Durham
Elizabeth A. Krupinski
Affiliation:
Emory University, Atlanta
Get access

Summary

A summary is not available for this content so a preview has been provided. Please use the Get access link above for information on how to access this content.
Image of the first page of this content. For PDF version, please use the ‘Save PDF’ preceeding this image.'
Type
Chapter
Information
The Handbook of Medical Image Perception and Techniques , pp. 245 - 262
Publisher: Cambridge University Press
Print publication year: 2018

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abbey, C.K., Samuelson, F.W., Gallas, B.D. (2013). Statistical power considerations for a utility endpoint in observer performance studies. Acad Radiol, 20, 798–806.Google Scholar
Abbey, C.K., Gallas, B.D., Boone, J.M., Niklason, L.T., Hadjiiski, L.M., Sahiner, B., Samuelson, F.W. (2014). Comparative statistical properties of expected utility and area under the ROC curve for laboratory studies of observer performance in screening mammography. Acad Radiol, 21, 481–490.CrossRefGoogle ScholarPubMed
Bamber, D. (1975). Area above ordinal dominance graph and area below receiver operating characteristic graph. J Math Psychol, 12, 387–415.CrossRefGoogle Scholar
DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44, 837–845.CrossRefGoogle ScholarPubMed
Dorfman, D.D., Berbaum, K.S. (2000a). A contaminated binormal model for ROC data – part III. Initial evaluation with detection ROC data. Acad Radiol, 7, 438–447.Google Scholar
Dorfman, D.D., Berbaum, K.S. (2000b). A contaminated binormal model for ROC data – part II. A formal model. Acad Radiol, 7, 427–437.Google Scholar
Dorfman, D.D., Berbaum, K.S., Metz, C.E. (1992). Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol, 27, 723–731.Google Scholar
Dorfman, D.D., Berbaum, K.S., Lenth, R.V., Chen, Y.F. (1999). Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: split plot experimental design. SPIE Proc Ser, 3663, 91–99.Google Scholar
Dorfman, D.D., Berbaum, K.S., Brandser, E.A. (2000). A contaminated binormal model for ROC data – part I. Some interesting examples of binormal degeneracy. Acad Radiol, 7, 420–426.Google Scholar
Gallas, B.D. (2006). One-shot estimate of MRMC variance: AUC. Acad Radiol, 13, 353–362.Google Scholar
Gallas, B.D. (2017). iMRMC v3p3 Application for Analyzing and Sizing MRMC Reader Studies (computer software). Division of Imaging, Diagnostics, and Software Reliability, OSEL/CDRH/FDA, Silver Spring, MD. Available for download from https://github.com/DIDSR/iMRMC/releases.Google Scholar
Gallas, B.D., Bandos, A., Samuelson, F.W., Wagner, R.F. (2009). A framework for random-effects ROC analysis: biases with the bootstrap and other variance estimators. Commun Statistics – Theory Methods, 38, 2586–2603.Google Scholar
Hanley, J.A., McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36.CrossRefGoogle Scholar
Hillis, S.L. (2007). A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Statistics Med, 26(3), 596–619.Google Scholar
Hillis, S.L. (2014). A marginal-mean ANOVA approach for analyzing multireader multicase radiological imaging data. Statistics Med, 33, 330–360.Google Scholar
Hillis, S.L., Berbaum, K.S. (2005). Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. Acad Radiol, 12, 1534–1542.Google Scholar
Hillis, S.L., Schartz, K.M. (2015). Demonstration of multi- and single-reader sample size program for diagnostic studies software. In: Mello-Thoms, C.R., Kupinski, M.A. (eds.) Medical Imaging 2015: Image Perception, Observer Performance, and Technology Assessment, Proc SPIE 9416 94160E.Google Scholar
Hillis, S.L., Obuchowski, N.A., Schartz, K.M., Berbaum, K.S. (2005). A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette methods for receiver operating characteristic (ROC) data. Statistics Med, 24, 1579–1607.Google Scholar
Hillis, S.L., Schartz, K.M., Berbaum, K.S. (2007). DBM MRMC 3.0 for SAS (computer software). Available at: http://perception.radiology.uiowa.edu.Google Scholar
Hillis, S.L., Berbaum, K.S., Metz, C.E. (2008). Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol, 15, 647–661.CrossRefGoogle ScholarPubMed
Hillis, S.L., Obuchowski, N.A., Berbaum, K.S. (2011). Power estimation for multireader ROC methods: an updated and unified approach. Acad Radiol, 18, 129–142.Google Scholar
Jiang, Y.L., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750.Google Scholar
McClish, D.K. (1989). Analyzing a portion of the ROC curve. Med Decis Making, 9, 190–195.Google Scholar
Metz, C.E., Pan, X.C. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol, 43, 1–33.CrossRefGoogle ScholarPubMed
Obuchowski, N.A. (1995). Multireader receiver operating characteristic studies: a comparison of study designs. Acad Radiol, 2, 709–716.CrossRefGoogle ScholarPubMed
Obuchowski, N.A. (2009). Reducing the number of reader interpretations in MRMC studies. Acad Radiol, 16, 209–217.Google Scholar
Obuchowski, N.A., Rockette, H.E. (1995). Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests: an ANOVA approach with dependent observations. Commun Statistics – Simul Comput, 24, 285–308.Google Scholar
Obuchowski, N.A., Gallas, B.D., Hillis, S.L. (2012). Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol, 19, 1508–1517.CrossRefGoogle ScholarPubMed
Pan, X.C., Metz, C.E. (1997). The ‘‘proper’’ binormal model: parametric receiver operating characteristic curve estimation with degenerate data. Acad Radiol, 4, 380–389.Google Scholar
Schartz, K.M., Hillis, S.L. (2018). Multi-reader sample size program for diagnostic studies (computer software). Available for download from http://perception.radiology.uiowa.edu.Google Scholar
Schartz, K.M., Hillis, S.L., Pesce, L.L., Berbaum, K.S. (2018). OR-DBM MRMC (Version 2.51) (computer software). Available for download from http://perception.radiology.uiowa.edu.Google Scholar
Shao, J., Dongshen, T. (1995). The Jackknife and Bootstrap. New York, NY: Springer-VerlagGoogle Scholar
Van Dyke, C.W., White, R.D., Obuchowski, N.A., Geisinger, M.A., Lorig, R.J., Meziane, M.A. (1993). Cine MRI in the diagnosis of thoracic aortic dissection. 79th RSNA Meetings, Chicago, IL, November 28–December 3.Google Scholar
Zeger, S.L., Liang, K.Y. (1986). Longitudinal data analysis for discrete and continuous outcomes. Biometrics, 42, 121–130.Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×