American College of Radiology. (2003). ACR BI-RADS: MRI. In: ACR BI-RADS: Breast Imaging Reporting and Data System: Breast Imaging Atlas. Reston, VA: ACR.Google Scholar

Barrett, H.H., Kupinski, M.A., Clarkson, E. (2005). Probabilistic foundations of the MRMC method. Proc SPIE, 5749, 21–31.CrossRefGoogle Scholar

Beiden, S.V., Wagner, R.F., Campbell, G. (2000). Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol, 7, 341–349.Google Scholar

Bird, R.E., Wallace, T.W., Yankaskas, B.C. (1992). Analysis of cancers missed at screening mammography. Radiology, 184, 613–617.Google Scholar

Birdwell, R.L., Bandodkar, P., Ikeda, D.M. (2005). Computer-aided detection with screening mammography in a university hospital setting. Radiology, 236, 451–457.Google Scholar

Bornefalk, H., Hermansson, A.B. (2005). On the comparison of FROC curves in mammography CAD systems. Med Phys, 32, 412–417.Google Scholar

Bunch, P.C., Hamilton, J.F., Sanderson, G.K., Simmons, A.H. (1977). A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE, 127, 124–135.CrossRefGoogle Scholar

Chakraborty, D.P. (2000). The FROC, AFROC and DROC variants of the ROC analysis. In: Beutel, J., Kundel, H., Van Metter, R. (eds.) Handbook of Medical Imaging, Volume 1. Physics and Psychophysics. Bellingham, WA: SPIE, pp. 771–798.Google Scholar

Chan, H.P., Sahiner, B., Helvie, M.A., et al. (1999). Improvement of radiologists’ characterization of mammographic masses by using computer-aided diagnosis: an ROC study. Radiology, 212, 817–827.Google Scholar

Chen, W., Giger, M.L., Bick, U. (2006a). A fuzzy c-means (FCM) based approach for computerized segmentation of breast lesions in dynamic contrast-enhanced MR images. Acad Radiol, 16, 63–72.Google Scholar

Chen, W., Giger, M.L., Bick, U., Newstead, G. (2006b). Automatic identification and classification of characteristic kinetic curves of breast lesions on DCE-MRI. Med Phys, 33, 2878–2887.Google Scholar

Clarkson, E., Kupinski, M.A., Barrett, H.H. (2006). A probabilistic model for the MRMC method. Part I. Theoretical development. Acad Radiol, 13, 1410–1421.Google Scholar

Cupples, T., Cunningham, J.E., Reynolds, J.C. (2005). Impact of computer-aided detection in a regional screening mammography program. AJR, 185, 944–950.Google Scholar

Dean, J.C., Iivento, C.C. (2006). Improved cancer detection using computer-aided detection with diagnostic and screening mammography: prospective study of 104 cancers. AJR, 187, 20–28.Google Scholar

Dorfman, D.D., Berbaum, K.S., Metz, C.E. (1992). Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol, 27, 723–731.CrossRefGoogle Scholar

Feig, S.A., Sickes, E.A., Evans, W.P., Linver, M.N. (2004). Re: Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 1260–1261.Google Scholar

Fenton, J.J., Taplin, S.H., Carney, P.A., et al. (2007). Influence of computer-aided detection on performance of screening mammography. N Engl J Med, 356, 1399–1409.Google Scholar

Flehinger, B.J., Kimmel, M., Melamed, M.R. (1992). The effect of surgical treatment on survival from early lung cancer. Implications for screening. Chest, 101, 1013–1018.CrossRefGoogle ScholarPubMed

Freedman, M., Lo, S., Lure, F., et al. (2001). Computer-aided detection of lung cancer on chest radiographs: algorithm performance vs. radiologists’ performance by size of cancer. Proc SPIE, 4319, 150–159.Google Scholar

Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection. prospective study of 12,860 patients in a community breast center. Radiology, 222, 781–786.Google Scholar

Gallas, B.D. (2006). One-shot estimate of MRMC variance: AUC. Acad Radiol, 13, 353–362.Google Scholar

Gallas, B.D., Brown, D.G. (2008). Reader studies for validation of CAD systems. Neural Networks, 21, 387–397.Google Scholar

Gallas, B.D., Pennello, G.A., Myers, K.J. (2007). Multi-reader multi-case variance analysis for binary data. J Opt Soc Am A, 24(12), B70–B80.Google Scholar

Giger, M.L., Huo, Z., Kupinski, M.A., Vyborny, C.J. (2000). Computer-aided diagnosis in mammography. In: Sonka, M., Fitzpatrick, M.J. (eds.) Handbook of Medical Imaging, Volume 2. Medical Imaging Processing and Analysis. Bellingham, WA: SPIE, pp. 915–1004.Google Scholar

Giger, M.L., Huo, Z., Vyborny, C.J., et al. (2003). Results of an observer study with an intelligent mammographic workstation for CAD. In: Peitgen, H.-O. (ed.) Digital Mammography, IWDM 2002. Berlin: Springer, pp. 297–303.Google Scholar

Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. Am J Roentgenol, 190(4), 854–859.Google Scholar

Gur, D., Sumkin, J.H., Rockette, H.E., et al. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.Google Scholar

Gur, D., Bandos, A.I., Cohen, C.S., et al. (2008). The “laboratory” effect: comparing radiologists’ performance and variability during prospective clinical and laboratory mammography interpretations. Radiology, 249, 47–53.Google Scholar

Hara, A.K., Johnson, C.D., Reed, J.E., et al. (1997). Detection of colorectal polyps with CT colography: initial assessment of sensitivity and specificity. Radiology, 205, 59–65.Google Scholar

Helvie, M., Hadjiiski, L., Makariou, E., et al. (2004). Sensitivity of noncommercial computer-aided detection system for mammographic breast cancer detection: pilot clinical trial. Radiology, 231, 208–214.Google Scholar

Henschke, C.I., Naidich, D.P., Yankelevitz, D.F., et al. (2001). Early lung cancer action project: initial findings on repeat screenings. Cancer, 92, 153–159.3.0.CO;2-S>CrossRefGoogle ScholarPubMed

Horsch, K., Giger, M.L., Vyborny, C.J., et al. (2006). Multi-modality computer-aided diagnosis for the classification of breast lesions: observer study results on an independent clinical dataset. Radiology, 240, 357–368.Google Scholar

Huo, Z., Giger, M.L., Vyborny, C.J., et al. (1995). Analysis of spiculation in the computerized classification of mammographic masses. Med Phys, 22, 1569–1579.Google Scholar

Huo, Z., Giger, M.L., Vyborny, C.J., et al. (2002). Effectiveness of CAD in the diagnosis of breast cancer: an observer study on an independent database of mammograms. Radiology, 224, 560–568.Google Scholar

Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750.Google Scholar

Jiang, Y., Nishikawa, R.M., Schmidt, R.A., et al. (1999). Improving breast cancer diagnosis with computer-aided diagnosis. Acad Radiol, 6, 22.Google Scholar

Jiang, Y., Nishikawa, R.M., Schmidt, R.A., et al. (2001). Potential of computer-aided diagnosis to reduce variability in radiologists’ interpretations of mammograms depicting microcalcifications. Radiology, 220, 787–794.Google Scholar

Khoo, L.A.L., Taylor, P., Given-Wilson, R.M. (2005). Computer detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.Google Scholar

Krupinski, E.A., Kundel, H.L., Judy, P.F., et al. (1998). Key issues for image perception research. Radiology, 209, 611–612.Google Scholar

Kuhl, C.K., Mielcareck, P., Klaschik, S., et al. (1999). Dynamic breast MR imaging: are signal intensity time course data useful for differential diagnosis of enhancing lesions? Radiology, 211, 101–110.Google Scholar

Kundel, H. (1975). Peripheral vision, structured noise and film reader error. Radiology, 114, 269–273.CrossRefGoogle ScholarPubMed

Kupinski, M.A., Giger, M.L. (1998). Automated seeded lesion segmentation on digital mammograms. IEEE Trans Med Imag, 17, 510–517.Google Scholar

Kupinski, M., Clarkson, E., Barrett, H. (2006). A probabilistic model for the MRMC method, part 2: validation and applications. Acad Radiol, 13, 1422–1430.CrossRefGoogle ScholarPubMed

McFarland, E.G., Brink, J.A., Pilgram, T.K., et al. (2001). Spiral CT colonography: reader agreement and diagnostic performance with two- and three-dimensional image-display techniques. Radiology, 218, 375–383.Google Scholar

Metz, C.E. (1978). Basic principles of ROC analysis. Semin Nucl Med, 8, 283–298.Google Scholar

Metz, C.E. (1986). ROC methodology in radiologic imaging. Invest Radiol, 21, 720–733.Google Scholar

Metz, C.E. (2000). Fundamental ROC analysis. In: Beutel, J., Kundel, H., Van Metter, R. (eds.) Handbook of Medical Imaging, Volume 1. Physics and Psychophysics. Bellingham, WA: SPIE, pp. 751–769.Google Scholar

Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar

Muramatsu, C., Li, Q., Suzuki, K., et al. (2005). Investigation of psychophysical measures for evaluation of similar images for mammographic masses: preliminary results. Med Phys, 32, 2295–2304.Google Scholar

Nappi, J., Yoshida, H. (2003). Feature-guided analysis for reduction of false positives in CAD of polyps for computed tomographic colonography. Med Phys, 30, 1592–1601.Google Scholar

Nishikawa, R.M., Giger, M.L., Vyborny, C.J., et al. (2001). Prospective computer analysis of cancers missed on screening mammography. In: Digital Mammography 2000, Proceedings of the 5th International Workshop on Digital Mammography. Madison, WI: Medical Physics, pp. 493–498.Google Scholar

Petrick, N., Haider, M., Summers, R.M., et al. (2008). CT colonography with computer-aided detection as a second reader: an observer performance study. Radiology, 246(1), 148–156.Google Scholar

Renfrew, D.L., Franken, E.A., Jr., Berbaum, K.S., Weigelt, F.H., AbuYousef, M.M. (1992). Error in radiology: classification and lessons in 182 cases presented at a problem case conference. Radiology, 183, 145–150.Google Scholar

Royster, A.P., Fenlon, H.M., Clarke, P.D., Nunes, D.P., Ferrucci, J.T. (1997). CT colonoscopy of colorectal neoplasms: two-dimensional and three-dimensional virtual-reality techniques with colonoscopic correlation. Am J Roentgenol, 169, 1237–1242.Google Scholar

Sahiner, B., Chan, H.-P., Petrick, N., Wagner, R.F., Hadjiiski, L. (2000). Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys, 27, 1509–1522.Google Scholar

Samuelson, F.W., Petrick, N. (2006). Comparing image detection algorithms using resampling. Proceedings of the 2006 IEEE International Symposium on Biomedical Imaging, pp. 1312–1315.Google Scholar

Shah, P.K., Austin, J.H., White, C.S., et al. (2003). Missed non-small cell lung cancer: radiographic findings of potentially resectable lesions evident only in retrospect. Radiology, 226, 235–241.Google Scholar

Sone, S., Li, F., Yang, Z.G., et al. (2001). Results of three-year mass screening programme for lung cancer using mobile low-dose spiral computed tomography scanner. Br J Cancer, 84, 25–32.Google Scholar

Sonka, M., Fitzpatrick, M.J. (eds.) (2000). Handbook of Medical Imaging, Volume 2. Medical Imaging Processing and Analysis. Bellingham, WA: SPIE.Google Scholar

Summers, R.M., Johnson, C.D., Pusanik, L.M., et al. (2001). Automated polyp detection at CT colonography: feasibility assessment in a human population. Radiology, 219, 51–59.Google Scholar

Summers, R.M., Yao, J., Pickhardt, P.J., et al. (2005). Computed tomographic virtual colonoscopy computer-aided polyp detection in a screening population. Gastroenterology, 129, 1832–1844.Google Scholar

Suzuki, K., Yoshida, H., Nappi, J., Dachman, A.H. (2006). Massive-training artificial neural network (MTANN) for reduction of false positives in computer-aided detection of polyps: suppression of rectal tubes. Med Phys, 33, 3814–3824.Google Scholar

Swensen, S.J., Jett, J.R., Hartman, T.E., et al. (2003). Lung cancer screening with CT: Mayo Clinic experience. Radiology, 226, 756–761.Google Scholar

Taylor, S.A., Charman, S.C., Lefere, P., et al. (2007). CT colonography: investigation of the optimum reader paradigm by using computer-aided detection software. Radiology, 246(2), 463–471.Google Scholar

Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar

Yoshida, H., Dachman, A. (2004). Computer-aided diagnosis for CT colonography. Semin Ultrasound CT MR, 25, 419–431.Google Scholar

Yoshida, H., Masutani, Y., MacEneaney, P., Rubin, D.T., Dachman, A.H. (2002). Computerized detection of colonic polyps at CT colonography on the basis of volumetric features: pilot study. Radiology, 222: 327–336.Google Scholar

Yousef, W.A., Wagner, R.F., Loew, M.H. (2005). Estimating the uncertainty in the estimated mean area under the ROC curve of a classifier. Patt Recog Lett, 26, 2600–2610.Google Scholar

Yousef, W.A., Wagner, R.F., Loew, M.H. (2006). Assessing classifiers from two independent data sets using ROC analysis: a nonpara-metric approach. IEEE Trans Patt Anal Mach Intell, 28, 1809–1817.Google Scholar

Yuan, Y., Giger, M.L., Li, H., Suzuki, K., Sennett, C. (2007). A dual-stage method for lesion segmentation on digital mammograms. Med Phys, 34, 4180–4193.Google Scholar

Zheng, B., Ganott, M.A., Britton, C.A., et al. (2001). Soft-copy mammographic reading with different computer-assisted detection cuing environments: preliminary findings. Radiology, 221, 633–640.Google Scholar

Zheng, B., Swensson, R.G., Golla, S., et al. (2004). Detection and classification performance levels of mammographic masses under different computer-aided detection cueing environments. Acad Radiol, 11, 396–406.Google Scholar

Zheng, B., Mello-Thoms, C.C., Wang, X.-H., et al. (2007). Interactive computer-aided diagnosis of breast masses: computerized selection of visually similar image sets from a reference library. Acad Radiol, 14, 917–927.Google Scholar

Alberdi, E., Povyakalo, A.A., Strigini, L., Ayton, P., Hartswood, M., Procter, R., Slack, R. (2005). Use of computer-aided detection (CAD) tools in screening mammography: a multidisciplinary investigation. Br J Radiol, 78 Spec No 1, S31–S40.Google Scholar

American College of Radiology (ACR) (2003). The Breast Imaging Reporting and Data System Atlas. Reston, VA: American College of Radiology.Google Scholar

Ballard-Barbash, R., Taplin, S.H., Yankaskas, B.C., Ernster, V.L., Rosenberg, R.D., Carney, P.A., Barlow, W.E., et al. (1997). Breast Cancer Surveillance Consortium: a national mammography screening and outcomes database. AJR Am J Roentgenol, 169, 1001–1008.Google Scholar

Barlow, W.E., Chi, C., Carney, P.A., Taplin, S.H., D’Orsi, C., Cutter, G., Hendrick, R.E., et al. (2004). Accuracy of screening mammography interpretation by characteristics of radiologists. J Natl Cancer Inst, 96, 1840–1850.Google Scholar

Beam, C.A., Layde, P.M., Sullivan, D.C. (1996). Variability in the interpretation of screening mammograms by US radiologists. Findings from a national sample. Arch Intern Med, 156, 209–213.Google Scholar

Beiden, S.V., Wagner, R.F., Doi, K., Nishikawa, R.M., Freedman, M., Lo, S.C., Xu, X.W. (2002). Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance. Acad Radiol, 9, 1036–1043.Google Scholar

Birdwell, R.L., Ikeda, D.M., O’Shaughnessy, K.F., Sickles, E.A. (2001). Mammographic characteristics of 115 missed cancers later detected with screening mammography and the potential utility of computer-aided detection. Radiology, 219, 192–202.Google Scholar

Birdwell, R.L., Bandodkar, P., Ikeda, D.M. (2005). Computer-aided detection with screening mammography in a university hospital setting. Radiology, 236, 451–457.Google Scholar

Brem, R.F., Baum, J., Lechner, M., Kaplan, S., Souders, S., Naul, L.G., Hoffmeister, J. (2003). Improvement in sensitivity of screening mammography with computer-aided detection: a multiinstitutional trial. AJR Am J Roentgenol, 181, 687–693.Google Scholar

Chan, H.P., Doi, K., Vyborny, C.J., Schmidt, R.A., Metz, C.A., Lam, K.L., Ogura, T., et al. (1990). Improvement in radiologists’ detection of clustered microcalcifications on mammograms. The potential of computer-aided diagnosis. Invest Radiol, 25, 1102–1110.Google Scholar

Ciatto, S., Ambrogetti, D., Collini, G., Cruciani, A., Ercolini, E., Risso, G., Rosselli Del Turco, M. (2006). Computer-aided detection (CAD) of cancers detected on double reading by one reader only. Breast, 15, 528–532.Google Scholar

Cupples, T.E., Cunningham, J.E., Reynolds, J.C. (2005). Impact of computer-aided detection in a regional screening mammography program. AJR Am J Roentgenol, 185, 944–950.Google Scholar

Dean, J.C., Ilvento, C.C. (2006). Improved cancer detection using computer-aided detection with diagnostic and screening mammography: prospective study of 104 cancers. AJR Am J Roentgenol, 187, 20–28.Google Scholar

Destounis, S.V., DiNitto, P., Logan-Young, W., Bonaccio, E., Zuley, M.L., Willison, K.M. (2004). Can computer-aided detection with double reading of screening mammograms help decrease the false-negative rate? Initial experience. Radiology, 232, 578–584.Google Scholar

Elmore, J.G., Wells, C.K., Howard, D.H. (1994). Variability in radiologists’ interpretations of mammograms. N Engl J Med, 331, 1493–1499.Google Scholar

Feig, S.A., Sickles, E.A., Evans, W.P., Linver, M.N. (2004). Re: Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 1260–1261; author reply 1261.Google Scholar

Fenton, J.J., Taplin, S.H., Carney, P.A., et al. (2007). Influence of computer-aided detection on performance of screening mammography. N Engl J Med, 356, 1399–1409.Google Scholar

Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology, 220, 781–786.Google Scholar

Gilbert, F.J., Astley, S.M., McGee, M.A., Gillan, M.G., Boggis, C.R., Griffiths, P.M., Duffy, S.W. (2006). Single reading with computer-aided detection and double reading of screening mammograms in the United Kingdom National Breast Screening Program. Radiology, 241, 47–53.Google Scholar

Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. AJR Am J Roentgenol, 190, 854–859.Google Scholar

Gur, D., Sumkin, J.H., Rockette, H.E., et al. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.Google Scholar

Gur, D., Bandos, A.I., Fuhrman, C.R., Klym, A.H., King, J.L., Rockette, H.E. (2007). The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol, 14, 49–53.Google Scholar

Helvie, M.A., Hadjiiski, L., Makariou, E., et al. (2004). Sensitivity of noncommercial computer-aided detection system for mammographic breast cancer detection: pilot clinical trial. Radiology, 231, 208–214.Google Scholar

Ikeda, D.M., Birdwell, R.L., O’Shaughnessy, K.F., Sickles, E.A., Brenner, R.J. (2004). Computer-aided detection output on 172 subtle findings on normal mammograms previously obtained in women with breast cancer detected at follow-up screening mammography. Radiology, 230, 811–819.Google Scholar

Jemal, A., Siegel, R., Ward, E., Hao, Y., Xu, J., Murray, T., Thun, M.J. (2008). Cancer statistics, 2008. CA Cancer J Clin, 58, 71–96.CrossRefGoogle Scholar

Jiang, Y., Miglioretti, D.L., Metz, C.E., Schmidt, R.A. (2007). Breast cancer detection rate: designing imaging trials to demonstrate improvements. Radiology, 243, 360–367.Google Scholar

Karssemeijer, N., Otten, J.D., Verbeek, A.L.M., Groenewoud, J.H., de Koning, H.J., Hendriks, J.H.C.L., Holland, R. (2003). Computer-aided detection versus independent double reading of masses on mammograms. Radiology, 227, 192–200.Google Scholar

Kegelmeyer, W.P., Jr., Pruneda, J.M., Bourland, P.D., Hillis, A., Riggs, M.W., Nipper, M.L. (1994). Computer-aided mammographic screening for spiculated lesions. Radiology, 191, 331–337.Google Scholar

Khoo, L.A., Taylor, P., Given-Wilson, R.M. (2005). Computer-aided detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.CrossRefGoogle ScholarPubMed

Ko, J.M., Nicholas, M.J., Mendel, J.B., Slanetz, P.J. (2006). Prospective assessment of computer-aided detection in interpretation of screening mammography. AJR Am J Roentgenol, 187, 1483–1491.Google Scholar

Kobayashi, T., Xu, X.W., MacMahon, H., Metz, C.E., Doi, K. (1996). Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on radiographs. Radiology, 199, 843–848.Google Scholar

Krupinski, E.A., Jiang, Y. (2008). Anniversary paper: evaluation of medical imaging systems. Med Phys, 35, 645–659.Google Scholar

Marx, C., Malich, A., Facius, M., Grebenstein, U., Sauner, D., Pfleiderer, S.O., Kaiser, W.A. (2004). Are unnecessary follow-up procedures induced by computer-aided diagnosis (CAD) in mammography? Comparison of mammographic diagnosis with and without use of CAD. Eur J Radiol, 51, 66–72.Google Scholar

Metz, C.E. (1978). Basic principles of ROC analysis. Semin Nucl Med, 8, 283–298.Google Scholar

Metz, C.E. (1989). Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol, 24, 234–245.Google Scholar

Moberg, K., Bjurstam, N., Wilczek, B., Rostgard, L., Egge, E., Muren, C. (2001). Computed assisted detection of interval breast cancers. Eur J Radiol, 39, 104–110.Google Scholar

Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar

Nishikawa, R.M. (2006). Modeling the effect of computer-aided detection on the sensitivity of screening mammography. In: Astley, S.M., Brady, M., Rose, C., Zwiggelaar, R. (eds.) Digital Mammography. London: Springer, pp. 46–53.CrossRefGoogle Scholar

Pisano, E.D., Gatsonis, C., Hendrick, E., et al. (2005). Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med, 353, 1773–1783.Google Scholar

Schmidt, R.A., Newstead, G.M., Linver, M.N., Eklund, G.W., Metz, C.E., Winkler, M.N., Nishikawa, R.M. (1998). Mammographic screening sensitivity of general radiologists. In: Karssemeijer, N., Thijssen, M., Hendriks, J., van Erning, L. (eds.) Digital Mammography. Dordrecht: Kluwer Academic, pp. 383–388.Google Scholar

Skaane, P., Kshirsagar, A., Stapleton, S., Young, K., Castellino, R.A. (2007). Effect of computer-aided detection on independent double reading of paired screen-film and full-field digital screening mammograms. AJR Am J Roentgenol, 188, 377–384.Google Scholar

Taplin, S.H., Rutter, C.M., Lehman, C.D. (2006). Testing the effect of computer-assisted detection on interpretive performance in screening mammography. AJR Am J Roentgenol, 187, 1475–1482.CrossRefGoogle ScholarPubMed

Taylor, P., Given-Wilson, R.M. (2005). Evaluation of computer-aided detection (CAD) devices. Br J Radiol, 78 Spec No 1, S26– S30.Google Scholar

te Brake, G.M., Karssemeijer, N., Hendriks, J.H. (1998). Automated detection of breast carcinomas not detected in a screening program. Radiology, 207, 465–471.Google Scholar

Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar

Warren Burhenne, L.J., Wood, S.A., D’Orsi, C.J., Feig, S.A., Kopans, D.B., O’Shaughnessy, K.F., Sickles, E.A., et al. (2000). Potential contribution of computer-aided detection to the sensitivity of screening mammography. Radiology, 215, 554–562.Google Scholar

Zheng, B., Shah, R., Wallace, L., Hakim, C., Ganott, M.A., Gur, D. (2002). Computer-aided detection in mammography: an assessment of performance on current and prior images. Acad Radiol, 9, 1245–1250.CrossRefGoogle ScholarPubMed

Abbey, C.K., Wu, Y., Burnside, E.S., Wunderlich, A., Samuelson, F.W., Boone, J.M. (2016). A utility/cost analysis of breast cancer risk prediction algorithms. Proc SPIE Med Imag, 9887, 97871J.Google Scholar

Adrion, W.R., Branstad, M.A., Cherniavsky, J.C. (1982). Validation, verification, and testing of computer software. ACM Comp Surv, 14, 159–192.Google Scholar

Aerts, H.J.W.L. (2016). The potential of radiomic-based phenotyping in precision medicine: a review. JAMA Oncol, 2, 1636–1642.Google Scholar

Agresti, A., Coull, B.A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. Am Statist, 52, 119–126.Google Scholar

Altman, D.G., Bland, J.M. (1999). Statistics notes – treatment allocation in controlled trials: why randomise? Br Med J, 318, 1209–1209.Google Scholar

Aoki, T., Oda, N., Yamashita, Y., Yamamoto, K., Korogi, Y. (2011). Usefulness of computerized method for lung nodule detection in digital chest radiographs using temporal subtraction images. Acad Radiol, 18, 1000–1005.Google Scholar

Armato, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P. et al. (2011). The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med Phys, 38, 915–931.Google Scholar

Bandos, A.I., Rockette, H.E., Song, T., Gur, D. (2009). Area under the free-response ROC curve (FROC) and a related summary index. Biometrics, 65, 247–256.Google Scholar

Belle, A., Thiagarajan, R., Soroushmehr, S.M.R., Navidi, F., Beard, D.A., Najarian, K. (2015). Big data analytics in healthcare. BioMed Res Intl, 2015, 1–16.Google Scholar

Bergstra, J., Yamins, D., Cox, D. (2013). Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, 28, 115–123.Google Scholar

Bogoni, L., Ko, J.P., Alpert, J., Anand, V., Fantauzzi, J., Florin, C.H., Koo, C.W., Mason, D., Rom, W., Shiau, M., Salganicoff, M., Naidich, D.P. (2012). Impact of a computer-aided detection (CAD) system integrated into a picture archiving and communication system (PACS) on reader sensitivity and efficiency for the detection of lung nodules in thoracic CT exams. J Digit Imag, 25, 771–781.Google Scholar

Boone, D., Mallett, S., McQuillan, J., Taylor, S.A., Altman, D.G., Halligan, S. (2015). Assessment of the incremental benefit of computer-aided detection (CAD) for interpretation of CT colonography by experienced and inexperienced readers. PLos One, 10.Google Scholar

Bornefalk, H. (2005). Estimation and comparison of CAD system performance in clinical settings. Acad Radiol, 12, 687–94.Google Scholar

Brown, L.D., Cai, T.T., DasGupta, A. (2001a). Interval estimation for a binomial proportion. Stat Sci, 16, 101–117.Google Scholar

Brown, L.D., Cai, T.T., DasGupta, A. (2001b). Interval estimation for a binomial proportion-comment-rejoiner. Stat Sci, 16, 117–133.Google Scholar

Brown, M.S., Goldin, J.G., Rogers, S., Kim, H.J., Suh, R.D., McNitt-Gray, M.F., Shah, S.K., Truong, D., Brown, K., Sayre, J.W., Gjertson, D.W., Batra, P., Aberle, D.R. (2005). Computer-aided lung nodule detection in CT: results of large-scale observer test. Acad Radiol, 12, 681–686.Google Scholar

Bunch, P.C., Hamilton, J.F., Sanderson, G.K., Simmons, A.H. (1978). A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng, 4, 166–171.Google Scholar

Byron, S.A., Van Keuren-Jensen, K.R., Engelthaler, D.M., Carpten, J.D., Craig, D.W. (2016). Translating RNA sequencing into clinical diagnostics: opportunities and challenges. Nat Rev Genet, 17, 257–271.Google Scholar

Chakraborty, D.P. (1989). Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys, 16, 561–568.Google Scholar

Chakraborty, D.P. (2006a). Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol, 13, 1187–1193.Google Scholar

Chakraborty, D.P. (2006b). A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol, 51, 3449–3462.Google Scholar

Chakraborty, D.P. (2008). Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol, 15, 1554–1566.Google Scholar

Chakraborty, D.P. (2017) FROC methodology website. Available at: www.devchakraborty.com/index.php (accessed November 10, 2017).Google Scholar

Chakraborty, D.P., Berbaum, K.S. (2004). Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys, 31, 2313–2330.Google Scholar

Chakraborty, D.P., Winter, L.H.L. (1990). Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology, 174, 873–881.Google Scholar

Chan, H.P., Wei, D., Helvie, M.A., Sahiner, B., Adler, D.D., Goodsitt, M.M., Petrick, N. (1995). Computer-aided classification of mammographic masses and normal tissue: linear discriminant analysis in texture feature space. Phys Med Biol, 40, 857.Google Scholar

Chan, H.P., Wei, J., Zhang, Y.H., Helvie, M.A., Moore, R.H., Sahiner, B., Hadjiiski, L., Kopans, D.B. (2008). Computer-aided detection of masses in digital tomosynthesis mammography: comparison of three approaches. Med Phys, 35, 4087–4095.Google Scholar

Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P. (2002). SMOTE: synthetic minority over-sampling technique. J Artif Intellig Res, 16, 321–357.CrossRefGoogle Scholar

Choudhury, K.R., Paik, D.S., Yi, C.A., Napel, S., Roos, J., Rubin, G.D. (2010). Assessing operating characteristics of CAD algorithms in the absence of a gold standard. Med Phys, 37, 1788–1795.Google Scholar

Dachman, A.H., Obuchowski, N.A., Hoffmeister, J.W., Hinshaw, J.L., Frew, M.I., Winter, T.C., Van Uitert, R.L., Periaswamy, S., Summers, R.M., Hillman, B.J. (2010). Effect of computer-aided detection for CT colonography in a multireader, multicase trial. Radiology, 256, 827–835.Google Scholar

De Boo, D.W., Uffmann, M., Weber, M., Bipat, S., Boorsma, E.F., Scheerder, M.J., Freling, N.J., Schaefer-Prokop, C.M. (2011). Computer-aided detection of small pulmonary nodules in chest radiographs: an observer study. Acad Radiol, 18, 1507–1514.Google Scholar

de Groot, J.A.H., Janssen, K.J.M., Zwinderman, A.H., Moons, K.G.M., Reitsma, J.B. (2008). Multiple imputation to correct for partial verification bias revisited. Stat Med, 27, 5880–5889.Google Scholar

de Hoop, B., De Boo, D.W., Gietema, H.A., van Hoorn, F., Mearadji, B., Schijf, L., van Ginneken, B., Prokop, M., Schaefer-Prokop, C. (2010). Computer-aided detection of lung cancer on chest radiographs: effect on observer performance. Radiology, 257, 532–540.Google Scholar

Dorfman, D.D., Alf, E. (1969). Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals – rating-method data. J Math Psych, 6, 487–496.Google Scholar

Dorfman, D.D., Berbaum, K.S., Metz, C.E., Lenth, R.V., Hanley, J.A., Dagga, H.A. (1997). Proper receiver operating characteristic analysis: the bigamma model. Acad Radiol, 4, 138–149.Google Scholar

Dwork, C. (2011). The promise of differential privacy a tutorial on algorithmic techniques. In: Ostrovsky, R. (ed.) 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science, pp. 1–2.Google Scholar

Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., Roth, A. (2015). The reusable holdout: preserving validity in adaptive data analysis. Science, 349, 636–638.Google Scholar

Edwards, D.C., Kupinski, M.A., Metz, C.E., Nishikawa, R.M. (2002). Maximum likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys, 29, 2861–2870.Google Scholar

Edwards, D.C., Metz, C.E., Kupinski, M.A. (2004). Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans Med Imag, 23, 891–895.Google Scholar

Efron, B. (1983). Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Statis Assn, 78, 316–331.Google Scholar

Efron, B., Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Stat Sci, 1, 54–75.Google Scholar

Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542, 115.Google Scholar

Evans, K.K., Birdwell, R.L., Wolfe, J.M. (2013). If you don’t find it often, you often don’t find it: why some cancers are missed in breast cancer screening. PLos One, 8.Google Scholar

Evans, B.J., Burke, W., Jarvik, G.P. (2015). The FDA and genomic tests – getting regulation right. N Engl J Med, 372, 2258–2264.Google Scholar

FDA (2012a). Guidance for industry and FDA staff: clinical performance assessment: considerations for computer-assisted detection devices applied to radiology images and radiology device data – premarket approval (PMA) and premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187315.pdf (accessed November 21, 2017).Google Scholar

FDA (2012b). Guidance for industry and FDA staff: computer-assisted detection devices applied to radiology images and radiology device data – premarket notification [510(k)] submissions. Available at: www.fda.gov/downloads/MedicalDevices/Device RegulationandGuidance/GuidanceDocuments/UCM187294.pdf (accessed November 21, 2017).Google Scholar

FDA (2013). Design considerations for pivotal clinical investigations for medical devices – guidance for industry, clinical investigators, institutional review boards and Food and Drug Administration staff. Available at: www.fda.gov/RegulatoryInformation/Guidances/ucm373750.htm (accessed November 21, 2017).Google Scholar

FDA (2017a). QuantX DEN170022 reclassification order. Available at: www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/denovo.cfm?ID=DEN170022 (accessed November 21, 2017).Google Scholar

FDA (2017b). iMRMC software. Available at: https://github.com/DIDSR/iMRMC (accessed November 10, 2017).Google Scholar

Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Trans Patt Anal Mach Intell, 32, 1627–1645.Google Scholar

Fenton, J.J., Abraham, L., Taplin, S.H., Geller, B.M., Carney, P.A., D’Orsi, C. et al. (2011). Effectiveness of computer-aided detection in community mammography practice. J Natl Cancer Inst, 103, 1152–1161.Google Scholar

Fisichella, V.A., Jaderling, F., Horvath, S., Stotzer, P.O., Kilander, A., Bath, M., Hellstrom, M. (2009). Computer-aided detection (CAD) as a second reader using perspective filet view at CT colonography: effect on performance of inexperienced readers. Clin Radiol, 64, 972–982.Google Scholar

Freer, T.W., Ulissey, M.J. (2001). Screening mammography with computer-aided detection: prospective study of 12,860 patients in a community breast center. Radiology, 220, 781–786.Google Scholar

Gallas, B.D., Brown, D.G. (2008). Reader studies for validation of CAD systems. Neural Networks, 21, 387–397.Google Scholar

Gallas, B.D., Chan, H.P., D’Orsi, C.J., Dodd, L.E., Giger, M.L., Gur, D., Krupinski, E.A., Metz, C.E., Myers, K. J., Obuchowski, N.A., Sahiner, B., Toledano, A.Y., Zuley, M.L. (2012). Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol, 19, 463–477.Google Scholar

Gallas, B.D., Pisano, E., Cole, E., Myers, K. (2017). Impact of different study populations on reader behavior and performance metrics: initial results. Proc SPIE Med Imag, 10136.Google Scholar

Gehan, E.A., Freireich, E.J. (1974). Non-randomized controls in cancer clinical trials. N Engl J Med, 290, 198–203.Google Scholar

Gilbert, F.J., Astley, S.M., Gillan, M.G.C., Agbaje, O.F., Wallis, M.G., James, J., Boggis, C.R.M., Duffy, S.W. (2008). Single reading with computer-aided detection for screening mammography. N Engl J Med, 359, 1675–1684.Google Scholar

Godoy, M.C.B., Kim, T.J., White, C.S., Bogoni, L., de Groot, P., Florin, C., Obuchowski, N., Babb, J.S., Salganicoff, M., Naidich, D.P., Anand, V., Park, S., Vlahos, I., Ko, J.P. (2013). Benefit of computer-aided detection analysis for the detection of subsolid and solid lung nodules on thin- and thick-section CT. Am J Roentgenol, 200, 74–83.Google Scholar

Gomez, S.S., Tabanera, M.T., Bolivar, A.V., Miranda, M.S., Mazo, A.B., Diaz, M.R., Miravete, P.M., Asturiano, E.L., Cacho, P.M., Macias, T.D. (2011). Impact of a CAD system in a screen-film mammography screening program: a prospective study. Eur J Radiol, 80, E317–E321.Google Scholar

Goo, J.M., Kim, H.Y., Lee, J. W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T., Goo, J.M., Kim, H.Y., Lee, J.W., Lee, H.J., Lee, C.H., Lee, K.W., Kim, T.J., Lim, K.Y., Park, S.H., Bae, K.T. (2008). Is the computer-aided detection scheme for lung nodule also useful in detecting lung cancer? J Comp Assist Tomogr, 32, 570–575.Google Scholar

Gossmann, A., Sahiner, B., Pezeshk, A. (2018). Test data reuse for evaluation of adaptive machine learning algorithms: over-fitting to a fixed “test” dataset and a potential solution. Proc SPIE Med Imag, 10577Google Scholar

Greenspan, H., van Ginneken, B., Summers, R.M. (2016). Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imag, 35, 1153–1159.Google Scholar

Gromet, M. (2008). Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. Am J Roentgenol, 190, 854–859.Google Scholar

Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P.C., Mega, J.L., Webster, R. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316, 2402–2410.Google Scholar

Gur, D., Rockette, H.E., Armfield, D.R., Blachar, A., Bogan, J.K., Brancatell, G., Britton, C.A., Brown, M.L., Davis, P.L., Ferris, J.V., Fuhrman, C.R., Golla, S.K., Katyal, S., Lacomis, J.M., McCook, B.M., Thaete, F.L., Warfel, T.E. (2003). Prevalence effect in a laboratory environment. Radiology, 228, 10–14.Google Scholar

Gur, D., Sumkin, J.H., Rockette, H.E., Ganott, M.A., Hakim, C., Hardesty, L.A., Poller, W.R., Shah, R., Wallace, L. (2004). Changes in breast cancer detection and mammography recall rates after the introduction of a computer-aided detection system. J Natl Cancer Inst, 96, 185–190.Google Scholar

Gur, D., Bandos, A.I., Fuhrman, C.R., Klym, A.H., King, J.L., Rockette, H.E. (2007). The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol, 14, 49–53.Google Scholar

Hadjiiski, L., Sahiner, B., Helvie, M.A., Chan, H.-P., Roubidoux, M.A., Paramagul, C., Blane, C., Petrick, N., Bailey, J., Klein, K. (2006). Breast masses: computer-aided diagnosis with serial mammograms. Radiology, 240, 343–356.Google Scholar

Hand, D.J., Till, R.J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learn, 45, 171–186.Google Scholar

Hanley, J.A. (1988). The robustness of the “binormal assumptions” used in fitting (ROC) curves. Med Decis Making, 8, 197–203.Google Scholar

Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K., Hirose, T., Nitta, N., Shiraishi, J., Nagatani, Y., Takahashi, M., Murata, K. (2008). Evaluation of computer-aided diagnosis (CAD) software for the detection of lung nodules on multidetector row computed tomography (MDCT): JAFROC study for the improvement in radiologists’ diagnostic accuracy. Acad Radiol, 15, 1505–1512.Google Scholar

Horsch, K., Giger, M.L., Metz, C.E. (2008). Potential effect of different radiologist reporting methods on studies showing benefit of CAD. Acad Radiol, 15, 139–152.Google Scholar

Huo, Z.M., Giger, M.L., Vyborny, C.J., Wolverton, D.E., Schmidt, R.A., Doi, K. (1998). Automated computerized classification of malignant and benign masses on digitized mammograms. Acad Radiol, 5, 155–168.Google Scholar

Hupse, R., Samulski, M., Lobbes, M., den Heeten, A., Imhof-Tas, M.W., Beijerinck, D., Pijnappel, R., Boetes, C., Karssemeijer, N. (2013). Standalone computer-aided detection compared to radiologists’ performance for the detection of mammographic masses. Eur Radiol, 23, 93–100.Google Scholar

ICRU. (2008). Receiver Operating Characteristic Analysis in Medical Imaging. Bethesda, MD: International Commission of Radiation Units and Measurements.Google Scholar

IMDRF SaMD Working Group, Software as a Medical device (SaMD). (2017) Clinical evaluation. Available at: www.imdrf.org/docs/imdrf/final/technical/imdrf-tech-170921-samd-n41-clinical-evaluation.pdf (accessed November 21, 2017).Google Scholar

Jesneck, J.L., Lo, J.Y., Baker, J.A. (2007). Breast mass lesions: computer-aided diagnosis models with mammographic and sonographic descriptors. Radiology, 244, 390–398.Google Scholar

Jiang, Y., Metz, C.E., Nishikawa, R.M. (1996). A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology, 201, 745–750.Google Scholar

Jiang, Y.L., Metz, C.E., Nishikawa, R.M., Schmidt, R.A. (2006). Comparison of independent double readings and computer-aided diagnosis (CAD) for the diagnosis of breast calcifications. Acad Radiol, 13, 84–94.Google Scholar

Kantarjian, H., Yu, P. (2015). Artificial intelligence, big data, and cancer. JAMA Oncol, 1, 573–574.Google Scholar

Karssemeijer, N., Otten, J.D.M., Verbeek, A.L.M., Groenewoud, J.H., de Koning, H.J., Hendriks, J.H.C.L., Holland, R. (2003). Computer-aided detection versus independent double reading of masses on mammograms. Radiology, 227, 192–200.Google Scholar

Kasai, S., Li, F., Shiraishi, J., Doi, K. (2008). Usefulness of computer-aided diagnosis schemes for vertebral fractures and lung nodules on chest radiographs. Am J Roentgenol, 191, 260–265.Google Scholar

Khoo, L.A.L., Taylor, P., Given-Wilson, R.M. (2005). Computer-aided detection in the United Kingdom national breast screening programme: prospective study. Radiology, 237, 444–449.Google Scholar

Kligerman, S., Cai, L., White, C.S. (2013). The effect of computer-aided detection on radiologist performance in the detection of lung cancers previously missed on a chest radiograph. J Thorac Imag, 28, 244–252.Google Scholar

Kooi, T., Litjens, G., van Ginneken, B., Gubern-Merida, A., Sancheza, C.I., Mann, R., den Heeten, A., Karssemeijer, N. (2017). Large scale deep learning for computer aided detection of mammographic lesions. Med Image Analysis, 35, 303–312.Google Scholar

Kosinski, A.S., Barnhart, H.X. (2003). A global sensitivity analysis of performance of a medical diagnostic test when verification bias is present. Stat Med, 22, 2711–2721.Google Scholar

Kotsiantis, S.B., Zaharakis, I.D., Pintelas, P.E. (2006). Machine learning: a review of classification and combining techniques. Artif Intell Rev, 26, 159–190.Google Scholar

Krizhevsky, A., Sutskever, I., Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. In: Pereira, F.C., Burges, J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, Vol. 25. Red Hook, NY: Curran Associates, pp. 1097–1105.Google Scholar

Kumar, V., Gu, Y., Basu, S., Berglund, A., Eschrich, S.A., Schabath, M.B., Forster, K., Aerts, H.J.W.L., Dekker, A., Fenstermacher, D., Goldgof, D.B., Hall, L.O., Lambin, P., Balagurunathan, Y., Gatenby, R.A., Gillies, R.J. (2012). Radiomics: the process and the challenges. Magn Reson Imag, 30, 1234–1248.Google Scholar

Lambin, P., Rios-Velazquez, E., Leijenaar, R., Carvalho, S., van Stiphout, R.G.P.M., Granton, P., Zegers, C.M.L., Gillies, R., Boellard, R., Dekker, A., Aerts, H.J.W.L. (2012). Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer, 48, 441–446.Google Scholar

LeCun, Y., Bengio, Y., Hinton, G. (2015). Deep learning. Nature, 521, 436–444.Google Scholar

Li, F., Arimura, H., Suzuki, K., Shiraishi, J., Li, Q., Abe, H., Engelmann, R., Sone, S., MacMahon, H., Doi, K. (2005). Computer-aided detection of peripheral lung cancers missed at CT: ROC analyses without and with localization. Radiology, 237, 684–690.Google Scholar

Li, Q., Gavrielides, M.A., Sahiner, B., Myers, K.J., Zeng, R., Petrick, N. (2015). Statistical analysis of lung nodule volume measurements with CT in a large-scale phantom study. Med Phys, 42, 3932–3947.Google Scholar

Metz, C. E., Pan, X. (1999). “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psych, 43, 1–33.Google Scholar

Metz, C.E., Wang, P.L., Kronman, H.B. (1984). A new approach for testing the significance for differences between ROC curves measured from correlated data. In: Deconinck, F. (ed.) Information Processing in Medical Imaging. The Hague, The Netherlands: Martinus Nijhoff, pp. 432–445.Google Scholar

Miller, D.P., O’Shaughnessy, K.F., Wood, S.A., Castellino, R.A. (2004). Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions. Proc SPIE Med Imag, 5372, 173–184.Google Scholar

Moin, P., Deshpande, R., Sayre, J., Messer, E., Gupte, S., Romsdahl, H., Hasegawa, A., Liu, B.J. (2011). An observer study for a computer-aided reading protocol (CARP) in the screening environment for digital mammography. Acad Radiol, 18, 1420–1429.Google Scholar

Morton, M.J., Whaley, D.H., Brandt, K.R., Amrami, K.K. (2006). Screening mammograms: interpretation with computer-aided detection – prospective evaluation. Radiology, 239, 375–383.Google Scholar

Mossman, D. (1999). Three-way ROCs. Med Decis Making, 19, 78–89.Google Scholar

Naaktgeboren, C.A., de Groot, J.A.H., Rutjes, A.W.S., Bossuyt, P.M.M., Reitsma, J.B., Moons, K.G.M. (2016). Anticipating missing reference standard data when planning diagnostic accuracy studies. Br Med J, 352.Google Scholar

Nakas, C.T., Yiannoutsos, C.T. (2004). Ordered multiple-class ROC analysis with continuous measurements. Stat Med, 23, 3437–3449.Google Scholar

Neuhaus, J.M., Kalbfleisch, J.D. (1998). Between- and within-cluster covariate effects in the analysis of clustered data. Biometrics, 54, 638–645.Google Scholar

Nietert, P.J., Ravenel, J.G., Taylor, K.K., Silvestri, G.A. (2011). Influence of nodule detection software on radiologists’ confidence in identifying pulmonary nodules with computed tomography. J Thorac Imag, 26, 48–53.Google Scholar

Nishikawa, R.M., Pesce, L.L. (2009). Computer-aided detection evaluation methods are not created equal. Radiology, 251, 634–636.Google Scholar

Obuchowski, N.A. (2005). ROC analysis [see comment]. Am J Roentgenol, 184, 364–72.Google Scholar

Obuchowski, N.A., Gallas, B.D., Hillis, S.L. (2012). Multi-reader ROC studies with split-plot designs: a comparison of statistical methods. Acad Radiol, 19, 1508–1517.Google Scholar

O’Keefe, R.M., O’Leary, D.E. (1993). Expert system verification and validation: a survey and tutorial. Artif Intell Rev, 7, 3–42.Google Scholar

Petrick, N., Haider, M., Summers, R.M., Yeshwant, S.C., Brown, L., Iuliano, E.M., Louie, A., Choi, J.R., Pickhardt, P.J. (2008). CT colonography with computer-aided detection as a second reader: observer performance study. Radiology, 246, 148–156.Google Scholar

Petrick, N., Sahiner, B., Armato, S.G., III, Bert, A., Correale, L., Delsanto, S., Freedman, M.T., Fryd, D., Gur, D., Hadjiiski, L., Huo, Z., Jiang, Y., Morra, L., Paquerault, S., Raykar, V., Salganicoff, M., Samuelson, F., Summers, R.M., Tourassi, G., Yoshida, H., Zheng, B., Zhou, C., Chan, H.-P. (2013). Evaluation of computer-aided detection and diagnosis systems. Med Phys, 40, 087001-1, 087001-17.Google Scholar

Pezeshk, A., Sahiner, B., Zeng, R.P., Wunderlich, A., Chen, W.J., Petrick, N. (2015). Seamless insertion of pulmonary nodules in chest CT images. IEEE Trans Biomed Eng, 62, 2812–2827.Google Scholar

Pezeshk, A., Petrick, N., Chen, W.J., Sahiner, B. (2017). Seamless lesion insertion for data augmentation in CAD training. IEEE Trans Med Imag, 36, 1005–1015.Google Scholar

Pisano, E.D., Gatsonis, C., Hendrick, E., Yaffe, M. (2005). Diagnostic performance of digital versus film mammography for breast-cancer screening. N Engl J Med, 353, 1773–1783.Google Scholar

Popescu, L.M. (2011). Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve. Med Phys, 38, 5690–5702.Google Scholar

Rao, J.N.K., Scott, A.J. (1992). A simple method for the analysis of clustered binary data. Biometrics, 48, 577–585.Google Scholar

Regge, D., Della Monica, P., Galatola, G., Laudi, C., Zambon, A., Correale, L., Asnaghi, R., Barbaro, B., Borghi, C., Campanella, D., Cassinis, M.C., Ferrari, R., Ferraris, A., Golfieri, R., Hassan, C., Iafrate, F., Iussich, G., Laghi, A., Massara, R., Neri, E., Sali, L., Venturini, S., Gandini, G. (2013). Efficacy of computer-aided detection as a second reader for 6–9-mm lesions at CT colonography: multicenter prospective trial. Radiology, 266, 168–176.Google Scholar

Sahiner, B., Chan, H.-P., Petrick, N., Wagner, R.F., Hadjiiski, L.M. (2000). Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample size. Med Phys, 27, 1509–1522.Google Scholar

Sahiner, B., Chan, H.-P., Roubidoux, M.A., Helvie, M.A., Hadjiiski, L.M., Ramachandran, A., Paramagul, C., LeCarpentier, G.L., Nees, A., Blane, C. (2004). Computerized characterization of breast masses on three-dimensional ultrasound volumes. Med Phys, 31, 744–754.Google Scholar

Sahiner, B., Chan, H.P., Roubidoux, M.A., Hadjiiski, L.M., Helvie, M.A., Paramagul, C., Bailey, J., Nees, A., Blane, C. (2007). Computer-aided diagnosis of malignant and benign breast masses in 3D ultrasound volumes: effect on radiologists’ accuracy. Radiology, 242, 716–724.Google Scholar

Sahiner, B., Chan, H.P., Hadjiiski, L.M. (2008). Performance analysis of three-class classifiers: properties of a 3-D ROC surface and the normalized volume under the surface for the ideal observer. IEEE Trans Med Imag, 27, 215–227.Google Scholar

Sahiner, B., Chan, H.-P., Hadjiiski, L.M., Cascade, P.N., Kazerooni, E.A., Chughtai, A.R., Poopat, C., Song, T., Frank, L., Stojanovska, J. (2009). Effect of CAD on radiologists’ detection of lung nodules on thoracic CT scans: analysis of an observer performance study by nodule size. Acad Radiol, 16, 1518–1530.Google Scholar

Samuelson, F.W., Abbey, C.K. (2017). The reproducibility of changes in diagnostic figures of merit across laboratory and clinical imaging reader studies. Acad Radiol, 24, 1436–1446.Google Scholar

Samuelson, F.W., Petrick, N. (2006). Comparing image detection algorithms using resampling. 3rd IEEE International Symposium on Biomedical Imaging: Nano to Macro, 1312–1315.Google Scholar

Schalekamp, S., van Ginneken, B., Heggelman, B.G.F., Imhof-Tas, M., Somers, I., Brink, M., Spee, M., Schaefer-Prokop, C.M., Karssemeijer, N. (2014). New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs. Br J Radiol, 87, 1036.Google Scholar

Scurfield, B.K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability. J Math Psych, 40, 253–269.Google Scholar

Shimauchi, A., Giger, M.L., Bhooshan, N., Lan, L., Pesce, L.L., Lee, J.K., Abe, H., Newstead, G.M. (2011). Evaluation of clinical breast MR imaging performed with prototype computer-aided diagnosis breast MR imaging workstation: reader study. Radiology, 258, 696–704.Google Scholar

Stages of Breast Cancer. (2017). Available from: www.breastcancer.org/symptoms/diagnosis/staging (accessed November 10, 2017).Google Scholar

Starr, S.J., Metz, C.E., Lusted, L.B., Goodenough, D.J. (1975). Visual detection and localization of radiographic images. Radiology, 116, 533–538.Google Scholar

Stepan-Buksakowska, I.L., Accurso, J.M., Diehn, F.E., Huston, J., Kaufmann, T.J., Luetmer, P.H., Wood, C.P., Yang, X., Blezek, D.J., Carter, R., Hagen, C., Horinek, D., Hejcl, A., Rocek, M., Erickson, B.J. (2014). Computer-aided diagnosis improves detection of small intracranial aneurysms on MRA in a clinical setting. Am J Neuroradiol, 35, 1897–1902.Google Scholar

Swensson, R.G. (1996). Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys, 23, 1709–1725.Google Scholar

Thrall, J.H. (2016). Trends and developments shaping the future of diagnostic medical imaging: 2015 annual oration in diagnostic radiology. Radiology, 279, 660–666.Google Scholar

Uchiyama, Y., Asano, T., Kato, H., Hara, T., Kanematsu, M., Hoshi, H., Iwama, T., Fujita, H. (2012). Computer-aided diagnosis for detection of lacunar infarcts on MR images: ROC analysis of radiologists’ performance. J Digit Imag, 25, 497–503.Google Scholar

University of Chicago. (2017). Metz ROC software. Available at: http://metz-roc.uchicago.edu/MetzROC (accessed November 10, 2017).Google Scholar

University of Iowa. (2017). ROC software. Available at: http://perception.radiology.uiowa.edu/ (accessed November 10, 2017).Google Scholar

van Beek, E.J.R., Mullan, B., Thompson, B. (2008). Evaluation of a real-time interactive pulmonary nodule analysis system on chest digital radiographic images: a prospective study. Acad Radiol, 15, 571–575.Google Scholar

Wagner, R.F., Beam, C.A., Beiden, S.V. (2004). Reader variability in mammography and its implications for expected utility over the population of readers and cases. Med Decis Making, 24, 561–572.Google Scholar

Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.Google Scholar

Way, T., Chan, H.P., Hadjiiski, L., Sahiner, B., Chughtai, A., Song, T.K., Poopat, C., Stojanovska, J., Frank, L., Attili, A., Bogot, N., Cascade, P.N., Kazerooni, E.A. (2010). Computer-aided diagnosis of lung nodules on CT scans: ROC study of its effect on radiologists’ performance. Acad Radiol, 17, 323–332.Google Scholar

Wei, J., Chan, H.P., Sahiner, B., Zhou, C., Hadjiiski, L.M., Roubidoux, M.A., Helvie, M.A. (2009). Computer-aided detection of breast masses on mammograms: dual system approach with two-view analysis. Med Phys, 36, 4451–4460.Google Scholar

Xin, H., Metz, C.E., Tsui, B.M.W., Links, J.M., Frey, E.C. (2006). Three-class ROC analysis – a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imag, 25, 571–581.Google Scholar

Yanagawa, M., Honda, O., Yoshida, S., Ono, Y., Inoue, A., Daimon, T., Sumikawa, H., Mihara, N., Johkoh, T., Tomiyama, N., Nakamura, H. (2009). Commercially available computer-aided detection system for pulmonary nodules on thin-section images using 64 detectors-row CT: preliminary study of 48 cases. Acad Radiol, 16, 924–933.Google Scholar

Yoon, H.J., Zheng, B., Sahiner, B., Chakraborty, D.P. (2007). Evaluating computer-aided detection algorithms. Med Phys, 34, 2024–2038.Google Scholar

Zeiler, M.D., Fergus, R. (2014). Visualizing and understanding convolutional networks. In: Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I. Cham: Springer International Publishing, pp. 818–833.Google Scholar

Zweig, M.H., Campbell, G. (1993). Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem, 39, 561–577.Google Scholar

Abramson, R.G., Su, P.F., Shyr, Y. (2012). Quantitative metrics in clinical radiology reporting: a snapshot perspective from a single mixed academic-community practice. Magn Reson Imag, 30(9), 1357–1366.Google Scholar

Birkelo, C.C., Chamberlain, W.E., Phelps, P.S., et al. (1947). Tuberculosis case finding: a comparison of the effectiveness of various roentgenographic and photofluorographic methods. JAMA, 133, 359–366.Google Scholar

Jaffe, T.A., Wickersham, N.W., Sullivan, D.C. (2010). Quantitative imaging in oncology patients: Part 2, oncologists’ opinions and expectations at major U.S. cancer centers. AJR Am J Roentgenol, 195(1), W19-W30.Google Scholar

Joint Committee for Guides in Metrology Working Group 1. (2008). Evaluation of measurement data – Supplement 1 to the Guide to the expression of uncertainty in measurement. Available at: www.bipm.org/utils/common/documents/jcgm/JCGM_100_2008_E.pdf (accessed December 21, 2017).Google Scholar

Kessler, L.G., Barnhart, H.X., Buckler, A.J., et al. (2015). The emerging science of quantitative imaging biomarkers terminology and definitions for scientific studies and regulatory submissions. Stat Methods Med Res, 24(1), 9–26.Google Scholar

Macari, M., Megibow, A.J. (2011). Focal cystic pancreatic lesions: variability in radiologists’ recommendations for follow-up imaging. Radiology, 259(1), 20–23.Google Scholar

Quantitative Imaging Biomarkers Alliance. Available online at: http://rsna.org/qiba (accessed December 21, 2017).Google Scholar

Raunig, D.L., McShane, L.M., Pennello, G., et al. (2015). Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment. Stat Methods Med Res, 24(1), 27–67.Google Scholar

Sullivan, D.C., Schwartz, L.H., Zhao, B. (2013). The imaging viewpoint: how imaging affects determination of progression-free survival. Clin Cancer Res, 19(10), 2621–2628.Google Scholar

Sullivan, D.C., Obuchowski, N.A., Kessler, L.G., Raunig, D.L., Gatsonis, C., Huang, E.P. et al. (2015). RSNA-QIBA metrology working group. Metrology standards for quantitative imaging biomarkers. Radiology, 277(3), 813–825.Google Scholar

Zhao, B., Tan, Y., Bell, D.J., Marley, S.E., Guo, P., Mann, H. et al. (2013). Exploring intra- and inter-reader variability in uni-dimensional, bi-dimensional, and volumetric measurements of solid tumors on CT scans reconstructed at different slice intervals. Eur J Radiol, 82(6), 959–968.Google Scholar