Hostname: page-component-745bb68f8f-hvd4g Total loading time: 0 Render date: 2025-01-08T10:20:21.781Z Has data issue: false hasContentIssue false
  • English
  • Français

Estimating the Accuracy of Dichotomous Judgments

Published online by Cambridge University Press:  01 January 2025

Joseph L. Fleiss
Joseph L. Fleiss*
Affiliation:
Biometrics Research, New York State Psychiatric Institute, and Columbia University
Get access Rights & Permissions [Opens in a new window]

Abstract

A reliability study is assumed to be carried out with each of a number of observers making a dichotomous judgment concerning each of a sample of subjects. A nonparametric model is proposed for the errors underlying such judgments, and conditions are given under which Cochran's Q statistic is valid for testing the hypothesis of no systematic differences among the judgments of the different observers. Inferences concerning the probabilities of error are shown to be possible in terms of the intraclass correlation coefficient. A numerical example is given.

Type
Original Paper
Information
Psychometrika , Volume 30 , Issue 4 , December 1965 , pp. 469 - 479
Copyright
Copyright © 1965 Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

This research was supported in part by NIMH grant MH-03546. I am indebted to Dr. E. I. Burdock, Associate Research Scientist, Biometrics Research, for his valuable suggestions and criticisms.

References

Burdock, E. I., Fleiss, J. L., and Hardesty, A. S. A new view of inter-observer agreement. Personnel Psychol., 1963, 16, 373–384.CrossRefGoogle Scholar
Cochran, W. G. The comparison of percentages in matched samples. Biometrika, 1950, 37, 256–266.CrossRefGoogle ScholarPubMed
Ebel, R. L. Estimation of the reliability of ratings. Psychometrika, 1951, 16, 407–424.CrossRefGoogle Scholar
Gulliksen, H. Theory of mental tests, New York: John Wiley, 1950.CrossRefGoogle Scholar
Maxwell, A. E. Analysing qualitative data, London: Methuen, 1961.Google Scholar
Spitzer, R. L., Fleiss, J. L., Kernohan, W., Lee, J., and Baldwin, I. T. The Mental Status Schedule: comparing Kentucky and New York schizophrenics. Arch. gen. Psychiat., 1965, 12, 448–455.CrossRefGoogle Scholar
Walsh, J. E. Concerning the effect of intraclass correlation on certain significance tests. Ann. math. Statist., 1947, 18, 88–96.Google Scholar