Hostname: page-component-745bb68f8f-d8cs5 Total loading time: 0 Render date: 2025-01-08T09:29:02.308Z Has data issue: false hasContentIssue false
  • English
  • Français

A More Powerful Method for Testing for Agreement between a Judge and a known Standard

Published online by Cambridge University Press:  01 January 2025

D. D. Wackerly  and
D. H. Robinson
D. D. Wackerly*
Affiliation:
University of Florida
D. H. Robinson
Affiliation:
University of Florida
*
Requests for reprints should be sent to D. D. Wackerly, Department of Statistics, Nuclear Sciences Center, University of Florida, Gainesville, Florida 32611.
Get access Rights & Permissions [Opens in a new window]

Abstract

We assume that a judge's task is to categorize each of N subjects into one of r known classes. The design of primary interest is employed if the judge is presented with s groups, each containing r subjects, such that each group of size r consists of exactly one subject of each of the r types. The probability distribution for the total number of correct choices is developed and used to test the null hypothesis that the judge is “guessing” in favor of the alternative that he or she is operating at a better than chance level. The power of the procedure is shown to be superior to two other procedures which appear in the literature.

Keywords

nominal datapower comparisonscomputer algorithmexact resultslarge sample approximation
Type
Original Paper
Information
Psychometrika , Volume 48 , Issue 2 , June 1983 , pp. 183 - 193
Copyright
Copyright © 1983 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The authors are grateful for the suggestions of the referees and for computer funding provided by the Northeast Regional Data Center at the University of Florida.

References

Birnbaum, A. Combining independent tests of significance. Journal of the American Statistical Association, 1954, 49, 559–575.Google Scholar
Blume, G. E. A comparative study of dreams and related fantasies. Unpublished doctoral dissertation, University of Florida, 1977.Google Scholar
Fisher, R. A. The design of experiments 9th ed., New York: Hafner Press, 1971.Google Scholar
Gridgeman, N. T. The lady tasting tea and allied topics. Journal of the American Statistical Association, 1959, 54, 776–783.CrossRefGoogle Scholar
Lancaster, H. O. The combination of probabilities arising in discrete distributions. Biometrika, 1949, 36, 370–382.CrossRefGoogle ScholarPubMed
Parzen, E. Modern probability theory and its applications, New York: John Wiley & Sons, 1960.CrossRefGoogle Scholar
Tocher, K. D. Extensions of the Neyman-Pearson theory to tests of discontinuous variates. Biometrika, 1950, 37, 130–144.CrossRefGoogle ScholarPubMed
Wackerly, D. D., McClave, J. T., & Rao, P. V. Measuring nominal scale agreement between a judge and a known standard. Psychometrika, 1978, 43, 213–223.CrossRefGoogle Scholar
Wallis, W. A. Compounding probabilities from independent significance tests. Econometrica, 1942, 10, 229–248.CrossRefGoogle Scholar