Hostname: page-component-cd9895bd7-fscjk Total loading time: 0 Render date: 2025-01-05T15:37:58.172Z Has data issue: false hasContentIssue false
  • English
  • Français

An Ordinal Coefficient of Relational Agreement for Multiple Judges

Published online by Cambridge University Press:  01 January 2025

Robert F. Fagot
Robert F. Fagot*
Affiliation:
University of Oregon
*
Send requests for reprints to Robert F. Fagot, Department of Psychology, University of Oregon, Eugene, OR 97403-1227.
Get access Rights & Permissions [Opens in a new window]

Abstract

In a recent article, Fagot proposed a generalized family of coefficients of relational agreement for multiple judges, focusing on the concept of empirically meaningful relationships. In this paper an ordinal coefficient of relational agreement, based on ranking data, is presented as a special case of the generalized family. It is shown that the proposed ordinal coefficient encompasses other ordinal coefficients, such as the Kendall coefficient of concordance, the average Spearman rank-order coefficient, and intraclass correlation based on ranks. It is also shown that the Kendall coefficient of concordance, corrected for chance agreement, is equivalent to the ordinal coefficient proposed in this paper.

Keywords

ordinal agreement coefficientsrank-order correlationconcordanceintraclass correlationeffect sizemeaningfulness theory
Type
Original Paper
Information
Psychometrika , Volume 59 , Issue 2 , June 1994 , pp. 241 - 251
Copyright
Copyright © 1994 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.CrossRefGoogle Scholar
Ehrenberg, A. S. C. (1952). On sampling from a population of rankers. Biometrika, 39, 82–87.CrossRefGoogle Scholar
Fagot, R. F. (1993). A generalized family of coefficients of relational agreement for numerical scales. Psychometrika, 58, 357–370.CrossRefGoogle Scholar
Fisher, R. A. (1924). The influence of rainfall on the yield of wheat at Rothamsted. Philosophical Transactions of the Royal Society of London, Series B, 213, 89–124.Google Scholar
Green, B. F. (1956). A method of scalogram analysis using summary statistics. Psychometrika, 21, 79–88.CrossRefGoogle Scholar
Haggard, E. A. (1958). Intraclass correlation and the analysis of variance, New York: Dryden Press.Google Scholar
Kendall, M., Gibbons, J. D. (1990). Rank correlation methods, New York: Oxford University Press.Google Scholar
Shrout, P. E., Fleiss, J. E. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.CrossRefGoogle ScholarPubMed
Siegel, S., Castellan, N. Jr. (1988). Nonparametric statistics for the behavioral sciences 2nd ed.,, New York: McGraw-Hill.Google Scholar
Stine, W. W. (1989). Interobserver relational agreement. Psychological Bulletin, 106, 341–347.CrossRefGoogle Scholar
Tanner, M. A., Young, M. A. (1985). Modeling ordinal scale disagreement. Psychological Bulletin, 98, 408–415.CrossRefGoogle ScholarPubMed
Zegers, F. E. (1986). A family of chance-corrected association coefficients for metric scales. Psychometrika, 51, 559–562.CrossRefGoogle Scholar
Zegers, F. E., ten Berge, J. M. F. (1985). A family of association coefficients for metric scales. Psychometrika, 50, 17–24.CrossRefGoogle Scholar