1. Introduction

The most popular measures of inter-rater agreement involve correction for chance agreement. These can be written on the form

(1.1) $\begin{array}{c}\frac{p_a-p_{\mathrm{ca}}}{1-p_{\mathrm{ca}}}=1-\frac{p_d}{p_{\mathrm{cd}}},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{p_{a}-p_{ca}}{1-p_{ca}}=1-\frac{p_{d}}{p_{cd}}, \end{aligned}$$\end{document}

where $p_a$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{a}$$\end{document} ( $p_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{d}$$\end{document} ) is the percentage agreement (disagreement) between the raters and $p_{\mathrm{ca}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{ca}$$\end{document} ( $p_{\mathrm{cd}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{cd}$$\end{document} ) is the chance agreement (disagreement) between the raters. Such measures are frequently called chance-corrected measures of agreement. Well-known examples of coefficients in this class are Cohen’s (Reference Cohen1960) kappa and its weighted variant (Reference Cohen1968), its multi-rater variant Conger’s kappa (Conger, Reference Conger1980; Light, Reference Light1971), Krippendorff’s (Reference Krippendorff1970) alpha, Scott’s (Reference Scott1955) pi, and Fleiss’ (Reference Fleiss1971) kappa. Some of these coefficients are defined only for two raters. The rest are defined in a pairwise manner, in the sense that they measure agreement between two raters at a time. However, not every proposed measure of agreement is defined on pairs of raters. The most famous is Hubert’s kappa (Reference Hubert1977), which was recently studied in detail by Martín Andrés and Álvarez Hernández (Reference Martín Andrés and Álvarez Hernández2020). Other agreement coefficients include the $A{C}_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$AC_{1}$$\end{document} coefficient (Gwet, Reference Gwet2008), the recent coefficient of van Oest (Reference van Oest2019), and a multitude of intraclass correlation coefficients (Gwet, Reference Gwet2014).

There is no consensus on how multi-rater agreement coefficients should be defined. Broadly speaking, two options are considered: pairwise coefficients and consensus coefficients. The pairwise coefficients measure the agreement between pairs of raters (Conger, Reference Conger1980), while the consensus coefficients measure the simultaneous agreement between all raters. In particular, consensus coefficients support the notion that “agreement occurs if and only if all raters agree on the categorization of an object” (Hubert, Reference Hubert1977). Both pairwise and consensus-based definitions of agreement are variants of g-wise measures of agreement (Conger, Reference Conger1980), where agreement is measured among g-tuples of raters. The case where $2<g<R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2<g<R$$\end{document} has received little attention in the literature (Warrens, Reference Warrens2012), and non-trivial ways to measure agreement are hard to invent in this case. However, we introduce a promising and general framework for handling g-wise measures of agreement based on the concept of Fréchet variances (Dubey and Müller, Reference Dubey and Müller2019). The Fréchet variances generalize the variance and the measures of agreement based on them generalize the nominal, linearly weighted, and quadratically weighted pairwise measures of agreement in a natural way. They are easily interpretable, as you measure how much the raters disagree with the generalized mean rater and then adjust for chance. For nominal data in particular, they measure how many raters disagree with the modal rater, with a resulting agreement measure less extreme than Hubert’s kappa.

We need inferential theory for the g-wise agreement coefficients to make them useful. Much work has been done on inference for agreement coefficients, but, to our knowledge, inference for g-wise agreement coefficients has yet to be studied. Assuming multivariate normality of the ratings, Lin (Reference Lin1989, Section 3) derived the asymptotic distribution of Cohen’s kappa with quadratic weights. Fleiss (Reference Fleiss1971) introduced a formula for the standard error of Fleiss’s kappa, but later showed that it was incorrect. Using the properties of the multinomial distribution and the delta method, Schouten (Reference Schouten1980) found the asymptotic variance of the weighted Fleiss’s kappa in the case when the number of categories is finite. Almost forty years later, Gwet (Reference Gwet2021) found a consistent estimator of the variance for the unweighted Fleiss’s kappa. We extend these results to the weighted g-wise Fleiss’s kappa for any number of categories below. In addition, we mention that bootstrap inference for Fleiss’s kappa and Krippendorff’s alpha was studied by Zapf et al. (Reference Zapf, Castell, Morawietz and Karch2016).

We begin the paper by providing the definitions of two kinds of chance-corrected agreement coefficients. Then, in Sect. 2, we establish connections between the multi-rater Cohen’s kappa, Fleiss’s kappa, Conger’s kappa, Krippendorff’s alpha, and Hubert’s kappa. We restrict ourselves to the context where every rater rates every item. In Sect. 3, we discuss the Fréchet variances mentioned above. Then we spell out the basic limit theory for this class agreement coefficients in Sect. 4, extending the results of Schouten (Reference Schouten1980), Schouten (Reference Schouten1982), and O’Connell and Dobson (Reference O’Connell and Dobson1984) to vector-valued items and g-wise coefficients. We do this using the theory of U-statistics (Lee, Reference Lee2019), but there are other ways to arrive at the same results. Then, in Sect. 5, we provide practical recommendations regarding the choice of confidence interval, obtained by comparing three confidence interval constructions: basic, arcsine transformed, and Fisher transformed. Using a simulation study, we find that the arcsine and Fisher intervals outperform the basic interval when n is small.

2. Measures of Agreement

Let $d(x_1,\dots ,x_g)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d(x_{1},\ldots ,x_{g})$$\end{document} be a disagreement function, a positive and symmetric function of g arguments that equals 0 when all $x_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{i}$$\end{document} s are equal, i.e., $d(x,\dots ,x)=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d(x,\ldots ,x)=0$$\end{document} . The disagreement function quantifies the disagreement between the ratings $x_1,\dots ,x_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{1},\ldots ,x_{g}$$\end{document} , where 0 is understood as complete agreement.

Most disagreement functions take two arguments. While there are infinitely many disagreement functions, the best-known belong to the class of $l_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{p}$$\end{document} quasi-norms, $p=0,1,2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p=0,1,2$$\end{document} , potentially raised to the pth power. The $l_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{p}$$\end{document} quasi-norms, $p\in [0,\infty ]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\in [0,\infty ]$$\end{document} in $R^k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbb {R}^{k}$$\end{document} are defined as

(2.1) $\begin{array}{c}{\Vert x\Vert }_p={\left(\sum _{i=1}^k,{\left|x_i\right|}^p\right)}^{1/p}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Vert x\Vert _{p}=\left( \sum _{i=1}^{k}|x_{i}|^{p}\right) ^{1/p}. \end{aligned}$$\end{document}

Here ${\left|\right|x\left|\right|}_0={\sum }_{i=1}^{k}1[x_i\ne 0]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$||x||_{0}=\sum _{i=1}^{k}1[x_{i}\ne 0]$$\end{document} and ${\left|\right|x\left|\right|}_{\infty }={sup}_i\left|x_i\right|$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$||x||_{\infty }=\sup _i |x_{i}|$$\end{document} , as can be verified by taking the limit of ${\left|\right|x\left|\right|}_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$||x||_{p}$$\end{document} as $p\to 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\rightarrow 0$$\end{document} and $p\to \infty$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\rightarrow \infty $$\end{document} , respectively. It is well known that ${\left|\right|x\left|\right|}_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$||x||_{p}$$\end{document} are proper norms if and only if $p\ge 1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\ge 1$$\end{document} , as the triangle inequality is violated when $1>p\ge 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1>p\ge 0$$\end{document} .

Now define the disagreement functions $d_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{p}$$\end{document} as the $l_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{p}$$\end{document} quasi-norm evaluated in $x_1-x_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{1}-x_{2}$$\end{document} , i.e.,

(2.2) $\begin{array}{c}{d}_p(x_1,x_2)=\left|\right|x_1-x_2{\left|\right|}_p.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} d_{p}(x_{1},x_{2})=||x_{1}-x_{2}||_{p}. \end{aligned}$$\end{document}

In the case of scalar values, $d_0(x_1,x_2)=1[x_1\ne x_2]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{0}(x_{1},x_{2})=1[x_{1}\ne x_{2}]$$\end{document} is known as the nominal disagreement function. For $p=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p=1$$\end{document} , the $l_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{p}$$\end{document} norm equals $d_1(x_1,x_2)=|x_1-x_2|$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{1}(x_{1},x_{2})=|x_{1}-x_{2}|$$\end{document} , which is known as the absolute value disagreement function (and sometimes the linear disagreement function). The quadratic disagreement function is $d_2^2(x_1,x_2)={(x_1-x_2)}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{2}^{2}(x_{1},x_{2})=(x_{1}-x_{2})^{2}$$\end{document} . Vector-valued variants of $d_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{p}$$\end{document} and $d_p^p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{p}^{p}$$\end{document} are much less common, but have been used by, e.g., Berry et al. (Reference Berry, Johnston and Mielke2008).

When the dimension of the disagreement function d is not equal to 2, we are mostly interested in the case where its dimension equals the number of raters R. In this case, the disagreement functions often measure the degree of consensus among the raters, with 0 reflecting complete consensus. The most obvious choice is the Hubert disagreement function,

(2.3) $\begin{array}{c}d(x_1,\dots ,x_g)=1-1[x_1=\cdots =x_g]\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} d(x_{1},\ldots ,x_{g})=1-1[x_{1}=\cdots =x_{g}] \end{aligned}$$\end{document}

which equals 0 if and only if every rater agrees on a rating. The disagreement function is employed in Hubert’s kappa (Hubert, Reference Hubert1977).

We present our results in terms of disagreement functions instead of the more popular agreement functions (i.e., positive symmetric functions bounded by 1 where 1 signifies maximal agreement, sometimes with the additional assumption that $a\ge 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a\ge 0$$\end{document} ). We do this mainly for mathematical convenience. Agreement functions and disagreement functions are closely related, for if a is an agreement function, then $d=1-a$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=1-a$$\end{document} is a disagreement function. Our results could have been framed in terms of agreement functions instead, though with some loss of generality. See Appendix (Sect. 6) for a short discussion.

Our results and definitions are framed in the following setup. Let R be the number of raters and n be the number of items rated. Moreover, let F be a fixed multivariate distribution function F so that all rating vectors $X_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{X}_{i}$$\end{document} are sampled independently from F. In symbols,

(2.4) $\begin{array}{c}{X}_1,X_2,\dots ,X_n\stackrel{\mathrm{iid}}{\sim }F.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \varvec{X}_{1},\varvec{X}_{2},\ldots ,\varvec{X}_{n} {\mathop {\sim }\limits ^{iid}} F. \end{aligned}$$\end{document}

There are no restrictions on the rating vector components $X_{\mathrm{ir}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{X}_{ir}$$\end{document} . They can be, e.g., categorical, real numbers, or vectors.

Equation (2.4) implies that every item is rated by exactly the same number of raters, which we refer to as the rectangular design assumption. The assumption is common in the literature,Footnote 1 but far from universal. It can be relaxed, but it is strictly required for the limit results. We sketch how to loosen it in Appendix (Sect. 6), but we have made no attempts at an inferential theory for non-rectangular designs.

There are two important special cases covered by equation (2.4). First, in the case of fixed raters, the same set of ordered raters rate every item. Having fixed raters is common in applications of Cohen’s kappa, Conger’s kappa, and the concordance correlation coefficient.Footnote 2 Having fixed raters ensures that F does not vary across different rating vectors, but F could potentially vary with the ratings when the raters are not fixed, provided we do not make further assumptions. And that leads us to the second case, that of exchangeable ratings given the item. Here, the rater identities do not affect the ratings given. The raters may be different for each item, but the distribution F will still be fixed. Exchangeable ratings occur when the ratings are identically distributed conditional on the item rated. Exchangeable ratings is an implicit assumption underlying most applications of Fleiss’ kappa, e.g., that of Fleiss (Reference Fleiss1971). In this case, the marginal distributions for all raters will be equal, which implies that the population value of the generalized Fleiss kappa equals the population value of the generalized Cohen’s kappa, both defined below. However, the sample Fleiss’s kappa is the preferred sample estimator, as it is invariant under changes of the raters’ identities.

We intend to collect the kappas of Cohen, Fleiss, Conger, Hubert, and so on, into a coherent framework of g-wise agreement coefficients. To do this, we will have to define some quantities. Let $x_i=(x_{i1},x_{i2},\dots ,x_{\mathrm{iR}})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{i}=(x_{i1},x_{i2},\ldots ,x_{iR})$$\end{document} be an R-dimensional vector of observed ratings, and recall that g is the dimension of our disagreement function d. The following definitions are natural population counterparts of sample definitions prevalent in the agreement literature.

1. (i) The disagreement at $x_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{1}$$\end{document} , as measured by d. The purpose of this quantity is to translate an arbitrary g-dimensional disagreement function d into a disagreement function taking an R-dimensional vector $x_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{1}$$\end{document} as input. It is defined as

   (2.5) $\begin{array}{c}{D}_d\left(x_1\right)={\left(\begin{array}{c}R\\ g\end{array}\right)}^{-1}\sum _{r_1,\dots ,r_g}d(x_{1r_1},\dots ,x_{1r_g}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{d}(\varvec{x}_{1})=\left( {\begin{array}{c}R\\ g\end{array}}\right) ^{-1} \sum _{r_{1},\ldots ,r_{g}}d(\varvec{x}_{1r_{1}},\ldots , \varvec{x}_{1r_{g}}), \end{aligned}$$\end{document}
   where the sum runs over all g-dimensional subsets of $\{1,\dots ,R\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{1,\ldots ,R\}$$\end{document} with order ignored, i.e., the g-combinations of R. The expression is simplified when $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} , as $D_d\left(x_1\right)=d(x_{11},\dots ,x_{1R})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{d}(\varvec{x}_{1})=d(\varvec{x}_{11},\ldots ,\varvec{x}_{1R})$$\end{document} in this case. To gain some intuition about this quantity, suppose that $g=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=2$$\end{document} , that $x_1,x_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{1},x_{2}$$\end{document} are scalars, and consider the nominal disagreement function $d_0(x_1,x_2)=1[x_1\ne x_2]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{0}(x_{1},x_{2})=1[x_{1}\ne x_{2}]$$\end{document} . Then $D_d\left(x_1\right)=2R^{-1}{(R-1)}^{-1}{\sum }_{r_1>r_2}1[x_{1r_1}\ne x_{1r_2}]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{d}(\varvec{x}_{1})=2R^{-1}(R-1)^{-1}\sum _{r_{1}>r_{2}}1[x_{1r_{1}}\ne x_{1r_{2}}]$$\end{document} is the percentage of times two distinct raters disagree on their rating.

2. (ii) The Cohen-type chance disagreement at $x_1,\dots ,x_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{1},\ldots ,\varvec{x}_{g}$$\end{document} , so called to differentiate it from the Fleiss-type chance disagreement. It is similar to the disagreement at $x_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{1}$$\end{document} , but this time the raters do not necessarily rate the same item, as one rater rates the first item (from $x_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{1}$$\end{document} ) another rater rates the second item (from $x_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{2}$$\end{document} ), and so on. We do not allow a rater to rate the same item more than once in a pass: Hence, we need to choose g raters from a set of R raters, and the chance disagreement is

   (2.6) $\begin{array}{c}{C}_d(x_1,\dots ,x_g)={\left(\begin{array}{c}R\\ g\end{array}\right)}^{-1}\sum _{r_1,\dots ,r_g}d(x_{1r_1},\dots ,x_{g{r}_g}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} C_{d}(\varvec{x}_{1},\ldots ,\varvec{x}_{g}) =\left( {\begin{array}{c}R\\ g\end{array}}\right) ^{-1}\sum _{r_{1},\ldots ,r_{g}}d(\varvec{x}_{1r_{1}}, \ldots ,\varvec{x}_{gr_{g}}), \end{aligned}$$\end{document}
   where the sum runs over all g-dimensional subsets of $\{1,\dots ,R\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{1,\ldots ,R\}$$\end{document} , i.e., the g-combinations of R. Observe that $D_d\left(x\right)=C_d(x,,\dots ,x)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{d}(\varvec{x})=C_{d}(\varvec{x},,\ldots ,\varvec{x})$$\end{document} . Since d is assumed to be symmetric, the expression is simplified to $d(x_{1r_1},\dots ,x_{R{r}_R})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d(\varvec{x}_{1r_{1}},\ldots ,\varvec{x}_{Rr_{R}})$$\end{document} when $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} . When $g=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=2$$\end{document} , $C_d(x_1,x_2)=R^{-1}{(R-1)}^{-1}{\sum }_{r_1\ne r_2}d(x_{1r_1},x_{2r_2})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{d}(\varvec{x}_{1},\varvec{x}_{2})=R^{-1}(R-1)^{-1} \sum _{r_{1}\ne r_{2}}d(\varvec{x}_{1r_{1}},\varvec{x}_{2r_{2}})$$\end{document} .

3. (iii) The Fleiss-type chance disagreement at $x_1,\dots ,x_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{x}_{1},\ldots ,\varvec{x}_{g}$$\end{document} is similar, but allows the same rater to rate an item multiple times. Its definition is

   (2.7) $\begin{array}{c}{F}_d(x_1,\dots ,x_g)=R^{-g}\sum _{r_1,\dots ,r_g}d(x_{1r_1},\dots ,x_{g{r}_g}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} F_{d}(\varvec{x}_{1},\ldots ,\varvec{x}_{g}) =R^{-g}\sum _{r_{1},\ldots ,r_{g}}d(\varvec{x}_{1r_{1}}, \ldots ,\varvec{x}_{gr_{g}}), \end{aligned}$$\end{document}
   where the sum runs over the product set $R^g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R^{g}$$\end{document} . The expression for $F_d(x_1,\dots ,x_g)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_{d}(\varvec{x}_{1},\ldots ,\varvec{x}_{g})$$\end{document} is not dramatically simplified when $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} . When $g=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=2$$\end{document} , $F_d(x_1,x_2)=R^{-2}{\sum }_{r_1,r_2}d(x_{1r_1},x_{2r_2})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_{d}(\varvec{x}_{1},\varvec{x}_{2})=R^{-2} \sum _{r_{1},r_{2}}d(\varvec{x}_{1r_{1}},\varvec{x}_{2r_{2}})$$\end{document} .

We will call the expected values of these quantities the mean disagreement, the mean Cohen-type chance disagreement, and the mean Fleiss-type chance disagreement. Slightly abusing notation, we denote them as

(2.8) $\begin{array}{c}{D}_d=E\left[D_d\left(X_1\right)\right],C_d=E\left[C_d(X_1,\dots ,X_g)\right],F_d=E\left[F_d(X_1,\dots ,X_g)\right],\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{d}=E[D_{d}(\varvec{X}_{1})],\quad C_{d}=E[C_{d}(\varvec{X}_{1},\ldots ,\varvec{X}_{g})],\quad F_{d}=E[F_{d}(\varvec{X}_{1},\ldots ,\varvec{X}_{g})], \end{aligned}$$\end{document}

where $X_1,\dots ,X_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{X}_{1},\ldots ,\varvec{X}_{g}$$\end{document} are independently sampled from the same distribution F. Discussions about the difference between $E\left[C_d(X_1,\dots ,X_g)\right]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E[C_{d}(\varvec{X}_{1},\ldots ,\varvec{X}_{g})]$$\end{document} and $E\left[F_d(X_1,\dots ,X_g)\right]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E[F_{d}(\varvec{X}_{1},\ldots ,\varvec{X}_{g})]$$\end{document} , and why to prefer one over the other, are abundant in the literature, often in the context of the so-called paradox of kappa (Cicchetti and Feinstein, Reference Cicchetti and Feinstein1990).

Definition 1

Let $X\sim F$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X\sim F$$\end{document} be a vector of R ratings and d be an agreement function with dimension g. Define the population values of the generalized Cohen’s kappa $\left({\kappa }_d\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\kappa _{d})$$\end{document} and Fleiss’s kappa $\left({\pi }_d\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(\pi _{d})$$\end{document} as

(2.9) $\begin{array}{c}{\kappa }_d=1-\frac{D_d}{C_d},{\pi }_d=1-\frac{D_d}{F_d}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \kappa _{d}=1-\frac{D_{d}}{C_{d}},\quad \pi _{d}=1 -\frac{D_{d}}{F_{d}}. \end{aligned}$$\end{document}

The generalized Fleiss’s kappa, denoted as ${\pi }_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi _{d}$$\end{document} since it generalizes of Scott’s pi (Scott, Reference Scott1955), is a straightforward generalization of the Fleiss kappa (Reference Fleiss1971) to hold for $2<g\le R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2< g\le R$$\end{document} . When $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} and d is the nominal disagreement, it equals Hubert’s kappa. Likewise, the generalized Cohen’s kappa is an extension of weighted Conger’s kappa to hold for $2\le g\le R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2\le g\le R$$\end{document} . When $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} , it equals the Schuster–Smith coefficient (Schuster & Smith, Reference Schuster and Smith2005, eq. 1).Footnote 3 It generalizes several other agreement coefficients as well. For instance, Berry and Mielke (Reference Berry and Mielke1988) discussed what we call ${\kappa }_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\kappa _{d}$$\end{document} for Euclidean weights between vector-valued ratings, while Janson and Olsson (Reference Janson and Olsson2001) extended it to squared Euclidean and nominal weights. The relationship between most of the mentioned agreement coefficients is summarized in Table 1.

Table 1 Weighted agreement coefficients.

*Lin’s concordance coefficient and the concordance correlation coefficient (CC) is defined for quadratic weights only.

${}^{†}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\dagger $$\end{document} Originally defined for nominal weights only.

3. Sample Estimates

Let $X_1,\dots ,X_n\sim F$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{X}_{1},\ldots ,\varvec{X}_{n}\sim F$$\end{document} be n iid vectors of ratings. Then there is a single natural sample estimator of $D_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{d}$$\end{document} , namely

(2.10) $\begin{array}{c}{\hat{D}}_d=n^{-1}\sum _{i=1}^n{D}_d\left(x_i\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{D}_{d}=n^{-1}\sum _{i=1}^nD_{d}(\varvec{x}_{i}). \end{aligned}$$\end{document}

There are, however, two natural estimators of the Cohen-type chance disagreement: one them a V-statistic (Lee, Reference Lee2019, Chapter 4.2) and the other a U-statistic (Lee, Reference Lee2019, Chapter 1),

(2.11) $\begin{array}{c}{\hat{C}}_d=n^{-g}\sum _{i_1,\dots ,i_g}{C}_d(x_{i_1},\dots ,x_{i_g})\text{and}{\hat{C}}_d^u={\left(\begin{array}{c}n\\ g\end{array}\right)}^{-1}\sum _{i_1,\dots ,i_g}{C}_d(x_{i_1},\dots ,x_{i_g}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{C}_{d} = n^{-g}\sum _{i_{1},\ldots ,i_{g}}C_{d} (\varvec{x}_{i_{1}},\ldots ,\varvec{x}_{i_g})\quad \text {and}\quad \hat{C}_{d}^{u}=\left( {\begin{array}{c}n\\ g\end{array}}\right) ^{-1}\sum _{i_{1}, \ldots ,i_{g}}C_{d}(\varvec{x}_{i_{1}},\ldots , \varvec{x}_{i_{g}}), \end{aligned}$$\end{document}

where the first estimator runs over all combinations with repetitions of $i_1,i_2,\dots ,i_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i_{1},i_{2},\ldots ,i_{g}$$\end{document} and the second estimator runs over the unordered combinations $i_1<i_2<\dots <i_g$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i_{1}<i_{2}<\ldots <i_{g}$$\end{document} . Using the basic results of U-statistics (Lee, Reference Lee2019, Chapter 1), we see that $C_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{d}^{u}$$\end{document} is the unique minimum-variance unbiased estimator of $C_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{d}$$\end{document} , which makes it attractive from a theoretical point of view. However, from a well-known correspondence between U-statistics and V-statistics, the asymptotic distributions of ${\hat{C}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{C}_{d}$$\end{document} coincide with the asymptotic distribution of ${\hat{C}}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{C}_{d}^{u}$$\end{document} (Lee, Reference Lee2019, Chapter 4, Theorem 1), so the choice between ${\hat{C}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{C}_{d}$$\end{document} and ${\hat{C}}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{C}_{d}^{u}$$\end{document} barely matters when n is sufficiently large.

Likewise, there are two natural estimators of the Fleiss-type weighted chance agreement,

(2.12) $\begin{array}{c}{\hat{F}}_d=n^{-g}\sum _{i_1,\dots ,i_g}{F}_d(x_{i_1},\dots ,x_{i_g})\text{and}{\hat{F}}_d^u={\left(\begin{array}{c}n\\ g\end{array}\right)}^{-1}\sum _{i_1,\dots ,i_g}{F}_d(x_{i_1},\dots ,x_{i_g}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{F}_{d} = n^{-g}\sum _{i_{1},\ldots ,i_{g}}F_{d}(\varvec{x}_{i_{1}},\ldots , \varvec{x}_{i_{g}})\quad \text {and}\quad \hat{F}_{d}^{u} =\left( {\begin{array}{c}n\\ g\end{array}}\right) ^{-1}\sum _{i_{1},\ldots ,i_{g}}F_{d}(\varvec{x}_{i_{1}},\ldots , \varvec{x}_{i_{g}}), \end{aligned}$$\end{document}

where the index sets are described above.

Now, we can define two sample variants of Cohen’s kappa (Fleiss’s kappa), depending on which one of ${\hat{C}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{C}_{d}$$\end{document} ( ${\hat{F}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{F}_{d}$$\end{document} ) and ${\hat{C}}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{C}_{d}^{u}$$\end{document} ( ${\hat{F}}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{F}_{d}^{u}$$\end{document} ) we choose to use. These are ${\hat{\kappa }}_d=1-{\hat{D}}_d/{\hat{C}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}=1-\hat{D}_{d}/\hat{C}_{d}$$\end{document} and ${\hat{\kappa }}_d^u=1-{\hat{D}}_d/{\hat{C}}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}^{u}=1-\hat{D}_{d}/\hat{C}_{d}^{u}$$\end{document} for Cohen’s kappa and ${\hat{\pi }}_d=1-{\hat{D}}_d/{\hat{F}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}=1-\hat{D}_{d}/\hat{F}_{d}$$\end{document} and ${\hat{\pi }}_d^u=1-{\hat{D}}_d/{\hat{F}}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}^{u}=1-\hat{D}_{d}/\hat{F}_{d}^{u}$$\end{document} for Fleiss’s kappa. The definition of the sample Cohen’s kappa (Cohen, Reference Cohen1968) agrees with ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} , not with ${\hat{\kappa }}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}^{u}$$\end{document} . Likewise, the sample Fleiss’s kappa has a definition agreeing with ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} (Fleiss, Reference Fleiss1971). Moreover, due to the possibility of binning data, ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} and ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} are faster to compute when the data is not continuous. Since the estimators are asymptotically equivalent in any case, we will stick to the V-statistics ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} and ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} for estimation, but use the U-statistic form when deriving limit distributions. We note that, since we need to compute strictly fewer combinations, ${\hat{\kappa }}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}^{u}$$\end{document} and ${\hat{\pi }}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}^{u}$$\end{document} are faster to compute when the data is continuous, which may be useful in some settings.

3. Fréchet Variances for g-Wise Agreement Coefficients

The most popular measures of agreement are defined only for $g=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=2$$\end{document} . It is easy to find reasonable disagreement measures in this case, as one can draw on the extensive literature on norms and distances. The $l_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_{p}$$\end{document} distances are the obvious choices, but there are many unexplored options, such as the Huber loss (Huber, Reference Huber1964) and the LINEX loss (Varian, Reference Varian1975).

In the setting of Hubert’s kappa and the Schuster–Smith coefficient, we have $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} , and it is not that easy to find reasonable disagreement functions anymore. The disagreement function used in Hubert’s kappa, $d(x_1,\dots ,x_R)=1-1[x_1=\cdots =x_R]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d(x_{1},\ldots ,x_{R})=1-1[x_{1}=\cdots =x_{R}]$$\end{document} , will penalize any number of discordant ratings equally, yielding the often undesirable outcome that most sets of ratings will be in complete disagreement. But there are less sensitive ways to count nominal disagreements. Consider the case of 10 raters with three ratings on an ordinal scale from 1–3, with 7 raters giving rating 1, 2 giving rating 2, and 1 giving rating 3. Then Hubert’s disagreement rating is 1, as the rating vector is not constant, and the pairwise disagreement is 46/100. But it sounds reasonable to pick the modal rating (in this case 1) and then report the number of raters that disagree with it, divided by the number of raters. In this case, the number of raters disagreeing with the modal rating is 3, and the “modal” disagreement equals 3/10.

Sometimes we wish to aggregate numerical ratings instead of categorical ratings. Consider the above case again but with the median (which is 1) instead of the mode. It is well known that the median of a vector x is equal to ${\text{argmin}}_{\mu }\frac{1}{R}{\sum }_{r=1}^R|x_r-\mu |$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\textrm{argmin}\,}}_{\mu }\frac{1}{R}\sum _{r=1}^R|x_{r}-\mu |$$\end{document} , so ${min}_{\mu }\frac{1}{R}{\sum }_{r=1}^R|x_r-\mu |$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\min _{\mu }\frac{1}{R}\sum _{r=1}^R|x_{r}-\mu |$$\end{document} (mean absolute deviation from the median) appears to be a reasonable measure of the mean disagreement when we use the median as the aggregation method. The resulting mean disagreement of the previous example is ${min}_{\mu }\frac{1}{R}{\sum }_{r=1}^R|x_r-\mu |=\frac{1}{10}{\sum }_{r=1}^{10}|x_r-1|=4/10$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\min _{\mu }\frac{1}{R}\sum _{r=1}^R|x_{r}-\mu |=\frac{1}{10} \sum _{r=1}^{10}|x_{r}-1|=4/10$$\end{document} .

The “modal” and “median” disagreement measures are instances of an intuitive generalization of the variance called the Fréchet variance (Dubey and Müller, Reference Dubey and Müller2019). Let l be a distance function satisfying $l(x,y)\ge 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l(x,y)\ge 0$$\end{document} and $l(x,x)=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l(x,x)=0$$\end{document} , and let $A=\{x_1,x_2,\dots ,x_R\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A=\{x_{1},x_{2},\ldots ,x_{R}\}$$\end{document} be a set of points. The sample Fréchet mean of A is defined as the (not necessarily unique) point ${\mu }_l$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _{l}$$\end{document} that minimizes the sum of distances to all points in A, that is,Footnote 4

(3.1) $\begin{array}{c}{\mu }_l\left[A\right]={\text{argmin}}_{\mu }\sum _{r=1}^{R}l(\mu ,x_r).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mu _{l}[A]={{\,\textrm{argmin}\,}}_{\mu }\sum _{r=1}^{R}l(\mu ,x_{r}). \end{aligned}$$\end{document}

Similarly, the sample Fréchet variance on A with distance function l is

(3.2) $\begin{array}{c}V\left(l\right)\left[A\right]=\underset{\mu }{min}\sum _{r=1}^R\frac{1}{R}l(\mu ,x_r)=\sum _{r=1}^R\frac{1}{R}l({\mu }_l\left[A\right],x_r).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} V(l)[A]=\min _{\mu }\sum _{r=1}^{R}\frac{1}{R}l(\mu ,x_{r}) =\sum _{r=1}^{R}\frac{1}{R}l(\mu _{l}[A],x_{r}). \end{aligned}$$\end{document}

The Fréchet mean (Fréchet, Reference Fréchet1948) is a generalization of centroids to arbitrary distance functions l; likewise, the Fréchet variance is a generalization of dispersion to any such distance function. They are best understood through a decision-theoretic lens: The Fréchet mean of A represents your best guess of the true classification or value of an item according to the distance l; the Fréchet variance V(l) is the decision-theoretic risk associated with the choice. See Cooil and Rust (Reference Cooil and Rust1994) for an investigation of a closely related idea in the context of agreement measures.

Define the g-dimensional disagreement based on l as

(3.3) $\begin{array}{c}d(x_1,\dots ,x_g)=V\left(l\right)\left[\{x_1,\dots ,x_g\}\right].\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} d(\varvec{x}_{1},\ldots ,\varvec{x}_{g})=V(l) [\{\varvec{x}_{1},\ldots ,\varvec{x}_{g}\}]. \end{aligned}$$\end{document}

The most important distance functions are:

1. (i) $d_0(x,y)=1[x\ne y]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{0}(x,y)=1[x\ne y]$$\end{document} . Generalizes the nominal distance. If the data are categorical, the Fréchet mean ${\mu }_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _{d}$$\end{document} equals the mode, and the Fréchet variance equals the percentage of observations different from the mode. If we are dealing with vector-valued data with I elements each, it might be preferable to use $I^{-1}{\sum }_{i=1}^{I}1[x_i\ne y_i]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I^{-1}\sum _{i=1}^{I}1[x_{i}\ne y_{i}]$$\end{document} instead, as it counts each dimension of the nominal data separately.

2. (ii) $d_1(x,y)=\left|\right|x-{y\left|\right|}_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{1}(x,y)=||x-y||_{1}$$\end{document} . For scalar ratings, the Fréchet mean is equal to the sample median. The Fréchet variance equals the sample mean absolute deviation from the median, i.e., $\frac{1}{R}{\sum }_{r=1}^R|x_r-{\mu }_d|$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{1}{R}\sum _{r=1}^{R}|x_{r}-\mu _d|$$\end{document} , where ${\mu }_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _d$$\end{document} is the sample median.

3. (iii) $d_2^2(x,y)=\left|\right|x-{y\left|\right|}_2^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{2}^{2}(x,y)=||x-y||_{2}^{2}$$\end{document} . For scalar ratings, the Fréchet mean is equal to the sample mean ${\mu }_d=\frac{1}{R}{\sum }_{r=1}^R{x}_r$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _d=\frac{1}{R}\sum _{r=1}^{R}x_{r}$$\end{document} , and the Fréchet variance is equal to the biased sample variance of $\{x_1,x_2,\dots ,x_R\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{x_{1},x_{2},\ldots ,x_{R}\}$$\end{document} , that is, $\frac{1}{R}{\sum }_{r=1}^R{(x_r-{\mu }_d)}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{1}{R}\sum _{r=1}^{R}(x_{r}-\mu _d)^{2}$$\end{document} .

4. (iv) $d_2(x,y)=\left|\right|x-{y\left|\right|}_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{2}(x,y)=||x-y||_{2}$$\end{document} . For vector-valued data, the Fréchet mean has no simple formula, but is known as the geometric median. If the data is scalar, $d_2=d_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{2}=d_{1}$$\end{document} , which implies that the Fréchet mean equals the median, hence the name. There is an extensive literature on the geometric median; see, e.g., Drezner et al. (Reference Drezner, Klamroth, Schöbel and Wesolowsky2002) for an overview and Cohen et al. (Reference Cohen, Lee, Miller, Pachocki and Sidford2016) for how to compute it. When the ratings are vector-valued, the geometric median is far more computationally expensive than the Fréchet mean based on $\left|\right|x-{y\left|\right|}_2^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$||x-y||_{2}^{2}$$\end{document} .

For any $p\in [0,\infty ]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p\in [0,\infty ]$$\end{document} and pair of vectors $x_1,x_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{1},x_{2}$$\end{document} , we have the following (proved in Appendix, Sect. 6):

(3.4) $\begin{array}{c}V\left(d_p\right)[x_1,x_2]=\frac{1}{2}{d}_p(x_1,x_2),V\left(d_p^p\right)[x_1,x_2]=\frac{1}{2^p}{d}_p^p(x_1,x_2).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} V(d_{p})[x_{1},x_{2}]=\frac{1}{2}d_{p}(x_{1},x_{2}),\quad V(d_{p}^{p})[x_{1},x_{2}]=\frac{1}{2^{p}}d_{p}^{p}(x_{1},x_{2}). \end{aligned}$$\end{document}

It follows that ${\kappa }_{d_p}={\kappa }_{V\left(d_p\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\kappa _{d_{p}}=\kappa _{V(d_{p})}$$\end{document} and ${\kappa }_{d_p^p}={\kappa }_{V\left(d_p^p\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\kappa _{d_{p}^{p}}=\kappa _{V(d_{p}^{p})}$$\end{document} when we are dealing with pairwise agreement. Thus, the Fréchet variances generalize the pairwise agreement for these distances to g-wise coefficients. But be aware that the particular case of $V\left(d_2^2\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(d_{2}^{2})$$\end{document} constitutes a trivial generalization, as it can be shown that the kappas do not vary with g when using the quadratic Fréchet variance $V\left(d_2^2\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(d_{2}^{2})$$\end{document} . It follows that ${\kappa }_V\left(d_2^2\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\kappa _V(d_2^2)$$\end{document} equals the concordance coefficient for every g.

Example 1

Suppose you have $R=5$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R=5$$\end{document} raters and 4 items, with ratings (1, 1, 2, 1, 1), (1, 2, 3, 2, 2), (2, 1, 1, 1, 1), (2, 3, 4, 4, 5). The Fréchet means using the distance $|x-y|$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|x-y|$$\end{document} equals the sample medians 1, 2, 1, 4. The Fréchet variances are $V\left(d_1\right)=(0.2,0.4,0.2,0.8)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(d_{1})=(0.2,0.4,0.2,0.8)$$\end{document} . To calculate the sample Cohen’s kappa with $d=V\left(d_1\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=V(d_{1})$$\end{document} , we first find the mean disagreement $\overline{V\left(d_1\right)}=0.4$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{V(d_{1})}=0.4$$\end{document} (2.10), then the mean Cohen disagreement, which is $\approx 0.73$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 0.73$$\end{document} (2.11). Thus, Cohen’s kappa is $1-0.4/0.73=0.45$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1-0.4/0.73=0.45$$\end{document} .

We believe the most useful distance measures will typically be $d_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{0}$$\end{document} for categorical data and $d_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{1}$$\end{document} for ordinal data, both using $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} . The quadratic distance $d_2^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{2}^{2}$$\end{document} could be used for ordinal data as well, but is harder to justify, as it is usually not obvious why we would be interested in the squared distance between two observations rather than just the distance itself. The distances $d_p,p\in (1,\infty ]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{p},p\in (1,\infty ]$$\end{document} , with $d_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d_{2}$$\end{document} included, are even harder to recommend, as they do not work in a coordinatewise manner for vector data. In any case, it seems most reasonable to go with the R-wise variants of these distance measures, as they make use of all the available information, but the g-wise agreement coefficients ( $g<R)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g<R)$$\end{document} do not.

Example 2

In the paper introducing what is now called Fleiss’s kappa, Fleiss (Reference Fleiss1971) discussed an example involving 5 different types of diagnoses, $n=30$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=30$$\end{document} patients, and $R=6$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R=6$$\end{document} psychiatrists. The data were originally from Sandifer et al. (Reference Sandifer, Hordern, Timbury and Green1968), but Fleiss removed some ratings to make the design rectangular. We use this data to illustrate the difference between Hubert’s kappa and the Fréchet variances when applied to nominal data with $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} .

Hubert’s kappa is $\pi =0.166$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi =0.166$$\end{document} while Fleiss’ kappa using $V\left(d_0\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(d_0)$$\end{document} is $\pi =0.486$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi =0.486$$\end{document} . The substantial difference suggests that a sizeable number of rating vectors contain at least one rating that disagrees with the others. Table 2 summarizes the relevant aspects of the data. The maximal agreement row could potentially go from 1 to 6, but the smallest number of raters agreeing on the classification of an item in this data set is 3. The count row counts the number of rows with the corresponding maximal agreements and distances. According to the Hubert distance, the raters disagree a lot, since only 5 items have a disagreement of 0 and the rest a disagreement of 1. On the other hand, $V\left(d_0\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(d_{0})$$\end{document} results in a much smaller overall disagreement, with all disagreements smaller than the maximum of 1.

Table 2 Maximal agreement for the data of Fleiss (Reference Fleiss1971).

*The largest number of raters that agree on the classification of an item. Both $V\left(d_0\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(d_{0})$$\end{document} and Hubert’s distance depend only on this when $g=R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=R$$\end{document} .

4. Inference

4.1. Limit Theory Using U-Statistics

Let $X_1,\dots ,X_n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{X}_{1},\ldots ,\varvec{X}_{n}$$\end{document} be independently and identically distributed and $\psi (x_1,\dots ,x_k)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi (x_{1},\ldots ,x_{k})$$\end{document} be a symmetric function. A U-statistic of order k with kernel $\psi$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi $$\end{document} is

(4.1) $\begin{array}{c}{U}_n={\left(\begin{array}{c}n\\ k\end{array}\right)}^{-1}\sum _{i_1,\dots ,i_k}\psi (X_{i_1},\dots ,X_{i_k}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} U_{n}=\left( {\begin{array}{c}n\\ k\end{array}}\right) ^{-1}\sum _{i_{1},\ldots ,i_{k}} \psi (\varvec{X}_{i_{1}},\ldots ,\varvec{X}_{i_{k}}), \end{aligned}$$\end{document}

where the sum extends over all k-dimensional tuples satisfying $1\le i_1<i_2<\cdots \le n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1\le i_{1}<i_{2}<\cdots \le n$$\end{document} .

The theory of U-statistics was established by Hoeffding (Reference Hoeffding1992); for an introduction, see, e.g., Chapter 6.1 of Lehmann (Reference Lehmann2004), Chapter 5 of Serfling (Reference Serfling1980), or the textbook of Lee (Reference Lee2019). These references handle U-statistics where the $X_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{i}$$\end{document} s are real-valued, but their results, including the simple results below, hold for vector-valued $X_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{i}$$\end{document} s as well (Korolyuk and Borovskich, Reference Korolyuk and Borovskich2013).

The weighted chance agreement of Fleiss-type (Cohen-type) is a U-statistic with kernel $F_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_{d}$$\end{document} ( $C_d)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$C_{d})$$\end{document} , of order g. The disagreement is a U-statistic with kernel $D_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{d}$$\end{document} , which has order 1. To find the asymptotic variance of the kappas, we will use formulas for the asymptotic covariance of U-statistics. Let $U_{1n}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_{1n}$$\end{document} and $U_{2n}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_{2n}$$\end{document} be two U-statistics of n observations with symmetric kernel functions ${\psi }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _{1}$$\end{document} , ${\psi }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\psi _{2}$$\end{document} of dimensions $k_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k_{1}$$\end{document} and $k_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k_{2}$$\end{document} . Define

$\begin{array}{cc}{\sigma }_1^2=& \text{Var}\left(E[{\psi }_1(X_1,\dots ,X_{k_1})\mid X_1)\right]),\\ {\sigma }_{12}=& \text{Cov}\left(E[{\psi }_1(X_1,\dots ,X_{k_1})\mid X_1)\right],E[{\psi }_2(X_1,\dots ,X_{k_2})\mid X_1)\left]\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sigma _{1}^{2}= & {} {{\,\textrm{Var}\,}}(E[\psi _{1}(\varvec{X}_{1},\ldots , \varvec{X}_{k_{1}})\mid \varvec{X}_{1})]),\\ \sigma _{12}= & {} {{\,\textrm{Cov}\,}}(E[\psi _{1}(\varvec{X}_{1},\ldots ,\varvec{X}_{k_{1}}) \mid \varvec{X}_{1})],E[\psi _{2}(\varvec{X}_{1}, \ldots ,\varvec{X}_{k_{2}})\mid \varvec{X}_{1})]). \end{aligned}$$\end{document}

Then we have $n\text{Cov}(U_{1n},U_{2n})\to k_1{k}_2{\sigma }_{12}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n{{\,\textrm{Cov}\,}}(U_{1n},U_{2n})\rightarrow k_{1}k_{2}\sigma _{12}$$\end{document} and $n\text{Var}\left(U_{1n}\right)\to k_1^2{\sigma }_1^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n{{\,\textrm{Var}\,}}(U_{1n})\rightarrow k_{1}^{2}\sigma _{1}^{2}$$\end{document} (Lee, Reference Lee2019, Theorem 2, p. 76)). It is also possible to calculate the exact covariances, which could potentially make the asymptotic variances for the kappas perform better. See Appendix, Sect. 6 for the formula for the exact covariance (Lee, Reference Lee2019, Theorem 2, p. 17)).

Lemma 1

Define the parameter vectors $p=(D_d,C_d,F_d)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varvec{p}=(D_{d},C_{d},F_{d})$$\end{document} and $\hat{p}=({\hat{D}}_d,{\hat{C}}_d,{\hat{F}}_d)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\varvec{p}}=(\hat{D}_{d},\hat{C}_{d},\hat{F}_{d})$$\end{document} . Then $\sqrt{n}(\hat{p}-p)\stackrel{d}{\to }N(0,\Sigma )$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{n}(\hat{\varvec{p}}-\varvec{p}){\mathop {\rightarrow }\limits ^{d}}N(0,\Sigma )$$\end{document} , where $\Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma $$\end{document} is the covariance matrix with elements

$\begin{array}{ccccccc}{\sigma }_{11}=& {\sigma }_D^2=& \text{Var}{D}_d\left(X_1\right)& ,& {\sigma }_{12}=& {\sigma }_{\mathrm{CD}}=& g\text{Cov}({\mu }_{\mathrm{dC}}\left(X_1\right),D_d\left(X_1\right)),\\ {\sigma }_{22}=& {\sigma }_C^2=& g^2\text{Var}{\mu }_{\mathrm{dC}}\left(X_1\right)& ,& {\sigma }_{13}=& {\sigma }_{\mathrm{FD}}=& g\text{Cov}({\mu }_{\mathrm{dF}}\left(X_1\right),D_d\left(X_1\right)),\\ {\sigma }_{33}=& {\sigma }_F^2=& g^2\text{Var}{\mu }_{\mathrm{dF}}\left(X_1\right)& ,& {\sigma }_{23}=& {\sigma }_{\mathrm{CF}}=& g\text{Cov}({\mu }_{\mathrm{dC}}\left(X_1\right),{\mu }_{\mathrm{dF}}\left(X_1\right)).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sigma _{11}= & {} \sigma _{D}^{2}= & {} {{\,\textrm{Var}\,}}D_{d}(\varvec{X}_{1})&,\quad&\sigma _{12}= & {} \sigma _{CD}= & {} g{{\,\textrm{Cov}\,}}(\mu _{dC}(\varvec{X}_{1}),D_{d}(\varvec{X}_{1})),\\ \sigma _{22}= & {} \sigma _{C}^{2}= & {} g^{2}{{\,\textrm{Var}\,}}\mu _{dC}(\varvec{X}_{1})&,\quad&\sigma _{13}= & {} \sigma _{FD}= & {} g{{\,\textrm{Cov}\,}}(\mu _{dF}(\varvec{X}_{1}),D_{d}(\varvec{X}_{1})),\\ \sigma _{33}= & {} \sigma _{F}^{2}= & {} g^{2}{{\,\textrm{Var}\,}}\mu _{dF}(\varvec{X}_{1})&,\quad&\sigma _{23}= & {} \sigma _{CF}= & {} g{{\,\textrm{Cov}\,}}(\mu _{dC}(\varvec{X}_{1}),\mu _{dF}(\varvec{X}_{1})). \end{aligned}$$\end{document}

Here the variable ${\mu }_{\mathrm{dC}}\left(X_1\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _{dC}(\varvec{X}_{1})$$\end{document} , and ${\mu }_{\mathrm{dF}}\left(X_1\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _{dF}(\varvec{X}_{1})$$\end{document} are defined as

$\begin{array}{c}{\mu }_{\mathrm{dC}}\left(X_1\right)=E[C_d(X_1,\dots ,X_g)\mid X_1]{\mu }_{\mathrm{dF}}\left(X_1\right)=E[F_d(X_1,\dots ,X_g)\mid X_1].\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mu _{dC}(\varvec{X}_{1})=E[C_{d}(\varvec{X}_{1}, \ldots ,\varvec{X}_{g})\mid \varvec{X}_{1}] \quad \mu _{dF}(\varvec{X}_{1}) =E[F_{d}(\varvec{X}_{1},\ldots ,\varvec{X}_{g}) \mid \varvec{X}_{1}]. \end{aligned}$$\end{document}

The form of the covariance matrix follows from the remarks preceding the lemma. Asymptotic normality follows from a general theorem about asymptotic normality of U-statistics, see, e.g., Theorem 2 of Lee (Reference Lee2019, p. 76).

We want to use Lemma 1 to find the limit distribution of the generalized Cohen’s kappa and Fleiss’s kappa. To this end, recall the multivariate delta method (see, e.g., Lehmann, Reference Lehmann2004, Theorem 5.2.3). Let $f:R^k\to R$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f:\mathbb {R}^{k}\rightarrow \mathbb {R}$$\end{document} be continuously differentiable at $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} and suppose that $\sqrt{n}(\hat{\theta }-\theta )\stackrel{d}{\to }N(0,\Sigma )$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{n}(\hat{\theta }-\theta ){\mathop {\rightarrow }\limits ^{d}}N(0,\Sigma )$$\end{document} . Then

(4.2) $\begin{array}{c}\sqrt{n}[f\left(\hat{\theta }\right)-f\left(\theta \right)]\stackrel{d}{\to }N(0,\nabla f{\left(\theta \right)}^T\Sigma \nabla f\left(\theta \right)),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sqrt{n}[f(\hat{\theta })-f(\theta )]{\mathop {\rightarrow }\limits ^{d}}N(0,\nabla f(\theta )^{T}\Sigma \nabla f(\theta )), \end{aligned}$$\end{document}

where $\nabla f\left(\theta \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nabla f(\theta )$$\end{document} denotes the gradient of f at $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} .

In the case of Cohen’s kappa and Fleiss’s kappa, we find that

(4.3) $\begin{array}{cc}\nabla {\kappa }_d=& \frac{1}{C_d}\left(-,1,,,\frac{D_d}{C_d}\right),\nabla {\pi }_d=\frac{1}{F_d}\left(-,1,,,\frac{D_d}{F_d}\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \nabla \kappa _{d}= & {} \frac{1}{C_{d}}\left( -1,\frac{D_{d}}{C_{d}}\right) , \quad \nabla \pi _{d}=\frac{1}{F_{d}}\left( -1,\frac{D_{d}}{F_{d}}\right) . \end{aligned}$$\end{document}

Using some algebra, the expressions for the asymptotic variances follow from Lemma 1 and the above gradients.

Proposition 1

Then Cohen’s kappa ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} and Fleiss’s kappa ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} are asymptotically normal, and their asymptotic variances are

(4.4) $\begin{array}{cc}{\sigma }_{\kappa }^2=& {\sigma }_D^2\frac{1}{C_d^2}-2{\sigma }_{\mathrm{CD}}\frac{D_d}{C_d^3}+{\sigma }_C^2\frac{D_d^2}{C_d^4},\\ {\sigma }_{\pi }^2=& {\sigma }_D^2\frac{1}{F_d^2}-2{\sigma }_{\mathrm{FD}}\frac{D_d}{F_d^3}+{\sigma }_F^2\frac{D_d^2}{F_d^4}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sigma _{\kappa }^{2}= & {} \sigma _{D}^{2}\frac{1}{C_{d}^{2}}-2\sigma _{CD} \frac{D_{d}}{C_{d}^{3}}+\sigma _{C}^{2}\frac{D_{d}^{2}}{C_{d}^{4}}, \nonumber \\ \sigma _{\pi }^{2}= & {} \sigma _{D}^{2}\frac{1}{F_{d}^{2}}-2\sigma _{FD}\frac{D_{d}}{F_{d}^{3}} +\sigma _{F}^{2}\frac{D_{d}^{2}}{F_{d}^{4}}. \end{aligned}$$\end{document}

These results are also valid for ${\hat{\kappa }}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}^{u}$$\end{document} and ${\hat{\pi }}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}^{u}$$\end{document} . Since the sample Krippendorff’s alpha (see note below) is equal to ${\hat{\alpha }}_d={\hat{\pi }}_d+\frac{1}{2Rn}(1-{\hat{\pi }}_d)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\alpha }_{d}=\hat{\pi }_{d}+\frac{1}{2Rn}(1-\hat{\pi }_{d})$$\end{document} , it is also asymptotically normal with asymptotic variance ${\sigma }_{\pi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{\pi }^{2}$$\end{document} .

With $g=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=2$$\end{document} and a finite number of categories, Schouten (Reference Schouten1980) derived ${\sigma }_{\pi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{\pi }^{2}$$\end{document} , while Schouten (Reference Schouten1982) and O’Connell and Dobson (Reference O’Connell and Dobson1984) derived ${\sigma }_{\kappa }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{\kappa }^{2}$$\end{document} . The result for Krippendorff’s alpha is, to our knowledge, new.

A brief aside on Krippendorff’s alpha Krippendorff’s alpha (Krippendorff, Reference Krippendorff1970) is an agreement coefficient especially popular in content analysis (Krippendorff, Reference Krippendorff2018). It has no population definition, but its sample definition equals ${\hat{\alpha }}_d={\hat{\pi }}_d+\frac{1}{N}(1-{\hat{\pi }}_d)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\alpha }_{d}=\hat{\pi }_{d}+\frac{1}{N}(1-\hat{\pi }_{d})$$\end{document} (the total sample size N equals 2Rn in the case of a rectangular design); see Proposition 3 in Appendix for a justification. For this reason, all of the results about the limit of ${\hat{\pi }}_d^u$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}^{u}$$\end{document} apply to Krippendorff’s alpha as well, as it is an asymptotically equivalent estimator of ${\pi }_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi _{d}$$\end{document} . Note, however, that Krippendorff (Reference Krippendorff2018) emphasizes the use of non-rectangular designs, and the limit results in the preceding section do not hold for such study designs.

4.2. Estimating the Variances

The unknown quantities ${\hat{D}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{D}_{d}$$\end{document} , ${\hat{C}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{C}_{d}$$\end{document} , and ${\hat{F}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{F}_{d}$$\end{document} can be estimated using their sample counterparts. The variances and covariances can be estimated using the empirical (co)variances of the estimated $\hat{\mu }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\mu }$$\end{document} s. These have formulas

(4.5) $\begin{array}{cc}{\hat{\mu }}_d\left(x_i\right)=& D_d\left(x_i\right),\\ {\hat{\mu }}_{\mathrm{dC}}\left(x_i\right)=& n^{-(g-1)}\sum _{i_1,\dots ,i_{g-1}}{C}_d(x_i,x_{i_1},\dots ,x_{i_{g-1}}),\\ {\hat{\mu }}_{\mathrm{dF}}\left(x_i\right)=& n^{-(g-1)}\sum _{i_1,\dots ,i_{g-1}}{F}_d(x_i,x_{i_1},\dots ,x_{i_{g-1}}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{\mu }_{d}(\varvec{x}_{i})= & {} D_{d}(\varvec{x}_{i}),\nonumber \\ \hat{\mu }_{dC}(\varvec{x}_{i})= & {} n^{-(g-1)}\sum _{i_{1},\ldots ,i_{g-1}}C_{d}(\varvec{x}_{i}, \varvec{x}_{i_{1}},\ldots ,\varvec{x}_{i_{g-1}}),\nonumber \\ \hat{\mu }_{dF}(\varvec{x}_{i})= & {} n^{-(g-1)}\sum _{i_{1},\ldots ,i_{g-1}}F_{d}(\varvec{x}_{i}, \varvec{x}_{i_{1}},\ldots ,\varvec{x}_{i_{g-1}}), \end{aligned}$$\end{document}

where the index sets run over all combinations with repetitions of $(i_1,i_2,\dots ,i_{g-1})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(i_{1},i_{2},\ldots ,i_{g-1})$$\end{document} .

Observe that estimating ${\hat{\mu }}_{\mathrm{dC}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\mu }_{dC}$$\end{document} and ${\hat{\mu }}_{\mathrm{dF}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\mu }_{dF}$$\end{document} directly is computationally very expensive, especially when done without binning, which cannot be done with continuous data. The obvious computation of all ${\hat{\mu }}_{\text{dC}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\mu }_\text {dC}$$\end{document} requires a number of operations on the order of $n^{g-1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n^{g-1}$$\end{document} , which is prohibitively expensive for large n and g. However, there are few applications of agreement measures with very large n and g, so this should not be a serious problem in practice. We note that less computationally demanding procedures are possible for the quadratic Fréchet variance $V\left(d_2^2\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$V(d_2^2)$$\end{document} , as it can be shown that its associated kappas are invariant under g. Thus, we may use the computationally very effective methods for the concordance coefficient outlined by, e.g., Carrasco and Jover (Reference Carrasco and Jover2003).

From the definitions of ${\hat{D}}_d,{\hat{C}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{D}_{d},\hat{C}_{d}$$\end{document} , and ${\hat{F}}_d,$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{F}_{d},$$\end{document} (4), we quickly deduce that $\overline{{\hat{\mu }}_d}={\hat{D}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\hat{\mu }_{d}}=\hat{D}_{d}$$\end{document} , $\overline{{\hat{\mu }}_{\mathrm{dC}}}={\hat{C}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\hat{\mu }_{dC}}=\hat{C}_{d}$$\end{document} and $\overline{{\hat{\mu }}_{\mathrm{dF}}}={\hat{F}}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\overline{\hat{\mu }_{dF}}=\hat{F}_{d}$$\end{document} . Using this fact, we can define the estimators

$\begin{array}{c}{\hat{\sigma }}_C^2=\frac{g^2}{n-1}\sum _{i=1}^n{({\hat{\mu }}_{\mathrm{dC}}\left(x_i\right)-{\hat{C}}_d)}^2,{\hat{\sigma }}_{\mathrm{CD}}^2=\frac{g}{n-1}\sum _{i=1}^n({\hat{\mu }}_{\mathrm{dC}}\left(x_i\right)-{\hat{C}}_d)({\hat{\mu }}_d\left(x_i\right)-{\hat{D}}_d),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \hat{\sigma }_{C}^{2}=\frac{g^2}{n-1}\sum _{i=1}^{n}(\hat{\mu }_{dC} (\varvec{x}_{i})-\hat{C}_{d})^{2},\quad \hat{\sigma }_{CD}^{2} =\frac{g}{n-1}\sum _{i=1}^{n}(\hat{\mu }_{dC}(\varvec{x}_{i}) -\hat{C}_{d})(\hat{\mu }_{d}(\varvec{x}_{i})-\hat{D}_{d}), \end{aligned}$$\end{document}

and ${\hat{\sigma }}_D^2=\frac{1}{n-1}{\sum }_{i=1}^n{({\hat{\mu }}_d\left(x_i\right)-{\hat{D}}_d)}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\sigma }_{D}^{2}=\frac{1}{n-1}\sum _{i=1}^{n}(\hat{\mu }_{d} (\varvec{x}_{i})-\hat{D}_{d})^{2}$$\end{document} . Moreover, we can estimate ${\hat{\sigma }}_F^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\sigma }_{F}^{2}$$\end{document} and ${\hat{\sigma }}_{\mathrm{FD}}^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\sigma }_{FD}^{2}$$\end{document} in the same way, substituting ${\hat{\mu }}_{\mathrm{dF}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\mu }_{dF}$$\end{document} for ${\hat{\mu }}_{\mathrm{dC}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\mu }_{dC}$$\end{document} . Using the formulas for the theoretical variances (4.4), we find the estimators

(4.6) $\begin{array}{cc}& {\hat{\sigma }}_{\kappa }^2={\hat{\sigma }}_D^2\frac{1}{{\hat{C}}_d^2}-2{\hat{\sigma }}_{\mathrm{CD}}\frac{{\hat{D}}_d}{{\hat{C}}_d^3}+{\hat{\sigma }}_C^2\frac{{\hat{D}}_d^2}{{\hat{C}}_d^4},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} \hat{\sigma }_{\kappa }^{2}=\hat{\sigma }_{D}^{2}\frac{1}{\hat{C}_{d}^{2}} -2\hat{\sigma }_{CD}\frac{\hat{D}_{d}}{\hat{C}_{d}^{3}} +\hat{\sigma }_{C}^{2}\frac{\hat{D}_{d}^{2}}{\hat{C}_{d}^{4}}, \end{aligned}$$\end{document}

(4.7) $\begin{array}{cc}& {\hat{\sigma }}_{\pi }^2={\hat{\sigma }}_D^2\frac{1}{{\hat{F}}_d^2}-2{\hat{\sigma }}_{\mathrm{FD}}\frac{{\hat{D}}_d}{{\hat{F}}_d^3}+{\hat{\sigma }}_F^2\frac{{\hat{D}}_d^2}{{\hat{F}}_d^4}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} \hat{\sigma }_{\pi }^{2}=\hat{\sigma }_{D}^{2}\frac{1}{\hat{F}_{d}^{2}} -2\hat{\sigma }_{FD}\frac{\hat{D}_{d}}{\hat{F}_{d}^{3}}+\hat{\sigma }_{F}^{2} \frac{\hat{D}_{d}^{2}}{\hat{F}_{d}^{4}}. \end{aligned}$$\end{document}

The variance estimator ${\hat{\sigma }}_{\pi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\sigma }_{\pi }^{2}$$\end{document} coincides with that of Gwet (Reference Gwet2021, equation 4) in the case of nominal weights; see Appendix (Sect. 6) for a proof sketch.

4.3. Improving Approximate Normality with the Arcsine and Fisher Transforms

It is well known that the Fisher transform (Fisher, Reference Fisher1915) improves the inference for the correlation coefficient. If r is the sample correlation, $\text{artanh}\left(r\right)=\frac{1}{2}log[(1+r)/(1-r)]$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\textrm{artanh}\,}}(r)=\frac{1}{2}\log [(1+r)/(1-r)]$$\end{document} has approximately the same variance for most r, and its distribution is closer to normal than that of the untransformed r, especially when the population correlation is close to $\pm 1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pm 1$$\end{document} . This transform makes sense outside the world of correlations; for instance, Lin (Reference Lin1989) used the Fisher transform to improve the normality of the quadratically weighted Cohen’s kappa.

The arcsine is another reasonable transformation of ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} and ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} . The arcsine is the inverse of the sine function and is defined as $arcsinx=\int 1/\sqrt{1-x^2}\text{d}x$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\arcsin x=\int 1/\sqrt{1-x^{2}}\textrm{d}x$$\end{document} . In ecology (Warton and Hui, Reference Warton and Hui2011), the arcsine transformation denotes $arcsin\sqrt{p}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\arcsin \sqrt{p}$$\end{document} , where p is a probability. We do not take square root, however, as ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} and ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} can be negative.

Calculating the limiting variance of $arcsin{\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\arcsin \hat{\kappa }_{d}$$\end{document} and $arcsin{\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\arcsin \hat{\pi }_{d}$$\end{document} requires an additional application of the delta method (4.2). Using that $\frac{\text{d}}{\text{d}x}arcsin\left(x\right)=1/\sqrt{1-x^2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{\textrm{d}}{\textrm{d}x}\arcsin (x)=1/\sqrt{1-x^{2}}$$\end{document} and $\frac{\text{d}}{\text{d}x}\text{artanh}\left(x\right)=1/(1-x^2)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{\textrm{d}}{\textrm{d}x}{{\,\textrm{artanh}\,}}(x)=1/(1-x^{2})$$\end{document} , we find

(4.8) $\begin{array}{cc}\sqrt{n}(arcsin{\hat{\kappa }}_d-arcsin{\kappa }_d)\to & N(0,{(1-{\kappa }_d^2)}^{-1}{\sigma }_{\kappa }^2),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sqrt{n}(\arcsin \hat{\kappa }_{d}-\arcsin \kappa _{d})\rightarrow & {} N(0,(1-\kappa _{d}^{2})^{-1}\sigma _{\kappa }^{2}), \end{aligned}$$\end{document}

(4.9) $\begin{array}{cc}\sqrt{n}(\text{artanh}{\hat{\kappa }}_d-\text{artanh}{\kappa }_d)\to & N(0,{(1-{\kappa }_d^2)}^{-2}{\sigma }_{\kappa }^2).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sqrt{n}({{\,\textrm{artanh}\,}}\hat{\kappa }_{d}-{{\,\textrm{artanh}\,}}\kappa _{d})\rightarrow & {} N(0,(1-\kappa _{d}^{2})^{-2}\sigma _{\kappa }^{2}). \end{aligned}$$\end{document}

Expressions for ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} can be found by swapping ${\kappa }_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\kappa _{d}$$\end{document} for ${\pi }_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\pi _{d}$$\end{document} and ${\sigma }_{\kappa }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{\kappa }^{2}$$\end{document} for ${\sigma }_{\pi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _{\pi }^{2}$$\end{document} .

Example 3

This example illustrates that the arcsine and Fisher transforms may make the sampling distribution closer to the normal distribution. Let the number of raters be $R=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R=3$$\end{document} , the disagreement function be quadratic (with $g=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$g=2$$\end{document} ), and the number of items be $n=20$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=20$$\end{document} . There are five categories and the true classification of an item is one of $\{1,2,3,4,5\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{1,2,3,4,5\}$$\end{document} with probability 1/5 each. Every rater knows the true classification of an item with probability 0.9. If they do not know the correct classification, they will guess a classification from $\{1,2,3,4,5\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{1,2,3,4,5\}$$\end{document} uniformly at random. One can show that the population value of the quadratically weighted Cohen’s kappa is 0.816 under these circumstances, following the arguments of Perreault and Leigh (Reference Perreault and Leigh1989). We simulate the value of ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} a total of $N=50,000$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=50,000$$\end{document} times and transform them using the identity transform, the arcsine transform, and the Fisher transform. The results are shown in Fig. 1. The arcsine transform appears to bring the sampling distribution of ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} closer to the normal distribution, with the Fisher transform also improving normality quite a bit.

Figure 1 Simulated sampling distribution of ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} for quadratic weights using three transformations, $n=20,R=3$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=20, R=3$$\end{document} . The simulation setup is described in Example 3. The arcsine transform makes the sampling distribution closest to the normal distribution.

5. Confidence Intervals

Using the methodology we have developed, we can easily construct confidence intervals for the agreement coefficients.

We describe our three confidence interval constructions only for Cohen’s kappa, as the intervals using Fleiss’ kappa can be found by replacing every instance ${\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\kappa }_{d}$$\end{document} with ${\hat{\pi }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\pi }_{d}$$\end{document} and ${\hat{\sigma }}_{\kappa }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\sigma }_{\kappa }^{2}$$\end{document} with ${\hat{\sigma }}_{\pi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\sigma }_{\pi }^{2}$$\end{document} . We use the two-sided t-distribution-based confidence intervals with nominal level $1-\alpha =0.95$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1-\alpha =0.95$$\end{document} . Let c be the $(1-\alpha /2)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(1-\alpha /2)$$\end{document} -quantile of the t distribution with $n-1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n-1$$\end{document} degrees of freedom. The basic interval is

(5.1) $\begin{array}{c}[{\hat{\kappa }}_d-c{\hat{\sigma }}_{\kappa }/\sqrt{n-1},{\hat{\kappa }}_d+c{\hat{\sigma }}_{\kappa }/\sqrt{n-1}],\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {[}\hat{\kappa }_{d}-c\hat{\sigma }_{\kappa }/\sqrt{n-1}, \hat{\kappa }_{d}+c\hat{\sigma }_{\kappa }/\sqrt{n-1}], \end{aligned}$$\end{document}

where ${\hat{\sigma }}_{\kappa }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\sigma }_\kappa $$\end{document} is the estimated variance described in equation (4.6).

The arcsine interval replaces the basic limits with

(5.2) $\begin{array}{c}sin\left(arcsin,{\hat{\kappa }}_d,\pm ,c,{(1-{\hat{\kappa }}_d^2)}^{-1/2},{\hat{\sigma }}_{\kappa },/,\sqrt{n-1}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sin \left( \arcsin \hat{\kappa }_{d}\pm c(1-\hat{\kappa }_{d}^{2})^{-1/2}\hat{\sigma }_{\kappa }/ \sqrt{n-1}\right) , \end{aligned}$$\end{document}

where ${(1-{\hat{\kappa }}_d^2)}^{-1}{\hat{\sigma }}_{\kappa }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(1-\hat{\kappa }_{d}^{2})^{-1}\hat{\sigma }_{\kappa }^{2}$$\end{document} is the asymptotic variance of $arcsin{\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\arcsin \hat{\kappa }_{d}$$\end{document} (4.8). The Fisher interval uses the area hyperbolic tangent,

(5.3) $\begin{array}{c}tanh\left(\text{artanh},{\hat{\kappa }}_d,\pm ,c,{(1-{\hat{\kappa }}_d^2)}^{-1},{\hat{\sigma }}_{\kappa },/,\sqrt{n-1}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tanh \left( {{\,\textrm{artanh}\,}}\hat{\kappa }_{d}\pm c(1-\hat{\kappa }_{d}^{2})^{-1}\hat{\sigma }_{\kappa }/ \sqrt{n-1}\right) , \end{aligned}$$\end{document}

where ${(1-{\hat{\kappa }}_d^2)}^{-2}{\hat{\sigma }}_{\kappa }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(1-\hat{\kappa }_{d}^{2})^{-2}\hat{\sigma }_{\kappa }^{2}$$\end{document} is the asymptotic variance of $\text{artanh}{\hat{\kappa }}_d$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{\,\textrm{artanh}\,}}\hat{\kappa }_{d}$$\end{document} (4.9).