1.1. Standardized Person-Fit Statistics

Consider the class of standardized person-fit statistics that was introduced by Snijders (Reference Snijders2001) and takes the form

(2) $\begin{array}{c}T\left(\theta \right)=\frac{W\left(\theta \right)}{\sqrt{\text{Var}\left(W\right(\theta \left)\right)}},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T(\theta )=\frac{W(\theta )}{\sqrt{\text {Var}(W(\theta ))}}, \end{aligned}$$\end{document}

where

(3) $\begin{array}{c}W\left(\theta \right)=\sum _{i=1}^n(X_i-p_i\left(\theta \right))w_i\left(\theta \right)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} W(\theta ) = \sum _{i=1}^n (X_i - p_i(\theta )) w_i(\theta ) \end{aligned}$$\end{document}

for some suitable weight function $w_i\left(\theta \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_i(\theta )$$\end{document} . For the standardized log-likelihood statistic $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} (Drasgow et al. Reference Drasgow, Levine and Williams1985), the weight function is given by

(4) $\begin{array}{c}{w}_i\left(\theta \right)=log\frac{p_i\left(\theta \right)}{q_i\left(\theta \right)},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w_i(\theta ) = \log \frac{p_i(\theta )}{q_i(\theta )}, \end{aligned}$$\end{document}

where $q_i\left(\theta \right)=1-p_i\left(\theta \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$q_i(\theta )=1-p_i(\theta )$$\end{document} . For the standardized extended caution indices ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} and ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} (Tatsuoka Reference Tatsuoka1984), the weight functions are given by

(5) $\begin{array}{c}{w}_i\left(\theta \right)=g-g_i\text{and}{w}_i\left(\theta \right)=h\left(\theta \right)-p_i\left(\theta \right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} w_i(\theta ) = g - g_i \text { and } w_i(\theta ) = h(\theta ) - p_i(\theta ), \end{aligned}$$\end{document}

respectively, for

$\begin{array}{cc}& g_i=\frac{1}{N}\sum _{v=1}^N{p}_i\left({\theta }_v\right),\\ & h\left({\theta }_v\right)=\frac{1}{n}\sum _{i=1}^n{p}_i\left({\theta }_v\right),\text{and}\\ & g=\frac{1}{n}\sum _{i=1}^n{g}_i=\frac{1}{N\times n}\sum _{v=1}^N\sum _{i=1}^n{p}_i\left({\theta }_v\right)=\frac{1}{N}\sum _{v=1}^{N}h\left({\theta }_v\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} g_i=\frac{1}{N} \sum _{v=1}^N p_i(\theta _v), \\{} & {} h(\theta _v)=\frac{1}{n} \sum _{i=1}^n p_i(\theta _v), \text { and } \\{} & {} g=\frac{1}{n} \sum _{i=1}^n g_i = \frac{1}{N \times n} \sum _{v=1}^N \sum _{i=1}^n p_i(\theta _v) = \frac{1}{N} \sum _{v=1}^N h(\theta _v), \end{aligned}$$\end{document}

where ${\theta }_v$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _v$$\end{document} is the ability parameter of examinee v, and N is the total number of examinees.

Equation 3 implies that

$\begin{array}{cc}E\left(W\right(\theta \left)\right)& =\mu \left(\theta \right),\\ \text{Var}\left(W\right(\theta \left)\right)& =E\left({(W\left(\theta \right)-\mu \left(\theta \right))}^2\right)={\sigma }^2\left(\theta \right),\\ \text{Skew}\left(W\right(\theta \left)\right)& =E\left({\left(\frac{W\left(\theta \right)-\mu \left(\theta \right)}{\sigma \left(\theta \right)}\right)}^3\right)=\gamma \left(\theta \right),\text{and}\\ \text{Kurt}\left(W\right(\theta \left)\right)& =E\left({\left(\frac{W\left(\theta \right)-\mu \left(\theta \right)}{\sigma \left(\theta \right)}\right)}^4\right)=\kappa \left(\theta \right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} E(W(\theta ))&= \mu (\theta ), \\ \text {Var}(W(\theta ))&= E\left[ (W(\theta )-\mu (\theta ))^2 \right] = \sigma ^2(\theta ), \\ \text {Skew}(W(\theta ))&= E\left[ \left( \frac{W(\theta )-\mu (\theta )}{\sigma (\theta )}\right) ^3 \right] = \gamma (\theta ), \text { and } \\ \text {Kurt}(W(\theta ))&= E\left[ \left( \frac{W(\theta )-\mu (\theta )}{\sigma (\theta )}\right) ^4 \right] = \kappa (\theta ), \end{aligned}$$\end{document}

where

(6) $\begin{array}{cc}\mu \left(\theta \right)& =0,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \mu (\theta )&= 0, \end{aligned}$$\end{document}

(7) $\begin{array}{cc}{\sigma }^2\left(\theta \right)& =\sum _{i=1}^n{p}_i\left(\theta \right)q_i\left(\theta \right)w_i^2\left(\theta \right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \sigma ^2(\theta )&= \sum _{i=1}^n p_i(\theta ) q_i(\theta ) w_i^2(\theta ), \end{aligned}$$\end{document}

(8) $\begin{array}{cc}\gamma \left(\theta \right)& =\frac{{\sum }_{i=1}^n{p}_i\left(\theta \right)q_i\left(\theta \right)(q_i\left(\theta \right)-p_i\left(\theta \right))w_i^3\left(\theta \right)}{{\sigma }^3\left(\theta \right)},\text{and}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \gamma (\theta )&= \frac{\sum _{i=1}^n p_i(\theta ) q_i(\theta ) (q_i(\theta ) - p_i(\theta )) w_i^3(\theta )}{\sigma ^3(\theta )}, \text { and} \end{aligned}$$\end{document}

(9) $\begin{array}{cc}\kappa \left(\theta \right)& =\frac{{\sum }_{i=1}^n{p}_i\left(\theta \right)q_i\left(\theta \right)(1-3p_i\left(\theta \right)q_i\left(\theta \right))w_i^4\left(\theta \right)}{{\sigma }^4\left(\theta \right)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \kappa (\theta )&= \frac{\sum _{i=1}^n p_i(\theta ) q_i(\theta )(1 - 3 p_i(\theta ) q_i(\theta )) w_i^4(\theta )}{\sigma ^4(\theta )}. \end{aligned}$$\end{document}

Therefore, the standardized person-fit statistic of Eq. 2 can be expressed as

(10) $\begin{array}{c}T\left(\theta \right)=\frac{W\left(\theta \right)-\mu \left(\theta \right)}{\sigma \left(\theta \right)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T(\theta ) = \frac{W(\theta ) - \mu (\theta )}{\sigma (\theta )}. \end{aligned}$$\end{document}

To classify a score pattern as aberrant, the significance probability p is computed as the probability that under the null distribution, the value of the test statistic T is equal to or exceeds the observed value t. That is,

(11) $\begin{array}{c}p=\left(\begin{array}{cc}P(T\le t)& \text{if extreme negative values of}t\text{indicate misfit,}\\ P(T\ge t)& \text{if extreme positive values of}t\text{indicate misfit.}\end{array}\right)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p = {\left\{ \begin{array}{ll} P(T \le t) &{}\text {if extreme negative values of }t\hbox { indicate misfit,} \\ P(T \ge t) &{}\text {if extreme positive values of }t\hbox { indicate misfit.} \end{array}\right. } \end{aligned}$$\end{document}

For the $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} statistic, extreme negative values indicate misfit. For the ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} and ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} statistics, extreme positive values indicate misfit.

Standardized person-fit statistics are assumed to have a standard normal null distribution. However, this distribution only holds when both of the following conditions are satisfied: (a) the true ability is known and is used to compute T and (b) an infinite number of items are available and are used to compute T. Figure 1 shows the null distribution of the $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} statistic when both conditions are (almost) satisfied—that is, when the true ability and 500 items are used to compute $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} . Observe that the null distribution (dashed black line) is very close to the theorized standard normal distribution (solid black line).

Figure 1 The null distributions of $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} .

However, when condition (a) is not satisfied—that is, when the true ability is unknown and an ability estimate is used instead—the null distribution of T has a variance smaller than 1, as indicated by the dashed gray line in Fig. 1. Thus, the assumption of a standard normal null distribution (that has a variance of 1) leads to a conservative assessment of person fit. When condition (b) is not satisfied—that is, when a finite number of items are used to compute T—the null distribution of T is skewed, as indicated by the dotted black line in Fig. 1, which represents a 12-item test. The distribution is negatively skewed if extreme negative values of the statistic indicate misfit (e.g., $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} ) or positively skewed if extreme positive values of the statistic indicate misfit (e.g., ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} , ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} ). Thus, the assumption of a standard normal null distribution (that is not skewed) leads to a liberal assessment of person fit. When both (a) and (b) are not satisfied, the null distribution of T is skewed, has a variance smaller than 1, and has a mean that differs slightly from 0, as indicated by the dotted gray line in Fig. 1. Thus, the assumption of a standard normal null distribution leads to either a liberal or a conservative assessment of person fit, depending on the chosen significance level: the use of smaller significance levels leads to a more liberal assessment of person fit, while the use of larger significance levels leads to a more conservative assessment.

In an effort to obtain a more accurate assessment of person fit, several researchers have proposed corrections for T. Mean and variance corrections have been suggested to account for the use of an estimated ability parameter. Skewness corrections have been suggested to account for the use of a finite number of items. The following subsections contain reviews of each of these corrections.

1.2. Mean and Variance Corrections

When the true ability is unknown and an ability estimate is used instead, a naïve approximation of $T\left(\theta \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T(\theta )$$\end{document} can be obtained by inserting $\hat{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\theta }$$\end{document} into Eq. 10. That is,

(12) $\begin{array}{c}T\left(\hat{\theta }\right)=\frac{W\left(\hat{\theta }\right)-\mu \left(\hat{\theta }\right)}{\sigma \left(\hat{\theta }\right)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T(\hat{\theta }) = \frac{W(\hat{\theta }) - \mu (\hat{\theta })}{\sigma (\hat{\theta })}. \end{aligned}$$\end{document}

However, Snijders (Reference Snijders2001) proved that replacing $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} with $\hat{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\theta }$$\end{document} has a non-negligible effect on the variance of W—and therefore, T—even when an infinite number of items are used. Thus, the assumption that $T\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T(\hat{\theta })$$\end{document} has a standard normal null distribution, even asymptotically, is incorrect. Snijders further showed that if $\hat{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\theta }$$\end{document} satisfies the condition

(13) $\begin{array}{c}{r}_0\left(\hat{\theta }\right)+\sum _{i=1}^n(X_i-p_i\left(\hat{\theta }\right))r_i\left(\hat{\theta }\right)=0\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_0(\hat{\theta }) + \sum _{i=1}^n (X_i - p_i(\hat{\theta })) r_i(\hat{\theta }) = 0 \end{aligned}$$\end{document}

for some functions $r_0\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_0(\hat{\theta })$$\end{document} and $r_i\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_i(\hat{\theta })$$\end{document} , then the mean and variance of $W\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W(\hat{\theta })$$\end{document} can be approximated using

$\begin{array}{cc}& E\left(W\left(\hat{\theta }\right)\right)\approx \tilde{\mu }\left(\hat{\theta }\right)\text{and}\\ & \text{Var}\left(W\left(\hat{\theta }\right)\right)\approx {\tilde{\sigma }}^2\left(\hat{\theta }\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} E(W(\hat{\theta })) \approx \tilde{\mu }(\hat{\theta }) \text { and }\\{} & {} \text {Var}(W(\hat{\theta })) \approx \tilde{\sigma }^2(\hat{\theta }), \end{aligned}$$\end{document}

respectively, where

(14) $\begin{array}{c}\tilde{\mu }\left(\hat{\theta }\right)=-c\left(\hat{\theta }\right)r_0\left(\hat{\theta }\right)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{\mu }(\hat{\theta }) = -c(\hat{\theta }) r_0(\hat{\theta }) \end{aligned}$$\end{document}

and

(15) $\begin{array}{c}{\tilde{\sigma }}^2\left(\hat{\theta }\right)=\sum _{i=1}^n{p}_i\left(\hat{\theta }\right)q_i\left(\hat{\theta }\right){\tilde{w}}_i^2\left(\hat{\theta }\right)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{\sigma }^2(\hat{\theta }) = \sum _{i=1}^n p_i(\hat{\theta }) q_i(\hat{\theta }) \tilde{w}_i^2(\hat{\theta }) \end{aligned}$$\end{document}

for

(16) $\begin{array}{cc}& c\left(\hat{\theta }\right)=\frac{{\sum }_{i=1}^n{p}_i^{\prime }\left(\hat{\theta }\right)w_i\left(\hat{\theta }\right)}{{\sum }_{i=1}^n{p}_i^{\prime }\left(\hat{\theta }\right)r_i\left(\hat{\theta }\right)},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} c(\hat{\theta }) = \frac{\sum _{i=1}^n p_i^\prime (\hat{\theta }) w_i(\hat{\theta })}{\sum _{i=1}^n p_i^\prime (\hat{\theta }) r_i(\hat{\theta })}, \end{aligned}$$\end{document}

where $p_i^{\prime }\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_i^\prime (\hat{\theta })$$\end{document} is the first derivative of $p_i\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_i(\hat{\theta })$$\end{document} with respect to $\hat{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\theta }$$\end{document} , and the modified weight function is given by

(17) $\begin{array}{c}{\tilde{w}}_i\left(\hat{\theta }\right)=w_i\left(\hat{\theta }\right)-c\left(\hat{\theta }\right)r_i\left(\hat{\theta }\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{w}_i(\hat{\theta }) = w_i(\hat{\theta }) - c(\hat{\theta }) r_i(\hat{\theta }). \end{aligned}$$\end{document}

Therefore, the asymptotically correct statistic $T^{\ast }\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*(\hat{\theta })$$\end{document} can be derived from $T\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T(\hat{\theta })$$\end{document} by adjusting both the mean and variance of the statistic. In other words,

(18) $\begin{array}{c}{T}^{\ast }\left(\hat{\theta }\right)=\frac{W\left(\hat{\theta }\right)-\tilde{\mu }\left(\hat{\theta }\right)}{\tilde{\sigma }\left(\hat{\theta }\right)}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T^*(\hat{\theta }) = \frac{W(\hat{\theta }) - \tilde{\mu }(\hat{\theta })}{\tilde{\sigma }(\hat{\theta })} \end{aligned}$$\end{document}

has an asymptotic standard normal null distribution.

The corrected statistic $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} can be computed using any ability estimate $\hat{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\theta }$$\end{document} that has functions $r_0\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_0(\hat{\theta })$$\end{document} and $r_i\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$r_i(\hat{\theta })$$\end{document} which satisfy Eq. 13. Magis et al. (Reference Magis, Raîche and Béland2012) showed that for the weighted likelihood (WL) estimate (Warm Reference Warm1989), maximum likelihood (ML) estimate, and maximum a posteriori (MAP) estimate, Eq. 13 is satisfied for

$\begin{array}{c}{r}_i\left(\hat{\theta }\right)=\frac{p_i^{\prime }\left(\hat{\theta }\right)}{p_i\left(\hat{\theta }\right)q_i\left(\hat{\theta }\right)}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_i(\hat{\theta }) = \frac{p_i^\prime (\hat{\theta })}{p_i(\hat{\theta }) q_i(\hat{\theta })} \end{aligned}$$\end{document}

and

$\begin{array}{c}{r}_0\left(\hat{\theta }\right)=\left(\begin{array}{cc}\frac{J\left(\hat{\theta }\right)}{2I\left(\hat{\theta }\right)}& \text{if}\hat{\theta }\text{is the WL estimate,}\\ 0& \text{if}\hat{\theta }\text{is the ML estimate,}\\ \frac{dlogf\left(\hat{\theta }\right)}{d\hat{\theta }}& \text{if}\hat{\theta }\text{is the MAP estimate,}\end{array}\right)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} r_0(\hat{\theta }) = {\left\{ \begin{array}{ll} \frac{J(\hat{\theta })}{2 I(\hat{\theta })} &{}\text {if }\hat{\theta }\hbox { is the WL estimate,} \\ 0 &{}\text {if }\hat{\theta }\text { is the ML estimate,} \\ \frac{d \log f(\hat{\theta })}{d \hat{\theta }} &{}\text {if } \hat{\theta }\hbox { is the MAP estimate,} \end{array}\right. } \end{aligned}$$\end{document}

where

$\begin{array}{cc}& J\left(\hat{\theta }\right)=\sum _{i=1}^n\frac{p_i^{\prime }\left(\hat{\theta }\right)p_i^{″}\left(\hat{\theta }\right)}{p_i\left(\hat{\theta }\right)q_i\left(\hat{\theta }\right)},\\ & I\left(\hat{\theta }\right)=\sum _{i=1}^n\frac{{\left[p_i^{\prime }\left(\hat{\theta }\right)\right]}^2}{p_i\left(\hat{\theta }\right)q_i\left(\hat{\theta }\right)},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} J(\hat{\theta }) = \sum _{i=1}^n \frac{p_i^\prime (\hat{\theta }) p_i^{\prime \prime }(\hat{\theta })}{p_i(\hat{\theta }) q_i(\hat{\theta })}, \\{} & {} I(\hat{\theta }) = \sum _{i=1}^n \frac{[p_i^\prime (\hat{\theta })]^2}{p_i(\hat{\theta }) q_i(\hat{\theta })}, \end{aligned}$$\end{document}

and $f(·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$f(\cdot )$$\end{document} is the prior distribution on $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} .

1.3. Skewness Corrections

Researchers such as Molenaar & Hoijtink (Reference Molenaar and Hoijtink1990), Noonan et al. (Reference Noonan, Boss and Gessaroli1992), and van Krimpen-Stoop & Meijer (Reference van Krimpen-Stoop and Meijer1999) have shown that the null distribution of T becomes more skewed as test length decreases. Therefore, the standard normal distribution (that is not skewed) provides a poor approximation for tests with fewer items. To obtain a more accurate approximation of the null distribution of T, several methods have been developed that take this skewness into account. These methods use naïve approximations of the mean, variance, and skewness of $W\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$W(\hat{\theta })$$\end{document} that are obtained by inserting $\hat{\theta }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\hat{\theta }$$\end{document} into Eqs. 6, 7, and 8, respectively. For example, the naïve approximation of skewness is given by

(19) $\begin{array}{cc}& \gamma \left(\hat{\theta }\right)=\frac{{\sum }_{i=1}^n{p}_i\left(\hat{\theta }\right)q_i\left(\hat{\theta }\right)(q_i\left(\hat{\theta }\right)-p_i\left(\hat{\theta }\right))w_i^3\left(\hat{\theta }\right)}{{\sigma }^3\left(\hat{\theta }\right)}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} \gamma (\hat{\theta }) = \frac{\sum _{i=1}^n p_i(\hat{\theta }) q_i(\hat{\theta }) (q_i(\hat{\theta }) - p_i(\hat{\theta })) w_i^3(\hat{\theta })}{\sigma ^3(\hat{\theta })}. \end{aligned}$$\end{document}

Molenaar & Hoijtink (Reference Molenaar and Hoijtink1990) suggested two methods to approximate the null distribution of T. The first method is based on the Cornish–Fisher expansion. The skewness-corrected statistic is given by

(20) $\begin{array}{cc}& T_{\text{CF}}\left(\hat{\theta }\right)=T\left(\hat{\theta }\right)-\frac{\gamma \left(\hat{\theta }\right)[{\left(T\left(\hat{\theta }\right)\right)}^2-1]}{12},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} T_\text {CF}(\hat{\theta }) = T(\hat{\theta }) - \frac{\gamma (\hat{\theta })[(T(\hat{\theta }))^2 - 1]}{12}, \end{aligned}$$\end{document}

which is equivalent to approximating the significance probability of Eq. 11 as

(21) $\begin{array}{c}{p}_{\text{CF}}=\left(\begin{array}{cc}\Phi \left(T_{\text{CF}},\left(\hat{\theta }\right)\right)& \text{if extreme negative values of}t\text{indicate misfit,}\\ 1-\Phi \left(T_{\text{CF}},\left(\hat{\theta }\right)\right)& \text{if extreme positive values of}t\text{indicate misfit,}\end{array}\right)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p_\text {CF} = {\left\{ \begin{array}{ll} \Phi \left( T_\text {CF}(\hat{\theta }) \right) &{}\text {if extreme negative values of }t\hbox { indicate misfit,} \\ 1 - \Phi \left( T_\text {CF}(\hat{\theta }) \right) &{}\text {if extreme positive values of }t\hbox { indicate misfit,} \end{array}\right. } \end{aligned}$$\end{document}

where $\Phi (·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi (\cdot )$$\end{document} denotes the cumulative distribution function (CDF) of the standard normal distribution.

The second method of Molenaar & Hoijtink (Reference Molenaar and Hoijtink1990) employs a higher-order approximation of the significance probability that is based on a ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} distribution with $\nu \left(\hat{\theta }\right)=\frac{8}{{\gamma }^2\left(\hat{\theta }\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nu (\hat{\theta })=\frac{8}{\gamma ^2(\hat{\theta })}$$\end{document} degrees of freedom. Thus, the significance probability is approximated as

(22) $\begin{array}{c}{p}_{{\chi }^2}=\left(\begin{array}{cc}P\left({\chi }^2,\left(\nu \left(\hat{\theta }\right)\right),\ge ,\frac{|W\left(\hat{\theta }\right)-\mu \left(\hat{\theta }\right)-a\left(\hat{\theta }\right)|}{b\left(\hat{\theta }\right)}\right)& \text{if extreme negative values of}t\text{indicate misfit,}\\ P\left({\chi }^2,\left(\nu \left(\hat{\theta }\right)\right),\ge ,\frac{|W\left(\hat{\theta }\right)-\mu \left(\hat{\theta }\right)+a\left(\hat{\theta }\right)|}{b\left(\hat{\theta }\right)}\right)& \text{if extreme positive values of}t\text{indicate misfit,}\end{array}\right)\\ \end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p_{\chi ^2} = {\left\{ \begin{array}{ll} P\left( \chi ^2(\nu (\hat{\theta })) \ge \frac{|W(\hat{\theta }) - \mu (\hat{\theta }) - a(\hat{\theta })|}{b(\hat{\theta })} \right) &{}\text {if extreme negative values of }t\text { indicate misfit,} \\ P\left( \chi ^2(\nu (\hat{\theta })) \ge \frac{|W(\hat{\theta }) - \mu (\hat{\theta }) + a(\hat{\theta })|}{b(\hat{\theta })} \right) &{}\text {if extreme positive values of }t\hbox { indicate misfit,} \end{array}\right. } \nonumber \\ \end{aligned}$$\end{document}

where

$\begin{array}{cc}& a\left(\hat{\theta }\right)=b\left(\hat{\theta }\right)\nu \left(\hat{\theta }\right)\text{and}\\ & b\left(\hat{\theta }\right)=\sqrt{\frac{{\sigma }^2\left(\hat{\theta }\right)}{2\nu \left(\hat{\theta }\right)}}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}{} & {} a(\hat{\theta })=b(\hat{\theta })\nu (\hat{\theta }) \text { and }\\{} & {} b(\hat{\theta })=\sqrt{\frac{\sigma ^2(\hat{\theta })}{2\nu (\hat{\theta })}}. \end{aligned}$$\end{document}

A third method was suggested by Bedrick (Reference Bedrick1997), who used the Edgeworth expansion to approximate the significance probability as

(23) $\begin{array}{c}{p}_{\text{EW}}=\left(\begin{array}{cc}\Phi \left(T\left(\hat{\theta }\right)\right)-\frac{\varphi \left(T\left(\hat{\theta }\right)\right)\gamma \left(\hat{\theta }\right)[{\left(T\left(\hat{\theta }\right)\right)}^2-1]}{6}& \text{if extreme negative values of}t\text{indicate misfit,}\\ 1-\left(\Phi ,\left(T\left(\hat{\theta }\right)\right),-,\frac{\varphi \left(T\left(\hat{\theta }\right)\right)\gamma \left(\hat{\theta }\right)[{\left(T\left(\hat{\theta }\right)\right)}^2-1]}{6}\right)& \text{if extreme positive values of}t\text{indicate misfit,}\end{array}\right)\\ \end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p_\text {EW} = {\left\{ \begin{array}{ll} \Phi (T(\hat{\theta })) - \frac{\phi (T(\hat{\theta })) \gamma (\hat{\theta })[(T(\hat{\theta }))^2 - 1]}{6} &{}\text {if extreme negative values of }t\text { indicate misfit,} \\ 1 - \left( \Phi (T(\hat{\theta })) - \frac{\phi (T(\hat{\theta })) \gamma (\hat{\theta })[(T(\hat{\theta }))^2 - 1]}{6} \right) &{}\text {if extreme positive values of }t\hbox { indicate misfit,} \end{array}\right. }\nonumber \\ \end{aligned}$$\end{document}

where $\varphi (·)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\phi (\cdot )$$\end{document} denotes the probability density function of the standard normal distribution. The Edgeworth expansion occasionally yields estimates of p that are smaller than 0 or larger than 1. In this paper, we replace such values with (traditional) estimates of p that are obtained by applying $T\left(\hat{\theta }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T(\hat{\theta })$$\end{document} under the assumption of a standard normal null distribution.

We note that it may be desirable to transform the significance probability approximations in Eqs. 22 and 23 to person-fit statistics that are approximately standard normally distributed. If extreme negative values of t indicate misfit, the significance probability approximations can be transformed using the inverse CDF of the standard normal distribution, ${\Phi }^{-1}\left(p\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi ^{-1}(p)$$\end{document} . If extreme positive values of t indicate misfit, the significance probability approximations can be transformed using ${\Phi }^{-1}(1-p)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Phi ^{-1}(1-p)$$\end{document} .

3.1. Design and Analysis

We conducted a detailed simulation study to (a) examine the null distributions and (b) compare the Type I error rates and power of the new and existing statistics. The simulations were designed to mimic realistic testing conditions—therefore, an estimated ability parameter and a finite number of items were used to compute the person-fit statistics.

Three test lengths (12, 36, 72) were studied to represent short, medium, and long tests. For each test length, 1 million (10,000 examinees $\times$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\times $$\end{document} 100 replications) score patterns were simulated. For each replication, new sets of person and item parameters were generated. As in Glas & Meijer (Reference Glas and Meijer2003), 90% of the examinees were non-aberrant—that is, they fit the model—and were used to study the null distributions and Type I error rates of the statistics. The remaining 10% of the examinees were divided equally into four groups of aberrant examinees and were used to study power. The four groups of aberrant examinees were characterized by the type of aberrant behavior (lack of motivation, item disclosure) and by the proportion of contaminated items ( $\frac{1}{6}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{1}{6}$$\end{document} , $\frac{1}{3}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{1}{3}$$\end{document} ).

Uncontaminated item scores were generated using the 3PLM. For each replication, the item parameters were sampled such that $a_i\sim Lognormal(0,0.{25}^2)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$a_i \sim Lognormal(0,0.25^2)$$\end{document} , $b_i\sim N(0,1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$b_i \sim \mathcal {N}(0,1)$$\end{document} , and $c_i\sim U(0.05,0.30)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$c_i \sim \mathcal {U}(0.05,0.30)$$\end{document} , as in Sinharay (Reference Sinharay2016b), and the person parameters were sampled such that ${\theta }_v\sim N(0,1)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _v \sim \mathcal {N}(0,1)$$\end{document} . Contaminated item scores were generated after manipulating the item success probabilities. As in Glas & Meijer (Reference Glas and Meijer2003), lack of motivation was simulated as random guessing on the easiest items, with a success probability equal to 0.2. Item disclosure was simulated as preknowledge of the most difficult items, with a success probability equal to 0.9. By simulating lack of motivation on the easiest items and item disclosure on the most difficult items, we studied conditions in which ability estimates would be severely impacted and are therefore important to detect.

After simulating the data, each of the score patterns was analyzed 72 times: once for each combination of two classes of person-fit statistics (T, $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} ), three weight functions ( $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} , ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} , ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} ), four skewness corrections (none, Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, Edgeworth expansion), and three ability estimates (WL, ML, MAP). In all conditions, the item parameters were treated as known. This assumption is common in person-fit research, as it prevents the null distribution of the statistics from being affected by any uncertainty in the item parameter estimates (e.g., Molenaar & Hoijtink Reference Molenaar and Hoijtink1990; Snijders Reference Snijders2001; van Krimpen-Stoop & Meijer Reference van Krimpen-Stoop and Meijer1999).

The ML and MAP estimates of ability were bounded between $-4$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-4$$\end{document} and 4. The standard normal distribution was used as the prior distribution for the MAP estimates. The standardized extended caution indices were computed after reversing the sign of the weights given in Eq. 5; thus, extreme negative values of the statistics indicated misfit. This adjustment was made to facilitate comparisons against the standardized log-likelihood statistics, for which extreme negative values also indicate misfit.

3.2. Results

The choice of ability estimate was found not to affect the relative performance of the person-fit statistics. Therefore, we focus on the WL estimates of ability (since they were computed without any bounds or prior distributions) and include results for the other ability estimates in “Appendix B”.

3.2.1. The Null Distributions of the Person-Fit Statistics

Figure 3 displays the first four moments of the null distributions of the person-fit statistics. Each row corresponds to a different moment (mean, variance, skewness, excess kurtosis), and each column corresponds to a different test length (12, 36, 72). Note that excess kurtosis is defined as the kurtosis minus 3. Horizontal dotted lines are used to indicate the values that are expected under the theoretical null distribution. It is desirable for the moments of the empirical null distributions to be as close to these values as possible.

Figure 3 reveals that the null distribution of T (i.e., the uncorrected statistic given by Eq. 12) is negatively skewed, has a variance smaller than 1, and has a mean that is slightly larger than 0. Similar results are shown in Table 1 of Li & Olejnik (Reference Li and Olejnik1997), Table 3 of Reise (Reference Reise1995), and Table 1 of van Krimpen-Stoop & Meijer (Reference van Krimpen-Stoop and Meijer1999). The skewness-corrected statistics $T_{\text{CF}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}$$\end{document} , $T_{{\chi }^2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}$$\end{document} , and $T_{\text{EW}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}$$\end{document} return values of skewness that are closer to 0, but do not offer much improvement in terms of the mean or variance. Conversely, the mean and variance-corrected statistic $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} (that is given by Eq. 18) has a variance that is closer to 1, but does not offer much improvement in terms of skewness. Interestingly, although $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} has a mean that is closer to 0, ${\zeta }_1^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1^*$$\end{document} and ${\zeta }_2^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2^*$$\end{document} have means that are farther from 0. Similar results are shown in Tables 1 and 2 of van Krimpen-Stoop & Meijer (Reference van Krimpen-Stoop and Meijer1999) for $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} , and in Table 1 of Sinharay (Reference Sinharay2016b) for ${\zeta }_1^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1^*$$\end{document} and ${\zeta }_2^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2^*$$\end{document} .

The newly proposed statistics $T_{\text{CF}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}^*$$\end{document} , $T_{{\chi }^2}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}^*$$\end{document} , and $T_{\text{EW}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}^*$$\end{document} are the only statistics to incorporate mean, variance, and skewness corrections. Therefore, it is not surprising to see that these are the only statistics that improve both the skewness and the variance. Figure 3 also reveals that although these statistics have similar means and variances, $T_{{\chi }^2}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}^*$$\end{document} is the most effective at reducing skewness, followed by $T_{\text{EW}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}^*$$\end{document} and then $T_{\text{CF}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}^*$$\end{document} . This finding parallels the results for the class of T statistics, as shown in Fig. 3 and in previous research (Bedrick Reference Bedrick1997; Molenaar & Hoijtink Reference Molenaar and Hoijtink1990; Santos et al. Reference Santos, de la Torre and von Davier2020; von Davier & Molenaar Reference von Davier and Molenaar2003).

Figure 3 Descriptive statistics of the null distributions of the person-fit statistics. CF Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, EW Edgeworth expansion.

3.2.2. Type I Error Rates

Figure 4 displays the Type I error rates of the person-fit statistics. Each row corresponds to a different significance level (.01,  .02,  .05,  .10), and each column corresponds to a different test length (12, 36, 72). Horizontal dotted lines are used to indicate the significance levels. It is desirable for the Type I error rates to be at or below these lines.

Figure 4 Type I error rates of the person-fit statistics. CF Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, EW Edgeworth expansion.

Figure 4 reveals that the Type I error rates of T (i.e., the uncorrected statistic) vary depending on the weight function that is used. For $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} , the Type I error rate is consistently smaller than the nominal level. This result can largely be attributed to the reduced variance of the statistic (see Fig. 3). In contrast, for ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} and ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} , the Type I error rates are slightly larger than the nominal level when $\alpha$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} is small and the test is short, but are close to or smaller than the nominal level in all other instances.

The Type I error rates of the skewness-corrected statistics $T_{\text{CF}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}$$\end{document} , $T_{{\chi }^2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}$$\end{document} , and $T_{\text{EW}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}$$\end{document} are always smaller than the Type I error rates of T. While this result is desirable in instances where the Type I error rates of T are inflated (e.g., when ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} and ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} are used with a small $\alpha$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} and a short test), it is undesirable in instances where T is already conservative, which seems to be the more common case. Therefore, $T_{\text{CF}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}$$\end{document} , $T_{{\chi }^2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}$$\end{document} , and $T_{\text{EW}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}$$\end{document} are of limited utility in the present context.

The Type I error rates of the mean and variance-corrected statistic $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} are always larger than the Type I error rates of T. When $\alpha =.10$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.10$$\end{document} , this result is useful, since the Type I error rates of $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} are closer to, but still do not exceed, the nominal level. In contrast, when $\alpha =.01$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.01$$\end{document} ,  .02, or  .05, the Type I error rates of $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} often exceed the nominal level, which is undesirable in practice.

The Type I error rates of the newly proposed statistics $T_{\text{CF}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}^*$$\end{document} , $T_{{\chi }^2}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}^*$$\end{document} , and $T_{\text{EW}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}^*$$\end{document} are generally quite favorable and seem to overcome the limitations of the other corrected statistics. That is, the Type I error rates tend to be close to the nominal level (unlike $T_{\text{CF}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}$$\end{document} , $T_{{\chi }^2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}$$\end{document} , and $T_{\text{EW}}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}$$\end{document} ), while still not exceeding it (unlike $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} ). Of the three new statistics, $T_{\text{CF}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {CF}^*$$\end{document} produces Type I error rates that are closest to the nominal level, followed by $T_{\text{EW}}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_\text {EW}^*$$\end{document} when $\alpha =.01$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.01$$\end{document} , or $T_{{\chi }^2}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T_{\chi ^2}^*$$\end{document} when $\alpha =.02$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.02$$\end{document} ,  .05, or  .10. These conclusions are the same regardless of test length or the choice of weight function.

Figure 5 displays the Type I error rates by quintile of the $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} and $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistics. (The results for ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} , ${\zeta }_1^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1^*$$\end{document} , ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} , and ${\zeta }_2^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2^*$$\end{document} are similar and are available upon request from the first author.) Quintiles were formed by separating examinees based on $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} , such that low-ability examinees were placed in Quintile 1, high-ability examinees were placed in Quintile 5, and the examinees in between were sorted accordingly. Notably, the Type I error rates of the newly proposed statistics are similar across ability levels.

3.2.3. Power

Table 1 displays the power of $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} and $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} at the $\alpha =.01$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.01$$\end{document} significance level. (The results for ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} , ${\zeta }_1^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1^*$$\end{document} , ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} , and ${\zeta }_2^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2^*$$\end{document} are similar and are available upon request.) Each row corresponds to a different combination of aberrance and test length, and each column corresponds to a different combination of person-fit statistic and skewness correction. As expected, power increases as test length increases and as the proportion of contaminated items increases. Furthermore, across all conditions, the $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistic with no skewness correction is the most powerful, followed closely by the $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistic corrected using the Cornish–Fisher expansion, the $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistic corrected using the Edgeworth expansion, and then the $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistic corrected using the ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation. Thus, it appears that the new statistics can be used without a significant loss in power. Similar results are shown in Table 2 at the $\alpha =.05$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.05$$\end{document} significance level.

Figure 5 Type I error rates by quintile of $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} and $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} . CF Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, EW Edgeworth expansion.

Table 1 Power of $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} and $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} ( $\alpha =.01$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.01$$\end{document} ).

CF Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, EW Edgeworth expansion.

Table 2 Power of $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} and $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} ( $\alpha =.05$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.05$$\end{document} ).

CF Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, EW Edgeworth expansion.

4.1. Data and Analysis

The data in this example originate from a single form of a licensure examination. The data have been studied in the context of person-fit assessment by researchers such as Sinharay (Reference Sinharay2016b), and in the context of preknowledge detection in several chapters of Cizek & Wollack (Reference Cizek and Wollack2017). Item scores are available for 1644 examinees on 170 scored items. Following a statistical analysis and careful investigative process, the testing program flagged 61 items as being compromised, and 48 examinees as likely having engaged in fraudulent behavior. The 48 flagged examinees can be considered truly aberrant for purposes of the present analysis. However, it is important to note that other types of aberrance may be present among some of the non-flagged examinees, as well.

The Rasch model parameter estimates provided by the testing program were treated as the true item parameters. Then, using the WL estimates of ability, each score pattern was analyzed 24 times: once for each combination of two classes of person-fit statistics (T, $T^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T^*$$\end{document} ), three weight functions ( $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} , ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} , ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} ), and four skewness corrections (none, Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, Edgeworth expansion).

4.2. Results

Tables 3 and 4 display the proportions of examinees classified as aberrant and the agreement rates for $l_z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z$$\end{document} and $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} at the $\alpha =.01$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.01$$\end{document} and $\alpha =.05$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.05$$\end{document} significance levels, respectively. (The results for ${\zeta }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1$$\end{document} , ${\zeta }_1^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _1^*$$\end{document} , ${\zeta }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2$$\end{document} , and ${\zeta }_2^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\zeta _2^*$$\end{document} are similar and are available upon request.) In each table, the proportions of examinees classified as aberrant are displayed in bold text along the diagonal, and the agreement rates are displayed in non-bold text in the off-diagonal. Agreement rate is defined as the proportion of times two statistics make the same classification decision (aberrant or non-aberrant).

Table 3 Proportions of examinees classified as aberrant (diagonal) and agreement rates (off-diagonal) ( $\alpha =.01$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.01$$\end{document} ).

CF Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, EW Edgeworth expansion.

Table 4 Proportions of examinees classified as aberrant (diagonal) and agreement rates (off-diagonal) ( $\alpha =.05$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.05$$\end{document} ).

CF Cornish–Fisher expansion, ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} ${\chi }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} approximation, EW Edgeworth expansion.

Across all person-fit statistics and skewness corrections, the proportions of flagged examinees classified as aberrant are much larger than the proportions of non-flagged examinees classified as aberrant. This result provides favorable evidence regarding the performance of the statistics. In addition, the proportions of non-flagged examinees classified as aberrant consistently exceed the significance levels. This result is interesting, as it implies that aberrance is present among some of the non-flagged examinees, as well. For example, some of the non-flagged examinees may have engaged in fraudulent behavior, but were mistakenly not flagged by the testing program. It is also possible that some of the non-flagged examinees had engaged in a different type of aberrant behavior altogether.

Consistent with the simulation results, the $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistic with no skewness correction classified the most examinees as aberrant, followed by the $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistic corrected using the Cornish–Fisher expansion. Notably, all four variants of the $l_z^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$l_z^*$$\end{document} statistic classified the same sets of flagged examinees as aberrant—however, the skewness-corrected statistics classified fewer non-flagged examinees as aberrant. This result shows that the new statistics may lead to noticeable differences in tests having as many as 170 items.