Despite its frequent consideration in test development, item complexity receives little attention in the psychometric modeling of item response data. In this address, I consider how variability in item complexity can be expected to emerge in the form of item characteristic curve (ICC) asymmetry, and how such effects may significantly influence applications of item response theory, especially those that assume interval-level properties of the latent proficiency metric and groups that vary substantially in mean proficiency. One application is the score gain deceleration phenomenon often observed in vertical scaling contexts, especially in subject areas like math or second language acquisition. It is demonstrated how the application of symmetric IRT models in the presence of complexity-induced positive ICC asymmetry can be a likely cause. A second application concerns the positive correlation between DIF and difficulty commonly seen in verbal proficiency (and other subject area) tests where problem-solving complexity is minimal and proficiency-related guessing effects are likely pronounced. Here we suggest negative ICC asymmetry as a probable cause and apply sensitivity analyses to demonstrate the ease with which such correlations disappear when allowing for negative ICC asymmetry. Unfortunately, the presence of systematic forms of ICC asymmetry is easily missed due to the considerable flexibility afforded by latent trait metrics in IRT. Speculation is provided regarding other applications for which attending to ICC asymmetry may prove useful.

Many of the models that have been proposed for response data share the assumptions that define the monotone homogeneity (MH) model. Observable properties that are implied by the MH model allow for these assumptions to be tested. For binary response data, the most restrictive of these properties is called conditional association (CA). All the other properties considered can be considered incomplete tests of CA that alleviate the practical limitations encountered when assessing the MH model assumptions using CA. It is found that the assessment of the MH model assumptions with an incomplete test of CA, rather than CA, is generally associated with a substantial loss of information. We also look at the sensitivity of the observable properties to model violation and discuss the implications of the results. It is argued that more research is required about the extent to which the assumptions and the model specifications influence the inferences made from response data.

A comprehensive class of models is proposed that can be used for continuous, binary, ordered categorical and count type responses. The difficulty of items is described by difficulty functions, which replace the item difficulty parameters that are typically used in item response models. They crucially determine the response distribution and make the models very flexible with regard to the range of distributions that are covered. The model class contains several widely used models as the binary Rasch model and the graded response model as special cases, allows for simplifications, and offers a distribution free alternative to count type items. A major strength of the models is that they can be used for mixed item formats, when different types of items are combined to measure abilities or attitudes. It is an immediate consequence of the comprehensive modeling approach that allows that difficulty functions automatically adapt to the response distribution. Basic properties of the model class are shown. Several real data sets are used to illustrate the flexibility of the models

In this paper, the notion of Markov move from algebraic statistics is used to analyze the weighted kappa indices in rater agreement problems. In particular, the problem of the maximum kappa and its dependence on the choice of the weighting schemes are discussed. The Markov moves are also used in a simulated annealing algorithm to actually find the configuration of maximum agreement.

Computerized assessment provides rich multidimensional data including trial-by-trial accuracy and response time (RT) measures. A key question in modeling this type of data is how to incorporate RT data, for example, in aid of ability estimation in item response theory (IRT) models. To address this, we propose a joint model consisting of a two-parameter IRT model for the dichotomous item response data, a log-normal model for the continuous RT data, and a normal model for corresponding paper-and-pencil scores. Then, we reformulate and reparameterize the model to capture the relationship between the model parameters, to facilitate the prior specification, and to make the Bayesian computation more efficient. Further, we propose several new model assessment criteria based on the decomposition of deviance information criterion (DIC) the logarithm of the pseudo-marginal likelihood (LPML). The proposed criteria can quantify the improvement in the fit of one part of the multidimensional data given the other parts. Finally, we have conducted several simulation studies to examine the empirical performance of the proposed model assessment criteria and have illustrated the application of these criteria using a real dataset from a computerized educational assessment program.

Reliability is a crucial concept in psychometrics. Although it is typically estimated as a single fixed quantity, previous work suggests that reliability can vary across persons, groups, and covariates. We propose a novel method for estimating and modeling case-specific reliability without repeated measurements or parallel tests. The proposed method employs a “Reliability Factor” that models the error variance of each case across multiple indicators, thereby producing case-specific reliability estimates. Additionally, we use Gaussian process modeling to estimate a nonlinear, non-monotonic function between the latent factor itself and the reliability of the measure, providing an analogue to test information functions in item response theory. The reliability factor model is a new tool for examining latent regions with poor conditional reliability, and correlates thereof, in a classical test theory framework.

Latent class models with covariates are widely used for psychological, social, and educational research. Yet the fundamental identifiability issue of these models has not been fully addressed. Among the previous research on the identifiability of latent class models with covariates, Huang and Bandeen-Roche (Psychometrika 69:5–32, 2004) studied the local identifiability conditions. However, motivated by recent advances in the identifiability of the restricted latent class models, particularly cognitive diagnosis models (CDMs), we show in this work that the conditions in Huang and Bandeen-Roche (Psychometrika 69:5–32, 2004) are only necessary but not sufficient to determine the local identifiability of the model parameters. To address the open identifiability issue for latent class models with covariates, this work establishes conditions to ensure the global identifiability of the model parameters in both strict and generic sense. Moreover, our results extend to the polytomous-response CDMs with covariates, which generalizes the existing identifiability results for CDMs.

Studies with sensitive questions should include a sufficient number of respondents to adequately address the research interest. While studies with an inadequate number of respondents may not yield significant conclusions, studies with an excess of respondents become wasteful of investigators’ budget. Therefore, it is an important step in survey sampling to determine the required number of participants. In this article, we derive sample size formulas based on confidence interval estimation of prevalence for four randomized response models, namely, the Warner’s randomized response model, unrelated question model, item count technique model and cheater detection model. Specifically, our sample size formulas control, with a given assurance probability, the width of a confidence interval within the planned range. Simulation results demonstrate that all formulas are accurate in terms of empirical coverage probabilities and empirical assurance probabilities. All formulas are illustrated using a real-life application about the use of unethical tactics in negotiation.

This paper proposes a novel collapsed Gibbs sampling algorithm that marginalizes model parameters and directly samples latent attribute mastery patterns in diagnostic classification models. This estimation method makes it possible to avoid boundary problems in the estimation of model item parameters by eliminating the need to estimate such parameters. A simulation study showed the collapsed Gibbs sampling algorithm can accurately recover the true attribute mastery status in various conditions. A second simulation showed the collapsed Gibbs sampling algorithm was computationally more efficient than another MCMC sampling algorithm, implemented by JAGS. In an analysis of real data, the collapsed Gibbs sampling algorithm indicated good classification agreement with results from a previous study.

The Procrustes-based perturbation model (Goodall in J R Stat Soc Ser B Methodol 53(2):285–321, 1991) allows minimization of the Frobenius distance between matrices by similarity transformation. However, it suffers from non-identifiability, critical interpretation of the transformed matrices, and inapplicability in high-dimensional data. We provide an extension of the perturbation model focused on the high-dimensional data framework, called the ProMises (Procrustes von Mises–Fisher) model. The ill-posed and interpretability problems are solved by imposing a proper prior distribution for the orthogonal matrix parameter (i.e., the von Mises–Fisher distribution) which is a conjugate prior, resulting in a fast estimation process. Furthermore, we present the Efficient ProMises model for the high-dimensional framework, useful in neuroimaging, where the problem has much more than three dimensions. We found a great improvement in functional magnetic resonance imaging connectivity analysis because the ProMises model permits incorporation of topological brain information in the alignment’s estimation process.

Personality tests employing comparative judgments have been proposed as an alternative to Likert-type rating scales. One of the main advantages of a comparative format is that it can reduce faking of responses in high-stakes situations. However, previous research has shown that it is highly difficult to obtain trait score estimates that are both faking resistant and sufficiently accurate for individual-level diagnostic decisions. With the goal of contributing to a solution, I study the information obtainable from comparative judgments analyzed by means of Thurstonian IRT models. First, I extend the mathematical theory of ordinal comparative judgments and corresponding models. Second, I provide optimal test designs for Thurstonian IRT models that maximize the accuracy of people’s trait score estimates from both frequentist and Bayesian statistical perspectives. Third, I derive analytic upper bounds for the accuracy of these trait estimates achievable through ordinal Thurstonian IRT models. Fourth, I perform numerical experiments that complement results obtained in earlier simulation studies. The combined analytical and numerical results suggest that it is indeed possible to design personality tests using comparative judgments that yield trait scores estimates sufficiently accurate for individual-level diagnostic decisions, while reducing faking in high-stakes situations. Recommendations for the practical application of comparative judgments for the measurement of personality, specifically in high-stakes situations, are given.

Latent variable models have been playing a central role in psychometrics and related fields. In many modern applications, the inference based on latent variable models involves one or several of the following features: (1) the presence of many latent variables, (2) the observed and latent variables being continuous, discrete, or a combination of both, (3) constraints on parameters, and (4) penalties on parameters to impose model parsimony. The estimation often involves maximizing an objective function based on a marginal likelihood/pseudo-likelihood, possibly with constraints and/or penalties on parameters. Solving this optimization problem is highly non-trivial, due to the complexities brought by the features mentioned above. Although several efficient algorithms have been proposed, there lacks a unified computational framework that takes all these features into account. In this paper, we fill the gap. Specifically, we provide a unified formulation for the optimization problem and then propose a quasi-Newton stochastic proximal algorithm. Theoretical properties of the proposed algorithms are established. The computational efficiency and robustness are shown by simulation studies under various settings for latent variable model estimation.

Component-based approaches have been regarded as a tool for dimension reduction to predict outcomes from observed variables in regression applications. Extended redundancy analysis (ERA) is one such component-based approach which reduces predictors to components explaining maximum variance in the outcome variables. In many instances, ERA can be extended to capture nonlinearity and interactions between observed and components, but only by specifying a priori functional form. Meanwhile, machine learning methods like neural networks are typically used in a data-driven manner to capture nonlinearity without specifying the exact functional form. In this paper, we introduce a new method that integrates neural networks algorithms into the framework of ERA, called NN-ERA, to capture any non-specified nonlinear relationships among multiple sets of observed variables for constructing components. Simulations and empirical datasets are used to demonstrate the usefulness of NN-ERA. The conclusion is that in social science datasets with unstructured data, where we expect nonlinear relationships that cannot be specified a priori, NN-ERA with its neural network algorithmic structure can serve as a useful tool to specify and test models otherwise not captured by the conventional component-based models.

Process data refer to data recorded in computer-based assessments (CBAs) that reflect respondents’ problem-solving processes and provide greater insight into how respondents solve problems, in addition to how well they solve them. Using the rich information contained in process data, this study proposed an item expansion method to analyze action-level process data from the perspective of diagnostic classification in order to comprehensively understand respondents’ problem-solving competence. The proposed method cannot only estimate respondents’ problem-solving ability along a continuum, but also classify respondents according to their problem-solving skills. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving item was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents’ problem-solving skills than the observed score categories; (b) although only one item was used, the estimated higher-order latent ability reflected the respondents’ problem-solving ability more accurately than the unidimensional latent ability estimated from the outcome data; and (c) interactions among problem-solving skills followed the conjunctive condensation rule, which indicated that the specific action sequence appeared only when a respondent mastered all required problem solving skills. In conclusion, the proposed diagnostic classification approach is feasible and promising analyzing process data.

A mixed-effects location scale model allows researchers to study within- and between-person variation in repeated measures. Key components of the model include separate variance models to study predictors of the within-person variance, as well as predictors of the between-person variance of a random effect, such as a random intercept. In this paper, a latent variable mixed-effects location scale model is developed that combines a longitudinal common factor model and a mixed-effects location scale model to characterize within- and between-person variation in a common factor. The model is illustrated using daily reports of positive affect and daily stressors for a large sample of adult women.