Hostname: page-component-745bb68f8f-b6zl4 Total loading time: 0 Render date: 2025-01-08T09:22:38.223Z Has data issue: false hasContentIssue false
  • English
  • Français

A Hierarchical Framework for Modeling Speed and Accuracy on Test Items

Published online by Cambridge University Press:  01 January 2025

Wim J. van der Linden
Wim J. van der Linden*
Affiliation:
University of Twente
*
Requests for reprints should be sent to W. J. van der Linden, Department of Research Methodology, Measurement, and Data Analysis, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands. E-mail:[email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Current modeling of response times on test items has been strongly influenced by the paradigm of experimental reaction-time research in psychology. For instance, some of the models have a parameter structure that was chosen to represent a speed-accuracy tradeoff, while others equate speed directly with response time. Also, several response-time models seem to be unclear as to the level of parametrization they represent. A hierarchical framework for modeling speed and accuracy on test items is presented as an alternative to these models. The framework allows a “plug-and-play approach” with alternative choices of models for the response and response-time distributions as well as the distributions of their parameters. Bayesian treatment of the framework with Markov chain Monte Carlo (MCMC) computation facilitates the approach. Use of the framework is illustrated for the choice of a normal-ogive response model, a lognormal model for the response times, and multivariate normal models for their parameters with Gibbs sampling from the joint posterior distribution.

Keywords

hierarchical modelingitem-response theoryGibbs samplerMarkov chain Monte Carlo estimationspeed-accuracy tradeoffresponse times
Type
Original Paper
Information
Psychometrika , Volume 72 , Issue 3 , September 2007 , pp. 287 - 308
Copyright
Copyright © 2007 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This study received funding from the Law School Admission Council (LSAC). The opinions and conclusions contained in this paper are those of the author and do not necessarily reflect the policy and position of LSAC. The author is indebted to the American Institute of Certified Public Accountants for the data set in the empirical example and to Rinke H. Klein Entink for his computational assistance

References

Albert, J.H. (1992). Bayesian estimation of normal-ogive item response curves using Gibbs sampling. Journal of Educational and Behavioral Statistics, 17, 261–269.Google Scholar
Béguin, A.A., Glas, C.A.W. (2001). MCMC estimation and some fit analysis of multidimensional IRT models. Psychometrika, 66, 541–562.CrossRefGoogle Scholar
Carlin, B.P., & Louis, T.A. (2000). Bayes and Empirical Bayes Methods for Data Analysis. Boca Raton, FL: Chapman & Hall.Google Scholar
Douglas, J., Kosorok, M., Chewning, B. (1999). A latent variable model for multivariate psychometric response times. Psychometrika, 64, 69–82.CrossRefGoogle Scholar
Dubey, S.D. (1969). A new derivation of the logistic distribution. Naval Research Logistics Quarterly, 16, 37–40.CrossRefGoogle Scholar
Fox, J.P., Glas, C.A.W. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 271–288.CrossRefGoogle Scholar
Glas, C.A.W., & van der Linden, W.J. (2006). Modeling variability in item parameters in item response models. Psychometrika. Submitted.Google Scholar
Jansen, M.G.H. (1986). A Bayesian version of Rasch’s multiplicative Poisson model for the number of errors on achievement tests. Journal of Educational Statistics, 11, 51–65.Google Scholar
Jansen, M.G.H. (1997a). Rasch model for speed tests and some extensions with applications to incomplete designs. Journal of Educational and Behavioral Statistics, 22, 125–140.CrossRefGoogle Scholar
Jansen, M.G.H. (1997b). Rasch’s model for reading speed with manifest exploratory variables. Psychometrika, 62, 393–409.CrossRefGoogle Scholar
Jansen, M.G.H., Duijn, M.A.J. (1992). Extensions of Rasch’s multiplicative Poisson model. Psychometrika, 57, 405–414.CrossRefGoogle Scholar
Johnson, V.E., Albert, J.H. (1999). Ordinal Data Modeling, New York: Springer-Verlag.CrossRefGoogle Scholar
Luce, R.D. (1986). Response times: Their Roles in Inferring Elementary Mental Organization. Oxford, UK: Oxford University Press.Google Scholar
Maris, E. (1993). Additive and multiplicative models for gamma distributed variables, and their application as psychometric models for response times. Psychometrika, 58, 445–469.CrossRefGoogle Scholar
Oosterloo, S.J. (1975). Modellen voor Reaktie-tijden [Models for Reaction Times]. Unpublished master’s thesis, Faculty of Psychology, University of Groningen, The Netherlands.Google Scholar
Patz, R.J., Junker, B.W. (1999). Applications and extensions of MCMC in IRT: Mulitple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 34, 342–366.CrossRefGoogle Scholar
Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: The University of Chicago Press. (Original published in 1960).Google Scholar
Roskam, E.E. (1987). Toward a psychometric theory of intelligence. In Roskam, E.E. & Suck, R. (Eds.), Progress in Mathematical Psychology (pp. 151–171). Amsterdam: North-Holland.Google Scholar
Roskam, E.E. (1997). Models for speed and time-limit tests. In van der Linden, W.J. & Hambleton, R.K. (Eds.), Handbook of Modern Item Response Theory (pp. 187–208). New York: Springer.CrossRefGoogle Scholar
Rouder, J.N., Sun, D., Speckman, P.L., Lu, J., Zhou, D. (2003). A hierarchical Bayesian statistical framework for response time distributions. Psychometrika, 68, 589–606.CrossRefGoogle Scholar
Scheiblechner, H. (1979). Specific objective stochastic latency mechanisms. Journal of Mathematical Psychology, 19, 18–38.CrossRefGoogle Scholar
Scheiblechner, H. (1985). Psychometric models for speed-test construction: The linear exponential model. In Embretson, S.E. (Ed.), Test design: Developments in psychology and education (pp. 219–244). New York: Academic Press.CrossRefGoogle Scholar
Schnipke, D.L., Scrams, D.J. (1997). Modeling response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213–232.CrossRefGoogle Scholar
Schnipke, D.L., & Scrams, D.J. (1999). Representing response time information in item banks (LSAC Computerized Testing Report No. 97-09). Newtown, PA: Law School Admission Council.Google Scholar
Schnipke, D.L., & Scrams, D.J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In Mills, C.N., Potenza, M., Fremer, J.J. & Ward, W. (Eds.), Computer-Based Testing: Building the Foundation for Future Assessments (pp. 237–266). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Swanson, D.B., Featherman, C.M., Case, S.M., Luecht, R.M., & Nungester, R. (1999, March). Relationship of response latency to test design, examinee proficiency and item difficulty in computer-based test administration. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.Google Scholar
Swanson, D.B., Case, S.E., Ripkey, D.R., Clauser, B.E., Holtman, M.C. (2001). Relationships among item characteristics, examinee characteristics, and response times on USMLE, Step 1. Academic Medicine, 76, 114–116.CrossRefGoogle Scholar
Tatsuoka, K.K., & Tatsuoka, M.M. (1980). A model for incorporating response-time data in scoring achievement tests. In Weiss, D.J. (Ed.), Proceedings of the 1979 Computerized Adaptive Testing Conference (pp. 236–256). Minneapolis, MN: University of Minnesota, Department of Psychology, Psychometric Methods Program.Google Scholar
Thissen, D. (1983). Timed testing: An approach using item response theory. In Weiss, D.J. (Ed.), New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing. New York: Academic Press.Google Scholar
Townsend, J.T., & Ashby, F.G. (1983). Stochastic Modeling of Elementary Psychological Processes. Cambridge, UK: Cambridge University Press.Google Scholar
van Breukelen, G.J.P. (2005). Psychometric modeling of response speed and accuracy with mixed and conditional regression. Psychometrika, 70, 359–376.CrossRefGoogle Scholar
van der Linden, W.J. (2005). Linear Models for Optimal Test Design, New York: Springer-Verlag.CrossRefGoogle Scholar
van der Linden, W.J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204.CrossRefGoogle Scholar
van der Linden, W.J. (2007a). Conceptual Issues in Response-Time Modeling. Submitted.Google Scholar
van der Linden, W.J. (2007b). Using response times for item selection in adaptive tests. Journal of Educational and Behavioral Statistics, 32.Google Scholar
van der Linden, W.J., & Guo, F. (2006). Two Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing. Manuscript submitted for publication.Google Scholar
van der Linden, W.J., Hambleton, R.K. (1997). Handbook of Modern Item Response Theory, New York: Springer-Verlag.CrossRefGoogle Scholar
van der Linden, W.J., Breithaupt, K., Chuah, S.C., & Zhang, Y. (2007). Detecting differential speededness in multistage testing. Journal of Educational Measurement, 44, in press.CrossRefGoogle Scholar
van der Linden, W.J., Klein Entink, R.H., & Fox, J.-P. (2006). IRT Parameter Estimation with Response Times as Collateral Information. Manuscript submitted for publication.Google Scholar
van der Linden, W.J., Scrams, D.J., Schnipke, D.L. (1999). Using response-time constraints to control for speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.CrossRefGoogle Scholar
Verhelst, N.D., Verstralen, H.H.F.M., & Jansen, M.G. (1997). A logistic model for time-limit tests. In van der Linden, W.J. & Hambleton, R.K. (Eds.), Handbook of Modern Item Response Theory (pp. 169–185). New York: Springer-Verlag.CrossRefGoogle Scholar