Hostname: page-component-745bb68f8f-s22k5 Total loading time: 0 Render date: 2025-01-23T06:03:54.982Z Has data issue: false hasContentIssue false
  • English
  • Français

Online Calibration Via Variable Length Computerized Adaptive Testing

Published online by Cambridge University Press:  01 January 2025

Yuan-chin Ivan Chang  and
Hung-Yi Lu
Yuan-chin Ivan Chang*
Affiliation:
Academia Sinica
Hung-Yi Lu
Affiliation:
Fu Jen Catholic University
*
Requests for reprints should be sent to Yuan-chin Ivan Chang, Academia Sinica, Taipei, Taiwan. E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Item calibration is an essential issue in modern item response theory based psychological or educational testing. Due to the popularity of computerized adaptive testing, methods to efficiently calibrate new items have become more important than that in the time when paper and pencil test administration is the norm. There are many calibration processes being proposed and discussed from both theoretical and practical perspectives. Among them, the online calibration may be one of the most cost effective processes. In this paper, under a variable length computerized adaptive testing scenario, we integrate the methods of adaptive design, sequential estimation, and measurement error models to solve online item calibration problems. The proposed sequential estimate of item parameters is shown to be strongly consistent and asymptotically normally distributed with a prechosen accuracy. Numerical results show that the proposed method is very promising in terms of both estimation accuracy and efficiency. The results of using calibrated items to estimate the latent trait levels are also reported.

Keywords

adaptive testingitem response theorylogistic regressionmeasurement errornonlinear designonline calibrationstopping rule
Type
Theory and Methods
Information
Psychometrika , Volume 75 , Issue 1 , March 2010 , pp. 140 - 157
Copyright
Copyright © 2010 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abdelbasit, K.M., & Plackett, R.L. (1983). Experimental design for binary data. Journal of the American Statistical Association, 78, 90–98.CrossRefGoogle Scholar
Baker, F.B. (1992). Item response theory: parameter estimation technique, New York: Marcel Dekker.Google Scholar
Berger, M.P.F. (1992). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57(4), 521–538.CrossRefGoogle Scholar
Berger, M.P.F. (1994). D-optimal sequential sampling designs for item response theory models. Journal of Educational Statistics, 19(1), 43–56.CrossRefGoogle Scholar
Berger, M.P.F., King, J., & Wong, W.K. (2000). Minimax d-optimal designs for item response theory models. Psychometrika, 65(3), 377–390.CrossRefGoogle Scholar
Bock, R., & Aitken, M. (1981). Marginal maximum likelihood estimation of item parameters: an application of the em algorithm. Psychometrika, 46, 443–460.CrossRefGoogle Scholar
Bock, R., & Mislevy, R. (1985). BILOG (Computer program). Scientific Software.Google Scholar
Chang, Y.-c.I. (1999). Strong consistency of maximum quasi-likelihood estimate in generalized linear models via a last time. Statistics and Probability Letters, 45, 237–246.CrossRefGoogle Scholar
Chang, Y.-c.I. (2001). Sequential confidence regions of generalized linear models with adaptive designs. Journal of Statistical Planning and Inference, 93 1–2277.CrossRefGoogle Scholar
Chang, Y.-c.I. (2006). Sequential estimation in generalized linear measurement-error models. Technical Report C-2006-13, Institute of Statistical Science, Academia Sinica.Google Scholar
Chang, Y.-c.I., & Martinsek, A. (1992). Fixed size confidence regions for parameters of a logistic regression model. Annals of Statistics, 20, 1953–1969.CrossRefGoogle Scholar
Chang, Y.-c.I., & Ying, Z. (2004). Sequential estimate in variable length computerized adaptive testing. Journal of Statistical Planning and Inference, 121, 249–264.CrossRefGoogle Scholar
Chen, K., Hu, I., & Ying, Z. (1999). Strong consistency of maximum quasi-likelihood estimators in generalized linear models with fixed and adaptive designs. Annals of Statistics, 27(4), 1155–1163.CrossRefGoogle Scholar
Chow, Y.S., & Robbins, H. (1965). On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Annals of Mathematical Statistics, 36(2), 457–462.CrossRefGoogle Scholar
Chow, Y.S., & Teicher, H. (1997). Probability theory, (3rd ed.). New York: Springer.CrossRefGoogle Scholar
Dmitrienko, A., & Govindarajulu, Z. (2000). Sequential confidence regions for maximum likelihood estimates. Annals of Statistics, 28(5), 1472–1501.CrossRefGoogle Scholar
Fedorov, V.V. (1972). Theory of optimal design, New York: Academic Press.Google Scholar
Ford, I. (1976). Optimal static and sequential design: a critical review. Doctoral Dissertation, University of Glasgow.Google Scholar
Ford, I., Titterington, D.M., & Kitsos, C.P. (1989). Recent advances in nonlinear experimental design. Technometrics, 31, 49–60.CrossRefGoogle Scholar
Grambsch, P. (1983). Sequential sampling based on the observed fisher information. Annals of Statistics, 11, 68–77.CrossRefGoogle Scholar
Grambsch, P. (1989). Sequential maximum likelihood estimation with applications to logistic regression in case-control studies. Journal of Statistical Planning and Inference, 22, 355–369.CrossRefGoogle Scholar
Heise, M.A., & Myers, R.H. (1996). Optimal designs for bivariate logistic regression. Biometrics, 52, 613–624.CrossRefGoogle Scholar
Jones, D.H., & Jin, Z. (1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59, 59–75.CrossRefGoogle Scholar
Kalish, L.A., & Rosenberger, J.L. (1978). Optimal designs for the estimation of the logistic function. Technical Report, Vol. 33, The Pennsylvania State University, Department of Statistics.Google Scholar
Khuri, A.I., Mukherjee, B., Sinha, B.K., & Ghosh, M. (2006). Design issues for generalized linear models, a review. Statistical Science, 21(3), 376–399.CrossRefGoogle Scholar
Lai, T.L., & Wei, C.Z. (1982). Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. Annals of Statistics, 10, 154–166.CrossRefGoogle Scholar
Li, T. (2002). Robust and consistent estimation of nonlinear errors-in-variables models. Journal of Econometrics, 110(1), 1–26.CrossRefGoogle Scholar
Li, T., & Hsiao, C. (2004). Robust estimation of generalized linear models with measurement errors. Journal of Econometrics, 118 1–251–65. Available at http://ideas.repec.org/a/eee/econom/v118y2004i1-2p51-65.htmlCrossRefGoogle Scholar
Lord, F.M. (1980). Applications of item response theory to practical testing problems, Hillsdale: Lawrence Erlbaum Associates.Google Scholar
Minkin, S. (1987). Optimal designs for binary data. Journal of the American Statistical Association, 82, 1098–1103.CrossRefGoogle Scholar
Patz, R.J., & Junker, B.W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146–178.CrossRefGoogle Scholar
Silvey, S.D. (1980). Optimal design, London: Chapman and Hall.CrossRefGoogle Scholar
Sitter, R.R. (1992). Robust designs for binary data. Biometrics, 48, 1145–1155.CrossRefGoogle Scholar
Stefanski, L.A., & Carroll, R.J. (1985). Covariate measurement error in logistic regression. Annals of Statistics, 13(4), 1335–1351.CrossRefGoogle Scholar
van der Linden, W.J., & Glas, C.A.W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13(1), 35–53.CrossRefGoogle Scholar
van der Linden, W.J., & Hambleton, R.K. (1997). Handbook of modern item response theory, Berlin: Springer.CrossRefGoogle Scholar
Wainer, H., & Mislevy, R. (2000). Computerized adaptive testing: a primer, (2nd ed.). New Jersey: Lawrence Erlbaum Association.CrossRefGoogle Scholar
Wu, C.F.J. (1985). Efficient sequential designs with binary data. Journal of American Statistical Association, 80, 974–984.CrossRefGoogle Scholar
Wynn, H.P. (1970). The sequential generation of d-optimum experimental designs. Annals of Mathematical Statistics, 41(5), 1655–1664.CrossRefGoogle Scholar
Ying, Z., & Wu, C.J. (1997). An asymptotic theory of sequential designs based on maximum likelihood recursions. Statistica Sinica, 7, 75–91.Google Scholar