Hostname: page-component-745bb68f8f-lrblm Total loading time: 0 Render date: 2025-01-07T22:43:40.456Z Has data issue: false hasContentIssue false
  • English
  • Français

Computerized adaptive testing under nonparametric IRT models

Published online by Cambridge University Press:  01 January 2025

Xueli Xu  and
Jeff Douglas
Xueli Xu*
Affiliation:
Educational Testing Service
Jeff Douglas
Affiliation:
University of Illinois at Urbana-Champaign
*
Requests for reprints should be sent to Xueli Xu, Rosedale Road MS 02-T, Princeton, NJ 08541, USA.
Get access Rights & Permissions [Opens in a new window]

Abstract

Nonparametric item response models have been developed as alternatives to the relatively inflexible parametric item response models. An open question is whether it is possible and practical to administer computerized adaptive testing with nonparametric models. This paper explores the possibility of computerized adaptive testing when using nonparametric item response models. A central issue is that the derivatives of item characteristic Curves may not be estimated well, which eliminates the availability of the standard maximum Fisher information criterion. As alternatives, procedures based on Shannon entropy and Kullback–Leibler information are proposed. For a long test, these procedures, which do not require the derivatives of the item characteristic eurves, become equivalent to the maximum Fisher information criterion. A simulation study is conducted to study the behavior of these two procedures, compared with random item selection. The study shows that the procedures based on Shannon entropy and Kullback–Leibler information perform similarly in terms of root mean square error, and perform much better than random item selection. The study also shows that item exposure rates need to be addressed for these methods to be practical.

Keywords

Shannon entropyKullback–Leibler informationnonparametric item response modelsitem response theory
Type
Original Paper
Information
Psychometrika , Volume 71 , Issue 1 , March 2006 , pp. 121 - 137
Copyright
Copyright © 2006 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The authors would like to thank Hua Chang for his help in conducting this research.

References

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord, F.M., & Novick, M.R. (Eds.), Statistical theories of mental test scores (pp. 397–479). Reading, MA: Addison-Wesley.Google Scholar
Chang, H.H., & Stout, W.F. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37–52.CrossRefGoogle Scholar
Chang, H.H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.CrossRefGoogle Scholar
Cover, T.M., & Thomas, J.A. (1991). Elements of information theory, New York: Wiley.Google Scholar
DeGroot, M.H. (1962). Uncertainty, information and sequential experiments. Annals of Mathematical Statistics, 33, 404–419.CrossRefGoogle Scholar
Douglas, J. (1997). Joint consistency of nonparametric item characteristic curve and ability estimation. Psychometrika, 62, 7–28.CrossRefGoogle Scholar
Eubank, R.L. (1988). Spline smoothing and nonparametric regression, New York: Marcel Dekker.Google Scholar
Grayson, D.A. (1988). Two-group classification in latent trait theory: Scores with monotone likelihood ratio. Psychometrika, 53, 383–392.CrossRefGoogle Scholar
He, X., & Ng, P. (1998). COBS: Qualitatively constrained smoothing via linear programming. Unpublished manual for SCOBS.Google Scholar
Nadaraya, E.A. (1964). On estimating regression. Probability Theory and its Applications, 9, 141–142.CrossRefGoogle Scholar
Ramsay, J.O. (1991). Kernel smoothing approaches to nonparametric item characteristic curve estimation. Psychometrika, 56, 611–630.CrossRefGoogle Scholar
Ramsay, J.O. (2000). TESTGRAF: A program for the graphical analysis of multiple choice test and questionnaire data [Computer Program], Montreal: McGill University.Google Scholar
Ramsay, J.O., & Abrahamowicz, M. (1989). Binomial regression with monotone splines: A psychometric application. Journal of the American Statistical Association, 84, 906–915.CrossRefGoogle Scholar
Ramsay, J.O., & Winsberg, S. (1991). Maximum marginal likelihood estimation for semiparametric item analysis. Psychometrika, 56, 365–379.CrossRefGoogle Scholar
Rossi, N., Wang, X., & Ramsay, J.O. (2002). Nonparametric item response function estimates with the EM algorithm. Journal of Educational and Behavioral Statistics, 27, 291–317.CrossRefGoogle Scholar
Shannon, C.E. (1948). A mathematical theory of communication. Bell Systems Techical Journal, 27, 379–423, 623–656CrossRefGoogle Scholar
Tatsuoka, C. (2002). Data analytic methods for latent parially ordered classification models. Journal of the Royal Statistical Society, Series C, 51, 337–350.CrossRefGoogle Scholar
Tatsuoka, C., & Ferguson, T. (2003). Sequential classification on patially ordered sets. Journal of Royal Statistical Society, Series B, 65, 143–158.CrossRefGoogle Scholar
van der Linden, W.J., & Glas, C.A.W. (2000). Computerized adaptive testing: Theory and practice, Dordrecht: Kluwer Academic.CrossRefGoogle Scholar
Walker, A.M. (1969). On the asymptotic behavior of posterior distributions. Journal of the Royal Statistical Society, Series B, 31, 80–88.CrossRefGoogle Scholar
Watson, G.S. (1964). Smooth regression analysis. Sankhya, Series A, 26, 359–372.Google Scholar
Xu, X., Chang, H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Presented at the Annual Meeting of the National Council of Measurement in Education, Chicago, April 2003.Google Scholar