Hostname: page-component-6bf8c574d5-w79xw Total loading time: 0 Render date: 2025-02-20T01:16:17.374Z Has data issue: false hasContentIssue false
  • English
  • Français

Hypergeometric Family and Item Overlap Rates in Computerized Adaptive Testing

Published online by Cambridge University Press:  01 January 2025

Hua-Hua Chang  and
Jinming Zhang
Hua-Hua Chang
Affiliation:
University of Texas at Austin
Jinming Zhang*
Affiliation:
Educational Testing Service
*
Requests for reprints should be sent to Jinming Zhang, MS 02-T, Educational Testing Service, Princeton, NJ 08541. E-Mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

A computerized adaptive test (CAT) is usually administered to small groups of examinees at frequent time intervals. It is often the case that examinees who take the test earlier share information with examinees who will take the test later, thus increasing the risk that many items may become known. Item overlap rate for a group of examinees refers to the number of overlapping items encountered by these examinees divided by the test length. For a specific item pool, different item selection algorithms may yield different item overlap rates. An important issue in designing a good CAT item selection algorithm is to keep item overlap rate below a preset level. In doing so, it is important to investigate what the lowest rate could be for all possible item selection algorithms. In this paper we rigorously prove that if every item has an equal possibility to be selected from the pool in a fixed-length CAT, the number of overlapping items among any α randomly sampled examinees follows the hypergeometric distribution family for α ≥ 1. Thus, the expected values of the number of overlapping items among any randomly sampled α examinees can be calculated precisely. These values may serve as benchmarks in controlling item overlap rates for fixed-length adaptive tests.

Keywords

computerized adaptive testinghypergeometric distributionitem exposure rateitem selectionstratificationSympson-Hetter methoditem overlap ratetest security
Type
Articles
Information
Psychometrika , Volume 67 , Issue 3 , September 2002 , pp. 387 - 398
Copyright
Copyright © 2002 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The research reported in this paper is collaborative in every respect, and the order of authorship is alphabetical. The authors wish to thank the three anonymous reviewers for their useful comments on earlier versions of this manuscript. The publication of this manuscript was supported in part by funds from Educational Testing Service and National Board of Medical Examiners.

References

Bickel, P.J., & Doksum, K.A. (1977). Mathematical statistics. San Francisco, CA: Holden-Day.Google Scholar
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In Lord, F.M., Novick, M.R. (Eds.), Statistical theories of mental test scores (pp. 395–479). Reading, MA: Addison-Wesley.Google Scholar
Chang, H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.CrossRefGoogle Scholar
Chang, H., & Ying, Z. (1999). A-stratified multistage in computerized adaptive testing. Applied Psychological Measurement, 23, 211–222.CrossRefGoogle Scholar
Chen, S., Ankenmann, R.D., & Spray, J.A. (1999, April). Exploring the relationship between item exposure rate and item overlap rate in computerized adaptive testing. Paper presented at the 1999 Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.CrossRefGoogle Scholar
Davey, T., & Parshall, C. (1995, April). New algorithms for item selection and exposure control with computerized adaptive testing. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.Google Scholar
Hulin, C.L., Drasgow, F., & Parsons, C.K. (1983). Item response theory: Application to psychological measurement. Homewood, IL: Dow Jones-Irwin.Google Scholar
Lord, F.M. (1970). Some test theory for tailored testing. In Holzman, W.H. (Eds.), Computer assisted instruction, testing, and guidance. New York, NY: Harper and Row.Google Scholar
Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
McBride, J.R., & Martin, J.T. (1983). Reliability and validity of adaptive ability tests in a military setting. In Weiss, D.J. (Eds.), New horizons in testing (pp. 223–226). New York, NY: Academic Press.Google Scholar
Owen, Z.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of American Statistical Association, 70, 351–356.CrossRefGoogle Scholar
Stocking, M.L. (1994). Three practical issues for modern adaptive testing item pools. Princeton, NJ: Educational Testing Service.Google Scholar
Stocking, M.L., & Lewis, C. (1995). A new method of controlling item exposure in computerized adaptive testing. Princeton, NJ: Educational Testing Service.CrossRefGoogle Scholar
Stocking, M.L., & Lewis, C. (1998). Controlling item exposure conditional on ability in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 23, 57–75.CrossRefGoogle Scholar
Sympson, J.B., & Hetter, R.D. (1985). Controlling item-exposure rates in computerized adaptive testing. Proceedings of the 27th Annual Meeting of the Military Testing Association (pp. 973–977). San Diego, CA: Navy Personnel Research and Development Center.Google Scholar
Thomasson, G.L. (1995, June). New item exposure control algorithms for computerized adaptive testing. Paper presented at the annual meeting of the Psychometric Society, Minneapolis, MN.Google Scholar
van der Linden, W.J., & Reese, L.M. (1998). A model for optimal constrained adaptive testing. Applied Psychological Measurement, 22, 259–270.CrossRefGoogle Scholar
Wainer, H. (1990). Computerized adaptive testing: A primer. Hillsdale, NJ: Lawrence Erlbaum Association.Google Scholar
Wainer, H. (2000). Rescuing computerized testing by breaking Zipf's law. Journal of Educational and Behavioral Statistics, 25, 203–224.CrossRefGoogle Scholar
Way, W.D. (1998). Protecting the integrity of computerized testing item pools. Educational Measurement: Issues and Practice, Winter, 17–27.CrossRefGoogle Scholar
Weiss, D.J. (1976). Adaptive testing research in Minnesota: Overview, recent results, and future directions. In Clark, C.L. (Eds.), Proceedings of the First Conference on Computerized Adaptive Testing (pp. 24–35). Washington, DC: United States Civil Service Commission.Google Scholar