Hostname: page-component-cd9895bd7-gxg78 Total loading time: 0 Render date: 2024-12-25T14:48:03.407Z Has data issue: false hasContentIssue false
  • English
  • Français

Rotating Item Banks versus Restriction of Maximum Exposure Rates in Computerized Adaptive Testing

Published online by Cambridge University Press:  10 April 2014

Juan Ramón Barrada ,
Julio Olea  and
Francisco José Abad
Juan Ramón Barrada*
Affiliation:
Universidad Autónoma de Barcelona (Spain)
Julio Olea
Affiliation:
Universidad Autónoma de Madrid (Spain)
Francisco José Abad
Affiliation:
Universidad Autónoma de Madrid (Spain)
*
Correspondence concerning this article should be addressed to Juan Ramón Barrada, Facultad de Psicología, Universidad Autonoma de Barcelona, 08193 Bellaterra (Spain). Phone: 00 34 93 581 32 63. E-mail: [email protected].
Get access Rights & Permissions [Opens in a new window]

Abstract

If examinees were to know, beforehand, part of the content of a computerized adaptive test, their estimated trait levels would then have a marked positive bias. One of the strategies to avoid this consists of dividing a large item bank into several sub-banks and rotating the sub-bank employed (Ariel, Veldkamp & van der Linden, 2004). This strategy permits substantial improvements in exposure control at little cost to measurement accuracy. However, we do not know whether this option provides better results than using the master bank with greater restriction in the maximum exposure rates (Sympson & Hetter, 1985). In order to investigate this issue, we worked with several simulated banks of 2100 items, comparing them, for RMSE and overlap rate, with the same banks divided in two, three… up to seven sub-banks. By means of extensive manipulation of the maximum exposure rate in each bank, we found that the option of rotating banks slightly outperformed the option of restricting maximum exposure rate of the master bank by means of the Sympson-Hetter method.

Si los examinandos conocieran de antemano una parte del contenido de un test adaptativo informatizado, sus niveles estimados de rasgo tendrían un marcado sesgo positivo. Una de las estrategias para evitar esto consiste en dividir un gran banco de ítems en varios sub-bancos y rotar el sub-banco empleado (Ariel, Veldkamp & van der Linden, 2004). Esta estrategia permite mejoras sustanciales en el control de la exposición con poca merma de la precisión de la medida. Sin embargo, no sabemos si esta opción proporciona mejores resultados que el uso del banco maestro con más restricción en la tasa máxima de exposición (Sympson & Hetter, 1985). Para investigar este problema, trabajamos con varios bancos simulados de 2100 ítems, comparándolos, en RMSE y en tasa de solapamiento, con los mismos bancos divididos en dos, tres… hasta siete sub-bancos. Mediante manipulación extensa de la tasa máxima de exposición en cada banco, encontramos que la opción de rotar los bancos ofrecía resultados ligeramente mejores que la opción de restringir la tasa máxima de exposición del banco maestro mediante el método Sympson-Hetter.

Keywords

item bank securityoverlap rateaccuracybank rotationSympson-Hetter methodseguridad del banco de ítemstasa de solapamientoprecisiónrotación del bancométodo Sympson-Hetter
Type
Articles
Information
The Spanish Journal of Psychology , Volume 11 , Issue 2 , November 2008 , pp. 618 - 625
Copyright
Copyright © Cambridge University Press 2008

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Ariel, A., Veldkamp, B. P., & van der Linden, W. J. (2004). Constructing rotating item pools for constrained adaptive testing. Journal of Educational Measurement, 41, 345–359.CrossRefGoogle Scholar
Barrada, J. R., Olea, J., & Ponsoda, V. (2007). Methods for restricting maximum exposure rate in computerized adaptive testing. Methodology, 3, 14–23.CrossRefGoogle Scholar
Barrada, J. R., Olea, J., Ponsoda, V., & Abad, F. J. (In press). Incorporating randomness to the Fisher information for improving item exposure control in CATs. British Journal of Mathematical and Statistical Psychology.Google Scholar
Barrada, J. R., Veldkamp, B. P., & Olea, J. (In press). Multiple maximum exposure rates in computerized adaptive testing. Applied Psychological Measurement.Google Scholar
Chang, H. H. (2004). Understanding computerized adaptive testing – From Robbins-Monro to Lord and beyond. In Kaplan, David (Ed.), The SAGE handbook of quantitative methodology for the social sciences (pp. 117–133). Thousand Oaks, CA: Sage.Google Scholar
Chang, H. H., & van der Linden, W. J. (2003). Optimal stratification of item pools in a -stratified computerized adaptive testing. Applied Psychological Measurement, 27, 262–274.CrossRefGoogle Scholar
Chang, H. H., & Ying, Z. (1999). α-Stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23, 211–222.CrossRefGoogle Scholar
Chen, S. Y., Ankenmann, R. D., & Spray, J. A. (2003). The relationship between item exposure and test overlap in computerized adaptive testing. Journal of Educational Measurement, 40, 129–145.CrossRefGoogle Scholar
Chen, S. Y., & Lei, P. W. (2005). Controlling item exposure and test overlap in computerized adaptive testing. Applied Psychological Measurement, 29, 204–217.CrossRefGoogle Scholar
Chen, S. Y., Lei, P. W., & Liao, W. H. (In press). Controlling item exposure and test overlap on the fly in computerized adaptive testing. British Journal of Mathematical and Statistical Psychology.Google Scholar
Davey, T., & Nering, N. (2002). Controlling item exposure and maintaining item security. In Mills, C. N., Potenza, M. T., Fremer, J. J., & Ward, W. C., (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 165–191). Mahwah, NJ: Erlbaum.Google Scholar
Dodd, B. G. (1990). The effect of item selection procedure and stepsize on computerized adaptive attitude measurement using the rating scale model. Applied Psychological Measurement, 14, 355–366.CrossRefGoogle Scholar
Lu, Y., & Hambleton, R. K. (2004). Statistics for detecting disclosed items in CAT environment. Metodologia de las Ciencias del Comportamiento, 5, 225–242.Google Scholar
Mills, G. N., & Steffen, M. (2000). The GRE computer adaptive test: Operation issues. In van der Linden, W. J. & Glas, C. A. W. (Eds.), Computerized adaptive testing: Theory and practice (pp. 75–100). Boston: Kluwer Academic Press.CrossRefGoogle Scholar
Revuelta, J., & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35, 311–327.CrossRefGoogle Scholar
Segall, D. O. (2004). A sharing item response theory model for computerized adaptive testing. Journal of Educational & Behavioral Statistics, 29, 439–460.CrossRefGoogle Scholar
Stocking, M. L. (1994). Three practical issues for modern adaptive testing item pools. ETS Research Report No 94-05. Princeton, NJ: Educational Testing Service.Google Scholar
Stocking, M. L., & Swanson, L. (1998). Optimal design of items banks for computerized adaptive tests. Applied Psychological Measurement, 22, 271–279.CrossRefGoogle Scholar
Sympson, J. B., & Hetter, R. D. (1985, October). Controlling item-exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the Military Testing Association (pp. 973–977). San Diego, CA: Navy Personnel Research and Development Center.Google Scholar
van der Linden, W. J. (2003). Some alternatives to Sympson-Hetter item-exposure control in computerized adaptive testing. Journal of Educational & Behavioral Statistics, 28, 249–265.CrossRefGoogle Scholar
van der Linden, W. J. (2005). Linear models for optimal test design. New York: Springer.CrossRefGoogle Scholar
van der Linden, W. J., & Veldkamp, B. P. (2004). Constraining item exposure in computerized adaptive testing with shadow tests. Journal of Educational & Behavioral Statistics, 29, 273–291.CrossRefGoogle Scholar
Veerkamp, W. & Glas, C.A.W. (2000). Detection of known items in adaptive testing with a statistical quality control method. Journal of Behavioral and Educational Statistics, 25, 373–389.CrossRefGoogle Scholar
Wainer, H. (2000). Rescuing computerized testing by breaking Zipf's law. Journal of Educational & Behavioral Statistics, 25, 203–224.CrossRefGoogle Scholar
Way, W. D. (1998). Protecting the integrity of computerized testing item pools. Educational Measurement: Issues and Practice, 17, 17–27.CrossRefGoogle Scholar
Wise, S. L., & Kingsbury, G. G. (2000). Practical issues in developing and maintaining a computerized adaptive testing program. Psicologica, 21, 135–155.Google Scholar