Hostname: page-component-745bb68f8f-d8cs5 Total loading time: 0 Render date: 2025-01-08T10:20:17.475Z Has data issue: false hasContentIssue false
  • English
  • Français

Conditional Covariance Theory and Detect for Polytomous Items

Published online by Cambridge University Press:  01 January 2025

Jinming Zhang
Jinming Zhang*
Affiliation:
Educational Testing Service
*
Requests for reprints should be sent to Jinming Zhang, Educational Testing Service, MS 02-T, Rosedale Road, Princeton, NJ 08541, USA. E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper extends the theory of conditional covariances to polytomous items. It has been proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, given an appropriately chosen composite is positive if, and only if, the two items measure similar constructs besides the composite. The theory provides a theoretical foundation for dimensionality assessment procedures based on conditional covariances or correlations, such as DETECT and DIMTEST, so that the performance of these procedures is theoretically justified when applied to response data with polytomous items. Various estimators of conditional covariances are constructed, and special attention is paid to the case of complex sampling data, such as those from the National Assessment of Educational Progress (NAEP). As such, the new version of DETECT can be applied to response data sets not only with polytomous items but also with missing values, either by design or at random. DETECT is then applied to analyze the dimensional structure of the 2002 NAEP reading samples of grades 4 and 8. The DETECT results show that the substantive test structure based on the purposes for reading is consistent with the statistical dimensional structure for either grade.

Keywords

item response theory (IRT)multidimensional item response theory (MIRT)dimensionalitymultidimensionalityPolyDETECT
Type
Original Paper
Information
Psychometrika , Volume 72 , Issue 1 , March 2007 , pp. 69 - 91
Copyright
Copyright © 2006 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This research was supported by the Educational Testing Service and the National Assessment of Educational Progress (Grant R902F980001), US Department of Education. The opinions expressed herein are solely those of the author and do not necessarily represent those of the Educational Testing Service. The author would like to thank Ting Lu, Paul Holland, Shelby Haberman, and Feng Yu for their comments and suggestions.

References

Allen, N., Carlson, J.E., & Zelenak, C. (1999). The NAEP 1996 technical report (NCES 1999-452). Washington, DC: Office of Educational Research and Improvement, US Department of Education.Google Scholar
Allen, N., Donoghue, J.R., & Schoeps, T.L. (2001). The NAEP 1998 technical report (NCES 2001-509). Washington, DC: Office of Educational Research and Improvement, US Department of Education.Google Scholar
Anderson, T.W. (1984). An introduction to multivariate statistical analysis 2nd ed.,, New York: Wiley.Google Scholar
Beaton, A.E., Johnson, E.G., & Ferris, J.J. (1987). The assignment of exercises to students. In Beaton, A.E. (Ed.), Implementing the new design: The NAEP 1983–84 technical report (pp. 97–118). Princeton, NJ: Educational Testing Service.Google Scholar
Douglas, J., Kim, H.R., & Stout, W.F. (1994). Exploring and explaining the lack of local independence through conditional covariance functions. Paper presented at the 1994 (April) annual meeting of the American Educational Research Association, New Orleans, LA.Google Scholar
Habing, B., Roussos, L.A. (2003). On the need for negative local item dependence. Psychometrika, 68, 435–451.CrossRefGoogle Scholar
Holland, P.W., Rosenbaum, P.R. (1986). Conditional association and unidimensionality in monotone latent variable models. The Annals of Statistics, 14, 1523–1543.CrossRefGoogle Scholar
Junker, B. (1993). Conditional association, essential independence and monotone unidimensional item response models. The Annals of Statistics, 21, 1359–1378.CrossRefGoogle Scholar
Kim, H.R. (1994). New techniques for the dimensionality assessment of standardized test data. Unpublished doctoral dissertation, Department of Statistics, University of Illinois at Urbana-Champaign.Google Scholar
McDonald, R.P. (1994). Testing for approximate dimensionality. In Laveault, D., Zumbo, B., Gessaroli, M., Boss, M. (Eds.), Modern theories of measurement: Problems and issues (pp. 63–85). Ottawa: University of Ottawa Press.Google Scholar
Mislevy, R., Bock, R.D. (1982). BILOG: Item analysis and test scoring with binary logistic models [Computer software], Mooresville, IN: Scientific Software.Google Scholar
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159–176.CrossRefGoogle Scholar
Muraki, E., Bock, R.D. (1997). PARSCALE: IRT item analysis and test scoring for rating scale data [Computer software], Chicago: Scientific Software.Google Scholar
Nandakumar, R., Stout, W.F. (1993). Refinement of Stout’s procedure for assessing latent trait essential unidimensionality. Journal of Educational Statistics, 18, 41–68.Google Scholar
National Assessment Governing Board (1992). Reading framework for the National Assessment of Educational Progress: 1992–2002, Washington, DC: National Assessment Governing Board.Google Scholar
National Assessment Governing Board (2002). Mathematics framework for the 2003 National Assessment of Educational Progress, Washington, DC: National Assessment Governing Board.Google Scholar
Oltman, P.K., Stricker, L.J., Barrows, T.S. (1990). Analyzing test structure by multidimensional scaling. Journal of Applied Psychology, 75, 21–27.CrossRefGoogle Scholar
Reckase, M.D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401–412.CrossRefGoogle Scholar
Reckase, M.D., McKinley, R.L. (1991). The discriminating power of items that measure more than one dimension. Applied Psychological Measurement, 15, 361–373.CrossRefGoogle Scholar
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, No. 17, Greensboro, NC: Psychometric Society.CrossRefGoogle Scholar
Samejima, F. (1972). A general model for free-response data. Psychometrika Monograph No. 18, Greensboro, NC: Psychometric Society.Google Scholar
Stout, W.F. (1987). A nonparametric approach for assessing latent trait dimensionality. Psychometrika, 52, 589–617.CrossRefGoogle Scholar
Stout, W.F., Habing, B., Douglas, J., Kim, H.R., Roussos, L.A., Zhang, J. (1996). Conditional covariance based nonparametric multidimensionality assessment. Applied Psychological Measurement, 20, 331–354.CrossRefGoogle Scholar
Van Abswoude, A.A.H., Van der Ark, L.A., Sijtsma, K. (2004). A comparative study on test dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28, 3–24.CrossRefGoogle Scholar
Wang, M. (1986). Fitting a unidimensional model on the multidimensional item response data (ONR Technical Report 87-1). Iowa City, IA: University of Iowa.Google Scholar
Yang, X., & Zhang, J. (2001). Construction and evaluation of bias-corrected estimators of DETECT dimensionality index. Paper presented at the 2001 (April) annual meeting of the American Educational Research Association, Seattle, WA.Google Scholar
Yu, F., Nandakumar, R. (2001). Poly-DETECT for quantifying the degree of multidimensionality of item response data. Journal of Educational Measurement, 38, 99–120.CrossRefGoogle Scholar
Zhang, J. (1996). Some fundamental issues in item response theory with applications. Unpublished doctoral dissertation, Department of Statistics, University of Illinois at Urbana-Champaign.Google Scholar
Zhang, J., Stout, W.F. (1999a). Conditional covariance structure of generalized compensatory multidimensional items. Psychometrika, 64, 129–152.CrossRefGoogle Scholar
Zhang, J., Stout, W.F. (1999b). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213–249.CrossRefGoogle Scholar
Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24, 293–308.CrossRefGoogle Scholar