Hostname: page-component-669899f699-rg895 Total loading time: 0 Render date: 2025-04-24T17:52:46.718Z Has data issue: false hasContentIssue false
  • English
  • Français

Diagnostic Classification Models for Testlets: Methods and Theory

Published online by Cambridge University Press:  01 January 2025

Xin Xu ,
Guanhua Fang ,
Jinxin Guo ,
Zhiliang Ying  and
Susu Zhang Open the ORCID record for Susu Zhang [Opens in a new window]
Xin Xu
Affiliation:
Beijing Normal University
Guanhua Fang
Affiliation:
Fudan University
Jinxin Guo
Affiliation:
Minzu University of China
Zhiliang Ying
Affiliation:
Columbia University
Susu Zhang*
Affiliation:
University of Illinois Urbana-Champaign
*
Correspondence should be made to Susu Zhang, University of Illinois Urbana-Champaign, Champaign, USA. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Diagnostic classification models (DCMs) have seen wide applications in educational and psychological measurement, especially in formative assessment. DCMs in the presence of testlets have been studied in recent literature. A key ingredient in the statistical modeling and analysis of testlet-based DCMs is the superposition of two latent structures, the attribute profile and the testlet effect. This paper extends the standard testlet DINA (T-DINA) model to accommodate the potential correlation between the two latent structures. Model identifiability is studied and a set of sufficient conditions are proposed. As a byproduct, the identifiability of the standard T-DINA is also established. The proposed model is applied to a dataset from the 2015 Programme for International Student Assessment. Comparisons are made with DINA and T-DINA, showing that there is substantial improvement in terms of the goodness of fit. Simulations are conducted to assess the performance of the new method under various settings.

Keywords

diagnostic classification modeltestlet DINAidentifiabilityPISAQ-matrixinteractionhypothesis testingmodel selection
Type
Theory & Methods
Information
Psychometrika , Volume 89 , Issue 3 , September 2024 , pp. 851 - 876
Copyright
Copyright © 2024 The Author(s), under exclusive licence to The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

References

Allman, E. S., Matias, C., Rhodes, J. A. (2009). Identifiability of parameters in latent structure models with many observed variables. The Annals of Statistics, 37 6A3099–3132.CrossRefGoogle Scholar
Bradlow, E. T., Wainer, H., Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64(2), 153–168.CrossRefGoogle Scholar
Cai, L. (2010). A two-tier full-information item factor analysis model with applications. Psychometrika, 75(4), 581–612.CrossRefGoogle Scholar
Chen, Y., Liu, J., Xu, G., Ying, Z. (2015). Statistical analysis of q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866.CrossRefGoogle Scholar
Chen, Y., Liu, Y., Xu, S. (2018). Mutual information reliability for latent class analysis. Applied Psychological Measurement, 42(6), 460–477.CrossRefGoogle ScholarPubMed
Chiu, C.-Y., Douglas, J. A., Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633–665.CrossRefGoogle Scholar
Coelho, P. S., Pereira, L. N. (2011). A spatial unit level model for small area estimation. REVSTAT-Statistical Journal, 9(2), 155–180.Google Scholar
Culpepper, S. A. (2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40(5), 454–476.CrossRefGoogle Scholar
Culpepper, S. A. (2019). An exploratory diagnostic model for ordinal responses with binary attributes: Identifiability and estimation. Psychometrika, 84(4), 921–940.CrossRefGoogle ScholarPubMed
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.CrossRefGoogle Scholar
de la Torre, J., Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.CrossRefGoogle Scholar
de La Torre, J., Karelitz, T. M. (2009). Impact of diagnosticity on the adequacy of models for cognitive diagnosis under a linear attribute structure: A simulation study. Journal of Educational Measurement, 46(4), 450–469.CrossRefGoogle Scholar
DeMars, C. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43(2), 145–168.CrossRefGoogle Scholar
Fang, G., Liu, J., Ying, Z. (2019). On the identifiability of diagnostic classification models. Psychometrika, 84(1), 19–40.CrossRefGoogle ScholarPubMed
Fang, G., Guo, J., Xu, X., Ying, Z., Zhang, S. (2021). Identifiability of bifactor models. Statistica Sinica, 31, 2309–2330.Google Scholar
Gibbons, R. D., Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57(3), 423–436.CrossRefGoogle Scholar
Gu, Y. (2020). Statistical analysis of structured latent attribute models (Unpublished doctoral dissertation).Google Scholar
Gu, Y., Xu, G. (2019). The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika, 84(2), 468–483.CrossRefGoogle ScholarPubMed
Gu, Y., Xu, G. (2020). Partial identifiability of restricted latent class models. The Annals of Statistics, 48(4), 2082–2107.CrossRefGoogle Scholar
Gu, Y., & Xu, G. (2021). Sufficient and necessary conditions for the identifiability of the q-matrix. Statistica Sinica.CrossRefGoogle Scholar
Gu, Y., & Xu, G. (2022). Generic identifiability of the DINA model and blessing of latent dependence. Psychometrika.Google Scholar
Hansen, M. (2013). Hierarchical item response models for cognitive diagnosis, University of California.CrossRefGoogle ScholarPubMed
Hansen, M., Cai, L., Monroe, S., Li, Z. (2016). Limited-information goodness-of-fit testing of diagnostic classification item response models. British Journal of Mathematical and Statistical Psychology, 69(3), 225–252.CrossRefGoogle ScholarPubMed
Henson, R., Templin, J., Willse, J. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210.CrossRefGoogle Scholar
Jennrich, R. I., Bentler, P. M. (2012). Exploratory bi-factor analysis: The oblique case. Psychometrika, 77(3), 442–454.CrossRefGoogle ScholarPubMed
Junker, B. W., Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.CrossRefGoogle Scholar
Kang, H.-A., Liu, J., & Ying, Z. (2017). A graphical diagnostic classification model. Preprint at arXiv:1707.06318.Google Scholar
Kish, L., Namboodiri, N. K., Pillai, R. K. (1962). The ratio bias in surveys. Journal of the American Statistical Association, 57(300), 863–876.CrossRefGoogle Scholar
Köhn, H.-F., Chiu, C.-Y. (2016). A proof of the duality of the DINA model and the DINO model. Journal of Classification, 33, 171–184.CrossRefGoogle Scholar
Ma, W. (2019). A diagnostic tree model for polytomous responses with multiple strategies. British Journal of Mathematical and Statistical Psychology, 72(1), 61–82.CrossRefGoogle ScholarPubMed
Ma, W., de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253–275.CrossRefGoogle ScholarPubMed
Ma, W., Wang, C., & Xiao, J. (2023). A testlet diagnostic classification model with attribute hierarchies. Applied Psychological Measurement, 01466216231165315.CrossRefGoogle Scholar
Macready, G. B., Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2(2), 99–120.CrossRefGoogle Scholar
Meng, X.-L. (1993). On the absolute bias ratio of ratio estimators. Statistics & Probability Letters, 18(5), 345–348.CrossRefGoogle Scholar
OECD. (2016). Pisa 2015 assessment and analytical framework: Science, reading, mathematic and financial literacy. Author Paris.CrossRefGoogle Scholar
Rupp, A. A., Templin, J., Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications, Guilford Press.Google Scholar
Sha, S. (2016). Nonparametric diagnostic classification analysis for testlet-based tests (Unpublished doctoral dissertation). The University of North Carolina at Greensboro.Google Scholar
Sireci, S. G., Thissen, D., Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28(3), 237–247.CrossRefGoogle Scholar
Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of educational measurement, 20(4), 345–354.CrossRefGoogle Scholar
Templin, J. L., Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287.CrossRefGoogle ScholarPubMed
Wainer, H., Bradlow, E. T., Wang, X. (2007). Testlet response theory and its applications, Cambridge University Press.CrossRefGoogle Scholar
Xu, G. (2017). Identifiability of restricted latent class models with binary responses. The Annals of Statistics, 45(2), 675–707.CrossRefGoogle Scholar
Xu, G., Zhang, S. (2016). Identifiability of diagnostic classification models. Psychometrika, 81(3), 625–649.CrossRefGoogle ScholarPubMed
Yavuz, E., Atar, H. Y. (2020). An examination of Turkish students’ PISA 2015 collaborative problem-solving competencies. International Journal of Assessment Tools in Education, 7(4), 588–606.CrossRefGoogle Scholar
Zhan, P., Li, X., Wang, W.-C., Bian, Y., & Wang, L. (2015). The multidimensional testlet-effect cognitive diagnostic models. Acta Psychologica Sinica.CrossRefGoogle Scholar
Zhan, P., Liao, M., Bian, Y. (2018). Joint testlet cognitive diagnosis modeling for paired local item dependence in response times and response accuracy. Frontiers in Psychology, 9, 607.CrossRefGoogle ScholarPubMed