Hostname: page-component-669899f699-7xsfk Total loading time: 0 Render date: 2025-04-25T19:14:43.479Z Has data issue: false hasContentIssue false
  • English
  • Français

A Diagnostic Facet Status Model (DFSM) for Extracting Instructionally Useful Information from Diagnostic Assessment

Published online by Cambridge University Press:  01 January 2025

Chun Wang Open the ORCID record for Chun Wang [Opens in a new window]
Chun Wang*
Affiliation:
University of Washington
*
Correspondence should bemade to ChunWang, 312E Miller Hall Measurement and Statistics, College of Education, University of Washington, 2012 Skagit Ln, Seattle, WA 98105, USA. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Modern assessment demands, resulting from educational reform efforts, call for strengthening diagnostic testing capabilities to identify not only the understanding of expected learning goals but also related intermediate understandings that are steppingstones on pathways to learning goals. An accurate and nuanced way of interpreting assessment results will allow subsequent instructional actions to be targeted. An appropriate psychometric model is indispensable in this regard. In this study, we developed a new psychometric model, namely, the diagnostic facet status model (DFSM), which belongs to the general class of cognitive diagnostic models (CDM), but with two notable features: (1) it simultaneously models students’ target understanding (i.e., goal facet) and intermediate understanding (i.e., intermediate facet); and (2) it models every response option, rather than merely right or wrong responses, so that each incorrect response uniquely contributes to discovering students’ facet status. Given that some combination of goal and intermediate facets may be impossible due to facet hierarchical relationships, a regularized expectation–maximization algorithm (REM) was developed for model estimation. A log-penalty was imposed on the mixing proportions to encourage sparsity. As a result, those impermissible latent classes had estimated mixing proportions equal to 0. A heuristic algorithm was proposed to infer a facet map from the estimated permissible classes. A simulation study was conducted to evaluate the performance of REM to recover facet model parameters and to identify permissible latent classes. A real data analysis was provided to show the feasibility of the model.

Keywords

Regularized expectation–maximization algorithmCognitive diagnostic modelFacet map
Type
Theory & Methods
Information
Psychometrika , Volume 89 , Issue 3 , September 2024 , pp. 747 - 773
Copyright
Copyright © 2024 The Author(s), under exclusive licence to The Psychometric Society, corrected publication

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The study is funded by IES R305D200015 and NSF EDU-CORE #2300382.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

References

Alonzo, A. C., Steedle, J. T. (2009). Developing and accessing a force and motion learning progression. Science Education, 93(3), 389–421.CrossRefGoogle Scholar
Bradshaw, L., Templin, J. (2014). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79(3), 403–425.CrossRefGoogle ScholarPubMed
Briggs, D. C., Alonzo, A. C., Schwab, C., Wilson, M. (2006). Diagnostic assessment with ordered multiple-choice items. Educational Assessment, 11(1), 33–63.CrossRefGoogle Scholar
Chattergoon, R. (2020). Using polytomous item response theory models to validate learning progressions. Univeristy of Colorado at Boulder, Doctoral Dissertation.Google Scholar
Chen, Y., Wang, S. (2023). Bayesian estimation of attribute hierarchy for cognitive diagnosis models. Journal of Educaitonal and Behavorial Statistics, .CrossRefGoogle Scholar
Cizek, G. J., Bunch, M. B., Koons, H. (2004). Setting performance standards: Contemporary methods. Educational Measurement: Issues and Practice, 23(4), 31–31.CrossRefGoogle Scholar
Clement, J. (1987). The use of analogies and anchoring intuitions to remediate misconceptions in mechanics. Paper presented at the Annual Meeting of American Educational Research Association, Washington, DC.Google Scholar
Corcoran, T. B., Mosher, F. A., & Rogat, A. (2009). Learning progressions in science: An evidence-based approach to reform. Report of the Center on Continuous Instructional Improvement, Teachers College, Columbia University, New York.CrossRefGoogle Scholar
Dadey, N. (2017). Reporting scores from NGSS assessments: Exploring scores & subscores, Portsmouth, NH: Reidy Interactive Lecture Series.Google Scholar
Dahlgren, M. A., Hult, H., Dahlgren, L. O., af Segerstad, H. H., & Johansson, K. (2006). From senior student to novice worker: Learning trajectories in political science, psychology and mechanical engineering. Studies in Higher Education, 31(5), 569–586.CrossRefGoogle Scholar
Daro, P., Mosher, F. A., & Corcoran, T. (2011). Learning Trajoctories in Mathematics: A Foundation for Standards, Curriculum, Assessment and Instruction. CPRE Research Report #RR-68. New York: Columbia University.Google Scholar
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.CrossRefGoogle Scholar
de la Torre, J., Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.CrossRefGoogle Scholar
DiBello, L. V., Henson, R. A., Stout, W. F. (2015). A family of generalized diagnostic classification models for multiple choice option-based scoring. Applied psychological measurement, 39(1), 62–79.CrossRefGoogle ScholarPubMed
diSessa, A. A. (1993). Toward an epistemology of physics. Cognition and Instruction, 10 2 &3165–255.CrossRefGoogle Scholar
diSessa, A. A. (2014a). A history of conceptual change research: Threads and fault lines. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences (2nd ed., pp. 88–108). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
diSessa, A. A. (2014b). The construction of causal schemes: Learning mechanisms at the knowledge level. Cognitive science, 38(5), 795–850.CrossRefGoogle Scholar
diSessa, A. A. (2017). Knowledge in pieces: An evolving framework for understanding knowing and learning. In Amin, T. G., Levrini, O. (Eds), Converging perspectives on conceptual change: Mapping an emerging paradigm in the learning sciences, London: Routledge 7–16.CrossRefGoogle Scholar
Doignon, J. P., Falmagne, J. C. (1999). Knowledge spaces, Berlin: Springer.CrossRefGoogle Scholar
Foisy, L-MB, Potvin, P., Riopel, M., Masson, S. (2015). Is inhibition involved in overcoming a common physics misconception in mechanics?. Trends in Neuroscience and Education, 4 1–226–36.CrossRefGoogle Scholar
Gu, Y., Xu, G. (2019). Learning attribute patterns in high-dimensional structured latent attribute models. Journal of Machine Learning, 20, 1–58.Google Scholar
Haberman, S., Sinharay, S., Puhan, G. (2009). Reporting subscores for institutions. British Journal of Mathematical and Statistical Psychology, 62(1), 79–95.CrossRefGoogle ScholarPubMed
Hadenfeldt, J. C., Bernholt, S., Liu, X., Neumann, K., Parchmann, I. (2013). Using ordered multiple-choice items to assess students’ understanding of the structure and composition of matter. Journal of Chemical Education, 90(12), 1602–1608.CrossRefGoogle Scholar
Henson, R. A., Templin, J. L., Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210.CrossRefGoogle Scholar
Kapon, S., diSessa, A. A. (2012). Reasoning through instructional analogies. Cognition and Instruction, 30(3), 261–310.CrossRefGoogle Scholar
Kluger, A. N., DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.CrossRefGoogle Scholar
Kuo, B. C., Chen, C. H., & de la Torre, J. (2018). A cognitive diagnosis model for identifying coexisting skills and misconceptions. Applied Psychological Measurement, 42(3), 179–191.CrossRefGoogle Scholar
Lanting, A. S. Y. (2000). An empirical study of a district-wide K–2 performance assessment program: Teacher practices, information gained, and use of assessment results. University of Illinois at Urbana-Champaign.Google Scholar
Levy, R. (2019). Dynamic Bayesian network modeling of game-based diagnostic assessments. Multivariate Behavioral Research, 54, 771–794.CrossRefGoogle ScholarPubMed
Liu, Y., Culpepper, S. A. (2023). Restricted latent class models for nominal response data: Identifiability and estimation. Psychometrika, .Google ScholarPubMed
Liu, R., Huggins-Manley, A. C., Bradshaw, L. (2017). The impact of Q-matrix designs on diagnostic classification accuracy in the presence of attribute hierarchies. Educational and Psychological Measurement, 77(2), 220–240.CrossRefGoogle ScholarPubMed
Minstrell, J. (1989). Teaching science for understanding. In Resnick, L. B., Klopfer, L. E. (Eds), Toward the thinking curriculum: Current cognitive research, Alexandria, VA: Association for Supervision and Curriculum Development 129–149.Google Scholar
Minstrell, J. (1991). Facets of students’ knowledge and relevant instruction. In R. Duit, F. Goldberg, & H. Niedderer (Eds.), Research in Physics Learning: Theoretical Issues and Empirical Studies, Proceedings of an International Workshop, Bremen, Germany, March 4–8, 1991 (IPN, Kiel Germany, 1992) pp. 110–128.Google Scholar
Minstrell, J. (1992). Facets of students’ knowledge and relevant instruction. In Duit, R., Goldberg, F., Niedderer, H. (Eds), Research in physics learning: Theoretical Issues and Empirical Studies, Kiel: IPN 110–128.Google Scholar
Minstrell, J. (2000). Student Thinking and Related Assessment: Creating a Facet Assessment-based Learning Environment. In Pellegrino, J. , Jones, L., and Mitchell, K. (Eds.) Grading the Nation’s Report Card: Research from the Evaluation of NAEP. Committee on the Evaluation of National and State Assessment of Educational Progress, Board of Testing and Assessment. Washington D.C.: National Academy Press.Google Scholar
Minstrell, J., Anderson, R., Li, M. (2015). Diagnostic instruction: Toward an integrated system for classroom assessment. In Duschl, Richard A, Bismark, Amber S (Eds), Reconceptualizing STEM education: The central role of practices, New York: Routledge.Google Scholar
Morphew, J. W., Mestre, J. P., Kang, H.-A., Chang, H.-H., Fabry, G. (2018). Using computer adaptive testing to assess physics proficiency and improve exam performance in an introductory physics course. Physical Review Physics Education Research, 14(2), .CrossRefGoogle Scholar
Naumenko, O. (2014). Comparison of various polytomous item response theory modeling approaches for task-based simulation CPA exam data. In AICPA 2014 summer internship project. Retrieved from naumenko-polytomous-2014.pdf (aicpa.org).Google Scholar
Pellegrino, J. W., Chudowsky, N., Glaser, R. (2001). Knowing what students know: The science and design of educational assessment, Washington, DC: National Academy Press.Google Scholar
Potvin, P., Masson, S., Lafortune, S., Cyr, G. (2015). Persistence of the intuitive conception that heavier objects sink more: A reaction time study with different levels of interference. International Journal of Science and Mathematics Education, 13(1), 21–43.CrossRefGoogle Scholar
Simon, M. A., Tzur, R. (2004). Explicating the role of mathematical tasks in conceptual learning: An elaboration of the hypothetical learning trajectory. Mathematical Thinking and Learning, 6(2), 91–104.CrossRefGoogle Scholar
Smolleck, L., & Hershberger, V. (2011). Playing with science: An investigation of young children’s science conceptions and misconceptions. Current Issues in Education, 14. Retrieved from http://cie.asu.edu/ojs/index.php/cieatasu/article/view.Google Scholar
Stavy, R., Babai, R., Tsamir, P., Tirosh, D., Lin, F.-L., McRobbie, C. (2006). Are intuitive rules universal?. International Journal of Science and Mathematics Education, 4, 417–436.CrossRefGoogle Scholar
Steedle, J. T., Shavelson, R. J. (2009). Supporting valid interpretations of learning progression level diagnoses. Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 46(6), 699–715.CrossRefGoogle Scholar
Su, S., Wang, C., Weiss, D. (2021). Performance of the S-χ2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\chi ^2$$\end{document} statistic for the multidimensional graded response model. Educational and Psychological Measurement, 81(3), 491–522.CrossRefGoogle Scholar
Tate, R. L. (2004). Implications of multidimensionality for total score and subscore performance. Applied Measurement in Education, 17(2), 89–112.CrossRefGoogle Scholar
Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.Google Scholar
Tatsuoka, K. K. (2009). Cognitive assessment: An introduction to the rule space method, New York: Routledge.CrossRefGoogle Scholar
Templin, J., Bradshaw, L. (2014). Hierarchical diagnostic classification models: A family of models for estimating and testing attribute hierarchies. Psychometrika, 79(2), 317–339.CrossRefGoogle ScholarPubMed
Thissen-Roe, A., Hunt, E., Minstrell, J. (2004). The diagnoser project: Combining assessment and learning. Behavior Research Methods, Instruments, & Computers, 36(2), 234–240.CrossRefGoogle ScholarPubMed
Thompson, F., Logue, S. (2006). An exploration of common student misconceptions in science. International Education Journal, 7(4), 553–559.Google Scholar
Tjoe, H., de la Torre, J. (2014). On recognizing proportionality: Does the ability to solve missing value proportional problems presuppose the conception of proportional reasoning?. The Journal of Mathematical Behavior, 33, 1–7.Google Scholar
Underwood, S., Posey, L., Herrington, D., Carmel, J., Cooper, M. (2018). Adapting assessment tasks to support three-dimensional learning. Journal of Chemical Education, 95, 207–217.CrossRefGoogle Scholar
Vosniadou, S., Verschaffel, L. (2004). Extending the conceptual change approach to mathematics learning and teaching. Learning and Instructions, 14, 445–451.CrossRefGoogle Scholar
Wang, C. (2021a). Using penalized EM algorithm to infer learning trajectories in latent transition CDM. Psychometrika, 86(1), 167–189.CrossRefGoogle Scholar
Wang, C. (2021b). On interim cognitive diagnostic computerized adaptive testing in learning context. Applied Psychological Measurement, 45, 235–252.CrossRefGoogle Scholar
Wang, C., Lu, J. (2021). Learning attribute hierarchies from data: Two exploratory approaches. Journal of Educational and Behavioral Statistics, 46, 58–84.Google Scholar