2 Main concepts in knowledge structure theory

In knowledge structure theory (KST, Doignon & Falmagne, Reference Doignon and Falmagne1999, Reference Doignon and Falmagne2012), the knowledge structure is a pair $(P,\mathcal {K})$ where P is a set named the domain of knowledge, and $\mathcal {K}$ is a family of subsets of P that contains, at least, the empty set $\emptyset $ and the domain P. Each subset $K \in \mathcal {K}$ is named a knowledge state. In concrete applications of the theory, the set P is regarded to be the collection of all the problems, questions, quizzes that can be formulated in a given area of knowledge (e.g., geometry). Then, the knowledge state of an individual is the subset $K \in \mathcal {K}$ of problems in P that the individual masters. It can be regarded as the possibly multidimensional “ability” of the individual.

It should be observed that, in general, the knowledge structure $\mathcal {K}$ does not contain all possible subsets of P. That is, in concrete applications, $\mathcal {K}$ is a strict subset of the entire power set on the set P. The method and criteria that are used for deciding whether the subset $X \subseteq P$ is a knowledge state or not can be either theoretical or data-driven, depending on the purpose of the application. The most elementary theoretical method consists in the specification of the so-called “surmise relation”, a quasi-order (reflexive and transitive) relation $\precsim $ of the set P of problems whose interpretation is as follows: given any two problems $p,q \in P$ , $p \precsim q$ if (excluding random error) failing p entails failing P.

Example 2.1. The following example is identical to that provided by Chiu et al. (Reference Chiu, Köhn and Ma2023). For $Q = \{a,b,c,d,e\}$ , consider the surmise relation $\precsim $ defined as follows:

$$\begin{align*}a \precsim c, \ b \precsim c, \ a \precsim d, \ b \precsim d, \ c \precsim e. \end{align*}$$

Notice that relations like $a \precsim a$ and $a \precsim e$ , that can be inferred by transitivity and or reflexivity are omitted. Not all subsets of P are “consistent” with this relation. A subset $K \subseteq P$ is consistent with $\precsim $ if for every problem $q \in K$ , all the predecessors of P are also in K, namely, the following implication must hold true for all pairs $p,q \in P$ of problems:

$$\begin{align*}\text{if } p \precsim q\quad \text{and}\quad q \in K \text{ then } p \in K. \end{align*}$$

Of the $2^5 = 32$ subsets of P, the only ones that are consistent with $\precsim $ are the following nine:

$$\begin{align*}\mathcal{K} = \{\emptyset, \{a\}, \{b\}, \{a, b\}, \{a, b, c\}, \{a, b, d\}, \{a, b, c, d\}, \{a, b, c, e\}, P\} \end{align*}$$

that form the knowledge structure $(P,\mathcal {K})$ derived from $\precsim $ (see Figure 1 for an illustration). This knowledge structure has two fundamental properties: the intersection of whatever subset $\mathcal {F} \subseteq \mathcal {K}$ of knowledge states is itself a knowledge state (closure under intersection). Similarly, the union of $\mathcal {F}$ is itself a knowledge state (closure under union). Knowledge structures of this type are named quasi-ordinal knowledge spaces. The relationship between the surmise relations of P and the quasi-ordinal knowledge spaces on P is one-to-one (Birkhoff, Reference Birkhoff1937; Doignon & Falmagne, Reference Doignon and Falmagne1999).

3 How KST and CD models are interlocked?

There is a close connection between KST and CD models, which has been spelled out and developed formally by Heller et al. (Reference Heller, Stefanutti, Anselmi and Robusto2015) for the case of two particular probabilistic models in the two theories: The so-called a competence-based extension of the BLIM (CBLIM) for KST, and the multiple strategy DINA model (de la Torre & Douglas, Reference de la Torre and Douglas2008) for CD models. It was shown that the two models at hand are formally equivalent. They are essentially the very same model, expressed by using different notations and different terminologies. In all probability, prior difficulties in recognizing such equivalence just laid in the substantially different formal and notational approaches followed by the two theories.

The present section summarizes the main theoretical results of Heller et al. (Reference Heller, Stefanutti, Anselmi and Robusto2015). The fundamental concept upon which a connection between the two theories has been built is rather simple. KST is mostly a set-theoretical theory. Its deterministic skeleton is based on assumptions, definitions, and results that largely draw upon set, order, and lattice theory. There is a reason for this: KST originated in an area of mathematical psychology that grew out of measurement theory as it was conceived, for instance, in the Foundations of Measurement (Krantz et al., Reference Krantz, Luce, Suppes and Tversky1971). In that particular area, the focus was especially on qualitative structures and on how to measure them through numbers. The whole machinery was set theoretically oriented.

On the other side, CD models were mostly developed under the item response theory (IRT) framework, which is inherently probabilistic and numerical. Numerical structures like vectors and matrices are the most common tools of the various models and methods developed within item response theory. The most relevant numerical structures that are of some interest here are the binary vectors and the binary matrices, not only because they populate a wide range of CD models that are around, but also because they are easily related to deterministic KST concepts.

Thus, at the ground of the connection between the two approaches there is the elementary observation that binary vectors of a given length n can be put in a bijective correspondence with subsets of a set $A = \{a_1,a_2,\ldots ,a_n\}$ . The bijection can be easily obtained by constructing, for each subset $X \subseteq A$ , its indicator vector, that is a binary vector $\iota (X)$ with one component for each element in A, where this component is one if the corresponding element of A is in X, and 0 if it is not. This simple observation makes it easy to switch from the primitive concepts of one theory to those of the other theory. This exercise was already carried out by Heller et al. (Reference Heller, Stefanutti, Anselmi and Robusto2015), who established some fundamental correspondences. Let $P=\{q_1,q_2,\ldots ,q_m\}$ be a set of items, and $S=\{s_1,s_2,\ldots ,s_n\}$ be a set of skills.

• • The observed response pattern is a binary vector $\mathbf {x} = (x_1,x_2,\ldots ,x_n)$ in CD models, and it is a subset $R \subseteq P$ in KST. In both cases, it represents the (dichotomous true/false) responses of an individual to a set P of m items. In particular, $\mathbf {x}$ and R represent the same response pattern if $\mathbf {x} = \iota (R)$ . Considering Example 2.1 on the domain $P=\{a,b,c,d,e\}$ of five items, the observed response pattern of an individual providing a correct answer for items a, c, and d is represented by the vector $\mathbf {x}=(1,0,1,1,0)$ in CD models, and by the set $R=\{a,c,d\}$ in KST.

• • In CD models, the notion of an ideal response pattern refers to the vector $\xi = (\xi _1,\xi _2,\ldots ,\xi _m)$ of dichomomous responses to the items that are expected if no error (e.g., in the form of guessing or slipping) occurs. In KST, this is called the knowledge state and it is represented as a subset $K \subseteq P$ of all the items that a person is capable of solving. In Example 2.1, the knowledge state $K=\{a,b,c\}$ of an individual corresponds to the ideal response pattern $\xi =(1,1,1,0,0)$ .

• • The attribute profile is, in CD models, a binary vector $\alpha =(\alpha _1,\alpha _2,\ldots ,\alpha _n)$ having length equal to the cardinality of the set S of skills. It represents the skills possessed by an individual. Let $C \subseteq S$ be the collection of all the skills possessed by an individual. Then in KST, C is called the competence state, and $\iota (C)$ is the attribute profile. In Example 2.1, suppose that mastering all skills in the set $S=\{s_1,s_2,s_3,s_4,s_5\}$ is required for solving the five items in P. Then, a plausible attribute profile of an individual is $\alpha =(0,0,1,1,1)$ in CD models and it is $C=\{s_3,s_4,s_5\}$ by KST.

It is worth noticing that, what in KST is called knowledge structure $\mathcal {K}$ corresponds, in CD models, to the collection of all the ideal response patterns. In CD models, this collection is represented by a binary matrix, where each row represents an ideal response pattern. In Example 2.1, the collection of all the ideal response patterns corresponding to knowledge structure $\mathcal {K}$ , is the binary matrix (rows are ordered like the elements in $\mathcal {K}$ ):

$$\begin{align*}\begin{pmatrix} 0 & 0 & 0 & 0 & 0 \\ 1 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 & 0 \\ 1 & 1 & 1 & 0 & 0 \\ 1 & 1 & 0 & 1 & 0 \\ 1 & 1 & 1 & 1 & 0 \\ 1 & 1 & 1 & 0 & 1 \\ 1 & 1 & 1 & 1 & 1 \\ \end{pmatrix}. \end{align*}$$

It is convenient to keep clearly distinct the level of the ideal response patterns from that of the attribute profiles. In KST, the former is referred to as the performance level, whereas the latter is referred to as the competence level. Notice that both levels represent latent constructs, which are distinguished from the observed response patterns.

In the beginning, KST was exclusively focused on the performance level of the knowledge states (ideal response patterns). From the perspective of cognitive diagnosis, ignoring the competence level of the skills may be seen as a disadvantage. If individual ability is represented by a subset of items, this representation provides no direct explanation or interpretation in terms of the psychological mechanisms underlying the response behavior. All in all, knowledge states are features of persons and as such, they should depend on skills/attributes. Since in KST the knowledge state is operationalized as a subset of items, it is item-dependent. Instead, in CD models, the notions of Q-matrix and attribute profile allow to separate individual skills from items.

Once the skill map (the Q-matrix) has been established, and its interpretation (e.g., conjunctive, rather than disjunctive) has been stated, each attribute profile delineates exactly one ideal response pattern (knowledge state). If the correspondence between attribute profiles and ideal response patterns is one-to-one, then there can be unique skill assessment (Heller et al., Reference Heller, Stefanutti, Anselmi and Robusto2015). In this case, the attribute profile and the ideal response pattern represent exactly the very same thing, namely a multidimensional individual ability. In CD models, the multidimensional ability is further decomposed into discrete skills.

However, if the relationship is not one-to-one, then the same ideal response pattern may be associated with more than one attribute profile. In this case there cannot be unique skill assessment. It means that there is no unique way of decomposing the multidimensional individual ability into discrete skills.

To summarize, if the performance and competence levels are in a one–to–one correspondence then skill assessment is unique. Thus, the knowledge state (ideal response pattern) depends on items in as much the same way as the competence state (attribute profile) depends on items. The only situation where the performance level of the ideal response pattern and the competence level of the attribute profile do not correspond with one another is when skill assessment is not unique. Here however the problem is located at the competence level, where we have more than a single interpretation (attribute profile) for the same ideal response pattern. At the performance level, the representation remains unique.

4 The BLIM and the DINA model

Chiu et al. (Reference Chiu, Köhn and Ma2023) claim that the BLIM is equivalent to a DINA model, to which certain modifications and restrictions apply. More precisely, their main statement is that the

However, regarding the BLIM as DINA model with single-attribute items is misleading, because the BLIM makes no assumptions about underlying attributes whatsoever. The very same issue arises when considering the PoLIM as a “single-attribute” version of some pre-existing polytomous DINA model. Both the BLIM and the PoLIM are totally agnostic to any psychological mechanism that might give rise to the knowledge structure they are based on. Singling out a particular link between items and attributes invites invalid conclusions.

The intimate relation between KST and CD notions in case of dichotomous items is fully characterized already by Heller et al. (Reference Heller, Stefanutti, Anselmi and Robusto2015), who prove various theorems on the existing correspondences. The BLIM is extended by introducing skills (attributes in CD terms), which are linked to the items via a so-called skill function (corresponding to a collection of Q-matrices in CD). Moreover, this approach is able to capture dependencies between skills through assuming an arbitrary competence structure, which amounts to considering an arbitrary subset of permissible attribute profiles in CD. The resulting model is called the competence-based BLIM (or, CBLIM).

As one of the main results of Heller et al. (Reference Heller, Stefanutti, Anselmi and Robusto2015) the CBLIM is shown to be equivalent to the multiple strategy DINA model (MS-DINA; De La Torre & Douglas, Reference De La Torre and Douglas2004). As the DINA model is a special case of the MS-DINA, it is to be conceived as a special case of the CBLIM which assumes a conjunctive rule operating on the skills/attributes. This shows that KST offers a framework for formulating models equivalent to the DINA model and special cases thereof, but these KST models cannot be identified with the BLIM.

Chiu et al. (Reference Chiu, Köhn and Ma2023) claim that “estimating BLIM using DINA requires that all BLIM-items are single-attribute items because only then $\boldsymbol {\alpha }=\boldsymbol {\xi }$ is true” (p. 658). Confining consideration to this very particular case may be fine if the aim of the paper was limited to exemplify that parameter estimation algorithms developed for the DINA model can be used for estimating the parameters of the BLIM, although the latter is equipped with its own algorithms (Heller & Wickelmaier, Reference Heller and Wickelmaier2013; Stefanutti & Robusto, Reference Stefanutti and Robusto2009) and publicly available routines (the R package pks; Heller & Wickelmaier, Reference Heller and Wickelmaier2013). However, the cited statement as such is wrong.

Beyond the fact that the equation $\boldsymbol {\alpha }=\boldsymbol {\xi }$ is problematic in itself (see below), the kind of constraint it implies is not necessary for aligning the predictions of a DINA model with those of a BLIM. For clarifying this issue we refer to the example of Chiu et al. (Reference Chiu, Köhn and Ma2023), which in turn was taken from Doignon and Falmagne (Reference Doignon and Falmagne1999, Example 7.1). Consider the knowledge structure

$$\begin{align*}\mathcal{K} = \{\emptyset, \{a\}, \{b\}, \{a, b\}, \{a, b, c\}, \{a, b, d\}, \{a, b, c, d\}, \{a, b, c, e\}, P\} \end{align*}$$

on the domain $P = \{a, b, c, d, e\}$ . On the left-hand side of Figure 1 the Hasse diagram of $\mathcal {K}$ is shown, with paths of ascending line segments representing set inclusion. In this subsequent plots we use the shorthand notation $abd$ to denote the set $\{a,b,d\}$ , for example.

Figure 1 Knowledge structure $\mathcal {K}$ on the set of items $P = \{a,b,c,d,e\}$ and isomorphic competence structure $\mathcal {C}$ on the set of skills/attributes $S = \{s,t,u,v,w\}$ .

Notice that the knowledge structure $\mathcal {K}$ is in one-to-one correspondence to the partial order on the domain of items illustrated by the Hasse diagram on the left-hand side in Figure 2, and thus is said to be an ordinal knowledge space (see, e.g.,Doignon & Falmagne, Reference Doignon and Falmagne1999).

Figure 2 Partial orders on the set of items $P = \{a,b,c,d,e\}$ and the set of skills/attributes $S = \{s,t,u,v,w\}$ corresponding to knowledge structure $\mathcal {K}$ and competence structure $\mathcal {C}$ , both illustrated in Figure 1.

The construction of Chiu et al. (Reference Chiu, Köhn and Ma2023) then proceeds by introducing a set of skills/attributes $S = \{s, t, u, v, w\}$ and the identity matrix

as the Q-matrix. This amounts to defining a bijection $f\colon P \to S$ such that $f (a) = s, f (b) = t, {\ldots }, f (e) = w$ . From a structural perspective the bijection f establishes an isomorphism between the knowledge structure $\mathcal {K}$ and a competence structure $\mathcal {C}$ defined by the equivalence

$$\begin{align*}C \in \mathcal{C} ~~\text{if and only if}~~ f^{-1} (C) \in \mathcal{K} \end{align*}$$

for all $C \subseteq S$ , where $f^{-1} (C)$ denotes the preimage of C under f. The isomorphism may be conceived as a mapping p from $\mathcal {C}$ onto $\mathcal {K}$ , which in KST is known as a problem function (e.g., Doignon & Falmagne, Reference Doignon and Falmagne1999). Comparing the Hasse diagram of the competence structure $\mathcal {C}$ on S to that of the knowledge structure $\mathcal {K}$ on P makes the isomorphism self-evident (see Figure 1). Using the notation $\boldsymbol {\alpha } = \boldsymbol {\xi }$ , equating attribute profiles $\boldsymbol {\alpha }$ (corresponding to the competence states $C \in \mathcal {C}$ ) and ideal response patterns $\boldsymbol {\xi }$ (corresponding to the knowledge states $K \in \mathcal {K}$ ), obscures the fact that there is no identity, but only an isomorphism. Choosing a competence structure $\mathcal {C}$ (i.e., the permissible attribute patterns) isomorphic to $\mathcal {K}$ amounts to inducing a hierarchical structure on the skills/attributes, which is isomorphic to the partial order on the items corresponding to $\mathcal {K}$ (see the Hasse diagram at the right-hand side of Figure 2). This goes unmentioned in Chiu et al. (Reference Chiu, Köhn and Ma2023), and ignoring this fact leads to a fundamental misunderstanding when they are dealing with identifiability issues (see Section 5).

Instead of imposing the above outlined constraints by inducing a hierarchical structure on the skills/attributes with the Q-matrix being the identity matrix, let all possible attribute profiles on the skills/attributes in S be permissible. Using the Q-matrix

(1)

the induced ideal response patterns are the vector representations of exactly the knowledge states in $\mathcal {K}$ (Heller, Reference Heller2022, Corollary 2). Notice that this Q-matrix is a densified version of the identity matrix in the sense of Gu and Xu (Reference Gu and Xu2020), and plays a prominent role in the so-called restricted Q-matrix design in the context of attribute hierarchy models (Leighton et al., Reference Leighton, Gierl and Hunka2004). This example makes clear that the very same knowledge structure can arise from substantially different assumptions on the underlying skills/attributes: while Chiu et al. (Reference Chiu, Köhn and Ma2023) assume single-attribute items, the Q-matrix of Equation (1) assumes multiple-attribute items. The equivalence of the two DINA models with respect to the ideal response patterns is due to the fact that the knowledge structure $\mathcal {K}$ is closed under intersection, a property which results as a necessary consequence of applying a conjunctive rule to an unstructured set of skills/attributes (Gediga & Düntsch, Reference Gediga and Düntsch2002; Ünlü et al., Reference Ünlü, Schrepp, Heller, Hockemeyer, Wesiak and Albert2013). Notice that in this case the problem function p from $2^S$ to $\mathcal {K}$ mapping competence states to knowledge states (i.e., attribute profiles to ideal response patterns) is not one-to-one, but many-to-one as illustrated in Figure 3. Gray circles are in one-to-one correspondence to the knowledge states in $\mathcal {K}$ shown in Figure 1, and all the competence states plotted in each of them form an equivalence class, as all of them are mapped to the respective knowledge state. For example, all the competence states included in $\{u,v,w\}$ are mapped to the empty knowledge state $\emptyset $ (i.e., $p (C) = \emptyset $ for all $C \subseteq \{u,v,w\}$ ). This means that the hierarchical structure of the skills/attributes can be represented in either the set of permissible attribute profiles (as in Figure 1), or in the Q-matrix (as in the densified version), or in both. In fact, given the Q-matrix of Equation (1), there are many more competence structures that induce the very same knowledge structure $\mathcal {K}$ . Take for example all the competence structures on S that include the structure $\mathcal {C}$ of Figure 1. Notice that the latter results from selecting the minimal competence states of each of the equivalence classes in Figure 3 (i.e., from each of the gray circles). All the instantiations of the DINA model built on these competence structures may be considered equivalent: not only because they induce the same knowledge structure, but also concerning the probabilistic framework defined on top of it. This is illustrated in the sequel.

Figure 3 Line diagram illustrating the many-to-one relationship between competence and knowledge states (i.e., attribute profiles and ideal response patterns) for an unstructured set of skills/attributes and the Q-matrix of Equation (1).

Let $\pi _{\mathcal {C}}$ denote the probability distribution over the competence states in $\mathcal {C}$ , $\pi _{\mathcal {K}}$ the probability distribution over the knowledge states in $\mathcal {K}$ , and $\pi _{\mathcal {R}}$ the probability distribution over the response patterns in $2^P$ . With $\beta $ and $\eta $ the vectors of careless error and lucky guess probabilities (collecting all the $\beta _q$ and $\eta _q$ , $q \in P$ ), the CBLIM is characterized by the parameters $(\pi _{\mathcal {C}}, \beta , \eta )$ , and the BLIM induced by the CBLIM (via $\mathcal {K} = p (\mathcal {C})$ ) is characterized by the parameters $(\pi _{\mathcal {K}}, \beta , \eta )$ . In general the relation between these two parameter spaces is given by the equation

(2) $$ \begin{align} \pi_{\mathcal{K}} (K) = \sum_{C \in \mathcal{C}:~ p(C) = K} \pi_{\mathcal{C}} (C). \end{align} $$

That is, we have the composition of the mappings

(3) $$ \begin{align} \overbrace{(\pi_{\mathcal{C}}, \beta, \eta) ~\mapsto~ \underbrace{(\pi_{\mathcal{K}}, \beta, \eta) ~\mapsto~ \pi_{\mathcal{R}}}_{BLIM}}^{CBLIM} \end{align} $$

In the situation illustrated in Figures 1 and 3 this means that the probability of the knowledge state $K = \{a, b\}$ , for example, is given by $\pi _{\mathcal {K}} (\{a, b\}) = \pi _{\mathcal {C}}(\{s, t\}) + \pi _{\mathcal {C}}(\{s, t, w\})$ .

While in any case the BLIM careless error and lucky guess parameters are identical to the DINA slipping and guessing parameters, the BLIM probability of knowledge state K in $\mathcal {K}$ is obtained by summing over the DINA probabilities of all attribute profiles $\boldsymbol {\alpha }$ that are mapped to the ideal response pattern $\boldsymbol {\xi }$ corresponding to the knowledge state K (Heller, Reference Heller2022; Heller et al., Reference Heller, Stefanutti, Anselmi and Robusto2015). This allows for fully recovering the BLIM parameter estimates, also in cases where the very same BLIM model is induced by distinct DINA models, which may be based on substantially different assumptions as outlined above.

Tables 1 and 2 show estimates of item parameters and probabilities of knowledge states and ideal response patterns, respectively, when replicating estimation for the example assuming a uniform initial distribution on all possible attribute profiles in the DINA model and the densified Q-matrix of Equation (1). The corresponding initial distribution on the knowledge states in the BLIM is derived using Equation (2). Otherwise, the same setup as described in Chiu et al. (Reference Chiu, Köhn and Ma2023) is used. Results again show that all the estimates of the BLIM and DINA parameters are essentially identical, although distinct from those listed in Chiu et al. (Reference Chiu, Köhn and Ma2023). The reason for the latter finding is discussed in Section 5.

Table 1 Item parameter estimates of the BLIM and the DINA model, with the Q-matrix of Equation (1) and all attribute profiles being permissible

Table 2 Results of estimating probabilities of the knowledge states via the BLIM and of the ideal response patterns via the DINA model, with the Q-matrix of Equation (1) and all attribute profiles being permissible

The coinciding parameter estimates demonstrate that the claim of Chiu et al. (Reference Chiu, Köhn and Ma2023), that for mimicking the BLIM with the DINA model, all items need to be conceived as single-attribute items, is wrong. In fact, we have shown above that there may be a variety of distinct DINA models that can mimic a given BLIM. The general mechanism for setting them up is to mirror the information captured by the BLIM’s knowledge structure at the skill/attribute level by coding it by either the set of permissible attribute patterns, or the Q-matrix, or even both. Doubling up the given behavioral information, however, cannot generate new insights. So, any interpretation of the Q-matrix (such as being composed of single- or multi-attribute items, for example) may be meaningless.

The BLIM does not need to come with the specification of any theoretical cognitive assumptions. As a marginal model, which is in line with a vast variety of cognitive theories, it is characterized by a high degree of flexibility. This allows for closely mirroring the structure underlying the observed responses. A knowledge state identifies the items that are mastered or not mastered by an individual, and a knowledge structure tells you whether the mastery of certain items implies the mastery of others. The knowledge structure underlying a BLIM may be established in a completely data-driven way, without the need of resorting to theoretical assumptions on the cognitive processing (e.g., Falmagne et al., Reference Falmagne, Albert, Doble, Eppstein and Hu2013, Section 9.3). This results in knowledge structures of high empirical validity and provides a basis for efficient and precise knowledge assessment as well as for personalized learning. This is demonstrated by highly successful large-scale applications (Falmagne et al., Reference Falmagne, Albert, Doble, Eppstein and Hu2013; Falmagne & Doignon, Reference Falmagne and Doignon2011, like the ALEKSFootnote ¹ system.

Of course, in a second step one may be interested in studying the cognitive mechanisms that bring about the knowledge structure, but this may introduce uncertainties. For example, if considering a DINA model with all attribute profiles being permissible and the Q-matrix of Equation (1) then, in case that more than one attribute profile is mapped to the same ideal response pattern (collections in each of the gray circles in Figure 3), the respective probabilities are not identifiable. In parameter estimation of a DINA model the probabilities of the attribute profiles in these collections usually turn out to be equal, but this is nothing more than an artifact of assuming a uniform initial distribution as the default. What in principle may be identifiable are the knowledge state probabilities of the BLIM, but see Section 5 for a discussion of the current example.

In conclusion, tying the BLIM to a very special case of the DINA model as in Chiu et al. (Reference Chiu, Köhn and Ma2023) creates a misleading perspective. The BLIM is to be conceived as a marginal model for a whole class of models that involve cognitive assumptions. There is previous theoretical work that elaborates on this point but found no mention in Chiu et al. (Reference Chiu, Köhn and Ma2023). The relation of the BLIM to its competence-based extension CBLIM as well as their correspondence to CD models was characterized in full generality by Heller et al. (Reference Heller, Stefanutti, Anselmi and Robusto2015, Reference Heller, Stefanutti, Anselmi and Robusto2016). See also Gu and Xu (Reference Gu and Xu2020), who introduced a conception equivalent to the BLIM in the CD context by considering what they called grouped population proportion parameters. On the basis of this concept they analyze the identifiability properties of the class of two-parameter Q-restricted latent class models (see Section 5 for further details).

5 Identifiability

Chiu et al. (Reference Chiu, Köhn and Ma2023) are right in observing that the BLIM is not identifiable for many types of knowledge structures. There is a broad KST literature which studies the kinds of non-identifiability that arise, and characterizes the conditions under which identifiability holds, or can be restored (see, e.g., Doignon et al., Reference Doignon, Heller and Stefanutti2018; Heller, Reference Heller2017; Spoto et al., Reference Spoto, Stefanutti and Vidotto2012, Reference Spoto, Stefanutti and Vidotto2013; Stefanutti et al., Reference Stefanutti, Heller, Anselmi and Robusto2012, Reference Stefanutti, Spoto and Vidotto2018). The authors refer to a similar characterization in CD by stating that “Gu and Xu (Reference Gu and Xu2019) showed that the DINA model is only identifiable if each attribute is used by at least three items (Condition 1 (ii); p. 471). By definition, this condition cannot be fulfilled by the DINA model with single-attribute items, as its Q-matrix is a $J{\times }J$ identity matrix—hence, due to the equivalence of the two models, identifiability cannot hold for BLIM either” (p. 665). A few remarks are in order here.

First, there is no basis for making inferences on the identifiability of the BLIM given that a single-attribute DINA is not identifiable. It was shown above that the two models cannot be conceived as being equivalent in any formal sense, but the BLIM may be seen as a marginal model for a whole class of CD models.

Second, the cited conclusion of Gu and Xu (Reference Gu and Xu2019) only holds for the case where all theoretically conceivable attribute profiles are permissible (see, e.g., Gu & Xu, Reference Gu and Xu2020, p. 2089). It has been made clear above that this is not the situation considered in Chiu et al. (Reference Chiu, Köhn and Ma2023), where a hierarchical structure on the skills/attributes is imposed by partially ordering them as illustrated in Figure 2, and thus the argument is vacuous. Moreover, the presented rationale would lead to the conclusion that any BLIM is not identifiable irrespective of the underlying knowledge structure, which is obviously false.

Third, as already mentioned, there are theoretical results for hierarchies of attributes in the CD context. Gu and Xu (Reference Gu and Xu2020) investigate identifiability of a class of CD models (the two-parameter Q-restricted latent class models quoted above) including the DINA model for general attribute structures (i.e., arbitrary subsets of attribute profiles are permissible). They define and characterize the concept of partial identifiability, which corresponds to identifiability of the BLIM induced by a CBLIM (Heller, Reference Heller2022). Considering the CBLIM (which includes the DINA model as a special case), it follows that the CBLIM is (locally) identifiable if and only if the composition of the mappings in Equation (3) is injective (Heller et al., Reference Heller, Stefanutti, Anselmi and Robusto2015, Corollary 1). It was shown that this holds if and only if the induced BLIM is identifiable and the problem function p is injective. Since p as an isomorphism is injective by the construction employed by Chiu et al. (Reference Chiu, Köhn and Ma2023), the CBLIM is (locally) identifiable if and only if the induced BLIM is (locally) identifiable. Recasting this in CD terms, the question actually is whether the DINA model is partially identifiable, or not.

The KST literature provides results showing that the BLIM defined on the considered knowledge structure $\mathcal {K}$ of Example 2.1 is not identifiable, both locally and globally (Heller, Reference Heller2017; Spoto et al., Reference Spoto, Stefanutti and Vidotto2012). The following notions are central to characterizing this case of non-identifiability. A knowledge structure $\mathcal {K}$ is said to be backward-graded in an item $q \in P$ if for each state $K \in \mathcal {K}$ the subset $K \setminus \{q\}$ is also a state in $\mathcal {K}$ , and it is said to be forward-graded in an item P if for each state $K \in \mathcal {K}$ the subset $K \cup \{q\}$ is also a state in $\mathcal {K}$ (Spoto et al., Reference Spoto, Stefanutti and Vidotto2012). The items in which $\mathcal {K}$ is backward- or forward-graded are easily identified by inspecting the corresponding partial order on P illustrated in Figure 2. Heller (Reference Heller2017, Corollary 1) shows that the ordinal knowledge space $\mathcal {K}$ is backward-graded in the maximal elements (i.e., in items d and e), and it is forward-graded in the minimal elements (i.e., in items a and b) of the corresponding partial order. Thus, the BLIM in the empirical demonstration of Chiu et al. (Reference Chiu, Köhn and Ma2023) indeed is not identifiable, and thus different initial parameter values in the estimation procedure of both the BLIM and the DINA model may lead to different final estimates. In particular, theoretical results of Spoto et al. (Reference Spoto, Stefanutti and Vidotto2012) and Heller (Reference Heller2017) predict that the affected item parameters are those of the items in which the knowledge structure $\mathcal {K}$ is backward- or forward-graded. Comparing Table 1 with the respective table in Chiu et al. (Reference Chiu, Köhn and Ma2023, p. 659) confirms this prediction. Discrepancies occur for the careless error and slipping probabilities (columns $\beta $ and s) of the maximal items d and e (previously 0.0798 and 0.0886), and for the guessing probabilities (columns $\eta $ and g) of the minimal items a and b (previously 0.1031 and 0.0951). These discrepancies are due to the differences in the initial knowledge state probabilities induced on the BLIM by the different instantiations of the DINA model. While in Chiu et al. (Reference Chiu, Köhn and Ma2023) the initial distribution is uniform, it results from Equation (2) via the assumption of a uniform distribution on all possible competence states (i.e., all permissible attribute profiles) in the above considered alternative model. For example, for the empty knowledge state $\emptyset $ the initial probability equals $1/9$ in the first case, and $8 \cdot 1/2^5 = 1/4$ in the second case. The obvious discrepancies of the estimated knowledge state probabilities in Table 2 from those listed in Chiu et al. (Reference Chiu, Köhn and Ma2023, p. 659) are explained by the parameter trade-offs theoretically characterized by Spoto et al. (Reference Spoto, Stefanutti and Vidotto2012, Theorems 1 and 3) and Heller (Reference Heller2017, Propositions 1 and 3) for global and local non-identifiability, respectively.

Pointing to a wealth of results available in both the KST and the CD literature, the discussion in the present section lets the concluding remark of Chiu et al. (Reference Chiu, Köhn and Ma2023) that “which models are equivalent when attributes have a hierarchy, and which Q-matrices lead to identical results is currently uncharted territory” (p. 665) appear rather strange. In fact, their application of the DINA model assumes an attribute hierarchy as illustrated in Figure 2, and theoretical results for this and even more general cases are readily available (Gu & Xu, Reference Gu and Xu2020; Heller, Reference Heller2022; Heller et al., Reference Heller, Stefanutti, Anselmi and Robusto2015).

6 PoLIM and the polytomous DINA model

An extension of the BLIM to items with more than two response alternatives was proposed by Stefanutti, de Chiusole, et al. (Reference Stefanutti, Anselmi, de Chiusole and Spoto2020). The provided model, named the PoLIM (politomous local independence model), is derived from assumptions that are minimally sufficient for extending the BLIM to polytomous items. The PoLIM can be applied to a very large variety of response formats. For instance, the item responses can be totally ordered, partially ordered or not ordered at all. Different items may have different numbers of response alternatives. There could be a mixture of dichotomous and polytomous items. No skills are assumed and no Q-matrices need be specified, although the model can be extended to either dichotomous or polytomous skills.

In this connection, a polytomous version of the DINA model was proposed by Chiu et al. (Reference Chiu, Köhn and Ma2023) in a commentary to Stefanutti, de Chiusole, et al. (Reference Stefanutti, Anselmi, de Chiusole and Spoto2020). Moreover, there exist many varieties of CD models for accommodating polytomous items, that were developed before the PoLIM (see, e.g., (Chen & de la Torre, Reference Chen and de la Torre2018; Chen & Zhou, Reference Chen and Zhou2017; von Davier, Reference von Davier2008)).

Chiu et al. (Reference Chiu, Köhn and Ma2023) try to generalize their base argument that the BLIM is a DINA model with single-attribute items to the polytomous case (i.e., to the PoLIM). In constructing a counterexample to their argument, like in the dichotomous case we delineate the same polytomous structure obtained by Chiu et al. (Reference Chiu, Köhn and Ma2023) with a different Q-matrix and a different set of attribute profiles. The aim of the example is to show that there are at least two different Q-matrices (the one in the example below and the one by Chiu et al. (Reference Chiu, Köhn and Ma2023)) that delineate exactly the same polytomous structure (set of polytomous ideal response patterns) at the performance level. There is no “standard” way of defining a Q-matrix, when the item responses are polytomous. The approach followed here relies upon recent developments described in Stefanutti et al. (Reference Stefanutti, Spoto, Anselmi and de Chiusole2023), but differs to some extent from the one described in the Commentary.

Chiu et al. (Reference Chiu, Köhn and Ma2023) considered a set $P=\{q_1, q_2, q_3, q_4, q_5\}$ of five items and a set $V=\{v_0, v_1, v_2\}$ of linearly ordered response values. The first four items in the example were dichotomous, whereas the last one was trichotomous. Here, we consider the Q-matrix displayed in Table 3, which assigns a subset of $S=\{s_1, s_2, s_3, s_4, s_5, s_6\}$ to each of the admissible item-response pairs for P and V listed in the first column.

Table 3 Q-matrix delineating the same set of ideal response patterns as in the example by Chiu et al. Reference Chiu, Köhn and Ma2023

It is worth noticing that, in the example by Chiu et al. (Reference Chiu, Köhn and Ma2023), the number of skills was five whereas in this example it is six. Moreover, in the example by Chiu et al. (Reference Chiu, Köhn and Ma2023), the Q-matrix can take on any value in V, whereas in the present counterexample it is dichotomous. Nonetheless, it produces polytomous ideal response patterns.

The full power set on the six skills is considered as the set of permissible attribute profiles, containing a total of $2^6=64$ attribute profiles. This choice is clearly different from the one in the example by Chiu and colleagues, which contains 11 attribute profiles in the whole. In the sequel, the notation $Q_{q,v,s}$ refers to the value of the Q-matrix for item q, response v, and attribute s. Thus, for instance, $Q_{q_1,v_1,s_1}=1$ , $Q_{q_1,v_1,s_2}=0$ , $Q_{q_2,v_1,s_1}=0$ , etc. Define then the function $\tau $ such that, for any item P, any response value $v \in V$ , and any attribute $s \in S$ , it holds that $s \in \tau (q,v)$ if and only if $Q_{q,v,s} = 1$ . Then, we first need to determine what the “ideal response” to item P will be when the attribute profile is $\boldsymbol {\alpha }$ , given that the set V contains more than two alternative answers. Such an ideal response is here defined as

$$\begin{align*}p_q(C) = \max\{v \in V: \tau(q,v) \subseteq C\}, \end{align*}$$

where $\max $ is the usual “maximum” function, and $C \subseteq S$ is the set representation of the binary vector $\boldsymbol {\alpha }$ (i.e., $\boldsymbol {\alpha } = \iota (C)$ ). Thus, $p_q(C)$ provides P with the maximum response value among all those response values that could be provided by an individual whose attribute profile is $\boldsymbol {\alpha }$ . The set representation (knowledge state) of the ideal response pattern “delineated” by $\boldsymbol {\alpha }$ is then obtained as

$$\begin{align*}p(C) = \{(q,p_q(C)): q \in P\}. \end{align*}$$

One should be careful with these definitions because, in general, $p_q(C)$ need not exist. The existence is only guaranteed if there is a maximum in the set $\{v \in V: \tau (q,v) \subseteq C\}$ . In our running example, however, this always holds true because the three response values in V are linearly ordered, and the minimum value is such that $\tau (q,v_0) = \emptyset $ for all items P (more general cases are considered in Stefanutti et al. (Reference Stefanutti, Spoto, Anselmi and de Chiusole2023)). To show how all of this applies concretely, suppose that the attribute profile is $\boldsymbol {\alpha } = (1,1,1,0,0,1)$ . Its set representation is $C = \{s_1,s_2,s_3,s_6\}$ . Then we have $p_{q_1}(C) = p_{q_2}(C) = p_{q_3}(C) = 1$ , $p_{q_4}(C) = p_{q_5}(C) = 0$ . This yields $p(C) = \{(1,1),(2,1),(3,1),(4,0),(5,0)\}$ , whose corresponding vector representation is $(1,1,1,0,0)$ . In the whole, by applying this method to each of the $2^6=64$ attribute profiles considered in this example, the obtained polytomous structure turns out to be identical to the one in the example provided by Chiu et al., that is:

$$\begin{align*}\mathcal{K}=\{00000,01000,10000,11000,11010,11100,11110,11101,11111,11102,11112\}, \end{align*}$$

whose elements are shorthand vector representations of the polytomous states. Like in the dichotomous example, there is an alternative skill interpretation of the PoLIM. In such interpretation there are items requiring more than a single skill, showing that the PoLIM is not a polytomous DINA model where each item requires a single skill.

As stated above, depending on how the elements in V are ordered, the mapping $p_q$ may be undefined for some attribute profiles. Suppose in fact that V is unordered (like with categorical or nominal responses). Then, for sure, $p_q(C)$ will be undefined because the set $\{v \in V: \tau (q,v) \subseteq C\}$ has not a maximum element. This fact does not affect the PoLIM, which remains applicable also with unordered and, more generally, with partially ordered response categories. Unfortunately, the same conclusion cannot be drawn with respect to the polytomous DINA model presented by Chiu and colleagues in the Appendix of their Commentary.

In Section 4.2., Chiu et al. (Reference Chiu, Köhn and Ma2023) claim that “… here, the general case is concerned that also includes non-ordered response categories; hence, the indices $l\, = 0, 1, \ldots , L_j$ should be merely interpreted as category labels”. However, if such an interpretation is applied, then the polytomous DINA model may provide meaningless ideal response patterns. The reason is in the way ideal response patterns are obtained from the attribute profiles. The equation that formally relates the former to the latter can be found in the Appendix of the cited article and it is repeated here for convenience (with the same notation used in the commentary):

(4) $$ \begin{align} \xi_{ij} = \max_{l \in \{0,1,\ldots,L_j\}} \left\{ l \prod_{a=1}^{A_j} I[\alpha^*_{ia} \ge q^*_{ja}] \right\}, \end{align} $$

where $\xi _{ij}$ is the jth entry in the ideal response pattern of individual i, $L_j$ is the highest level of the polytomous ideal response, $A_j$ is the total number of attributes for item j, $\alpha ^*_{ia}$ and $q^*_{ja}$ are, respectively, the rearranged attribute profile for individual i, and attribute a, and the rearranged polytomous Q-matrix entry for item j and attribute a.

As an example, suppose that in a study on preference, each item in a questionnaire asks the respondent to choose one element out of a set of three. There is no a-priori order on the three response alternatives, because it may vary from one individual to another according to preference. Clearly, in such a situation, the numbers in the set $\{0,1,\ldots ,L_j\}$ that appears as the subscript of the maximum in Equation (4) can only have nominal value (e.g., although one is less than two, the response alternative associated with the number one and that associated with the number two cannot be ordered in a unique way). In other words, with unordered response alternatives, although a maximum always exists in the finite set $\{0,1,\ldots ,L_j\}$ , it does not exist among the unordered response categories that are represented by those numbers.

Suppose then that $V = \{v_1,v_2,v_3,v_4\}$ is a set of four response categories (notice that V is a non-numerical set), and denote by $f:V \to N = \{0,1,2,3\}$ the mapping that assigns a number to each of the categories in V. Given the unordered nature of the elements in V, the mapping f can be any bijection. The numbers only serve as labels. Therefore, if $t:N \to N$ is a bijection, then the function composition $t \circ f$ is still a bijection between V and N, meaning that f and $t \circ f$ are equally good in representing V. If, like in our example, the choice of the numbers representing the elements in V is restricted to the set N, then $4!=24$ distinct bijections are possible. It is questioned whether the response category represented by $\xi _{ij}$ is invariant under bijective transformations $t:N \to N$ . We show that, unfortunately, the answer is negative.

Table 4 A polytomous Q-matrix for a set of four items and three polytomous skills

Note: There are three response categories for each item, taken from the unordered set $V=\{v_1,v_2,v_3,v_4\}$ .

Suppose that the questionnaire contains four polytomous items $a,b,c,d$ , each of which presents the participant with three of the four options in the set V (e.g., one of the items could present the participant with the options $v_1$ , $v_2$ and $v_4$ ). The response of the participant consists of choosing one of the three options. Suppose furthermore that three polytomous attributes $s_1, s_2, s_3$ of individuals are associated with the responses to the four items. Each of the three attributes can take on a value in the set $\{0,1,2\}$ . A polytomous Q-matrix that associates attributes to item-response pairs is displayed in Table 4. So, for instance, the Q-matrix predicts that, for responding $v_1$ to item a, an individual’s attribute profile must score at least two in attribute $s_1$ . For responding $v_4$ to item b the individual’s profile must score at least one in both attributes $s_1$ and $s_3$ , and so on.

With 3 trichotomous attributes, the total number of attribute profiles turns out to be $3^3=27$ . Equation (4) can now be applied to each of the 27 attribute profiles for obtaining an ideal response pattern, by applying the Q-matrix specified in Table 4. This, however can only be done after having chosen one of the 24 alternative bijective correspondences between the options in V and the numbers in N. Assume that the mapping f such that

$$\begin{align*}f(v_1)=0, \ f(v_2)=1, \ f(v_3)=2, \ f(v_4)=3 \end{align*}$$

is applied. In this case, the attribute profile $\alpha = (2,1,1)$ delineates the ideal response pattern $\xi = (1,3,3,3)$ . The details of the derivation are only described for item a. We have $q_{a}=(2,0,0)$ , and hence $I(\alpha _s^* \ge q_a^{*(l)})$ equals 1 for $l \in L_a = \{0,1\}$ and it equals 0 for $l=2$ . By taking the maximum (between 0 and 1), one obtains $\xi _a = 1$ . For the remaining three items a similar procedure is applied. At this point we can map the ideal response pattern $\xi $ back to the original options in V through the inverse bijection $f^{-1}$ . We obtain $f^{-1}(\xi ) = (v_2,v_4,v_4,v_4)$ . The conclusion is: Ideally, an individual with attribute profile $(2,1,1)$ will choose $v_2$ in item one, and $v_4$ in all other items.

Table 5 List of all the ideal response patterns $\xi _i$ , in terms of the non-numerical values in set V, that are obtainable through the application of the mappings f (columns 1 to 4) and g (columns 5 to 8)

Note: The two mappings generate totally different sets of ideal response patterns.

Suppose now that the mapping g such that

$$\begin{align*}g(v_1)=3, \ g(v_2)=2, \ g(v_3)=1, \ g(v_4)=0 \end{align*}$$

is applied, rather than f. Of course between the two there exists a bijection. In this alternative case, by keeping every other aspect identical, the attribute profile $\alpha =(2,1,1)$ delineates the ideal response pattern $\xi '=(3,3,3,2) \ne \xi $ . This different ideal response pattern is mapped back to $g^{-1}(\xi ')=(v_1,v_1,v_1,v_2)$ . A totally different set of choices, compared to $f^{-1}(\xi )$ . However, the only difference between the two cases is in the choice of the bijective mapping from options in V to numbers in N. For the sake of completeness, Table 5 lists the complete sets of ideal response patterns that are obtained with bijections f (columns 1 to 4) and g (columns 5 to 8). As it can be seen, totally different ideal response patterns are obtained. The two sets do not even match in size. Concluding, contrary to what Chiu et al. (Reference Chiu, Köhn and Ma2023) claim, the PoLIM is not a paraphrase of a DINA model in cognitive diagnosis for polytomous items. The former cannot be obtained from the latter, in general. This fact becomes particularly evident when the response categories do not form a totally ordered set. To conclude, like in the BLIM case, the PoLIM is better understood as a marginal model for an entire class of CD models.