To create a standardized scoring procedure to evaluate open-ended responses as part of a novel questionnaire (Measure of Metamemory; MoM) designed to assess declarative metamemory in youth. Metamemory is an aspect of metacognition that is one’s knowledge of the factors related to storage and retrieval of information (Flavell 1971; Kreutzer et al., 1975), and includes both declarative metamemory (i.e., one’s knowledge about factors influencing memory) and procedural metamemory (i.e., one’s understanding of their own memory performance).

Fourteen short vignettes related to memory were administered to 100 participants (age 6-12) with questions such as, “Two children hear a story they must remember. The first person is 5 years old. The second person is 12 years old. Who is most likely to remember it best?” After answering, they were then prompted to explain their answer (i.e., “Why?”) and their responses were recorded verbatim. To develop standardized and objective criteria for each of the 14 open-ended responses, responses from a subsample of 20 youth were collectively examined by the study team and a scoring structure similar to open-ended items on common intelligence tests (e.g., WISC-V/WAIS-IV) was created. Two points (full credit) were awarded for complete and thorough understanding of memory processes related to the question; 1 point was given for partially accurate or incomplete understanding of the related memory process; and 0 points for an inability to correctly express an understanding of relevant memory concepts. This scoring guide was then applied independently by each of the six raters to an additional 25 participants (ages 6-12 mean age (SD)). To assess the interrater reliability of this 3-point ordinal scoring system, we examined both Fleiss’ kappa and 2-way random-effects, single-rater, absolute agreement Intra-Class Correlations (ICC).

Across the six independent raters, reliability coefficients for each of the 14 items ranged from (Fleiss') k =. 277 to .792 (ICC ranged from .481 to .880). Of these 14 items, the kappa value was classified (using interpretation rules for Cohen’s kappa) as “substantial” for 10 items, “moderate” for 3 items, and “fair” for 1 item. Based on these lower inter-rater reliabilities, two items were subsequently removed from the measure to create the 12-item open-ended measure of metamemory, the MoM-12, with reliable scoring for youth as young as 6 years old.

A consensus process established a quantifiable scoring procedure to assess open-ended responses related to youth’s knowledge of memory (e.g., metamemory). Reliability metrics identified acceptable interrater reliabilities in 12 of the 14 original items. Further examination of psychometric properties, including internal consistency and lines of evidence for validity is needed. The successful crafting of a scoring procedure is a first step towards developing a psychometrically sound measure (the MoM) to evaluate and study metamemory concepts objectively and reliably in youth.