Demystifying a Q-Matrix for Making Diagnostic Inferences About L2 Reading Skills

Abstract

In researching the potential of cognitive diagnostic assessment, researchers concur that the quality of diagnostic inferences is subject to the extent to which the construct representations based on cognitive skills are theoretically compelling, empirically sound, and relevant to test use. In this paper, I argue that the construction of a Q matrix requires multiple sources of evidence supporting the representation of the construct with well-defined cognitive skills and their explicit links to item characteristics. I illustrate the process of constructing and refining a Q matrix using the results from a large-scale study that examined the validity of applying cognitive diagnosis assessment approaches to LanguEdge reading comprehension tests. I focus on the characteristics of reading skills identified from verbal protocols along with the analyses of text and items and discuss issues related to identifying reading skills and determining the granularity for diagnostic inferences. By demonstrating the process of refining a Q matrix, I discuss fundamental issues arising from the application of cognitive diagnostic assessment through retrofitting. I believe that the paper provides useful guidelines for those who are interested in designing a systematic cognitive diagnostic assessment for L2 reading comprehension abilities.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 75 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.