A validity-based approach to quality control and assurance of automated scoring

Abstract

Automated scoring of constructed responses is already operational in several testing programmes. However, as the methodology matures and the demand for the utilisation of constructed responses increases, the volume of automated scoring is likely to increase at a fast pace. Quality assurance and control of the scoring process will likely be more prominent as a result. A validity-based approach to ensure the quality of scores based on automated means is proposed. First, an argument is made for aligning quality assurance/control with validity, rather than viewing them as separate processes. Second, to pre-emptively avoid quality and design defects that would erode the validity of scores, it is argued that quality must be ‘designed in’ through the assessment design process by attending to the interdependencies among the different components of an assessment, including scoring. Third, key elements of the design of scoring engines, evidence extraction and evidence synthesis, are discussed as a further avenue for maintaining score quality.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 42 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.