Comparing holistic and analytic scoring methods: issues of validity and reliability

Abstract

This paper explores issues of rating quality when assessing writing in a level-specific approach, i.e. tasks targeting specific proficiency levels. We investigate whether holistic scores could mask deviances in how underlying descriptors are interpreted and applied by the raters, as such deviances could compromise rating validity. We conducted a study with six raters, comparing a holistic approach with a combined approach whereby an analytic score for each descriptor was collected complementary to the holistic judgements. The results confirmed the initial hypothesis that holistic scores may mask deviances in how descriptors are applied. To monitor rating quality and enhance rating validity, we therefore recommend a complementary approach combining holistic scores with analytic, descriptor-focused scores. We illustrate the applicability of this approach by a successful implementation during rater training. Our findings contribute towards enhancing rater consistency and improving rating validity.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 63 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.