On Rater Agreement and Rater Training

Abstract

This paper first analyzed two studies on rater factors and rating criteria to raise the problem of rater agreement. After that the author reveals the causes of discrepencies in rating administration by discussing rater variability and rater bias. The author argues that rater bias can not be eliminated completely, we can only reduce the error to a cetain degree by training raters. The study on rater factors can help us better understand rater variability and rater bias, train raters more effectively and find out ways to modify the scores given by raters. The author suggests that rater files which contain rater information including each rater bias tendency should be established and kept so that information can be retrieved about the selection of raters, the interpretation as well as the modification of the scores given by raters. Raters need to receive pre-service training, on-service training and pilot -on-task training.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 16 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.