Identifying Rater Effects Using Latent Trait Models

Abstract

This study describes how latent trait models, specifically the multi-faceted Rasch model, may be applied to identifying rater effects in performance ratings. Three general types of rater effects are presented (Accuracy/Inaccuracy, Severity/Leniency, and Centrality/ Extremism), and indices that are useful indicators of those rater effects are identified. Each of these indices is examined in the context of ratings assigned by highly trained raters to essays written for an Advanced Placement examination, and individual raters suspected of exhibiting these various rater effects are identified. Each of these rater effects exists in the data in both non-ignorable rates and magnitudes.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 82 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.