Interrater Agreement Reconsidered: An Alternative to the rwg Indices

Abstract

For continuous constructs, the most frequently used index of interrater agreement (r wg(1))can be problematic. Typically, rwg(1) is estimated with the assumption that a uniform distribution represents no agreement. The authors review the limitations of this uniform nullr wg(1) index and discuss alternative methods for measuring interrater agreement. A new interrater agreement statistic,a wg(1), is proposed. The authors derive thea wg(1) statistic and demonstrate thatawg(1) is an analogue to Cohen’s kappa, an interrater agreement index for nominal data. A comparison is made between agreement estimates based on the uniformr wg(1) and a wg(1), and issues such as minimum sample size and practical significance levels are discussed. The authors close with recommendations regarding the use ofr wg(1) /rwg(J) when a uniform null is assumed,r wg(1) /rwg(J) indices that do not assume a uniform null,awg(1) / a wg(J) indices, and generalizability estimates of interrater agreement.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 299 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.