Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity

Abstract

Researchershavecriticizedchance-correctedagreementstatistics,particularly theKappastatistic,asbeingverysensitivetoraters'classiflcationprobabilities(marginal probabilities) and to trait prevalence in the subject population. Consequently, several authorshavesuggestedthatmarginalprobabilitiesbetestedforhomogeneityandthatany comparison between reliability studies be preceded by an assessment of trait prevalence amongsubjects.Theobjectiveofthispaperisthreefold:(i)todemonstratethatmarginal homogeneitytestingdoesnotpreventtheunpredictableresultsoftenobtainedwithsome ofthemostpopularagreementstatistics,(ii)topresentasimpleandreliableinter-rater agreement statistic, and (iii) to gain further insight into the dependency of agreement statistics upon trait prevalence.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 199 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.