On Assessing Interrater Agreement for Multiple Attribute Responses

Abstract

New methods are developed for assessing the extent of interrater agreement when each unit to be rated is characterized by a (possibly empty) subset of a specified set of distinct nominal attributes. For such multiple attribute response data, a two-rater concordance statistic is derived, and associated statistical inference-making procedures are provided. This concordance statistic is corrected for chance agreement by using an underlying hypergeometric model. Numerical examples are given to illustrate the proposed methodology, and comparisons to other agreement statistics (e.g., kappa) are made.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 49 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.