Statistical inference of Gwet’s AC 1 coefficient for multiple raters and binary outcomes

Abstract

Cohen’s kappa and intraclass kappa are widely used for assessing the degree of agreement between two raters with binary outcomes. However, many authors have pointed out its paradoxical behavior, that comes from the dependence on the prevalence of a trait under study. To overcome the limitation, Gwet (2008 Gwet, K. L. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61 (1):29–48. doi:10.1348/000711006X126600.[Crossref], [PubMed], [Web of Science ®] , [Google Scholar]) proposed an alternative and more stable agreement coefficient referred to as the AC1. In this paper, we discuss a likelihood-based inference of the AC1 in the case of multiple raters and binary outcomes. Construction of confidence intervals is mainly discussed. In addition, hypothesis testing, sample size estimation, and the method of assessing the effect of subject covariates on agreement are also presented. The performance of the estimator of AC1 and its confidence intervals are investigated in a simulation study, and an example is presented.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 24 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.