Can One Use Cohen’s Kappa to Examine Disagreement?

Abstract

Abstract. This research discusses the use of Cohen’s κ (kappa), Brennan and Prediger’s κn, and the coefficient of raw agreement for the examination of disagreement. Three scenarios are considered. The first involves all disagreement cells in a rater × rater cross-tabulation. The second involves one of the triangles of disagreement cells. The third involves the cells that indicate disagreement by one (ordinal) scale unit. For each of these three scenarios, coefficients of disagreement in the form of κ equivalents are derived. The behavior of the coefficients of disagreement in the three situations is studied. The first and the third case pose no particular problem. The κ equivalents and the other coefficients can be interpreted as usual. In the second case, problems arise such that the range of disagreement κs is restricted because the tables are incomplete. Thus, the standard log-frequency model of rater independence is no longer applicable. When the more general models of quasi-independence are used, negative degrees of freedom can result for smaller tables. Simulation results illustrate the characteristics of the coefficients of disagreement for each of the three scenarios. Empirical data examples are given.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 17 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.