A note on interval estimation of kappa in a series of 2×2 tables

Abstract

When there are confounders in reliability studies, failing to stratify data to account for these confounding effects may produce a misleading estimate of the interrater agreement. In this paper, we focus discussion on interval estimation of kappa for measuring agreement between two raters with stratified data. Using Monte Carlo simulation, we compare four asymptotic interval estimators of kappa for stratified data: estimator (1), a weighted average of the stratum-specific kappa estimates with weights equal to the inverse of the estimated asymptotic variances of these estimates; the two estimators, (2) and (3), with use of the logarithmic and the square root transformations, respectively, and a principle similar as used in estimator (1); estimator (4), a weighted average of the stratum-specific kappa estimates with weights equal to stratum sizes. We find that while the coverage probability of the first three interval estimators (1)-(3) can often be less than the desired confidence level, the fourth interval estimator (4) consistently performs well in all the situations considered here. We further find that when the underlying kappa is moderate (0.30</=kappa</=0.50), we can substantially improve the performance of the first three estimators by using a point estimator recently proposed elsewhere for kappa in estimation of weights. Because interval estimator (4) outperforms the other three estimators in a variety of situations, we recommend this estimator for general use.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 8 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.