HOW reliable are change‐corrected measures of agreement?

I. Guggenmoos-Holzmann

85 citations0/0 files storedStatistics in Medicine

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Reliability and Agreement in Measurement Psychometric Methodologies and Testing Meta-analysis and systematic reviews

Abstract

Chance-corrected measures of agreement are prone to exhibit paradoxical and counter-intuitive results when used as measures of reliability. It is demonstrated that these problems arise with Cohen's kappa as well as with Aickin's alpha. They are the consequence of an analogy to Simpson's paradox in mixed populations. It is further shown that chance-corrected measures of agreement may yield misleading values for binary ratings. It is concluded that improvements in the design and the analysis of reliability studies are a prerequisite for valid and pertinent results.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 85 citations exist upstream.

References

1960A Coefficient of Agreement for Nominal Scales 1990High agreement but low Kappa: I. the problems of two paradoxes 1955Reliability of Content Analysis: The Case of Nominal Scale Coding 1979Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm 1981Coefficient Kappa: Some Uses, Misuses, and Alternatives 1981Clinical biostatistics: LIV. The biostatistics of concordance 1987MISINTERPRETATION AND MISUSE OF THE KAPPA STATISTIC 1975The Reliability of Clinical Methods, Data and Judgments

Cited by

1999Validity of a Self-reported History of Doctor-diagnosed Angina 2014Observer agreement paradoxes in 2x2 tables: comparison of agreement measures 2000Statistical description of interrater variability in ordinal ratings 2007Assessing observer agreement when describing and classifying functioning with the International Classification of Functioning, Disability and Health 2012Factors affecting intercoder reliability: a Monte Carlo experiment

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.