Measuring interrater reliability among multiple raters: An example of methods for nominal data

Karen L. Posner · Paul D. Sampson · Robert A. Caplan · Richard John Ward · Frederick W. Cheney

229 citations0/0 files storedStatistics in Medicine

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Reliability and Agreement in Measurement Psychometric Methodologies and Testing Statistical Methods in Epidemiology

Abstract

This paper reviews and critiques various approaches to the measurement of reliability among multiple raters in the case of nominal data. We consider measurement of the overall reliability of a group of raters (using kappa-like statistics) as well as the reliability of individual raters with respect to a group. We introduce modifications of previously published estimators appropriate for measurement of reliability in the case of stratified sampling frames and we interpret these measures in view of standard errors computed using the jackknife. Analyses of a set of 48 anaesthesia case histories in which 42 anaesthesiologists independently rated the appropriateness of care on a nominal scale serve as an example.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 229 citations exist upstream.

References

1977The Measurement of Observer Agreement for Categorical Data 1960A Coefficient of Agreement for Nominal Scales 1971Measuring nominal scale agreement among many raters.1977An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers 1973The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability 1971Measures of response agreement for qualitative data: Some generalizations and alternatives.1980Integration and generalization of kappas for multiple raters.1977A One-Way Components of Variance Model for Categorical Data

Cited by

1992The Reliability of Peer Assessments of Quality of Care 2015The Application of Interrater Reliability as a Solidification Instrument in a Phenomenological Study 2000Statistical methods in epidemiology. v. Towards an understanding of the kappa coefficient 1996Intra-observer variation in the histopathological assessment of chronic viral hepatitis 2013Interrater reliability assessment using the Test of Gross Motor Development-2

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.