Inference Procedures for Assessing Interobserver Agreement among Multiple Raters

Mekibib Altaye · Allan Dormer · Neil S. Klar

36 citations0/0 files storedBiometrics

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Reliability and Agreement in Measurement Probability and Statistical Research Statistical Methods in Epidemiology

Abstract

We propose a new procedure for constructing inferences about a measure of interobserver agreement in studies involving a binary outcome and multiple raters. The proposed procedure, based on a chi-square goodness-of-fit test as applied to the correlated binomial model (Bahadur, 1961, in Studies in Item Analysis and Prediction, 158-176), is an extension of the goodness-of-fit procedure developed by Donner and Eliasziw (1992, Statistics in Medicine 11, 1511-1519) for the case of two raters. The new procedure is shown to provide confidence-interval coverage levels that are close to nominal over a wide range of parameter combinations. The procedure also provides a sample-size formula that may be used to determine the required number of subjects and raters for such studies.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 36 citations exist upstream.

References

1977The Measurement of Observer Agreement for Categorical Data 1959A REPRESENTATION OF THE JOINT DISTRIBUTION OF RESPONSES TO N DICHOTOMOUS ITEMS 19892 x 2 Kappa Coefficients: Measures of Agreement or Association 1992A goodness‐of‐fit approach to inference procedures for the kappa statistic: Confidence interval construction, significance‐testing and sample size estimation 1985Modeling Agreement Among Raters 1988Analysing Intraclass Correlation for Dichotomous Variables 1993Interval Estimation under Two Study Designs for Kappa with Binary Classifications

Cited by

2008Assessing Agreement between Multiple Raters with Missing Rating Information, Applied to Breast Cancer Tumour Grading 2012Bias-corrected estimator for intraclass correlation coefficient in the balanced one-way random effects model 2010Sample Size Requirements for Training to a κ Agreement Criterion on Clinical Dementia Ratings 2020Statistical inference of Gwet’s AC 1 coefficient for multiple raters and binary outcomes 2005Assessing the reliability of ordered categorical scales using kappa-type statistics

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.