Assessing rater agreement using marginal association models

Susan M. Perkins · Mark P. Becker

23 citations0/0 files storedStatistics in Medicine

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Reliability and Agreement in Measurement Probability and Statistical Research Psychometric Methodologies and Testing

Abstract

New models that are useful in the assessment of rater agreement, particularly when the rating scale is ordered or partially ordered, are presented. The models are parameterized to address two important aspects of rater agreement: (i) agreement in terms of the overall frequency in which raters assign categories; and (ii) the extent to which raters agree on the category assigned to individual subjects or items. We present methodology for the simultaneous modelling of univariate marginal responses and bivariate marginal associations in the K-way contingency table representing the joint distribution of K rater responses. The univariate marginal responses provide information for evaluating agreement in terms of the overall frequency of responses, and the bivariate marginal associations provide information on category-wise agreement among pairs of raters. In addition, estimated scores within a generalized log non-linear model for bivariate associations facilitate the assessment of category distinguishability.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 23 citations exist upstream.

References

1977The Measurement of Observer Agreement for Categorical Data 1960A Coefficient of Agreement for Nominal Scales 1968Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.1977An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers 1973The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability 1990High agreement but low Kappa: I. the problems of two paradoxes 1988A Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple Raters 1985Modeling Agreement Among Raters

Cited by

2005A Fortran Program for Computing the Exact Variance of Weighted Kappa 2006Log‐linear non‐uniform association models for agreement between two ratings on an ordinal scale 2012Hierarchical modeling of agreement 2009Agreement between raters and groups of raters 2005Exact and Resampling Probability Values for Weighted Kappa

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.