Agreement and Information in the Reliability of Coding

Klaus Krippendorff

453 citations0/0 files storedCommunication Methods and Measures

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Reliability and Agreement in Measurement Psychometric Methodologies and Testing Computational and Text Analysis Methods

Abstract

Coefficients that assess the reliability of data-making processes, that is, coding text, transcribing interviews, or categorizing observations into analyzable terms, are mostly conceptualized in terms of the agreement a set of coders, observers, judges, or measuring instruments exhibit. When variation is low, reliability coefficients reveal their dependency on an often neglected phenomenon, the amount of information that reliability data provide about the reliability of the coding process or the data it generates. This paper explores the concept of reliability, simple agreement, three conceptions of chance to correct that agreement, and sources of information deficiency, and it develops two measures of information about reliability, akin to the power of a statistical test, intended as a companion to traditional reliability coefficients, especially Krippendorff's (2004 Krippendorff, K. 2004. Content analysis: An introduction to its methodology, 2nd, Thousand Oaks, CA: Sage. [Google Scholar], pp. 221–250; Hayes & Krippendorff, 2007 Hayes, A. F. and Krippendorff, K. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1: 77–89. [Google Scholar]) alpha.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 453 citations exist upstream.

References

1951Coefficient Alpha and the Internal Structure of Tests 1960A Coefficient of Agreement for Nominal Scales 1984Content Analysis: An Introduction to its Methodology.1989A Concordance Correlation Coefficient to Evaluate Reproducibility 2007Answering the Call for a Standard Reliability Measure for Coding Data 1955Reliability of Content Analysis: The Case of Nominal Scale Coding 1989Reliability of Nominal Data Based on Qualitative Judgments 1981Coefficient Kappa: Some Uses, Misuses, and Alternatives

Cited by

2014The Distress Analysis Interview Corpus of human and computer interviews 2014Measuring and Reporting Intercoder Reliability in Plan Quality Evaluation Research 2017A Pluralistic Perspective to Overcome Major Blind Spots in Research on Interorganizational Relationships 2019Transparency, Interactivity, Diversity, and Information Provenance in Everyday Data Journalism 2019Understanding emotional customer experience and co-creation behaviours in luxury hotels

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.