Assessing Rater Performance without a "Gold Standard" Using Consensus Theory

Susan C. Weller · N. Clay Mann

110 citations0/0 files storedMedical Decision Making

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Reliability and Agreement in Measurement Radiology practices and education Clinical Reasoning and Diagnostic Skills

Abstract

This study illustrates the use of consensus theory to assess the diagnostic performances of raters and to estimate case diagnoses in the absence of a criterion or "gold" standard. A description is provided of how consensus theory "pools" information provided by raters, estimating rater competencies and differentially weighting their responses. Although the model assumes that raters respond without bias (i.e., sensitivity = specificity), a Monte Carlo simulation with 1,200 data sets shows that model estimates appear to be robust even with bias. The model is illustrated on a set of elbow radiographs, and consensus-model estimates are compared with those obtained from follow-up data. Results indicate that with high rater competencies, the model retrieves accurate estimates of competency and case diagnoses even when raters' responses are biased.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 110 citations exist upstream.

References

1987Latent Class Analysis 1986Culture as Consensus: A Theory of Culture and Informant Accuracy 1974Exploratory Latent Structure Analysis Using Both Identifiable and Unidentifiable Models 2005Clinical decision analysis 1980Clinical Decision Analysis 1988Systematic Data Collection 1910CORRELATION CALCULATED FROM FAULTY DATA 1910SOME EXPERIMENTAL RESULTS IN THE CORRELATION OF MENTAL ABILITIES 1

Cited by

2010The Cultural Category of Cooperation: A Cultural Consensus Model Analysis for China and the United States 2005Improving Decision Making in Forensic Child Sexual Abuse Evaluations.2002Study Design for Concurrent Development, Assessment, and Implementation of New Diagnostic Imaging Technology 2019How social networks facilitate collective responses to organizational paradoxes 2013Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.