The quality of expert probability judgement: issues and analysis

Abstract

Abstract: Many decision‐aiding technologies require valid probability judgements to be elicited from domain experts. But how valid are experts' probability judgements? We describe two approaches to the assessment of quality of probability judgement—calibration and coherence—and review the research findings following from these two approaches. In many cases, expert probability judgement has been found to lack validity and this sub‐optimality has largely been attributed to computational errors on the part of the expert. The preferred solution to poor validity in probability judgement has therefore been to reduce the amount of computation performed by the expert. Complex probabilities can be calculated mechanically from simple probability judgements elicited from the expert. We present evidence which suggests that this recomposition technique doesn't guarantee valid probabilities. Our explanation for this finding is that there are various problems concerned with eliciting even the simple probabilities which are necessary for subsequent recomposition. We conclude by proposing some solutions to these elicitation problems which should help ensure that probability judgements of increased validity are available to those attempting to capture subjective assessments for input into decision support systems.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 6 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.