Assessing the merits and faults of holistic and disaggregated judgments

Abstract

Abstract Three studies explored both the advantages of and subjects' preferences for a disaggregated judgment procedure and a holistic one. The task in our first two studies consisted of evaluating colleges; the third study asked participants to evaluate job applicants. Holistic ratings consisted of providing an overall evaluation while considering all of the characteristics of the evaluation objects; disaggregated ratings consisted of evaluating each cue independently. Participants also made paired comparisons of the evaluation objects. We constructed preference orders for the disaggregated method by aggregating these ratings (unweighted or weighted characteristics). To compare the holistic, disaggregated, and weighted‐disaggregated method we regressed the four cues on the participant's holistic rating, on the linearly aggregated disaggregated ratings, and on the average weighted disaggregated rating, using the participant's “importance points” for each cue as weights. Both types of combined disaggregated ratings related more closely to the cues in terms of proportion of variance accounted for in Experiments 1 and 2. In addition, the disaggregated ratings were more closely related to the paired‐comparison orderings, but Experiment 2 showed that this was true for a small set (10) but not a large set (60) of evaluation objects. Experiment 3 tested the “gamesmanship” hypothesis: People prefer holistic ratings because it is easier to incorporate illegitimate but appealing criteria into one's judgment. The results suggested that the disaggregated procedure generally produced sharper distinctions between the most relevant and least relevant cues. Participants in all three of these studies preferred the holistic ratings despite their statistical inferiority. Copyright © 2009 John Wiley & Sons, Ltd.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 26 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.