Applying many-facet Rasch modeling in the assessment of creativity.

Abstract

Creativity assessment with open-ended production tasks relies heavily on scoring the quality of a subject's ideas.This creates a faceted measurement structure involving persons, tasks (and ideas within tasks), and raters.Most studies, however, don't model possible systematic differences among raters.The present study examines the impact of individual rater differences in the reliability and validity of creativity assessments.It applies the Many-Facet-Rasch-Model (MFRM) to model and correct for these differences.We reanalyzed data from two studies (Ns=132 and 298) where subjects produced metaphors, alternate uses for common objects, and creative instances.Each idea was scored by two to four raters.We simulated several conditions of reduced load on raters where they scored subsets of responses.We then compared the reliability and validity of IRT estimated scores (original versus IRT adjusted) on various conditions of missing data.Results show that (a) raters vary substantially on the lenient-severity dimension, so rater differences should be modeled; (b) when different combinations of raters assess different subsets of ideas, systematic rater differences confound subjects' scores, increasing measurement error and lowering validity; and (c) MFRM adjustments effectively correct for rater effects, thus increasing validity.We conclude that MFRM is a powerful means to model rater differences and reduce rater load in creativity research.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 45 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.