An Evaluation of Item Response Theory Classification Accuracy and Consistency Indices

Abstract

This article introduces two new classification consistency indices that can be used when item response theory (IRT) models have been applied. The new indices are shown to be related to Rudner’s classification accuracy index and Guo’s classification accuracy index. The Rudner- and Guo-based classification accuracy and consistency indices are evaluated and compared with estimates from the more commonly applied IRT-recursive procedure using a simulation study and data from two large-scale assessments. Results from the simulation study and practical examples suggested that the Guo- and Rudner-based indices tended to produce estimates that were closer to the simulated values and exceeded those from the IRT-recursive-based procedure. However, results did suggest that the Rudner- and Guo-based indices can have some undesirable features that are important to keep in mind when applying them in practice. The values of the classification accuracy and consistency indices appeared to be affected by a number of factors including the distribution of examinees, test length, the placement of the cut-scores, and the proficiency estimators applied to estimate examinee ability. Suggestions are made that an important part of investigations evaluating classification accuracy and consistency indices should be the creation of figures that show the value of the classification accuracy and classification consistency for individual examinees across the range of possible scores as these figures can help provide indications into subtle and important differences between indices.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 27 citations exist upstream.