Influence of Type of Judge, Normative Information, and Discussion on Standards Recommended for the National Teacher Examinations

John Christian Busch · Richard M. Jaeger

66 citations0/0 files storedJournal of Educational Measurement

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Psychometric Methodologies and Testing Educational Assessment and Pedagogy Teacher Education and Assessments

Abstract

There are few empirical investigations of the consequences of using widely recommended data collection procedures in conjunction with a specific standardsetting method such as the Angoff (1971) procedure. Such recommendations include the use of several types of judges, the provision of normative information on examinees' test performance, and the opportunity to discuss and reconsider initial recommendations in an iterative standard‐setting procedure. This study of 236 expert judges investigated the effects of using these recommended procedures on (a) average recommended test standards, (b) the variability of recommended test standards, and (c) the reliability of recommended standards for seven subtests of the National Teacher Examinations Communication Skills and General Knowledge Tests. Small, but sometimes statistically significant, changes in mean recommended test standards were observed when judges were allowed to reconsider their initial recommendations following review of normative information and discussion. Means for public school judges changed more than did those for college or university judges. In addition, there was a significant reduction in the within‐group variability of standards recommended for several subtests. Methods for estimating the reliability of recommended test standards proposed by Kane and Wilson (1984) were applied, and their hypothesis of positive covariation between empirical item difficulties and mean recommended standards was confirmed. The data collection procedures examined in this study resulted in substantial increases in the reliability of recommended test standards.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 66 citations exist upstream.

References

1952Educational Measurement 1951Estimation of the Reliability of Ratings 1986A Consumer's Guide to Setting Performance Standards on Criterion-Referenced Tests 1989Certification of student competence.1980Standard Setting Issues and Methods 1980A Comparison of the Nedelsky and Angoff Cutting Score Procedures Using Generalizability Theory 1982An Iterative Structured Judgment Process for Establishing Standards on Competency Tests: Theory and Application 1978ON THE USE OF CUT‐OFF SCORES WITH CRITERION‐REFERENCED TESTS IN INSTRUCTIONAL SETTINGS

Cited by

2009Judges' Use of Examinee Performance Data in an Angoff Standard‐Setting Exercise for a Medical Licensing Examination: An Experimental Study 1999How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory Study 2016Insights into the Angoff method: results from a simulation study 2002Combining Tests and Setting Standards 2008An Empirical Examination of the Impact of Group Discussion and Examinee Performance Information on Judgments Made in the Angoff Standard-Setting Procedure

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.