Investigating the effectiveness of individualized feedback to rating behavior — a longitudinal study

Ute Knoch

70 citations0/0 files storedLanguage Testing

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Psychometric Methodologies and Testing EFL/ESL Teaching and Learning Student Assessment and Feedback

Abstract

The effectiveness of individualized feedback on rater behavior has been investigated in several previous studies (e.g. Elder, Knoch, Barkhuizen, & von Randow, 2005; Lunt, Morton, & Wigglesworth, 1994; O’Sullivan & Rignall, 2007; Wigglesworth, 1993). The findings of these studies are somewhat inconclusive. However, all of these studies investigated the value of such feedback on a one-off basis, and it is thus not clear how raters use this type of feedback over several administrations of a test. Furthermore, previous research has focused only on one language skill at a time, and we do not know if raters can incorporate the feedback in a similar way when rating speaking and writing. This study tracks the rating behavior of 19 raters assessing a large-scale English for specific purposes (ESP) assessment for the health professions over eight administrations. After each administration, raters received detailed performance profiles of their rating behavior which were generated using many-faceted Rasch measurement. Raters also completed a questionnaire and a subset of raters were interviewed to ascertain their views on the effectiveness of this feedback. The findings showed that the raters were rating no better when receiving the feedback than when they had not received feedback and neither speaking nor writing raters were able to incorporate the feedback more successfully. Although the raters were generally positive about the success of the feedback, there was no relationship between their perceptions of the feedback and the success of the feedback.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 70 citations exist upstream.

References

1990How to Design and Evaluate Research in Education 2000Item response theory for psychologists 1998Measuring Second Language Performance 2002Assessing Writing 1998Using FACETS to model rater training effects 2002Assessment criteria in a large-scale writing test: what do they really mean to the raters?1990Expertise in evaluating second language compositions 2008Rater types in writing performance assessments: A classification approach to rater variability

Cited by

2015Building a better rubric: Mixed methods rubric revision 2015A comparison of newly-trained and experienced raters on a standardized writing assessment 2015Investigating rater severity/leniency in interpreter performance testing 2013How invariant and accurate are domain ratings in writing assessment?2014Assessing learners’ writing skills in a SLA study: Validating the rating process across tasks, scales and languages 2017From Diagnosis Toward Academic Support

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.