Rater bias patterns in an EFL writing assessment

Edward Schaefer

162 citations0/0 files storedLanguage Testing

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Psychometric Methodologies and Testing EFL/ESL Teaching and Learning Student Assessment and Feedback

Abstract

The present study employed multi-faceted Rasch measurement (MFRM) to explore the rater bias patterns of native English-speaker (NES) raters when they rate EFL essays. Forty NES raters rated 40 essays written by female Japanese university students on a single topic adapted from the TOEFL Test of Written English (TWE). The essays were assessed using a six-category rating scale (Content, Organization, Style and Quality of Expression, Language Use, Mechanics, and Fluency). MFRM revealed several recurring bias patterns among rater subgroups. In rater—category bias interactions, if Content and/or Organization were rated severely, then Language Use and/or Mechanics were rated leniently, and vice versa. In rater—writer bias interactions, there tended to be more severe or lenient bias towards higher ability writers than lower ability writers. Some raters also rated higher ability writers more severely and lower ability writers more leniently than expected. This study has implications for issues of rater training in L2 writing assessment.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 162 citations exist upstream.

References

1998Measuring Second Language Performance 1981Testing Esl Composition: A Practical Approach 2003Detecting and measuring rater effects using many-facet Rasch measurement: Part II.1998Using FACETS to model rater training effects 1995Rater characteristics and rater bias: implications for training 2002Assessment criteria in a large-scale writing test: what do they really mean to the raters?1994Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch Model 2012Second language writing: assessment issues

Cited by

2017Validation of rating processes within an argument-based framework 2012Operational Rater Types in Writing Assessment: Linking Rater Cognition to Rater Behavior 2015Building a better rubric: Mixed methods rubric revision 2018Recent Issues in the Use of Signed Language Assessments for Diagnosis of Language Disorders in Signing Deaf and Hard of Hearing Children 2015Investigating rater severity/leniency in interpreter performance testing

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.