Still a fallible tool? Revisiting effects of automated writing evaluation from activity theory perspective

Abstract

Despite the popularity of automated writing evaluation (AWE) that has provoked an increased scholarly interest, synthesized research to comprehensively understand its pedagogical effects is still in paucity. To fill the gap, this study aims to meta‐analyse the overall effect of AWE on learners' writing skill development and whether the effect differs from how it has been used. From activity theory (AT) perspective, this study reported results based on a meta‐analysis of 33 valid effect sizes involving 2995 participants from 25 experimental and quasi‐experimental studies published during 2000–22. The results of the study showed that the overall effect was g = 0.429, 95% CI [0.266, 0.592], suggesting that AWE for writing skill development is more effective than traditional methods, such as traditional classroom interaction, teacher‐led interaction and peer interaction. Regarding AT‐related moderators for the overall effect, different moderator effects of educational levels, proficiency levels, sample size types, target language types, intervention durations, text genre types, human–computer interaction types, feedback types and program types were reported. Informed by the results obtained, this study also provided some insights into the pedagogical implications for practice. Practitioner notes What is already known about this topic The facilitative effects and educational affordances of automated writing evaluation (AWE) for writing skill development have been well‐documented. Existing studies have been helpful in affording insights into the use of AWE for writing skill development. Few studies to date published have directly and quantitatively meta‐analyse the use of AWE for writing skill development. It is difficult to understand the effect size and potential AT‐related moderators of AWE for writing skill development. What this paper adds A thorough AT‐driven meta‐analysis of the publications on AWE for writing skill development was conducted. AWE is more effective than traditional methods for writing skill development with an effect size of g = 0.429. The pedagogical effect of AWE is affected by such moderators as educational levels, intervention durations, human–computer interaction types and feedback types. Implications for practice and/or policy Pedagogical potentials of AWE should be encouraged to explore among learners of different educational levels. When using AWE, a combination of human interaction with AWE is necessary. Researchers should understand how such factors as learner individual differences, writing assignment, AWE program and research design may impact the pedagogical effects of AWE.

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 51 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.