BLEU is Not Suitable for the Evaluation of Text Simplification

Elior Sulem · Omri Abend · Ari Rappoport

168 citations0/0 files storedConference on Empirical Methods in Natural Language Processing (EMNLP)

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Text Readability and Simplification Natural Language Processing Techniques Topic Modeling

Abstract

BLEU is widely considered to be an informative metric for text-to-text generation, including Text Simplification (TS).TS includes both lexical and structural aspects.In this paper we show that BLEU is not suitable for the evaluation of sentence splitting, the major structural simplification operation.We manually compiled a sentence splitting gold standard corpus containing multiple structural paraphrases, and performed a correlation analysis with human judgments.1 We find low or no correlation between BLEU and the grammaticality and meaning preservation parameters where sentence splitting is involved.Moreover, BLEU often negatively correlates with simplicity, essentially penalizing simpler sentences.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 168 citations exist upstream.

References

1968Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.1975Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel 2016Optimizing Statistical Machine Translation for Text Simplification 2017Sentence Simplification with Deep Reinforcement Learning 2006Syntactic Simplification and Text Cohesion 2012Sentence Simplification by Monolingual Machine Translation 2017Exploring Neural Text Simplification Models 2011Learning to Simplify Sentences with Quasi-Synchronous Grammar and Integer Programming

Cited by

2019EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing 2020Data-Driven Sentence Simplification: Survey and Benchmark

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.