How Machine Translation Quality Is Scored: BLEU, COMET and Human Error Marking
When a company says its machine translation is "better" or "human-level", the claim rests on a score. This explains how the main scores are computed, from BLEU's word-overlap count to expert error marking, what each is known to miss, and what that means for claims about translators' work.
Checked against the original BLEU (2002), chrF (2015), COMET (2020), MQM-study (2021) and error-span-annotation (2024) papers and the WMT24 findings
As of · Sources: the papers that introduced or evaluated each method1234578
| Method | Type | Introduced | What it measures | Stated strength | Stated limit | Source |
|---|---|---|---|---|---|---|
| BLEU | Automatic | 2002 | Counts how many word sequences (n-grams) of the machine output also appear in one or more human reference translations, with a penalty for outputs that are too short | Quick, inexpensive and language-independent, and the authors say it correlates highly with human evaluation | Sole use of BLEU "impeded the development of improved models" and led to bad deployment decisions (Kocmi et al., 2021) | 1, 5 |
| chrF | Automatic | 2015 | F-score on character n-grams against a reference; needs no extra tools or tokenisation | Simple and language-independent; reported to correlate well with human rankings on WMT 2012 to 2014 data | Evaluated against human rankings on older shared-task data only | 2 |
| COMET | Automatic (trained model) | 2020 | A neural model trained on human judgements that reads the source text and a reference translation to predict quality | State-of-the-art correlation with human judgements on the WMT 2019 metrics task | Needs human judgements to train; learned scores are harder to inspect than a count | 3 |
| Direct assessment (DA) | Human | Not stated | Raters give each translation an overall score | Simpler and faster, and can be done by translators of any level | Less reliable than error-based methods (Kocmi et al., 2024) | 7 |
| MQM | Human (experts) | 2021 study | Professional translators mark each error by category and severity; the study weights a Minor error 1, a Major error 5 and a non-translation 25 | Expert error analysis; its 2021 study found a substantially different system ranking from crowd workers | Expensive, time-consuming, and limited by the availability of experts (Kocmi et al., 2024) | 4, 7 |
| Error span annotation (ESA) | Human | 2024 | Annotators mark error spans and severity while also giving a continuous score | Faster and cheaper than MQM at the same quality level in a 12-system English-to-German test, without MQM experts | Validated on one language pair and test set in the paper cited | 7, 8 |
The strengths and limits are what the cited papers state about themselves or about each other, not our own tests; they were run on specific test sets and language pairs, so they may not hold elsewhere. The year for DA is not given in the abstracts we opened. No score in this table measures how a translated text performs for a reader or client.
How it works
BLEU: counting overlap with a human translation
BLEU, introduced in 2002 by IBM researchers, was built because human evaluation of machine translation is "extensive but expensive" and can take months; the authors proposed an automatic method as an "understudy" to skilled human judges.1 Its central idea is that the closer a machine translation is to a professional human translation, the better it is.1 It compares n-grams, runs of words, in the machine output with those in one or more human reference translations and counts matches, but it clips each word’s count to its maximum in any single reference: the nonsense output “the the the the the the the” scores a modified unigram precision of 2/7, not 7/7, against a reference containing “the” twice.1 The paper also adds a brevity penalty so that very short outputs do not score well on precision alone.1
chrF: the same idea on characters
chrF applies an F-score to character n-grams instead of words, which the author says makes it simple, language-independent and independent of tokenisation; it was checked against human rankings on the WMT 2012 to 2014 shared-task data.2
COMET: a model trained to predict human judgements
COMET trains a neural model on human quality judgements, including direct assessments, post-editing effort and MQM annotations, and uses both the source text and a reference translation to predict quality; its authors report state-of-the-art correlation with human judgements on the WMT 2019 metrics task.3 It is still a prediction of what humans would score, so it depends on the judgements it was trained on.
Human evaluation: ratings and error marking
Human methods differ in how much expertise and time they need. Direct assessment asks raters for an overall score, which is simpler and faster but less reliable.7 MQM (Multidimensional Quality Metrics) has professional translators mark each error by category and severity. In the largest MQM study at its 2021 publication, Google researchers weighted a Minor error at 1, a Major error at 5 (they tested Major weights from 1 to 10, the MQM standard’s recommendation being 10) and a non-translation at 25, and capped annotation at five errors per segment.4 Error span annotation, from 2024, combines a continuous score with marking error spans and severity; its authors found it faster and cheaper than MQM at the same quality level for English-to-German.7 In 2024 the WMT general translation task evaluated system outputs with professional annotators using ESA.8
What the scores miss
Different scores can rank the same systems differently. The 2021 MQM study found a "substantially different ranking" of the top WMT 2020 systems from the one given by crowd workers, with a clear preference for human over machine output, and found that metrics based on pre-trained embeddings could outperform the crowd workers.4 A 2021 study of 2.3 million sentence-level human judgements concluded that using BLEU alone impeded the development of improved models and led to bad deployment decisions.5 And an earlier claim that machine translation had reached human parity on Chinese-to-English news was reassessed in 2018: using original source text only, and professional translators as evaluators, the authors found evidence that parity had not been achieved.6
What it means for claims about translators’ work
A score tells you how a system performed on one test set under one method. It does not tell you whether a text is fit for a particular legal, medical or marketing purpose, which is the judgment professional translators and their clients make; O*NET lists the occupation as Interpreters and Translators, SOC 27-3091. For what machine translation and AI interpreting actually do in the occupation, with each case graded by its evidence, see how machine translation and AI interpreting work, and what is left for translators.
Sources
- Papineni, Roukos, Ward and Zhu (IBM), Bleu: a Method for Automatic Evaluation of Machine Translation, ACL 2002, pp. 311–318 (full text read at aclanthology.org/P02-1040.pdf).
- Popović M., chrF: character n-gram F-score for automatic MT evaluation, WMT 2015, pp. 392–395.
- Rei, Stewart, Farinha and Lavie, COMET: A Neural Framework for MT Evaluation, EMNLP 2020, arXiv:2009.09025.
- Freitag, Foster, Grangier, Ratnakar, Tan and Macherey (Google), Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation, TACL 2021, arXiv:2104.14478.
- Kocmi, Federmann, Grundkiewicz, Junczys-Dowmunt, Matsushita and Menezes, To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation, WMT 2021, arXiv:2107.10821.
- Toral, Castilho, Hu and Way, Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation, WMT 2018, arXiv:1808.10432.
- Kocmi, Zouhar, Federmann and Post, Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation, arXiv:2406.11580 (v2, October 2024).
- Kocmi et al., Findings of the WMT24 General Machine Translation Shared Task, WMT 2024, November 2024.