A translation model returns 0.87 on the evaluation set. Six weeks later a customer in Osaka points at a paragraph in the Japanese help centre that describes a cancellation window the company does not offer. The sentence is grammatical, idiomatic and completely invented. Nothing in the evaluation flagged it, and nothing in that evaluation ever would have, because the metric was scoring how much the output reads like a translation rather than whether it is one.
That is the specific way large language models break translation evaluation. The metric stack most teams inherited was built to score neural machine translation, which fails visibly: garbled word order, dropped clauses, obviously wrong terms. LLM output fails in the opposite direction. It is fluent, confident and paraphrastic, which is precisely the profile these metrics were trained to reward. The failure mode they were designed to catch is not the failure mode they are now being asked to catch.
What follows is how to evaluate LLM translation in a way that survives contact with more than one language: what each metric is actually good for, why a single threshold across a locale set is not one standard, and a six step protocol that produces comparable numbers.
Why LLM translation output breaks the metrics built for neural MT
Automatic translation metrics reward fluency and semantic plausibility, and LLM output is fluent and semantically plausible even when it is factually wrong. The score rises for the same property that makes the error dangerous.
The research on this is uncomfortably specific. COMET, the most widely deployed neural metric, assigns generous scores to confident but unfaithful translations, because its training data rarely contains blatant hallucinations labelled with low scores. An empty translation can score around 0.32. Roughly 0.25 percent of real translations in one audit scored lower than an empty string. Most telling: when Chinese text was substituted into an English to Russian evaluation, it still scored 0.655, against 0.463 for genuinely random sentences.
Sit with that last number. A metric that cannot reliably distinguish the wrong language from the right one is not going to distinguish an invented refund policy from a real one. This does not make the metric useless, it makes it narrow. Treat the automatic score as a fluency and adequacy proxy that detects drift between builds, and never as a check on whether the content is true.
Reference-based, reference-free, and the case where you have no reference
Reference-based metrics compare output against a trusted human translation. Reference-free metrics, usually called quality estimation, score output against the source alone. In production LLM translation you almost never have a reference, which makes quality estimation the working default and changes what you are allowed to conclude from the number.
Reference dependency creates a second problem specific to LLM output. A single reference is one valid translation among many, and reference-based metrics penalise legitimate paraphrase and synonym choice. LLMs generate paraphrase constantly, so a lower reference-based score on LLM output can mean the model diverged from one translator’s wording rather than that it produced worse translation. Teams comparing an LLM against their existing engine on a reference-based metric are frequently measuring stylistic distance from a reference and reading it as quality.
Choose the path from what you actually have rather than from what the tooling defaults to. If a trusted reference exists, reference-based scoring is informative. If it does not, quality estimation is the only automatic option, and it carries wider error bars. For anything carrying real business risk, neither is sufficient on its own.
What each automatic metric is actually good for
Each metric answers a narrower question than its single number suggests, and the gap between what it measures and what a buyer assumes it measures is where bad decisions come from.
| Metric | Needs a reference | What it catches | Known blind spot |
|---|---|---|---|
| BLEU / chrF | Yes | Surface overlap with the reference | Penalises valid paraphrase, near-useless on LLM output |
| COMET / COMET-22 | Yes | Semantic adequacy and fluency | Scores confident hallucinations generously |
| CometKiwi (QE) | No | Estimated quality from source alone | Weakest human correlation of the neural family |
| xCOMET / xCOMET-QE | Either | Error spans with minor, major, critical severity | Coverage thins outside high-resource pairs |
| GEMBA-MQM | Yes | LLM-detected error spans by MQM category | Inherits judge bias, degrades in low-resource languages |
Each metric answers a narrower question than its single number suggests. None of them answers whether the content is true.
Independent benchmarking makes the spread visible. When the localization platform Intento compared current metrics against its own human quality review, GEMBA-MQM correlated at roughly 0.5 to 0.6, XCOMET-XL at 0.4 to 0.6, CometKiwi-XL at about 0.3, and MetricX-23 at nearly zero. None of them were adopted for production use, on grounds of limited language coverage, poor interpretability, cost, and speed (MetricX-23 was clocked at 1.5 seconds per segment). Separately, WMT24 results put COMET-22’s system-level correlation with human Multidimensional Quality Metrics (MQM) ratings at about 0.69, with weaker agreement at the segment level.
The practical reading of those two findings together is that automatic metrics are reasonable for ranking systems against each other and unreliable for judging any individual segment. Use them to answer which engine is better across a corpus. Do not use them to answer whether this string is safe to publish. The scoring frameworks underneath all of this are worth understanding how an MQM score is actually calculated, and the wider method sits in the complete guide to AI translation quality.
Why the same score means different things in different languages
Automatic scores are not calibrated across translation directions. A flat threshold applied to every language in your locale set is not one standard, it is a different standard per language wearing the same number.
The underlying reason is in the training data. Human assessment scores used to build these metrics range from roughly 0.51 to 0.91 depending on the translation direction, which bakes a systematic per-direction bias into the output scale. English to German and English to Tamil do not start from the same place, so holding both to 0.85 sets a lenient bar for one and an unreachable one for the other. Teams then conclude the Tamil engine is failing when what is failing is the comparison.
Version drift compounds it. The same English to German test set scored 0.796 under one library version and 0.837 under another, with no change to the translations at all, and at least 12 percent of published papers using COMET never state which version produced their numbers. A score of 0.85 without a direction and a version attached is not a measurement, it is a rumour. Set per-direction thresholds anchored on human scores for that direction, and record the metric version in the same row as the score.
How to use an LLM as a translation judge without trusting it blindly
An LLM judge earns its place when it is asked to name error spans and severities, not when it is asked to produce a score. The difference is auditability.
GEMBA-MQM, the method behind the strongest correlation in the Intento comparison, prompts a frontier model with few-shot examples to identify error spans in a translation and classify them by MQM category and severity. What comes back is a located, categorised list of problems rather than a number. A reviewer can check a flagged span in seconds and either confirm it or dismiss it. A reviewer presented with a bare 0.82 has nothing to check and no choice but to trust it.
The judge’s known weaknesses still apply, and they apply hardest where coverage is thinnest. Reliability degrades sharply outside high-resource languages, so a span flagged in Spanish deserves more confidence than one flagged in Telugu, and an absence of flags in a low-resource language is close to uninformative. Ask for spans and severities, keep the human verdict on anything that ships, and read the output as a prioritised queue rather than a result. The broader case for that posture is set out in the risks of AI-only translation.
A working protocol for evaluating LLM translation across languages
Six steps produce numbers you can compare across languages and across quarters: pin the version, build a native test set per direction, calibrate a human-anchored baseline, screen everything with quality estimation, route flagged output to humans by risk, and track results per direction rather than blended.
Steps one and two are hygiene. Record the metric name, its version and the model build alongside every score, and build the test set for each direction from content authored in that language rather than translated into it. Step three is the one that makes the other five mean anything: have native linguists score a fixed sample per direction, then anchor the automatic scale to those human scores so each direction gets its own pass band. Skip it and you are back to comparing German and Tamil on a number that does not mean the same thing in both.
Steps four and six are the ones that turn a one-off assessment into a running signal. Screening runs quality estimation over everything the model produces, which is affordable precisely because it needs no reference, and its job is to rank output by suspicion rather than to pass or fail it. Tracking then keeps the result per direction and never blends the locale set into a single headline number, because a blended score moves when the traffic mix moves and tells you nothing about the model. A quality figure that improves because a strong locale grew faster than a weak one is worse than no figure at all.
Step five is where cost is decided, and the routing logic should follow risk rather than flag volume. The NEX Translation Matrix™ scores five inputs per piece of content, content type, business risk, customer impact, regulatory requirement and quality expectation, and returns one of three outcomes: automatic scoring is enough, a native linguist refines meaning, or an independent reviewer signs off before publication. The rule that makes it usable is that it routes content, not projects. One release produces all three outcomes at once, so routing runs per string or per document, never per job.
In practice the second and third outcomes map onto work a language services partner already does. Refining a draft is machine translation post-editing (MTPE). An independent sign-off before release is linguistic quality assurance (LQA), which is also the layer that catches the fluent invention an automatic score waved through. Verification is a budget line rather than an afterthought, and it is worth sizing early: NexTranslate pricing publishes per-word rates with human proofreading included at every tier, which makes the verification share of an LLM translation program straightforward to model before committing to a locale count.
| Failure mode | What it looks like in output | Which check catches it |
|---|---|---|
| Confident hallucination | Fluent sentence stating a fact that is not in the source | Human review, or an LLM judge returning error spans |
| Wrong register | Correct meaning, wrong formality or honorific level | Native reviewer. No automatic metric catches this |
| Silent omission | Clause dropped, remaining text still reads cleanly | Source-to-target span alignment, then human check |
| Terminology drift | Product terms translated differently across strings | Glossary check against translation memory (TM) |
| Wrong language | Output partly or wholly in an unintended language | Language identification on output, before scoring |
Automatic scoring catches the bottom two reliably and the top three poorly. Route review budget accordingly.
The pattern underneath that table is that the failures automatic scoring misses are the ones a reader notices immediately. That is the argument for keeping a human layer even on content the model handles well, and it is the same argument behind how LQA protects a brand on human-written translation. Teams that need this run end to end rather than assembled in-house typically buy it as managed AI evaluation services sitting alongside the translation workflow itself.
Frequently asked questions
How do you evaluate LLM translation quality?
Evaluate it per language direction, using automatic quality estimation to screen everything and native human review on a risk-selected sample. The automatic score identifies drift and ranks systems. The human review decides whether individual output is publishable. Record the metric version and the translation direction alongside every number, because neither score is interpretable without them.
Is COMET reliable for evaluating LLM translation output?
It is reliable for comparing systems across a corpus and unreliable for judging individual segments. WMT24 put COMET-22’s system-level correlation with human MQM ratings at about 0.69, with weaker segment-level agreement. It also scores confident hallucinations generously and has been shown to rate wrong-language output well above random text, so it cannot serve as a faithfulness check.
What is the difference between reference-based and reference-free evaluation?
Reference-based metrics compare output against a trusted human translation. Reference-free metrics, known as quality estimation, score output against the source alone. Production LLM translation rarely has a reference, so quality estimation is usually the only automatic option available, and it carries wider error bars than the reference-based equivalent.
Can one quality threshold be used across all languages?
No. Human assessment scores underlying these metrics span roughly 0.51 to 0.91 depending on translation direction, so one flat cut-off is lenient in some directions and unreachable in others. Anchor a separate pass band per direction against human scores for that direction before comparing any two languages.
Can an LLM judge replace human review of translations?
Not for anything that ships. An LLM judge is most useful when prompted to return error spans and severities rather than a score, because a reviewer can verify a flagged span quickly. Its reliability drops sharply in low-resource languages, where an absence of flags carries almost no information.
How much LLM translation output needs human review?
Enough that every high-risk category is covered, which is a question about risk rather than percentage. Regulated, legal and safety-critical content warrants independent sign-off regardless of its automatic score. High-volume low-risk content can run on automatic screening with periodic sampled audits to confirm the screen is still working.
Conclusion: a score without a direction and a version is not a measurement
A translation score without a direction and a version attached is not a measurement. It is a number that felt like one. The teams getting this right are not running more sophisticated metrics than everyone else, they are running the same metrics with the bookkeeping attached: which direction, which version, which human baseline the scale was anchored against, and which segments a person actually looked at.
The uncomfortable part is that LLM translation made evaluation harder at the exact moment it made translation look easier. Output that reads beautifully in a language nobody on the team speaks is not evidence of quality, and the metric that scored it was built to reward exactly that quality of reading. Fluency was never the hard part.
If you are shipping model output into languages your team cannot read, the question worth answering this quarter is not which metric to adopt. It is which of your directions currently has a human-anchored baseline, and which are running on a number nobody has calibrated. We can put that baseline in place per direction and size the verification layer around it. Get a quote and we will start with your actual locale list.
Written by Karuppusamy Arunachalam, Founder, NexTranslate
Published September 2026 · Filed under AI & LLM Evaluation
