The Complete Guide to Multilingual AI and LLM Evaluation: Benchmarking Models Across Languages
A product team ships a support assistant into eight languages. The evaluation suite reports 91 percent. Three weeks later the Japanese market lead escalates: the assistant is polite in the wrong register, it has invented a refund window that does not exist in that market, and the score has not moved a single point. It has not moved because nothing in the suite was written in Japanese by a Japanese speaker.
That is the default failure mode of multilingual AI. Evaluation gets built in English, then extended to other languages by translating the test set and reusing the same automated judge. Both shortcuts feel rigorous. Neither one measures what the model actually does for a user in Osaka, São Paulo or Riyadh. The result is a number that is technically real and practically worthless, and teams only discover the gap when a customer finds it first.
This guide covers what multilingual AI evaluation measures, why translated benchmarks produce the wrong ranking, where automated judges stop being trustworthy, and how to build a program that produces a number you can defend to a regulator, a board or an enterprise buyer.
What multilingual AI evaluation measures that an English score does not
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.Multilingual AI evaluation measures whether a model’s output is accurate, fluent, culturally appropriate, helpful and safe in each target language independently. An English score measures one language and predicts nothing reliable about the others.
The five criteria are not interchangeable and they fail separately. Accuracy is whether the content is factually right. Fluency is whether a native speaker would accept the phrasing. Cultural relevance is whether the example, the register and the reference make sense in that market. Helpfulness is whether the response resolves the request. Safety is a pass or fail on toxic, biased or misleading output. A model can hold accuracy while collapsing on register, which is exactly the failure that reaches a customer without ever reaching a dashboard.
Degradation is also uneven by task and by conversation length. Research comparing frontier models on multi-turn dialogue found English accuracy holding steady near 55 percent across six to ten turns while Arabic and Korean fell to roughly 41 percent over the same span. Short prompt tests would have missed it entirely. Treat every language you ship as its own system under test, with its own thresholds and its own regression history.
That is the premise behind dedicated multilingual AI evaluation services: a rubric applied by native speakers, per language, against output the model produced in that language rather than against a translation of an English test.
Why translated benchmarks give you the wrong ranking
Translating an English benchmark does not produce a multilingual benchmark. It produces an English benchmark written in another language, and it carries the original’s cultural assumptions across intact.
The Global MMLU research team audited the most widely cited general knowledge benchmark and found that 28 percent of its questions require culturally sensitive knowledge to answer, and that 84.9 percent of its geography questions concern North America or Europe. Their conclusion was blunt: simply translating a dataset is insufficient. Model rankings shifted materially between the culturally sensitive and culturally agnostic subsets, which means a leaderboard built on translated questions can rank models by Western general knowledge rather than by language capability. Their corrected set spans 42 languages and was verified by paid professional and community annotators rather than by machine translation.
The practical consequence is a procurement problem. A team selecting a model for Hindi or Vietnamese on the strength of a translated leaderboard may have selected for something it never intended to buy. The fix is not to abandon translated sets, which are still useful for regression testing, but to author a portion of the evaluation set natively in each target locale and treat that portion as the one that decides.
Where LLM-as-a-judge stops being trustworthy
An automated judge is dependable for triage in high-resource languages and unreliable as a final verdict anywhere else. Use it to narrow the pile, never to sign off on the result.
An empirical study of multilingual LLM-as-a-judge reliability measured an average Fleiss’ Kappa of roughly 0.3 across languages, against an ideal of 1.0. The strongest single model reached 0.54 on one dataset. Spanish and German landed between 0.30 and 0.61 on question answering, while Telugu and Arabic approached zero, with one model scoring as low as 0.002 on mathematical questions. Two findings matter more than the headline number. Neither larger model scale nor multilingual-specific training improved judgment reliability, so waiting for a better model is not a plan. And graded assessments were substantially less consistent than binary ones.
Two design rules follow directly. Ask the judge yes or no questions rather than requesting a score out of five, because the binary form is where consistency survives. And where an automated judge must carry load, run an ensemble and take a majority vote, which the same study found lifted consistency by roughly 0.1 to 0.2 Kappa points over the weakest single model. This is the same lesson that shows up in production translation, and it is worth reading alongside where AI-only translation breaks down in other contexts.
The four layers of a multilingual evaluation stack
A workable multilingual evaluation stack has four layers: automatic metrics for coverage, an automated judge for triage, native human rubric scoring for the verdict, and independent expert adjudication for anything regulated or high-exposure. Each layer exists to reduce the volume reaching the one below it.
Layer one runs on everything. Automatic metrics such as COMET and chrF are cheap enough to score every output you generate, which makes them the right tool for detecting drift between releases. They are also blind to register and cultural fit, so a stable metric score is evidence of no regression, not evidence of quality. Layer two narrows the pile with binary judge calls, subject to the limits in the previous section. Layer three is where the actual verdict is produced, by native speakers scoring against a written rubric. Layer four is a second, independent reviewer for output that carries legal, medical, financial or brand risk, which is the same structure as linguistic quality assurance (LQA) on human translation, where an independent linguist signs off before release.
| Method | What it measures | Where it breaks in multilingual | Use it for |
|---|---|---|---|
| Automatic metrics (COMET, chrF) | Similarity to a reference output | No reference exists for open-ended AI output; blind to register | Release-over-release drift detection |
| LLM-as-a-judge | Relative or rubric-scored preference | Agreement collapses below high-resource languages | Triage and binary pass or fail screening |
| Native human rubric scoring | Accuracy, fluency, culture, helpfulness, safety | Needs calibration or annotators diverge | The verdict on production readiness |
| Independent expert adjudication | Second opinion on contested or high-risk output | Cost, so it cannot run on everything | Regulated and high-exposure output |
The question that decides the budget is what share of output reaches layer three. There is no universal answer, but there is a workable default: sample heavily at launch and taper as the error rate stabilises, rather than setting a flat percentage and holding it forever. A locale in its first month deserves a sample large enough to characterise the failure pattern. A locale that has held its threshold for two quarters needs enough to catch drift and no more. Flat sampling rates are how teams end up over-reviewing a stable market while a new one ships blind.
The error typology matters as much as the layer count. Scoring against a structured framework such as MQM gives you a repeatable error category and severity rather than a general impression, and ISO 5060, published specifically to govern human evaluation of translation output, gives you a standard to point at when a client asks how the number was produced. The full method sits in the guide to AI translation quality, and the scoring frameworks themselves are covered in detail in how translation quality is measured.
How to route AI output to the right review depth
Route AI output to a review depth by the risk the output carries, not by the model that produced it or the project it belongs to. One release almost always contains all three depths at once.
The NEX Translation Matrix™ scores five inputs for each piece of content: content type, business risk, customer impact, regulatory requirement and quality expectation. It returns one of three outcomes. Automatic scoring is sufficient for low-risk, high-volume output such as internal summaries or bulk product descriptions. Native human review applies where meaning, tone or terminology need a linguist on top of the draft. Independent adjudication before release applies to regulated, legal, safety-critical or high-exposure output, where a second reviewer signs off.
The rule that makes it work is that the matrix routes content, not projects. A single product release typically produces all three outcomes simultaneously: the changelog is automatic, the onboarding flow is human review, and the terms of service is adjudication. Routing runs per string or per document, never per job. Teams that route per job either overpay for review on everything or underpay on the one string that ends up in front of a regulator. The same logic governs where machine translation post-editing (MTPE) sits relative to full human translation in a production pipeline.
Worked through on a real release, the split is obvious once the inputs are scored. An in-app changelog is low risk, low regulatory weight and high volume, so it clears on automatic scoring. The onboarding sequence carries high customer impact and a tone expectation the brand cares about, so it goes to native human review. A revised data processing notice carries a regulatory requirement, so it goes to adjudication regardless of how short it is or how confident the model was. Three destinations, one release, one afternoon of scoring.
What belongs on a per-language evaluation scorecard
A per-language scorecard needs one row per language carrying five criterion scores, the sample size behind them, the inter-annotator agreement, and a pass threshold that was set before any scoring began. Drop any of the four and the row stops being defensible.
Inter-annotator agreement is the number most teams omit, and omitting it is what turns an evaluation into an opinion. If two qualified native speakers score the same hundred outputs and agree only two thirds of the time, the problem is the rubric, not the model, and no amount of additional sampling will fix it. Report agreement next to the score, always. Sample size deserves the same treatment: forty samples in Thai and four hundred in German are not comparable numbers, and presenting them in adjacent columns without the counts invites a bad decision.
Set the pass threshold first. A threshold chosen after seeing results is a description of the results. Write down what each score band means with a worked example for each, so a new annotator can reproduce the scale, and keep the failing samples themselves rather than only the aggregate, because the error pattern is the part engineering can act on. This is the same discipline that governs what an LQA pass actually covers on human-written translation.
Record these per language, every cycle:
- Score per criterion: accuracy, fluency, cultural relevance, helpfulness, safety
- Sample size, and how samples were selected
- Inter-annotator agreement, reported next to the score
- Pass threshold, fixed before scoring started
- The failing samples themselves, with error category and severity
Language resource tiers and the review each one needs
Group target languages into resource tiers and set the minimum human review by tier, not by traffic. The languages where models degrade most are usually the ones receiving the least scrutiny.
The common allocation is backwards. Review budget follows revenue per locale, so German and French get deep review while Telugu, Vietnamese and Indonesian get spot checks, which is precisely the inverse of where model reliability sits. At launch, revenue per locale is a forecast, not a fact, and under-reviewing a low-resource market is how a locale gets written off as having no demand when it actually had a broken product.
| Resource tier | Example languages | Typical model behaviour | Minimum human review |
|---|---|---|---|
| High resource | English, Spanish, German, French | Stable across tasks and turn counts | Sampled rubric scoring, judge triage acceptable |
| Mid resource | Japanese, Korean, Arabic, Hindi | Register and multi-turn coherence degrade first | Full native rubric scoring, no judge-only sign-off |
| Low resource | Tamil, Telugu, Vietnamese, Swahili | Fluent-sounding output with unreliable meaning | Native scoring plus adjudication on anything customer-facing |
Tier is a useful proxy but it is not the whole picture, because some failures track writing system and grammar rather than training data volume. Right-to-left scripts expose layout and bidirectional text handling that no rubric will catch unless the evaluation renders output in the actual interface. Languages with grammatical honorifics, Japanese and Korean among them, fail on register in ways that read as fluent and score well on accuracy. Agglutinative and heavily inflected languages produce string lengths and compound forms that break truncation assumptions built around English. Build at least one check for each of these into the rubric for the locales where they apply, rather than trusting a general fluency score to surface them.
Tiering also gives you a defensible way to say no. When a stakeholder asks for thirty languages at once, the tier table converts that request into a review cost and a calendar, which is a more productive conversation than a flat refusal.
How to stand up an evaluation program in ninety days
Stand up the program in four stages over roughly ninety days: define the rubric, pilot two contrasting languages, calibrate annotator agreement, then scale to the full locale set. The sequence matters more than the timeline.
Stage one is the rubric, with five criteria and a worked example for every score band. Stage two pilots exactly two languages, deliberately chosen as one high-resource and one low-resource, because a pilot run on two European languages will tell you the program works and then fail the moment it meets Arabic script or Japanese register. Stage three is calibration: the same annotators double-score a shared set until agreement holds, and the rubric gets rewritten wherever it does not.
Stage three is the one teams skip, and skipping it invalidates everything downstream, because scores produced by uncalibrated annotators cannot be compared across languages or across quarters. Stage four scales out, with the sampling rate per locale set by risk rather than by volume. Run the whole program as a standing cadence rather than a launch gate, in the same way a hybrid AI plus human workflow runs continuously rather than as a one-time pass.
What multilingual AI evaluation costs and how to scope it
The cost of multilingual AI evaluation is driven by four things: the number of languages, the sample size per language, the review depth each sample needs, and whether independent adjudication is required. Model size is not one of them.
That is useful because three of the four are yours to set. A program covering ten languages at two hundred samples each, with human review on half and adjudication on the regulated tenth, is a very different budget from the same ten languages reviewed uniformly at maximum depth, and it produces a better answer because the scrutiny landed where the risk was. Scope the sampling by risk first and the total falls out of that.
Evaluation work and production language work share the same linguist pool and the same tier logic, which is why the economics are easier to model than most teams expect. NexTranslate publishes its rates openly, and the transparent translation pricing page shows how review depth maps to cost per word across the three tiers. The same three-tier structure that governs translation delivery governs evaluation depth, so a team already buying at Professional tier has a reasonable starting estimate for what rubric-scored evaluation in the same locales will run.
Three questions separate an evaluation vendor from a labelling vendor. Ask who writes the evaluation content, because a vendor that only scores what you supply cannot help you when the problem is that your test set was translated. Ask how inter-annotator agreement is measured and what happens when it falls below threshold, because a vendor without an answer is selling throughput rather than reliability. And ask whether the annotators scoring your Korean output are Korean, which sounds like an insulting question and is answered in the negative more often than the industry admits.
Frequently Asked Questions
1. What is multilingual AI evaluation?
Multilingual AI evaluation is the structured assessment of an AI system’s output in every language it serves, scored against defined criteria by native speakers rather than inferred from English performance. It typically covers accuracy, fluency, cultural relevance, helpfulness and safety, and it produces a separate result per language rather than a single blended score.
2. Can I just translate my English evaluation set?
No, not as your primary set. A translated benchmark carries the cultural assumptions of the original, and research on the widely used MMLU benchmark found that 28 percent of its questions require culturally sensitive knowledge. Translated sets remain useful for regression testing between releases, but the set that decides whether a model is ready for a market should be authored in that market.
3. Is LLM-as-a-judge good enough for multilingual evaluation?
It is good enough for triage in high-resource languages and not good enough as a final verdict. Published reliability studies report average agreement around 0.3 Fleiss’ Kappa across languages, with low-resource languages approaching zero. Use binary judge questions rather than graded scores, ensemble several models where you rely on one, and keep a human verdict on anything that ships.
4. How many samples do I need per language?
Enough that the sample size can survive being printed next to the score, which in practice means a few hundred for a language carrying real risk and a smaller regression set for a language you are only monitoring. The number matters less than reporting it, because a score without its sample size and its inter-annotator agreement is not comparable to anything.
5. Which languages should be evaluated first?
Pilot one high-resource and one low-resource language together rather than starting with the two largest markets. A program validated only on European languages tends to break on the first right-to-left script or honorific system it meets, and finding that in a two-language pilot is far cheaper than finding it after rollout.
6. How is this different from translation quality assurance?
Translation quality assurance evaluates a translation against a source text, so there is a correct answer to compare with. AI evaluation frequently has no reference output at all, which is why it relies on rubric scoring and pairwise comparison rather than error counting alone. The linguist skill set overlaps heavily, the methodology does not.
Conclusion: a multilingual quality claim is only as good as the program behind it
A multilingual quality claim is only as strong as the program that produced it. An English benchmark score, a translated test set and an automated judge can all be assembled in an afternoon, and together they will tell a team what it wants to hear in every language it does not speak. The alternative is not more expensive so much as more deliberate: native-authored evaluation content, a written rubric, calibrated annotators, agreement reported alongside every score, and review depth routed by risk rather than by revenue.
The teams that get this right treat evaluation as infrastructure rather than as a launch checkbox. They can say which languages are production-ready and which are not, and they can show the working. That is the claim buyers, regulators and boards are actually asking for.
If you are shipping AI into languages your team does not speak and cannot yet say which of those languages are production-ready, that is the gap worth closing first. We can scope a rubric, a pilot pair and a sampling plan against your actual locale list. Get a quote and we will come back with a program, not a proposal template.
Written by Karuppusamy Arunachalam, Founder, NexTranslate
Published September 2026 · Filed under AI & LLM Evaluation