AI translation quality assurance: cut false positives, keep the real errors

What over-flagging costs in AI translation quality assurance

An English nonfiction book, translated into Latin American Spanish, comes back from AI translation quality assurance with 18 flagged errors. A linguist checks each one: only four are right as filed. Sorting out the other 14 is paid review time.

Eleven flag correct Spanish: an accepted spelling variant, an optional “que”, a biblical quotation in its traditional form, a spacing “rule” the model invented. Three point at a real issue but with the wrong category or an inflated severity.

That is about 22% clean precision, or 39% (seven of 18) if you count the miscoded three. It happened on a translation quality platform our engineers built with a wider product team for a language services provider. Two AI agents logged errors for MQM scoring, then linguists reviewed them.

MQM (Multidimensional Quality Metrics) is a shared list of error types and severities. Precision is the share of flags that are correct; recall is the share of real errors that get flagged. Low precision means linguists stop trusting the tool, and scores move for reasons nobody can defend.

Why more prompt rules did not stop the false positives

The prompts already said “paraphrase is not an error.” So this was a behavior failure, not a missing rule. We traced the false positives to six causes:

• Hypercorrection: flagging valid variants, like “consciencia” and “conciencia”, both accepted by the Royal Spanish Academy in this sense.
• Paraphrase read as omission: a meaning-preserving idiom coded as missing content.
• Genre-blind rules: register rules applied to a fixed quotation or a legitimately plural address.
• Invented micro-rules: typography no style authority defines.
• Severity inflation: Major where the reader understands fine.
• A validator that waved things through: the second agent passed all 11.

The usual reaction is one more exception per complaint. Mechanical ones (guillemets, non-breaking spaces) worked and stayed. But the list kept growing, and in one round fixing a percent case broke how a book title was handled.

How to reduce false positives in AI translation QA

The first agent reads one segment, logs MQM errors and proposes a corrected target. The second agent works as an LLM judge, accepting, editing or rejecting each flag. We rewrote the first agent’s prompt around four principles:

1. Presume the target is acceptable. Depart only on objective evidence of a defect.
2. Evidence, not preference. Paraphrase, word order and style variation are not defects alone.
3. Improvement is not error. Being able to word it better proves nothing.
4. The independent-linguist test. Would another experienced linguist flag this exact issue? If it is mainly taste, skip it.

The second agent starts from the same presumption, keeps a flag only with concrete evidence, still adds errors the first agent missed, and downgrades inflated severity.

A better alternative does not make the current translation wrong.

How we test it: a gold set with a recall guard

The linguist’s verdicts on those 18 flags became a gold set: 4 real, 11 false, 3 miscoded. A short script scores what the pipeline keeps, lists surviving false positives, and counts kept flags nobody has labeled, which stay out of precision until labeled.

Precision alone is a trap: flag nothing and you score perfectly. So the script also requires all four real errors to survive. Run comparisons at temperature zero where the provider allows; that reduces run-to-run noise but does not remove it.

The target for this file: all 14 bad flags gone or corrected, all four real errors kept. But 18 flags from one book can steer a prompt change, not back a “90% across all books” claim, so we advised more reviewed files before quoting a figure.

Enforce glossary terms and translation memory in code, not prompts

Whether an error is Minor or Major is judgment. Whether a protected brand name appears verbatim is computable. Yet glossary and do-not-translate (DNT) terms lived only in the prompt, so a mandated term could pass one run and fail the next.

A terminology gate with no model call

• DNT tier: a protected source term must appear verbatim in the target, case-sensitive, because wrong case on a brand is the violation. Severity Major.
• Glossary tier: a source glossary term needs an approved rendering from any glossary on the job. Severity Minor.
• Boundary-aware matching: a substring test finds “he” inside “when”, and the regex word boundary fails on “C++” and “.NET”, so we check the characters on either side (lookarounds).
• Honest silence: for Chinese, Japanese, Korean, Thai, Lao, Khmer and Burmese, the glossary tier switches off rather than guess at word edges.

On the first live run, 16 of 40 gate findings duplicated an error the agents had already flagged, a case unit tests could not see. The gate now skips segments that already carry a terminology error and never displaces an agent’s finding.

Translation memory outranks the model

Translation memory (TM) is the client’s library of approved translations. When a target matches an approved entry, translation memory suppression soft-flags the AI’s errors there: hidden from reviewers and scores, kept for audit with a link to the entry. Matching checks only the target, as the client chose, so a short “Bonjour” can rescue an unrelated segment.

Exact matching first failed on curly apostrophes and non-breaking spaces from a spreadsheet export, so both sides now get the same normalization (Unicode NFKC, punctuation folding, whitespace collapse).

The order matters. One wrong glossary term across a 5,000-segment file is hundreds of findings; running the gate before the repeat downgrade keeps every copy from costing full points. A file fully rescued by TM skips the human stages.

A separate pass for book translation consistency

The pipeline ran many segments at once, but each call saw only one: the model judging segment 35 had never seen segment 34.

English “you” can be singular or plural, so a Spanish narrator can slip between “tú” and “ustedes” for the same reader, sometimes in the same breath. Each segment is valid alone; the error exists only across them. A brand name rendered five ways is the same defect.

For other content, a separate opt-in engine we built reports drift only with evidence from at least two segments, at exact Unicode code-point offsets, and reports too little evidence as unknown, never as consistent.

We chunk even when a long-context model could take the whole book: when most of a chunk had drifted, 460 segments in one call returned no findings, and a 30-segment chunk returned 10 correct ones. For a 60,000-word book we estimated 220,000 to 250,000 extra input tokens, which belongs in your AI batch job cost estimate.

Where humans still decide, and what changed

Findings then pass through up to four people: reviewer, translator, second reviewer, arbitrator. Humans disagree too: in one study, trained MQM annotators averaged a segment-level Cohen’s kappa of 0.321, where 1 is perfect agreement and 0 is chance. Arbitration is unavoidable, so keep it small.

• Only fresh disagreement escalates. A second reviewer repeating the first reviewer’s rejection is not a new dispute. A new rejection or any edit is.
• Empty stages skip themselves. No one assigned, or nothing left to decide, means no queue.
• One count everywhere. Vendor lists, badges and notifications share one filter, so nobody sees “12 segments” when two need work.

Staffing those stages is its own problem: see how we manage translation vendors.

What changed, and what we would harden next

Reviewers stopped seeing client-approved translations flagged, and every vendor-facing count agrees. Where a client enables the gate, a DNT breach is checked by code on every run. Prompt changes can be scored against the gold set before they ship.

Not everything held. Client feedback on eight segments showed preference flags still getting through: the validator, on a small model, accepted every one. We added a stronger model for the validator, and code now drops any error whose own note admits the original is correct (65% of notes in one job did). Re-running one French book, those self-refuting notes fell from 7 to 1, and edits no logged error explained fell from 45 to 0.

An AI translation QA checklist for your next file

When a vendor quotes agreement with human reviewers, ask for precision and recall on your own files.

Our localization and LSP software page and the quality platform case study show the wider build. If you are weighing a custom QA pipeline against an off-the-shelf LQA tool, book a free 30-minute discovery call and bring one noisy file. We will show you where precision leaks.