Translation vendor management: vet linguists fairly and catch bot claims

Where translation vendor management breaks at scale

Picture a book-length review job entering the claim pool. Two seconds later a script has claimed it, and the linguist who has covered that language pair for years refreshes to find nothing, again. This is what translation vendor management looks like when it breaks.

Meanwhile, test grading runs weeks behind, and two new applicants share one agency’s phone number.

Vetting decides who gets in, and dispatch decides who gets paid. Our engineers built both for a language services provider: a testing and onboarding platform for freelance linguists, and a review marketplace where linguists claim book translations. The marketplace runs in production, alongside our wider localization platform work.

Pressure comes from three sides: fragile freelance supply, cheaper fraud and tighter rules for AI in hiring.

So we had four goals: vet faster, keep honest freelancers, catch cheats, and never let a model quietly turn a person away.

Why the usual ways to vet and dispatch linguists fail

Grade everything by hand. Accurate, but the work grows with every ad campaign. Our client had found it too costly to do at all.

Let an LLM score everything. That was the original plan. But a model asked twice can give two answers, and a rejected candidate deserves the same explanation every time.

First come, first served. Simple, and it feels fair. In practice it rewards whoever refreshes fastest, and scripts refresh faster than people.

How to vet freelance linguists: deterministic scoring first

The qualification test is real work: a candidate transcribes a clip, and we compare it with a verified transcript the client trusts (the gold output).

This platform recruits transcribers, annotators and aligners, so its tests are transcription. To vet freelance translators, keep the layering but swap the metric, because one gold text penalizes valid rewording. Check numbers, names, tags and terminology exactly, and review meaning with a rubric.

The core metric is an error rate: the fewest swaps, deletions or insertions that turn the candidate’s text into the closest accepted gold version, divided by the gold length. It is word error rate (WER), except punctuation marks and annotation tags count as words too, so a missing tag cannot hide.

The judge is boxed in. It sees transcript text and speaker labels, never identity, audio or file names. It returns just a score and one of three codes: same meaning, minor meaning change or major meaning change.

It stays off until three settings are true: AI enabled, legal sign-off, and confirmation that the provider keeps no data. Until then, humans review the middle band.

We inverted the LLM-first plan for reproducibility, not cost. Deterministic layers give the same answer every time, so a disputed score can be explained years later without a provider account. The same principle drives our AI translation quality assurance work: let rules settle what they can.

For the judge, we store its score, rationale code, model and a hash of the exact prompt, because asking again gives a second opinion, not an explanation.

The model call can take seconds, so the final save refuses the score if the key or answers changed meanwhile.

AI candidate screening with a human in the loop

A score can rank candidates; only a named person can turn one away.

In our AI candidate screening, machine results are recommendations. The scorer can only mark an exact match as a pass. Everything else lands as pending for a recruiter, who can pass it, fail it or issue a retake with a logged reason.

The design reserves a third outcome, trainable, for borderline people. The database accepts it, but version one does not write it yet.

Rejections need a person: a database rule (a CHECK constraint) refuses any manual fail without the decider’s id, a timestamp and a reason code. Bulk rejection skips anyone within 5 points of the pass mark, a server-side margin the recruiter’s request cannot change, so “reject all” cannot sweep up borderline people.

The website can’t read the answers, even by mistake. Multiple-choice keys sit in a table the web app’s database account has no permission to read, and a deployment check fails if any runtime role can.

Transcription gold is encrypted at rest and decrypted only inside the scoring service. The server also owns the clock and the question order, so any sitting can be rebuilt exactly for an appeal, and late answers are refused.

Catching cheats and duplicate applicants

Duplicate applicant detection follows one rule: flag, never block, never show the candidate. Email, phone and name-plus-country checks run inside the submit transaction, and the CV-author check stays “not evaluated” until the virus scan finishes, so “not checked yet” never reads as “clean.”

Agencies legitimately submit many people from one number. Two hundred applications on one phone mean 19,900 match rows, so we store at most 20 per signal plus the true count: one badge with “+N more,” not 199 chips.

Fair work allocation for freelancers: checks, then one atomic claim

In a separate review marketplace, linguists claim books from a shared pool. Before any claim, the server checks, in order:

1. The session belongs to this linguist, and the account is active.
2. The AI pre-check is done, and no high error count holds the book for a project manager.
3. A project manager has not released this linguist from this job before.
4. Source and target languages match at the base-language level.
5. No urgent or high-priority job is waiting that this linguist could claim instead.

The claim itself is one conditional UPDATE: assign this job to me only if it is still pending, unassigned, ready and not released for me. If two people click at once, one write wins and the other sees “no longer available.” No explicit locks, no double assignment.

An atomic claim makes the race clean, not fair: a script still clicks first. Fairness has to come from the checks before the claim and the flags after it.

Workload is advice, not a gate. We count business days (weekends and holidays skipped) to the tightest deadline, at a default 12,500 words per reviewer per day. If it doesn’t fit, the linguist sees “Tight schedule” and can still claim.

It used to block long books until project managers asked us to stop.

Catching bots that claim jobs in seconds

A database trigger stamps when a book becomes claimable, on every path including direct database edits, and no request can rewrite it. Each claim logs its elapsed time and reported browser:

• Fast claim: within 5 seconds of availability by default, configurable from 1 to 3,600 seconds.
• Headless browser: the browser calls itself “headless,” meaning a script drives it with no screen, checked on blocked claims too.
• Unknown start: if the availability time is missing, nothing is guessed and no fast-claim flag is raised (the headless check still runs).

Logging never fails the claim. Afterward, reviewer checks alert the project manager: no activity 24 business hours after claiming, over 5,000 words per hour, or under 25% progress by business day 2.

What it cost, and what we would harden next

Today no book can be assigned twice, claims are logged for review, and the scorer cannot reject anyone. The costs:

• Thresholds start as estimates. The 0.45 fail line, the 65/35 blend and the review margin are unvalidated until a calibration run on labeled submissions (150 pairs planned).
• People become the ceiling. A human on every rejection makes recruiter capacity a hard limit on campaign size.
• Signals are self-reported. A user agent can be faked, so the headless flag catches careless scripts, not determined ones.
• Advisory caps allow over-commitment. Some linguists will claim more than fits, and alerts catch it later, not at the door.

Next, we would add per-account throttling for repeat fast claims and exact checks on numbers and names. We would also admit a small random sample regardless of score, to measure who the process wrongly turns away. The onboarding platform was in implementation hardening at the time of writing.

A translation vendor management checklist

If your vendor managers are buried in test grading, or your best linguists keep losing jobs to faster clicks, we have built this before. See our localization technology and custom business systems work, then book a free 30-minute discovery call to map where your vetting and dispatch break first.