LLM cost estimation for AI batch jobs: quotes that hold and runs that resume
Most organizations misestimate AI costs, and batch jobs that die halfway can pay twice for finished work. How we make AI batch jobs quotable, capped and resumable.
On this page
- Where LLM cost estimation breaks on real batch jobs
- Why words-times-rate quotes and blind retries fail
- LLM cost estimation per chunk: plan the real requests first
- Turn the estimate into a fixed AI quote that cannot drift
- LLM retry logic that resumes a failed batch job without paying twice
- Keep Celery long-running tasks alive through crashes and deploys
- What we would do again, and a checklist for your next AI batch job
- Questions we hear
The short version
- Reliable LLM cost estimation prices the run you will make: the real chunk plan, instructions once per chunk, and each pass separately.
- A quote holds when its inputs are hashed, it expires after 30 minutes, and its prompt and model versions are pinned.
- Chunks commit only after their output validates, so a failed job resumes where it broke and never repays finished work.
- Retries back off with jitter, honor Retry-After, skip errors a retry cannot fix, and reach a fallback model last.
- Long jobs survive crashes and deploys: stale workers cannot finish a job, orphaned jobs return to the queue, and deploys pause work.
Where LLM cost estimation breaks on real batch jobs
A client accepts an AI price to translate three supplier contracts. Forty minutes in, the provider starts answering HTTP 429 (“too many requests”) and the job dies at chunk 31 of 60. A naive rerun pays again for the 30 chunks that were fine, so the LLM cost estimation behind that price is already wrong.
Then a release ships, the worker restarts, and the queue says “processing” while nothing is. On Monday, finance asks why the invoice and the provider bill disagree.
This is the scene we design against. Once accepted, a price is a promise. Usage-based AI loses money in two places: a bad estimate before the run, and waste during it.
of organizations misestimate AI costs by more than 10%
of all AI spend is wasted, by organizations’ own estimate
Here is how we closed that gap on two systems for language services providers. One is an AI translation portal we built as a working showcase of a client’s written specification. The other is a translation quality platform we helped build, with hour-long AI jobs in production.
Why words-times-rate quotes and blind retries fail
The usual estimate is words times a rate. It misses three costs that still show up on the bill:
- Instructions repeat. The system prompt goes out with every chunk, so you pay for it many times per file.
- Structure costs tokens. The JSON schema for structured output rides along with each call.
- Failed calls are billed. A malformed response is still charged, even though you discard it.
Blind retries make it worse: they repay finished work, ignore a provider’s request to wait, and let a stale worker overwrite newer results.
- Priced from word count only
- Quote can be accepted days later
- Any failure reruns the whole job
- Retries fire immediately
- Any worker can mark the job done
- Priced per pass from the real chunk plan
- Quote expires after 30 minutes
- Only failed chunks rerun
- Backoff honors Retry-After
- Only the current attempt can finish
LLM cost estimation per chunk: plan the real requests first
A token is the unit providers bill by, roughly four characters of English. Cost is input tokens times the input price, plus output tokens times the output price, which is usually higher. Our rule: size each request from the exact text the provider will receive, and price from the same chunk plan.
In the portal, the file is split into chunks: runs of consecutive segments (sentences or short paragraphs) sent in one request. A chunk defaults to about 2,000 input tokens and 40 segments at most, plus two read-only neighbors per side.
The same code that builds each request sizes it:
- Instructions, including output rules that code appends after any editable prompt.
- The JSON schema the model must answer in.
- Glossary terms that appear in the chunk or its context.
- Neighboring context, capped at half the context limit.
- A response reserve: 1.5 times each segment’s text, plus 32 tokens per segment for translation or 512 for a review pass that explains its edits.
The request must fit within 90% of the context limit, leaving 10% headroom for estimation error. Too big, and read-only context is dropped first. Still too big, and the job stops before sending it, so that request is never billed.
The quote counts chunks with the same code, so instructions are priced once per chunk. It then adds a set number of input and output tokens per word and per segment for each AI pass: translation, a first review and a second review.
The second review sees only flagged segments, so we price it at an assumed 30% of them.
Turn the estimate into a fixed AI quote that cannot drift
An estimate becomes a quote when a client can accept it. The hard part is proving that what they accepted is exactly what runs.
A fingerprint is a short code computed from inputs: change any input and it changes. We hash the files, languages, workflow, versions, glossary, estimates and client rates with SHA-256.
- Analyze
Files are converted, chunked and priced, with no AI call.
- Fingerprint
A SHA-256 hash binds the price to its exact inputs.
- Hold 30 minutes
The quote stays valid for 30 minutes, then must be redone.
- Accept
Hash, expiry, versions and rates are re-checked, then one order is written.
- Pin and run
The job runs on the prompt, model and settings it was priced with.
A one-order-per-project rule in the database means two clicks in two tabs cannot create two orders. If a prompt version or the client’s price list changed after analysis, acceptance fails and asks for a fresh estimate.
A quote is only a promise if the system can prove what was promised.
Before accepting, the client sees a short notice: this is the maximum charge, and any change needs a new estimate. Provider cost is logged per attempt, so it can be compared with the quote.
LLM retry logic that resumes a failed batch job without paying twice
A checkpoint is a saved point you can restart from. In the portal it is the chunk, and a chunk commits only after its output validates.
Validation runs in code: same segment count, known unique IDs, no empty translations, inline tags in order, and placeholders such as {0} or %1$s intact. Results and the “completed” status share one transaction, so a half-written chunk cannot exist.
Runs stop at the first failure, so finished chunks run from chunk 1 with no gaps. On retry they keep their exact boundaries, even if a deploy changed the planning code in between. Only the rest is replanned.
When to wait, stop or switch
- Back off with jitter. Up to three retries at 2, 8 and 30 seconds, plus up to a second of randomness so parallel work does not retry in lockstep.
- Honor Retry-After. When the provider names a wait, in a header or the error body, the longer wait wins, up to 15 minutes.
- Fail fast on permanent errors. Authentication and configuration errors skip the remaining retries instead of burning attempts.
- Fall back last. A second model runs only after the primary has used every attempt or hit an error a retry cannot fix.
Keep Celery long-running tasks alive through crashes and deploys
Checkpoints solve half the problem. The other half is the worker.
On the quality platform (the one behind our post on AI review precision), jobs run on Celery, a widely used Python job queue, with Redis and Postgres. A large file can take hours.
The first trap is a default. Celery’s Redis visibility timeout is one hour, after which an unacknowledged task is redelivered and run again.
Our tasks are acknowledged only when they finish, so we set it to seven hours, above a five-hour soft limit that checkpoints gracefully and a six-hour hard limit. Lost tasks then wait longer for redelivery, so a reconciler recovers them instead.
- One active slot. One job runs at a time, so jobs do not compete for provider quota. Connection-error recovery sends up to three requests at once, then one at a time after a rate limit.
- A fencing token per attempt. Think of it as a ticket number for one run: completion writes must match the current attempt ID, so a stale worker changes nothing.
- Heartbeats and a reconciler. The task reports every 30 seconds. Every 60 seconds, a reconciler asks Celery which tasks are alive. A job whose task is gone and silent for five minutes returns to the queue with its checkpoints.
Deploys get their own path. A pause flag asks the running job to checkpoint and return, keeping its slot until the old task exits. The reconciler then resumes it on the new worker.
What we would do again, and a checklist for your next AI batch job
Replay the opening scene with these controls. The 429 costs a wait, and if retries run out, the rerun starts at chunk 31. The release pauses the job instead of killing it, and finance can check the provider bill against the cost logged per attempt.
The tradeoffs we accepted
- Retries come out of your margin. The quote is the client’s maximum charge, so price that risk into your rates.
- Ratios can be wrong. If well over 30% of segments get flagged, the second pass costs you more than you priced, so compare logged usage with estimates.
- Serial jobs wait. One slot is simple, but a big job delays the next. Bounded concurrency per provider would be the next step.
- A showcase worker is not a queue. The portal’s in-process worker is a deliberate simplification. A hosted product needs the queue above.
- Work that can wait is cheaper. One provider’s batch API cuts cost by 50%. Most batches finish within an hour, but some take up to 24: fine overnight, wrong for a client watching a progress bar.
- Price from the real chunk plan, and size requests from the exact text sent.
- Keep 10% context headroom and reject oversized requests before any call.
- Hash every quote input, pin versions, and expire quotes after about 30 minutes.
- Commit each chunk only after validation, in one transaction with its status.
- Back off with jitter, honor Retry-After, and never retry authentication errors.
- Log tokens and cost per attempt, without storing customer text.
- Fence attempts, send heartbeats, and pause jobs before deploys.
- Set the broker visibility timeout above your longest job, and pair it with a reconciler.
If your AI feature is moving from prototype to paid production, we build applied AI systems with these controls from the first release. Book a free 30-minute discovery call, bring your current cost model, and we will show you where it leaks.
Questions we hear
How do you estimate LLM costs before running a job?
Multiply each request’s input tokens by the input price and its expected output tokens by the output price, then add up every request. Plan with the same chunking code the run will use, price instructions once per chunk and each pass separately, and log actual usage per attempt.
How should an LLM pipeline handle 429 rate limit errors?
Retry with exponential backoff and jitter, and let the provider’s Retry-After win when it asks for longer. We use 2, 8 and 30 seconds plus jitter and cap provider-requested waits at 15 minutes. Where requests run in parallel, drop recovery retries to one at a time after a rate limit. Never retry authentication errors.
How do you resume a failed LLM batch job without starting over?
Checkpoint per chunk and commit each chunk only after its output validates. Stop at the first failure so finished chunks run from the start with no gaps, keep their boundaries on retry, and replan only the rest. In a job queue, fence each attempt so a stale worker cannot overwrite the job.
How do you give a client a fixed price for usage-based AI work?
Treat the quote as a ceiling bound to its inputs. Hash files, languages, model and prompt versions, glossary and rates into a fingerprint, expire it after about 30 minutes, and re-check everything at acceptance. Pin those versions for the run and absorb retries in your margin.