Your pipeline cloud bill went up again. What does one accepted sample cost?

The bill went up. Nobody can say what a sample costs.

The September cloud invoice arrives at $41,200. In June it was $29,800, and sample volume rose only 12%. Finance asks the head of bioinformatics what one sample costs. The honest answer is a range, a shrug and a promise to look into it.

Everyone has a theory. The new variant-calling step is heavy. Someone reran a whole cohort after a reference fix. The spot queue was slow, so jobs moved to on-demand. Each may be true, and none is a number anyone can check.

A monthly bill tells you what you spent. It cannot tell you what you got for it.

The figures in this guide are illustrative, not client data. The pattern is common: orchestration works, results arrive, and cost hides in a total nobody can attribute to a run, a project or an accepted output.

Measure cost per accepted sample, not per job

Cost per job looks tidy and misleads. It ignores the runs that failed, the samples that failed QC and the storage nobody deleted. The useful number is total cost for a period or project divided by samples your scientists actually accepted.

Take one project from that September bill. 400 samples were submitted and 360 accepted; the rest failed QC or are waiting for a rerun.

• Compute attributed to the project’s runs, retries included: $21,600.
• Storage for work directories and results: $3,900.
• Data transfer out of the region: $900.
• Platform fees allocated to the project: $1,500.
• Total $27,900, divided by 360 accepted samples: $77.50 per accepted sample.

A common shortcut is compute divided by submitted samples: $54. The gap of more than 40% is failed work, storage, transfer and fees. That gap is where decisions get made.

Measure turnaround the same way, from sample delivery to accepted result, and split it. Nextflow’s execution timeline and trace file split each task’s time into scheduling (submit to start) and real execution. If most of the delay is waiting for capacity, a faster instance will not help.

Check what Seqera and your cloud already report

Before building anything, read what your tools already give you, and what each number leaves out. Seqera Platform shows an estimated cost per run and per task. Its own cloud cost documentation says the estimate covers compute only, excludes storage, network and the head job that orchestrates the run, and can double-count cached tasks across resumed attempts. Seqera recommends resource labels plus your provider’s native cost tools for real accounting.

On AWS, cost allocation tags must be activated before they appear in Cost Explorer or the cost allocation report, and can take up to 24 hours to show. On Google Cloud, Batch job labels can filter Cloud Billing reports. Both providers offer budget alerts natively. Use them first.

Untagged spend is a finding; report it as its own line. On AWS Batch, one instance often runs tasks from several runs, so instance tags alone cannot split it. Use per-project compute environments, or apportion instance cost by each task’s CPU and memory share from the trace.

Find where the money hides

Once cost is attributed, the same few causes usually explain the gap. Look for each one in your own runs:

• Failed and retried tasks. Every attempt is billed. A task that fails twice for lack of memory costs three runs.
• Oversized requests. A process that requests 64 GB and peaks at 9 GB still reserves, and pays for, all 64 GB.
• Spot interruptions. AWS gives a two-minute interruption notice. A long task that cannot checkpoint starts again from zero.
• Whole reruns. Restarting a cohort instead of resuming pays again for every finished task.
• Intermediate files kept forever. Work directories grow with every run and are rarely anyone’s job to delete.
• Egress. Moving results across regions or out of the cloud adds charges that appear nowhere near the pipeline.

Resume only works when the right work files survive, so retention and recovery have to be decided together. Our pipeline handover guide covers proving resume; here the question is how long to pay for those files. Once a run is accepted and its recovery window has passed, tag its work directory so a lifecycle rule expires it, and keep final results under their own retention.

Right-size from traces, with the scientific owner

Nextflow’s execution report plots CPU, memory, duration and disk I/O per process, both raw and as a percentage of what was requested. That is the evidence for right-sizing. Seqera’s pipeline optimization builds a profile from the maximum use per process in a previous successful run, and warns it only suits similar runs. If input sizes vary widely, Seqera suggests separate profiles for size bins.

Tighter requests make out-of-memory failures more likely, so pair them with a retry rule. Nextflow’s process directives can raise memory on each attempt and cap retries, and Nextflow 24.10 and later can size a retry from the previous attempt’s trace. For spot capacity, treat an interruption as retryable, cap the attempts and run deadline-critical work on on-demand capacity.

Change resources, not the method. A cheaper run whose outputs fail the scientist’s acceptance criteria is not a saving. Keep method changes as a separate decision with their own review.

Build only the gap: a run ledger, alerts and a runbook

What platforms rarely give you is the join between billed cost and accepted output. A run ledger holds one row per run: project, workflow revision, samples submitted and accepted, attempts, queue time, run time, and attributed compute, storage and transfer. Alerts then watch outcomes, not totals: cost per accepted sample above its band, a run with unusual retries, or work directories past their window.

Before trusting the ledger, test it against cases that break cost reports:

• Launch a tagged test run. Its cost appears against the right project once billing data arrives.
• Resume a failed run. Cached tasks are not counted twice.
• Interrupt a spot task. The retry is recorded and its cost included.
• Fail a run completely. It shows real cost and zero accepted samples.
• Reject samples at QC. The accepted count falls and cost per sample rises.

The runbook says who reads the alert, what they check first and when to stop a run rather than retry it.

Benchmark one pipeline before changing anything

Request a call from this article and tell us what last month’s bill looked like and which pipeline you suspect. The first conversation needs no data; confidential files can wait for an NDA or your approved access process. See how we work with biotech teams.