Your model scored 0.89. Will it rank next quarter’s compounds?

The model scored 0.89. Then the new series arrived.

An ML lead presents an ADMET liability classifier with an AUC of 0.89 on a random 80/20 split. The chemistry team uses it to triage next quarter’s compounds, most of them from a new series. When the assay results come back, the model ranks that series no better than a simple fingerprint baseline. The numbers here are illustrative; the pattern is common.

Nobody cheated. The test answered a different question. A random split asked how well the model predicts compounds similar to those it has seen. The project needed to know how well it ranks chemistry it has not.

A benchmark score answers the question its split asked, not the one your project asks.

A Nature Methods perspective on data leakage describes the mechanism: information shared between training and test data produces performance estimates far better than the model achieves in its intended application, and biological dependencies make that sharing hard to see.

Where the optimism comes from

Most inflated scores trace back to a handful of causes. Each one is ordinary, and each one is testable:

• Near-duplicates and shared scaffolds. Close analogs, or proteins from one sequence family, land on both sides of the split.
• Time leakage. Measurements made after the test compounds help predict them.
• Assay and batch effects. The model learns a plate, site or protocol signature that the random split shares between train and test.
• Replicates and label noise. Repeat measurements of one compound appear in both sets, and noisy labels cap what any model can achieve.
• The wrong distribution. A public benchmark’s chemistry looks little like your next series.
• The wrong metric. Overall AUC or R² weighs every rank or error equally; chemists act only on the top of the list.
• No lineage. Nobody can say which data snapshot, split and model version produced the headline score.

The literature has made this point for years. MoleculeNet recommends scaffold splits for several datasets, noting that separating structurally different molecules is a greater challenge than a random split. A time-split study found random selection too optimistic and leave-class-out too pessimistic compared with true prospective prediction. Therapeutics Data Commons ships meaningful splits and reports that strong algorithms still fall short under real distributional shifts.

Start from the decision, not the leaderboard

Write the decision down in one sentence. For example: rank 400 proposed compounds in a new series so chemists can choose 40 to make. That sentence sets the test set (unseen series), the metric (how many genuinely good compounds reach the top 40) and the segment that matters (the new series, not the whole history).

Regulators use the same idea. FDA’s draft guidance on AI for regulatory decision-making assesses credibility for a defined context of use. Its scope is information supporting decisions about safety, effectiveness or quality, so most discovery models sit outside it, but the discipline transfers.

Build a test set that looks like next quarter

Group before you split. For small molecules, cluster by scaffold or fingerprint similarity; for proteins, by sequence identity. Keep every replicate of a compound in one group. Then hold out whole groups: the newest series, the most recent quarter or an entire assay batch. Scientists approve the grouping, because similarity is a scientific definition, not a library default.

Keep the held-out set out of tuning. If a team checks it after every hyperparameter change, it slowly becomes training data. Record who opened it and when.

Report uncertainty, segments and a baseline

A held-out series may hold only a few dozen compounds, so a single score can swing widely. Report an interval, for example by bootstrapping whole series or clusters rather than single compounds, and show results by series, assay and time window. An overall gain can hide a loss on the segment you care about.

Always include simple baselines: a nearest-neighbor model, a tuned tree ensemble, a model that predicts the series mean. In the DrEval study of drug response models, deep learning models barely outperformed a naive mean-effects model, and no complex model beat properly tuned tree-based ensembles in the relevant settings. Your result may differ, but you only learn that by running the comparison.

A drop on the honest split is a finding, not a failed project. It tells you where the model can be trusted and where chemists should rely on judgment.

Make every score reproducible

Every reported score should carry its dataset version, split definition, code commit, model artifact and metric configuration. MLflow, DVC or your platform’s own tracking can hold these; the requirement is that someone can rerun last quarter’s number and get it back.

Then test the harness with the failures it exists to catch:

• Rerun a published score from its record. It matches within the agreed tolerance.
• Copy a test compound into training. The overlap check stops the run.
• Split a compound’s replicates across sets. Grouping keeps them together.
• Add a new assay batch. Its results appear as a separate segment.
• Submit a model that loses to the baseline on the target series. It is flagged, not promoted.

Reuse benchmark tools; build the local harness

Benchmark suites, split utilities and experiment trackers already exist. Use them. The gap is usually local: your intended-use split, overlap checks against your own chemistry, the lineage glue between data platform and tracker, a per-segment report scientists can read and a recorded acceptance decision.

If your question is whether LLM-written summaries match their source documents, that is a different problem; see the AI evidence review guide.

Scope one model and one decision

Request a call from this article and tell us, in a few lines, which model you trust least and the decision it informs. The first conversation needs no data; confidential files can wait for an NDA or your approved access process. See how we work with biotech teams.