Your assay team renamed one column. Forty submissions loaded without it.

The column that went quietly empty

In March the assay team publishes template v3 for its binding assay. “Conc” becomes “Concentration (µM)”, cell line becomes required and the readout list replaces “Lum” with “Luminescence”. Half the lab keeps last quarter’s v2 file on their desktops.

The importer matches columns by header. It finds no “Concentration (µM)” in the v2 files and loads them anyway, with the field blank. By June forty submissions sit in the warehouse without a concentration, and an analyst finds out when a dose-response fit refuses to run. Nothing failed loudly.

A template change is a release. Unversioned, it breaks data without breaking anything that alerts you.

The assay, template versions and field names in this guide are illustrative, not client data.

Two ways an importer fails a template change

Either it rejects everything that does not match the newest template, so scientists email files to a curator and the workaround becomes the process. Or it accepts what it can parse, which is worse: values land in the wrong field or no field, and downstream analysis silently loses a column.

Spreadsheets are not the villain. A Scientific Data paper on HuBMAP, an NIH tissue-atlas program, lists the usual errors: missing required fields, typos, formatting errors and values outside the agreed value set. Fixing them often means a curator contacting the submitter. The cure is an explicit contract around the spreadsheet, not banning it.

Write the contract down and version it

Put a version marker inside every template, in a protected cell or file property, and keep each version’s schema in source control. The importer reads the marker and applies that version’s rules. A file without one is returned, not guessed at.

• Separate required, optional and conditionally required fields. JSON Schema’s conditional keywords express rules such as “a dose unit is required when a dose is present”.
• Store the vocabulary identifier beside the display label, so renaming a label never changes what earlier records mean.
• Keep the submitted value beside any normalized value, so a reviewer can see exactly what the scientist typed.
• Author the schema once. LinkML describes data structures in YAML and generates other formats, including JSON Schema, from one definition.

This is FAIR practice, applied to one assay. The FAIR Guiding Principles ask for rich metadata, vocabularies that are themselves FAIR, and domain-relevant community standards. Where a standard such as ISA already describes your investigation, study and assay structure, map to it. Do not invent a new ontology for one template.

Validate where scientists enter data, in their language

“Column 7: enum mismatch” sends a scientist to a curator. “Readout ‘Lum’ was retired in v3. Use ‘Luminescence’, or ask the readout vocabulary owner” lets them fix the file in a minute. Name the row, the field, the allowed values and who decides.

CEDAR builds metadata templates with controlled terms and, in HuBMAP, generates the spreadsheets investigators fill in and validates incoming datasets. The HuBMAP paper adds a useful detail: curators use the validator as a teaching aid in submission office hours. Good messages reduce the next round of errors, not just this one.

Accept two versions, with written migration rules

Support the current version and the one before it. A scientist halfway through a study should not have to switch mid-run, and each extra version multiplies the tests.

Write a rule for each change from v2 to v3. A rename is mechanical: “Conc” maps to “Concentration (µM)” when the v2 file states micromolar, and waits for review when it does not. A new required field cannot be invented; the submission returns, or the field owner approves an explicit “not recorded” value. A retired term maps to its successor only when the vocabulary owner says the meaning is unchanged.

Agree what counts as a major change. HuBMAP released a new versioned template for renamed fields or new required fields, and reissued spreadsheets without a version change for minor, backwards-compatible revisions. Your rule can differ, as long as it is written down.

Treat a template change like a release

Keep a field register: each field’s owner, definition, allowed values, the template versions that contain it and the analyses that read it. When someone proposes renaming “Conc”, the register shows which dashboards and models need the same change on the same day.

Check what your ELN or LIMS already does before building any of this. If you use Benchling, its schema documentation notes that changing a schema’s field definitions can change data in the objects that use it, and that schema fields are archived rather than deleted. Native required fields and dropdowns may cover part of the contract. Build only the gap: version detection, migration rules, readable errors and the tests.

Test changes against submissions you already have

Synthetic files test the rules you thought of. Last quarter’s real submissions test the ones you forgot. Before v3 ships, it should pass checks like these:

• Load a v2 file with “Conc” and no unit. It waits for review instead of loading a blank or a guessed concentration.
• Submit v3 without a cell line. The scientist sees the rows, the field and who to ask.
• Use the retired term “Lum”. The message names its successor and the original value is kept.
• Submit a file with no version marker, or a retired version. It is returned with a link to the current template, and nothing loads partially.
• Replay last quarter’s accepted submissions. Each produces the same records or a listed, approved difference.
• Compare fields accepted at entry with the analysis table. No column disappears between them.

Scope one submission path you can test

Request a call from this article and describe the last template change that broke an import. The first conversation needs no data; confidential files can wait for an NDA or your approved access process. See how we work with biotech teams. If the files come from a partner rather than your own lab, read the CRO data integration guide.