Benchmark reality for bio foundation models: how data leakage, missing baselines and metric choice inflate “beats baseline” claims

Evidence-first notes on bioscience and deep tech, at the edge of the lab and the market. Information only — not investment advice.

The 30-second version

• What. A leaderboard number for a bio foundation model (bio-FM) is the product of model ability and how forgiving the evaluation is. Data leakage is close to the default, not the exception: in a meta-analysis of genomics ML studies, 44% were inflated by feature-selection leakage, raising AUROC by about 0.18 on average (Barnett et al. 2023, attributed). In protein-protein interaction (PPI) benchmarks, random splits produced a leakage rate up to 86%, which drops to roughly 0% under interface-similarity splitting (MLSB@ICLR 2024).
• So what. The single largest lever after leakage is a missing strong baseline: if the comparison is an out-of-the-box old method, the reported margin measures the baseline’s handicap, not the model. Temporal (time-based) splitting is the separator between memorization and generalization — LiveProteinBench shows performance falls systematically as release dates get more recent.
• Now what. Do not over-generalize: leakage is a per-benchmark measurement, not a blanket assumption. One genomics eQTL benchmark was 99.7% novel (only 12 of 4,297 test items matched by full-sequence identity), i.e. genuinely clean. Treat vendor leaderboards as claims and independent leakage-controlled benchmarks as evidence. Verification here is PARTIAL: 12 confirmed / 2 corrected (boundary) / 3 unverified.

[benchmark-vs-deployment gap] This is the AI form of the demo/deployment gap: a benchmark win is not the same as generalization to a novel, out-of-distribution target. Where evidence is single-source or vendor-reported (e.g. an Inductive Bio blog claim, or vendor “+50% / 2× DEG” figures), it is labeled unverified and is not headlined.

The five-minute read

A leaderboard number is model ability times evaluation leniency

One figure on a bio-FM leaderboard (PoseBusters 84%, AUROC 0.87, “+50% better”) compresses two things that are rarely separated: what the model can do, and how generous the test setup is. The thesis of this part: bio-FM benchmarks are systematically biased upward by four overlapping levers — (1) train/test leakage, (2) absence of temporal splitting, (3) missing strong baselines, and (4) metric and subset cherry-picking. When these stack, a “specialized baseline is beaten” claim is almost guaranteed. The purpose here is not to disparage any one model, but to sharpen the honest evaluation frame the firm applies before treating any such claim as an asset.

Leakage is close to the default — and random splits manufacture it

Leakage is information that should be inaccessible at evaluation time bleeding into training or preprocessing. In biology it is especially insidious because two sequences or structures can look like “different data points” while being evolutionarily or structurally near-identical — effectively copies of the same problem. A PPI benchmark audit (MLSB@ICLR 2024) quantified leakage by split strategy over ~50,000 PDB interactions: PPI-code-based 86%, PDB-code 65%, deposition-time 61%, sequence-similarity (MMseqs2) 30%, and interface-similarity (iDist) near 0%. The closer to a random split, the more “easy twins” remain in the test set. A related signal: in an AlphaFold leakage study (Brief Bioinform 2025), the optimal pLDDT threshold for disorder prediction shifted from 0.89 to 0.69 (KS-test P<0.0001) on a decontaminated set — evidence that benchmarks routinely leave training near-duplicates in place.

The missing strong baseline is the biggest single lever

If a FM paper’s baseline is a handicapped, out-of-box version, the margin measures the handicap. In single-cell perturbation prediction, this is the strongest signal: Nature Methods 2025 reports that deep-learning gene-perturbation prediction still does not beat a deliberately simple linear baseline (scGPT and scFoundation compared against GEARS, CPA and linear regression). For cell embeddings, a zero-shot evaluation (Genome Biology 2025) found single-cell FMs consistently fail to surpass classical HVG+PCA pipelines, with batch effects remaining a fundamental barrier to any “universal embedding.” The heuristic: the real control is the specialized baseline a skilled domain expert can build in half a day — linear regression, ensemble docking, HVG+PCA, a conservation score. If the FM cannot beat that, the scale-and-parameters narrative is advertising.

[diagram: PPI benchmark leakage rate by split strategy]

  Share of test pairs that are "leaked" near-duplicates
  Random / PPI-code   ██████████████████████  86%
  PDB-code            ████████████████        65%
  Deposition-time     ███████████████         61%
  Sequence-similarity ███████                  30%
  Interface-similarity ▏                       ~0%
  ───────────────────────────────────────────────────
  Anchor: genomics meta-analysis — 44% of studies inflated,
          +0.18 AUROC average, from feature-selection leakage
  Boundary: one eQTL benchmark was 99.7% novel (12/4,297 matched)

Deep dive

1. Background — how benchmarks inflate “superiority”

Where Part 0 of this series observed that “the answer differs by axis” (protein, genome, cell), this part dissects the common machinery that distorts that answer across all three axes: data leakage, missing strong baselines, and metric selection. A quantitative anchor sets the scale — in a meta-analysis of genomics ML studies (Barnett et al. 2023, IEEE/ACM TCBB, via a Front Bioinform 2024 review), 44% of surveyed genomics studies were inflated by feature-selection leakage, with mean AUROC rising by 0.18. An AUROC delta of 0.18 is the difference between “usable” and “SOTA.” Leakage is therefore not a rare accident but contamination close to the default.

2. What this deep dive newly establishes

Core answer: across all three axes, the “beats a specialized baseline” claim is systematically biased upward by four levers, and none of the axes has yet cleanly cleared leakage control + strong baseline + multi-metric + reproducibility simultaneously.

• Leakage / random splits — CONFIRMED (independent/high): PPI random-split leakage up to 86%, dropping to ~0% under interface-similarity splitting; genomics 44% inflated at +0.18 AUROC. Manifests on all axes — protein (ligand near-duplicates), genome (feature-selection), cell (test sets included in pretraining, e.g. Tabula Sapiens / Immune inside scGPT training). Fix: similarity-aware or temporal splitting with the leakage rate reported. DataSAIL (Nat Commun 2025) is one similarity-aware “cold split” tool (quantitative detail behind an auth-wall, unverified).
• Absence of temporal splitting — CONFIRMED (scope caveat): LiveProteinBench (arXiv 2512.22257) uses UniProt entries new since 2025-01-01 and shows performance falls systematically as release dates become more recent — the memorization-vs-generalization separator. Caveat: LiveProteinBench is a general-LLM protein-text-reasoning benchmark, not a sequence-FM structure benchmark; the temporal-split methodology transfers, but the numbers are not directly comparable to AF3/ESM3 structure prediction.
• Missing strong baseline — CONFIRMED, strongest on the cell axis (primary/high): Nature Methods 2025 shows DL perturbation prediction still does not beat a linear baseline. On the protein axis, Inductive Bio reports that a ~100-line strong baseline (ensemble conformation + Gnina rescoring) beats blind AF3 by +4.2% (and is −7.1% versus pocket-specified AF3) — single blog source, no independent replication (blog/med, unverified).
• Metric / subset cherry-picking — CONFIRMED (independent/high): AUROC is lenient on imbalanced / small-sample data and treats false positives and false negatives alike (Front Bioinform 2024 recommends MCC or AUPRC). Strong subsets (e.g. Evo2 has a short-range blind spot; nucleic-acid LDDT drops to DNA 0.53 / RNA 0.61 versus monomer 0.88) are easily promoted into overall headlines — the variance hides behind the average.

3. Strengths and limits of the methodology

Strengths: each lever is grounded in primary methodology literature, and the levers are cleanly separable (split strategy, contamination audit, baseline strength, metric fit). The boundary case is preserved honestly rather than swept into a blanket claim — the genomics eQTL benchmark in the OCR modeling study (arXiv 2602.02014) was 99.7% novel (only 12 of 4,297 test items matched by 100% full-sequence identity), so some benchmarks are genuinely clean. Limits: several key quantitative claims remain unverified — the DataSAIL random-split over-estimation figure (auth-wall), the Inductive Bio “+4.2% over blind AF3” (single blog, no replication), and vendor figures (“+50% / 2× DEG / 1 trillion teraflops”). These are held as claims and not headlined.

4. Neighbouring domains

The evaluation discipline transfers directly. Open-weight models (ESM3-open, Evo2, Geneformer) are structurally reproducible and sit at a higher verification tier; closed APIs and internal-engine claims sit at a lower tier by construction — so the open-vs-closed divide is also an evaluation-trust divide. For cardio-renal-metabolic (CKM) work, the same rules port onto clinical biomarker and drug-response prediction models: the Barnett 44% leakage warning applies to clinical prediction models too, and this checklist is the seed of a due-diligence frame for CKM predictive FMs.

5. Commercialization and market context (verification tiers, companies)

Company context (factual, neutral, no buy/sell implication): the discussion touches listed companies (Alphabet/DeepMind, NVIDIA, Recursion) and private ones (Isomorphic Labs, EvolutionaryScale, Arc Institute). Benchmarks have become marketing assets: if a vendor simultaneously selects (a) a favorable split, (b) a weak baseline, (c) a lenient metric and (d) a strong subset, “SOTA” is nearly guaranteed. The firm treats vendor leaderboards as claims and independent leakage-controlled benchmarks (FoldBench, LiveProteinBench-type) as evidence. All vendor quantitative figures are attributed and held as unverified until independently reproduced. No single-winner conclusion is asserted.

6. The skeptic’s bottom line

• verified-clean: none. No axis has yet cleanly cleared leakage control + strong baseline + multi-metric + reproducibility at once. There is no confirmed, contamination-free superiority.
• proceed-with-caveats (conditional): the baseline-superiority claims on the protein and genome axes pass conditionally only if the leakage / temporal-split / strong-baseline caveats are placed in the 30-second summary, and the benchmark-vs-deployment gap is stated explicitly.
• hold: “virtual cell beats baseline”-type claims on the cell axis are held for publication and escalated, as long as the Nat Methods 2025 linear-baseline refutation and the Genome Biology 2025 zero-shot limits stand. Vendor “+50% / 2× DEG” figures are not to be cited before independent replication.
• Firm’s seven-point honest-evaluation checklist (any failure lowers the tier and attaches a caveat): (1) Split — is it random (auto-penalty), or similarity-aware / temporal, with the leakage rate measured and reported? (2) Contamination audit — confirmed that test items are not in the FM’s pretraining corpus, with a cutoff date? (3) Strong baseline — is the control a “half-day specialized baseline,” not an out-of-box old version? (4) Metric fit — not AUROC alone on imbalanced/small data; cross-checked with MCC/AUPRC; no strong subset promoted to the headline? (5) Reproducibility — code, weights, eval scripts, split seed, license public; third-party reproduction exists? (6) Uncertainty — confidence intervals / multiple seeds / statistical tests, not a single point estimate? (7) Deployment gap — does the benchmark edge hold on novel (out-of-distribution) targets, or only on near-duplicates?

7. What to watch (falsifiable predictions)

1. If a standard protocol combining leakage control (similarity/temporal split) with a strong baseline spreads, much of the current bio-FM leaderboard margin should shrink significantly (with cases where the margin halves in protein-ligand and genomic variant classification). Falsifiable via follow-on leakage-controlled benchmarks.
2. Cell perturbation / embedding FMs will not significantly beat linear or HVG+PCA baselines across all tasks in published benchmarks within the next 12–18 months. Falsifiable via public results such as the Virtual Cell Challenge.
3. The gap between vendor-internal “superiority” figures and independent leakage-controlled benchmark numbers will be systematically smaller on the independent side (vendor claim > independent reproduction). Falsifiable via FoldBench / LiveProteinBench-type replications.

References

• Whalen, Sean, et al. 2022. “Navigating the pitfalls of applying machine learning in genomics.” Nature Reviews Genetics. https://www.nature.com/articles/s41576-021-00434-9
• Model-evaluation-metrics review. 2024. Frontiers in Bioinformatics (cites Barnett et al. 2023, IEEE/ACM TCBB, for the 44% / +0.18 AUROC figure). https://www.frontiersin.org/journals/bioinformatics/articles/10.3389/fbinf.2024.1457619
• “Revealing data leakage in protein interaction benchmarks.” 2024. MLSB@ICLR. arXiv:2404.10457. https://arxiv.org/html/2404.10457v1
• DataSAIL: leakage-avoiding data splitting. 2025. Nature Communications. https://www.nature.com/articles/s41467-025-58606-8
• LiveProteinBench: contamination-free benchmark. 2025. arXiv:2512.22257. https://arxiv.org/abs/2512.22257
• “Zero-shot evaluation reveals limits of single-cell FMs.” 2025. Genome Biology. https://genomebiology.biomedcentral.com/articles/10.1186/s13059-025-03574-x
• “Batch effects as a barrier to universal embeddings.” 2025. bioRxiv 2025.12.19.695371. https://www.biorxiv.org/content/10.64898/2025.12.19.695371
• “Deep-learning perturbation prediction vs. linear baseline.” 2025. Nature Methods. https://www.nature.com/articles/s41592-025-02772-6
• “AlphaFold data leakage.” 2025. Briefings in Bioinformatics bbaf104 (pLDDT threshold 0.89→0.69). https://academic.oup.com/bib/article/26/2/bbaf104/8069415
• “Genomic modeling via OCR — eQTL overlap analysis.” 2026. arXiv:2602.02014 (eQTL benchmark 99.7% novel). https://arxiv.org/pdf/2602.02014
• Inductive Bio. “A strong baseline for AlphaFold 3 docking” (blog, single-source, unverified). https://www.inductive.bio/blog/strong-baseline-for-alphafold-3-docking

Source knowledge asset: knowledge-base/deep-dives/bio-foundation-models/part4-benchmark-reality.md (generated 2026-07-12, verification PARTIAL — 12 confirmed / 2 corrected(boundary) / 3 unverified). This draft inherits the figures and source attributions of the original part and creates no new figures or sources.

Disclosure

This post is for information only and is not investment advice. The author holds no position in, and no financial interest in, the listed companies mentioned (Alphabet, NVIDIA, Recursion and others).

COI note: This document describes listed companies (Alphabet/DeepMind, NVIDIA, Recursion) and private ones (Isomorphic Labs, EvolutionaryScale, Arc Institute) in a factual, neutral technology/evaluation context. There is no buy/sell implication. All vendor and single-source quantitative claims (e.g. “+50% / 2× DEG / 1 trillion teraflops,” the Inductive Bio “+4.2%” blog figure, DataSAIL quantitative details) are attributed as “vendor / author / preprint claims” and marked unverified pending independent reproduction.