Have cell / virtual-cell foundation models beaten the linear baseline? An honest read (Part 3)

Evidence-first notes on bioscience and deep tech, at the edge of the lab and the market. Information only — not investment advice.

The 30-second version

• What. This is the most skeptical axis of the bio-foundation-model series. As of 2026-07, cell / virtual-cell foundation models (scGPT, Geneformer, scFoundation, Arc STATE and others) do not yet consistently beat deliberately simple baselines (no-change, additive, linear/ridge, PCA) on their actual tasks — and this is stated by primary literature, not reviewer opinion. Four independent angles converge on the same answer.
• So what. The convergence is unusually hard to dismiss: (1) Nature Methods 2025 puts “does not yet outperform simple linear baselines” in the title across 7 models; (2) an independent benchmark (Mao et al., arXiv) finds Arc’s STATE gives “relatively weak results”; (3) Arc’s own Virtual Cell Challenge 2025 officially concedes models “do not yet consistently beat naive baselines”; (4) zero-shot representation learning also underperforms HVG/PCA/scVI/Harmony. The vendor headline for STATE (perturbation discrimination “+50%”, true DEGs “2×”) is a self-benchmark claim we place on HOLD — it is contradicted by the independent benchmark and must not be cited as demonstrated fact.
• Now what. Read the verdict narrowly: this concerns the predictor role, not the atlas / representation / data-infrastructure role, which can carry value if kept separate (and even representation zero-shot underperforms, so caveats remain). The metric to watch is not headline benchmark wins but generalization under leakage-controlled splits (cell-state-similarity splits), where nearly every method’s performance collapses today.

[demo-gap note] This axis is the least mature of the three in the series. Most systems (scGPT, Geneformer, STATE) are research demos / open weights, not products. Product claims such as “drug-target prioritization” or “in-silico screening” presuppose a baseline win that is currently unmet. The vendor “+50% / DEG 2×” figure for Arc STATE is a self-benchmark (Arc’s own Cell_Eval) claim, refuted as “weak” by an independent benchmark — held, not published as evidence.

The five-minute read

A milestone framing that fails the “does it beat baseline?” test

Where proteins (Part 1) and genomes (Part 2) returned a “partial yes,” the cell axis returns “not yet.” The sharpest statement stands in a title: Nature Methods 2025, “Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines” (Ahlmann-Eltze, Huber, Anders, 2025-08-04). The core tension of this axis is that virtual-cell vendor headlines (“+50% better perturbation discrimination; 2× true DEGs”) and independent evaluations point in opposite directions. The firm’s discipline — does it actually beat the baseline? — applies here most sharply.

Deliberately simple baselines that are hard to lose to

The Nature Methods study pitted five FM-family models (scGPT, scFoundation, Geneformer, UCE, scBERT) plus two task-specific DL models (GEARS, CPA) against baselines that were intentionally simple: “no change” (predict the unperturbed control), “additive” (sum of individual log-fold-changes for double perturbations), “mean” (training-set average) and linear/ridge regression on low-dimensional embeddings. On double perturbations, every DL model showed substantially higher error than the additive baseline, and failed to beat even “no change” on genetic-interaction detection. On single perturbations, the FMs failed to consistently beat mean/linear; scGPT, UCE and scBERT showed almost no variation across perturbations — functionally collapsing to “no change.” The robustness comes from the setup: the baselines were built to be simple (the loser cannot cry foul), the reproduction code is public, and seven models under one protocol rule out a single bad implementation.

Vendor headline vs. independent benchmark, and the host’s own contest

Arc Institute’s STATE (bioRxiv, 2025-06-23) is a two-module virtual-cell model trained on ~170M observed cells plus >100M perturbation cells across 70 cell lines. Its vendor claims (+50% discrimination, 2× DEGs) rest on Arc’s own Cell_Eval framework. The independent benchmark by Mao et al. (Shanghai AI Lab, arXiv:2604.27646, 2026-04-30) reverses this: under a cell-state-similarity split (forcing generalization rather than memorization), nearly all methods’ performance collapses, linear methods rank best on unseen cells, and STATE gives, in the authors’ words, “relatively weak results.” Even Arc’s own Virtual Cell Challenge 2025 (114 countries, 1,200+ teams) officially concluded that models “do not yet consistently beat naive baselines,” and the winning entries were DL + classical statistical hybrids, not pure end-to-end FMs.

[diagram: vendor headline vs. four independent angles]

  "Does the cell/virtual-cell FM beat a simple baseline?"

  VENDOR HEADLINE (Arc STATE, self-benchmark)
    +50% perturbation discrimination · 2x true DEGs   → [HOLD: unverified]

  ──────────── independent evidence, all one direction ────────────
  1  Nature Methods 2025   7 models vs no-change/additive/linear   → NO (title)
  2  Mao et al. (indep.)   cell-state split; linear best; STATE    → "weak"
  3  VCC 2025 (Arc's own)  MAE mostly below baseline; hybrids win   → NO (consistent)
  4  Genome Biology 2025   zero-shot < HVG/PCA/scVI/Harmony         → NO

  Verdict: predictor role = "not yet"; wins cluster on
           pretraining-set-overlap (leakage-suspect) data.

Deep dive

1. Background — the sharpest skeptic axis of the series

Part 0 flagged the cell axis as the one with “the most negative answer.” The framing must be precise: the claim is not “cell FMs are useless.” Value as representation / atlas / data infrastructure and performance as a quantitative perturbation predictor must be separated. This part’s verdict is limited to the latter. The through-line — benchmark win ≠ generalization — holds most dramatically here.

2. What this deep dive newly establishes

Core answer: across perturbation prediction and zero-shot representation learning, cell/virtual-cell FMs fail to consistently beat simple baselines, and the vendor claim that STATE beats baselines “+50%” is contradicted by independent evaluation.

• Linear-baseline refutation (primary) — CONFIRMED: Nature Methods 2025-08-04 compared 7 models against 4 deliberately simple baselines on Norman / Replogle / Adamson data. Double perturbations: all DL substantially worse than additive; single perturbations: FMs fail to beat mean/linear, several collapsing to “no change.” Reproduction code public (github.com/const-ae).
• Scope honesty (primary) — CONFIRMED: the authors state their own limits — only 4 datasets, all cancer cell lines, no QC filtering of perturbations that did not affect their target gene, single transcriptome modality, and non-perturbation tasks (clustering / annotation) not evaluated. So the reading is narrow: “for perturbation prediction, right now, DL does not beat linear,” not “cell FMs are void.” That honesty raises, not lowers, the refutation’s credibility.
• Vendor claim vs. independent benchmark — Mao et al. (arXiv:2604.27646, 2026-04-30) is CONFIRMED/independent: under cell-state-similarity split nearly all methods collapse, linear ranks best on unseen cells, STATE is “relatively weak,” and the paper notes “model performance is often overestimated” while the commonly used Pearson correlation has low discriminative power and masks real weaknesses. The Arc STATE “+50% / DEG 2×” figures are UNVERIFIED/vendor (self-preprint, self-benchmark) and refuted in this independent test — cited here only as an attributed vendor claim, never as demonstrated fact.
• Host’s own contest concedes it (primary) — CONFIRMED: Virtual Cell Challenge 2025 (wrap-up 2025-12-06) officially found perturbation models “do not yet consistently beat naive baselines across all metrics” (with improvement on discrimination / DEG identification), with most models below baseline on MAE — prompting a new 7-metric Generalist Prize. Winners were DL + classical hybrids (BioMap’s xTrimoSCPerturb; Altos Labs’ flow-matching generalist), and the top team stated “Purely AI-based approaches did not consistently outperform statistical baselines.”
• Zero-shot representation also underperforms (independent) — CONFIRMED: Genome Biology 2025-04-18 (Kedzierska et al.) found Geneformer and scGPT zero-shot worse than simple HVG selection / scVI / Harmony on cell-type clustering, with Geneformer consistently worst on batch integration. Crucially, the cases where scGPT led (Tabula Sapiens, Immune) involve datasets that were in scGPT’s pretraining set — a leakage confound.

3. Strengths and limits of the methodology

Strengths: the baselines are deliberately simple (a hard bar to lose to), reproduction code is public, seven models share one protocol, and vendor claims are attributed and isolated from independent evidence. Limits (scope honesty): the Nature Methods data are only 4 datasets, all cancer cell lines, with no QC filtering and a single modality; the verdict is narrow to perturbation prediction, not “cell FMs are void.” One supporting item, “one PCA still rules them all” (arXiv:2410.13956), is confirmed only at title/abstract level (PDF not parsed) and is deferred to Part 4. The deeper lesson is that evaluation methodology dominates the result: the split (cell-state-similarity), the confound (pretraining-set leakage) and the metric choice (MAE vs. discrimination/DEG scores) each flip conclusions.

4. Neighbouring domains

The refutations feed directly into Part 4 (benchmarks / leakage / generalization): Mao et al.’s cell-state split collapse, the zero-shot leakage confound (scGPT-Tabula Sapiens), and VCC’s metric-driven ranking flips are its core material. Across the series, three axes (protein memorization, genome short-range blind spot, cell split-sensitivity) converge on a shared bottleneck — benchmark advantage ≠ generalization. This also connects to convergence signal L-S01 (“treat AI as a hypothesis generator, not a predictor”): cell FMs fall short as predictors but are being reframed as atlas / representation / hypothesis-generation infrastructure. For CKM (cardio-renal-metabolic), transplanting perturbation prediction into drug-target prioritization presupposes a baseline win — currently unmet.

5. Commercialization and market context (actors, platforms — not a clinical angle)

Deployment readiness: the least mature of the three axes. Most systems are research demos / open weights (scGPT, Geneformer, STATE). Product claims (drug-target prioritization, in-silico screening) presuppose a baseline win that is currently unmet. Commercial value is being realized first in atlas / data / representation infrastructure (Arc Virtual Cell Atlas, 300M+ cells), which should be read separately from any predictive-SOTA narrative. No single-winner conclusion is asserted; all vendor quantitative claims are attributed as such.

6. The skeptic’s bottom line

• verified-clean: none. No model established a clean baseline win under leakage control and generalization splits.
• proceed-with-caveats (conditional): the atlas / representation / data-infrastructure value of cell FMs may be described if separated from predictive performance. Because representation also underperforms zero-shot, a caveat is mandatory.
• HOLD: quantitative claims that “virtual-cell FMs beat the baseline” or STATE’s “+50%” — held for publication, escalated. As long as Nature Methods 2025, Mao et al.’s independent benchmark, and the VCC 2025 official conclusion all point the other way, content that presents the vendor headline as demonstrated evidence is held.
• Verdict (inherited): the firm’s view — “cell FMs do not yet beat the linear baseline” — is robust via four-angle convergence (proceed-with-caveats). The vendor superiority claim specifically is HOLD, escalated to skeptic and Principal.

7. What to watch (falsifiable predictions)

1. Over the next 12–18 months, no cell perturbation FM will significantly beat additive/linear baselines on a published, leakage-controlled (cell-state split) benchmark across all tasks. Falsifiable by VCC 2026 or a follow-up independent benchmark.
2. Data-scale expansion alone (Tahoe-100M scale) will not cross the threshold — even STATE (170M + 100M cells) was “weak” independently, implying the bottleneck is evaluation / problem formulation, not scale. A 2×-scale model that beats baselines would falsify this.
3. DL + classical hybrids will keep beating pure end-to-end FMs (the VCC 2025 pattern). A pure FM winning the next contest over hybrids would falsify this.

References

• Ahlmann-Eltze, Constantin, Wolfgang Huber, and Simon Anders. 2025. “Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines.” Nature Methods, 2025-08-04. https://www.nature.com/articles/s41592-025-02772-6 (PMC full text: PMC12328236; reproduction code: github.com/const-ae)
• Kedzierska, Katarzyna Z., Lorin Crawford, Ava P. Amini, and Alex X. Lu. 2025. “Assessing the limits of zero-shot foundation models in single-cell biology.” Genome Biology, 2025-04-18. https://link.springer.com/article/10.1186/s13059-025-03574-x (PMC: PMC12007350)
• Mao, et al. (Shanghai AI Lab). 2026. “Benchmarking in-the-wild perturbation response prediction.” arXiv:2604.27646, 2026-04-30. https://arxiv.org/abs/2604.27646
• “One PCA still rules them all” (perturbation transcriptomics FM benchmark). arXiv:2410.13956. https://arxiv.org/abs/2410.13956 (title/abstract level; body deferred to Part 4)
• Arc Institute. 2025. “STATE” virtual-cell model. bioRxiv, 2025-06-23. https://arcinstitute.org/manuscripts/State (“+50% / 2× DEG” — vendor self-benchmark claim: news/virtual-cell-model-state)
• Arc Institute. 2025. “Virtual Cell Challenge 2025 wrap-up.” 2025-12-06. https://arcinstitute.org/news/virtual-cell-challenge-2025-wrap-up (framing: Cell 2025, S0092-8674(25)00675-0)

Source knowledge asset: knowledge-base/deep-dives/bio-foundation-models/part3-cell-fm.md (generated 2026-07-12, VERIFIED — 9 confirmed / 1 corrected(refined) / 2 unverified-vendor). This draft inherits the figures and source attributions of the original part and creates no new figures or sources.

Disclosure

This post is for information only and is not investment advice. The author holds no position in, and no financial interest in, the entities mentioned (Arc Institute, CZI, Altos Labs, BioMap, Tahoe Therapeutics, NVIDIA and others).

COI note: This document describes non-profit institutes (Arc Institute, CZI), private companies (Altos Labs, BioMap, Tahoe Therapeutics) and one listed company (NVIDIA) in a factual, neutral technology context, with no buy/sell implication. All vendor quantitative claims (Arc STATE “+50% / 2× DEG”) are attributed as self-benchmark preprint claims and kept separate from independent evaluation, which refutes them as “weak”; they are not presented as demonstrated fact. The firm view that “cell FMs do not yet beat the linear baseline” is a firm-specific position built on four-angle convergence, framed neutrally so as not to read as a negative characterization of any single actor.