A Preprint Benchmark of Six Single-Cell and Spatial Transcriptomics Foundation Models Finds No Overall Winner, on Curated Datasets and Without a Reported Simple-Baseline Comparison

Analysis based on information available as of 2026-08-02.

30-second summary

• What: A preprint (arXiv:2607.17227, submitted 2026-07-19, q-bio.GN) evaluates six single-cell and spatial transcriptomics foundation models (FMs) under one harmonised framework: Nicheformer, CellPLM, scGPT-spatial, GenePT, scELMo and Novae. Tasks were clustering, supervised annotation, marker-gene concordance and perturbation prediction, across scRNA-seq, spatial transcriptomics and Perturb-seq data. The authors report that no model dominated across tasks and that rankings shifted with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice.
• So what: This is a not-yet-peer-reviewed preprint, and the results are metrics on benchmark datasets the authors selected, not performance in a clinical or drug-discovery deployment. The paper compares the six FMs with each other, not with simple baselines such as PCA or linear models. The finding that no single FM wins is therefore not a judgement on the FM approach itself, and the common claim that FMs beat simpler methods is likewise not confirmed by this paper. We read only the abstract; the methods section was not read.
• Now what: Practitioners may take it as support for choosing models per task and modality. Whether the changing rankings reveal real evaluation instability or are an artifact of the harmonisation itself cannot yet be told apart, because the preprocessing, data-split and metric details and the availability of code were not verified. TRL 3 (evaluation infrastructure). Not peer reviewed.

5-minute read

What was benchmarked

Over the past two years, many foundation models have been proposed for single-cell and spatial transcriptomics. They differ in architecture: cell-level transformers, graph-aware spatial models, and models that build gene representations from language-model embeddings of gene descriptions. Their original papers each report results under their own conditions, which makes side-by-side comparison hard. This preprint proposes a harmonised evaluation framework and runs six models through it. It does not introduce a new model; its contribution is the evaluation setup. The quantitative results are the authors’ own runs rather than numbers self-reported by the model developers.

What the abstract reports

The central sentence is: “No model dominated across tasks, and rankings shifted with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice.” Beyond that, the abstract describes three tendencies, which we paraphrase: cell-level transformers trained on expression data did best on many cell-identity tasks; spatial and graph-aware models were stronger at preserving tissue architecture; and gene embeddings derived from language models were competitive on some perturbation-response metrics. The authors conclude that future models should be judged on biological generalisation, interpretability and perturbation-grounded validity, “not by scale or leaderboard performance alone”.

How to read it

Two limits matter. First, the claim is about the six models relative to one another. It says nothing about whether a plain PCA-plus-nearest-neighbour pipeline would match them, a question that is often raised about single-cell FMs. Second, a benchmark paper is only as informative as its methods. If fixing preprocessing and tokenisation made the rankings stable, the “no winner” result would reflect incomplete harmonisation, not a property of the models. The abstract cannot settle which is the case.

Deep dive

1. Background

Foundation models for single-cell biology borrow the pretrain-then-adapt recipe from language modelling. A model is pretrained on a very large corpus of cells and then used zero-shot or fine-tuned for downstream tasks such as cell-type annotation, clustering or predicting the response to a genetic perturbation. Spatial transcriptomics adds tissue coordinates and neighbourhood structure. According to public descriptions that we did not re-verify against the original papers, Nicheformer combines dissociated and spatial data in pretraining, scGPT-spatial continues pretraining of scGPT on spatial corpora, CellPLM encodes cell positions, Novae is graph-based, and GenePT and scELMo derive gene or cell representations from language-model embeddings. Their heterogeneity (different tokenisation, different priors, different training data) is what makes a harmonised comparison both valuable and hard to design.

2. What this study newly establishes

Design (from the abstract). Six models, four task families (zero-shot and continually pretrained clustering, supervised annotation, marker-gene concordance, perturbation prediction) and three modalities (scRNA-seq, spatial transcriptomics, Perturb-seq) under one framework.

Results (from the abstract). No overall winner; rankings change with modality, preprocessing, tokenisation, biological prior, domain shift and metric choice. Expression-trained cell-level transformers were best on many cell-identity tasks, spatial or graph-aware models were strong on tissue architecture, and language-derived gene embeddings were competitive on some perturbation-response metrics. No numerical results are quoted here because only the abstract was read.

What the paper does not establish. This paper compares the six FMs with each other, not with PCA, linear models or other simple baselines. The “no model dominated” result is therefore not a judgement on whether the FM approach is valid, and the conventional narrative that FMs outperform simpler methods is also not confirmed here. The abstract mentions no non-FM baseline; whether the full text does is unverified.

Reading status. The abstract could not tell us whether the evaluation data are held-out clinical cohorts or reuses of widely circulated public benchmarks. We therefore describe the datasets as selected by the authors and do not claim real-world downstream performance.

3. Strengths and limitations of the method

Strengths.

• A single framework applied to heterogeneous models addresses the incomparability of results reported separately in each model’s own paper.
• The quantitative results come from the authors’ own runs, not developer self-reports, which lowers vendor self-report risk. It is not an independent third-party replication.
• The task set spans cell-identity, tissue-structure and perturbation questions, and the authors state explicitly that evaluation should go beyond scale and leaderboard position.

Limitations.

• Preprint, not peer reviewed. Methods reproducibility (preprocessing pipeline, metric definitions, data splits) is unverified because the full text was not read.
• Curated datasets. The results are metrics on selected benchmark data. They are not evidence about clinical diagnosis or target discovery.
• Harmonisation unknown. How differences in tokenisation were absorbed, and how leakage between pretraining corpora and test data was controlled, are not visible in the abstract. Benchmark contamination cannot be assessed.
• No simple-baseline comparison reported in the abstract. See above; this limits what can be inferred about the FM approach as a whole.
• Code and data availability. Not stated in the abstract or on the abstract page (unverified). For a benchmark paper, a closed evaluation harness would make the conclusions hard to check.
• Interpretation fork. The result could be a finding of evaluation instability or a limitation of the harmonisation. Our reading is that the two are not yet distinguishable.

4. Connections to neighboring domains

General LLM benchmarking. Rankings that flip with prompt format, evaluation harness or metric are a familiar problem for general-purpose language-model leaderboards, including benchmark saturation and contamination debates. The structure is similar: the winner depends on how the test is run. The bio setting adds domain-specific axes, namely biological prior knowledge and perturbation ground truth, which general LLM benchmarks lack. We consider this analogy structural rather than evidential; it adds little new information beyond the paper’s own argument.

AI-enabled drug discovery. If it is still unsettled which architecture suits which downstream task, that is one upstream reason why data-infrastructure-layer tools have not yet translated into pipeline-level outcomes. This is an indirect inference and a weak link.

5. Commercialization and investment perspective

TRL 3. The paper is evaluation infrastructure at the methods-validation stage. The six models are mostly research artifacts released for academic use (code and weight availability for each was not verified here). Some of the scGPT family are reused across institutions and may be closer to TRL 4, but this preprint provides no evidence for raising that level. If anything it argues against general-purpose trust in any single model.

Commercial timeline. The benchmark does not point to a specific product. Single-cell and spatial platform vendors and bio-FM startups could plausibly use such comparisons when choosing models; no listed or private company is named in the paper, and we found no company link in the source. We draw no investment inference.

6. Counterarguments

• The missing baseline. The abstract compares FMs with FMs. Without PCA or linear baselines, one cannot say whether the FMs as a class add value on these tasks. Readers should not convert “no model dominated” into either “FMs are overhyped” or “FMs are validated”.
• Finding or artifact. The methods section decides whether changing rankings are a real property of the models or a result of imperfect harmonisation. If fixing preprocessing and tokenisation stabilised the rankings, the conclusion would be reattributed to harmonisation failure rather than evaluation instability. We have not read the methods.
• No reproducibility evidence. Code, data and harness availability are unconfirmed, the work is not peer reviewed, and no independent replication exists. This reflects the state of the paper, not a defect of any particular reading.
• Affiliations unverified. The abstract page lists 15 authors but does not show affiliations. Whether any author overlaps with the teams that built the six models is unchecked. The risk direction appears benign, since the authors ran all models themselves, but this is not confirmed.

7. Indicators to watch

• The full text: preprocessing and tokenisation unification, data splits, metric definitions, and whether non-FM baselines such as PCA or linear models appear.
• Whether rankings remain unstable when preprocessing and tokenisation are fixed, which would separate evaluation instability from harmonisation failure.
• Release of the evaluation code, data and harness, and any independent re-run.
• Peer-reviewed publication and whether the abstract’s wording changes.
• Follow-up benchmarks that use held-out cohorts or perturbation-grounded ground truth, which the authors themselves propose as the right yardstick.

References

• Chen, S., R. Zahedi, L. Chhuo, et al. (15 authors). 2026. “Harmonised benchmarking of foundation models for single-cell and spatial transcriptomics reveals context-dependent generalisation.” arXiv:2607.17227, submitted July 19, 2026 (preprint, not peer reviewed; no DOI assigned). https://arxiv.org/abs/2607.17227
• The original papers describing the six evaluated models (Nicheformer, CellPLM, scGPT-spatial, GenePT, scELMo, Novae) are not itemised here because their bibliographic details were not verified in the underlying analysis; descriptions of them above rely on public summaries and were not re-checked.

Disclosure

• This article is for informational purposes only and is not investment advice.
• The author holds no position in any company named in this post.
• Conflict-of-interest notes:
  • The preprint lists 15 authors; affiliations are not shown on the arXiv abstract page and are therefore unverified.
  • Whether the authors overlap with the development teams of the six evaluated models (Nicheformer, CellPLM, scGPT-spatial, GenePT, scELMo, Novae) was not checked, so any incentive to favour or disfavour a particular model cannot be assessed.
  • The quantitative benchmark results are the authors’ preprint claims from their own runs, not developer-reported figures and not an independent replication.
  • No listed company is connected to the paper in the material we reviewed.
• Our reading was limited to the abstract and the abstract page; the full text was not read.
• The figure in this article is an original drawing; no figure from the source was reproduced.