Uthando Nethemba

Protein foundation models (AlphaFold3, ESM3): does beating the specialist baseline mean generalization — or memorization?

—

by

itembalabantu
in

Evidence-first notes on bioscience and deep tech, at the edge of the lab and the market. Information only — not investment advice.

The 30-second version

  • What. On FoldBench, a leakage-controlled independent benchmark, AlphaFold3 (AF3) leads on most tasks — but its score splits sharply by axis. It is robust on protein monomers (LDDT 0.88) and protein–protein interfaces (DockQ success 72.9%), yet on antibody–antigen it reaches only 47.9% and on nucleic acids it collapses (DNA LDDT 0.53, RNA 0.61), where any baseline advantage disappears.
  • So what. Where AF3 does lead on protein–ligand (64.9%), the lead is partly explained by memorization, not generalization: FoldBench finds accuracy rises with ligand similarity to the training set. A ~100-line open-source strong baseline (ligand-ensemble + Gnina rescoring) beats blind AF3 by +4.2 percentage points — so the docking “win” partly reflects a weak-baseline artifact, not a robust one (AF3 still leads by 7.1 points when given pocket information).
  • Now what. Data leakage is quantified, not merely suspected: decontaminating the benchmark shifts the optimal pLDDT confidence threshold from 0.89 to 0.69 (KS-test P<0.0001). ESM3’s headline design, esmGFP, is a proof-of-concept, not proven de novo generalization — its design pipeline leaned on existing crystal structures (1QY3/1EMA) and the protein matures 1 day–1 week versus minutes for standard GFP. The metric to watch is not “does the model predict everything” but performance under similarity/leakage control, per axis.

[demo-gap note] A benchmark number (64.9%) and real-world trust on a novel target in blind docking are different things. esmGFP is a single wet-lab proof-of-concept — one bright well (C10) reached after a first-round design (well B8) that was ~50× dimmer than natural GFP — and does not establish that the model beats directed-evolution/Rosetta design in general.

The five-minute read

Two questions, kept separate: does it win, and why does it win?

The protein foundation-model story usually stops at “does AF3 beat the specialist tools?” One layer in, two questions must be separated. (1) Does it win? On FoldBench — a benchmark curated from post-2023 PDB entries with low homology to the training set — AF3 does lead on most tasks. That much is confirmed. (2) Why does it win? The lead splits by axis, and where AF3 leads on protein–ligand, accuracy rises the more the ligand resembles the training set. The through-line: beating a baseline is only true per axis and per condition, and the margin wobbles once leakage is controlled.

The score splits by axis

“AF3 predicts every molecule” holds only for monomers and protein–protein interfaces (the mature AF2-lineage territory). On antibody–antigen it reaches under half (47.9%; an independent mAbs 2025 study reaches the same “>50% failure rate” conclusion), and on nucleic acids it drops to LDDT 0.53 (DNA) and 0.61 (RNA). Nor is there a single all-conquering model: HelixFold3 leads AF3 on protein–peptide (89.5%). The headline of universal dominance does not survive contact with the per-axis table.

Where it does lead, memorization is doing part of the work

Two lines of evidence erode the protein–ligand “lead” (64.9% vs. Boltz-1 ~55%). First, ligand-similarity dependence: FoldBench reports that higher similarity to training ligands tracks better modeling — and success collapses for unseen (low-Tanimoto) ligands across all models. That is training-set proximity, not generalization into new chemical space. Second, a ~100-line open-source strong baseline (Inductive Bio): a ligand conformation ensemble plus Gnina (CNN) rescoring raised PoseBusters success from 59.7% to 79.5%, beating blind/standard AF3 by +4.2 points (though trailing pocket-informed AF3 by 7.1). The implication is that AF3’s docking margin was inflated by comparison against a weak out-of-box baseline (AutoDock Vina).

[diagram: AF3 wins by axis — robust vs. eroded]

  FoldBench (post-2023, low-homology, leakage-controlled)
  protein monomer    ██████████████████ LDDT 0.88   ROBUST
  protein-protein    ███████████████    DockQ 72.9%  ROBUST
  protein-ligand     █████████████      64.9%        LEAD, but memorization + strong-baseline caveat
  antibody-antigen   █████████          47.9%        NO WIN (>50% failure)
  DNA monomer        ██████             LDDT 0.53    COLLAPSE
  RNA monomer        ███████            LDDT 0.61    COLLAPSE
  ──────────────────────────────────────────────────────────
  Leakage is quantified: decontamination shifts optimal
  pLDDT threshold 0.89 -> 0.69 (KS-test P<0.0001)
Beating the baseline is conditional. AF3 is robust on monomer/PPI, leads-but-contested on protein–ligand (ligand-similarity memorization; a ~100-line strong baseline beats blind AF3 by +4.2 pts), and shows no durable advantage on antibody or nucleic acids. Redrawn from FoldBench (Nat Commun 2025) figures; no source figures reproduced.

Deep dive

1. Background — the real question for protein foundation models

This part fixes axis A (protein) of the bio-foundation-models series through three-vote adversarial verification. The animating question is not whether AF3 and ESM3 are impressive, but whether they beat domain-specialist baselines (docking programs, alignment-based predictors) — and if so, whether that win is generalization to novel targets or memorization of the training set. The through-line — headline capability ≠ generalization — is the same discipline this series applies to genomics and cell models in later parts.

2. What this deep dive newly establishes

Core answer: AF3 does lead on most FoldBench tasks, but the lead is robust only on monomer/PPI; on protein–ligand it is partly a memorization/weak-baseline effect, and on antibody and nucleic acids the baseline advantage is gone.

  • Per-axis split on an independent, leakage-controlled benchmark — CONFIRMED (independent/high): FoldBench (bioRxiv 2025.05.22.655600 → Nat Commun 2025-12-04; BEAM-Labs, data/eval/predictions public) covers 1,522 biological assemblies from post-2023 PDB, curated low-homology to AF3’s training set. AF3 leads across five predictors (Boltz-1, Chai-1, HelixFold3, Protenix), but scores range from LDDT 0.88 (monomer) to 0.53 (DNA).
  • Memorization signal on protein–ligand — CONFIRMED (independent/high): FoldBench finds “higher similarity → better ligand modeling,” with success collapsing for low-Tanimoto ligands across all models. The protein–ligand lead is substantially explained by training-set proximity, not new-chemistry generalization.
  • A ~100-line strong baseline beats blind AF3 — CONFIRMED (blog/medium, code public): Inductive Bio’s ligand-ensemble + Gnina rescoring raised PoseBusters success 59.7% → 79.5%, +4.2 pts over blind/standard AF3 (−7.1 pts vs. pocket-informed AF3). Attributed to the authors; reproducible via public GitHub.
  • Leakage is quantified — CONFIRMED (Brief Bioinform 2025, bbaf104): on a decontaminated set (BETA), the optimal pLDDT threshold for intrinsically disordered region prediction shifts 0.89 → 0.69 (KS-test P<0.0001). The “confidence” baseline in contaminated benchmarks was itself inflated.
  • esmGFP is a proof-of-concept, not proven generalization — CONFIRMED as PoC (primary/high + critique/medium): ESM3 (Science 2025, ads0018; bioRxiv 2024.07.01.600583) generated a GFP 58% sequence-divergent from the nearest natural fluorescent protein, via a first round of 96 designs (well B8, ~50× dimmer than natural GFP, delayed maturation) then 96 more (well C10 = esmGFP). Critiques note the design pipeline’s three filters relied on existing GFP crystal structures (1QY3/1EMA), both esmGFP and B8 take 1 day–1 week to mature (vs. minutes for standard GFP), and generated sequences partly overlap natural ones (Naughton).

3. Strengths and limits of the methodology

Strengths: FoldBench isolates leakage by construction (post-cutoff, low-homology) and publishes data, evaluation and predictions, so its per-axis and similarity findings are checkable. The Inductive Bio baseline is open-source. Limits: what is confirmed is mostly “AF3 leads on this benchmark,” not that the lead reflects transferable skill. The memorization signal is a correlation (similarity ↔ accuracy), and stratified controls are still needed to size the effect. Vendor framings around ESM3 — “1 trillion teraflops” and “500 million years of evolution” — are [unverified vendor] marketing conversions (the former is not a standard unit; the latter assumes a uniform mutation rate over 58% divergence) and are isolated from the technical claims. ESM3’s compute (1.07×10²⁴ FLOPs, 2.78×10⁸ proteins, 98B params / public 1.4B–7B variants) is reported by the vendor.

4. Neighbouring domains

Three hooks carry into the genomics axis (Part 2). (1) Leakage-controlled benchmarking: FoldBench’s post-cutoff/low-homology principle should apply to zero-shot variant-effect evaluation (e.g., Evo2). (2) Generation-leakage: just as esmGFP’s design leaned on existing structures, “de novo” claims for genome generative models need training-proximity control. (3) Similarity = memorization: AF3’s ligand-similarity pattern becomes a lens to test whether genome models are strong only near their training species/regions.

5. Commercialization and market context (TRL, companies)

Sub-axisHeadline metricSpecialist baseline beaten?Real bottleneck / weaknessEvidence tier
protein monomer / PPILDDT 0.88 · DockQ 72.9%YesMature AF2-lineage territory; little disputeindependent/high
protein–ligand64.9% (Boltz-1 ~55%)Partly (unstable)Ligand-similarity memorization; ~100-line strong baseline beats blind AF3independent/high + blog/med
antibody–antigen47.9%No (effectively unsolved)>50% failure rate; baseline advantage goneindependent/high
nucleic acids (DNA/RNA)LDDT 0.53 / 0.61NoSharp drop vs. monomerindependent/high
design (ESM3)esmGFP 58% divergentUndeterminedDesign-leakage (1QY3/1EMA); maturation 1 day–1 week; sequence overlapprimary/high + critique/med

Company context (factual, neutral, no buy/sell implication): the AF3 engine is the drug-design core of Isomorphic Labs (private, an Alphabet company; publicly reported partnerships with Lilly and Novartis). ESM3 is from EvolutionaryScale (private; open 1.4B/7B weights plus API, with an AWS partnership). The two distribution models — a closed commercial engine versus open-weight-plus-API — sit in contrast. Any equity implication belongs to the investment layer; here only facts are stated, and the conditional nature of “AF3 beats the baseline” (memorization, per-axis variance) is kept separate from the headline.

6. The skeptic’s bottom line

  • Adversarial verification complete (17 confirmed / 1 corrected / 3 unverified-vendor), but no verified-clean item. There is no confirmed case of a flawless “generalization advantage” on a forward-looking, leakage-controlled benchmark. Misreading warning: “AF3/ESM3 beat the baseline across proteins” is true only per axis and per condition, and the margin wobbles once leakage is controlled.
  • demo-gap: A benchmark score is not blind real-world trust; esmGFP is a single wet-lab proof-of-concept with a maturation defect and structure-dependent design filters, not proof of de novo generalization.
  • Memorization vs. generalization: protein–ligand accuracy tracks ligand similarity to training data, and a ~100-line strong baseline beats blind AF3 — so part of the “win” is data proximity and weak-baseline comparison, not transferable skill.
  • hold-level claims: “AF3/ESM3 beat the baseline across all of protein space” and “esmGFP proves de novo generalization” should be withheld and escalated while antibody/nucleic-acid collapse and design-leakage stand.

7. What to watch (falsifiable predictions)

  1. If ligand-similarity is stratified and controlled, AF3’s protein–ligand margin should shrink significantly (the 64.9% lead narrowing to <5 pts over Boltz-1 in the low-Tanimoto stratum) — testable via FoldBench stratified follow-ups and strong-baseline reproduction.
  2. The Inductive Bio strong baseline (59.7 → 79.5) should reproduce independently, reclassifying the AF3 docking “win” narrative as a weak-baseline artifact — falsifiable by re-running the public GitHub code.
  3. Within 12–18 months, ESM3-family design should not significantly beat a directed-evolution baseline in published evaluations under a predicted-structure-only condition (no crystal-structure filters) — falsifiable by follow-up wet-lab redesign papers.

References

Source knowledge asset: knowledge-base/deep-dives/bio-foundation-models/part1-protein-fm.md (generated 2026-07-12, VERIFIED — 17 confirmed / 1 corrected / 3 unverified-vendor). This draft inherits the figures and source attributions of the original part and creates no new figures or sources.

Disclosure

This post is for information only and is not investment advice. The author holds no position in, and no financial interest in, the companies mentioned.

COI note: This document describes listed companies (Alphabet/DeepMind, NVIDIA, Amazon/AWS) and private ones (Isomorphic Labs, EvolutionaryScale) in a factual, neutral technology/roadmap context. There is no buy/sell implication. Vendor quantitative and marketing expressions — ESM3’s “1 trillion teraflops” and “500 million years of evolution” — are isolated as unverified vendor framings, and all reported model/eval numbers are attributed to their vendor, author or preprint source.

Comments

Leave a comment