The memory wall and advanced packaging: the new Moore’s Law is bandwidth, not transistors

Evidence-first notes on bioscience and deep tech, at the edge of the lab and the market. Information only — not investment advice.

The 30-second version

• What. Inside the AI accelerator, the rate-limiting step is no longer logic (FLOPs) but memory bandwidth and the packaging that attaches it. If Moore’s Law was a transistor-density game, the real scaling axis in 2026 is the HBM stack, hybrid bonding and CoWoS. Adversarial verification confirms the core numbers (HBM4’s 2,048-bit interface, ~2 TB/s per stack, a logic base die) at 3-0.
• So what. Headlines still read “N-PFLOPs accelerator,” but in real service (LLM decoding) the limiter is bandwidth, not FLOPs. The genuine constraint is not the bandwidth number — it is thermals, yield and packaging capacity. Verification killed three vendor-timeline overstatements (mass production is not “development complete”).
• Now what. HBM4 arrives with Nvidia Rubin / AMD MI400 in 2026 (SK Hynix mass-production target Q3 2026 — a vendor milestone, not shipping today), and CoWoS is the gating bottleneck for AI-accelerator supply. The metric to watch is not “TB/s” but stack-height thermals, KGD yield economics, and packaging capacity (CoPoS 2028–29). Verification grade: VERIFIED-partial.

[demo-gap note] CXL memory pooling, co-packaged optics (CPO) and analog compute-in-memory (CIM) are largely at the pilot / roadmap stage, not large-scale deployed proof. Digital compute-near-memory (CNM) is commercialized (UPMEM, Samsung, SK Hynix); analog CIM remains research-only. Vendor “development complete” is not mass production.

The five-minute read

The scaling axis moved from transistors to bandwidth and packaging

If Part 1 argued that “the bottleneck of AI compute is power, not the chip,” Part 2’s thesis sits one layer inward: even inside the chip, the bottleneck is not logic (FLOPs) but memory bandwidth and the packaging that binds it. The headline is still “N-PFLOPs accelerator,” but the rate-limiting step for real-world performance is the bandwidth, packaging capacity and heat next to it. This follows directly from the Part 0 result, confirmed under adversarial verification, that moving data costs 10–100× more energy than a logic operation.

The memory wall, quantified

The bottleneck of LLM inference flips stage to stage (CONFIRMED 3-0, arXiv:2402.16363). The prefill stage is compute-bound (arithmetic intensity ~1,024–1,215), but the decode (token-generation) stage is memory-bound (AI ~1.0), sitting far below the GPU’s compute ceiling — because every token reloads the decoder weights from memory. In real service (decoding), the limiter is bandwidth, not FLOPs. On a roofline view, low-arithmetic-intensity workloads are trapped in the bandwidth-sloped region, and LLM decoding (per-token KV-cache and weight reads) is the archetype.

HBM and packaging as the real scaling game

HBM4 doubles the interface to 2,048-bit (versus HBM3E’s 1,024-bit) for an official maximum of ~2 TB/s per stack (JEDEC, 2025-04). Structurally, the base die shifts from a memory node to a foundry logic node, turning the memory stack into a de-facto co-processor and dissolving the memory/logic/packaging boundary. In parallel, advanced packaging (CoWoS, hybrid bonding, chiplets/UCIe) becomes the new scaling axis — and its physical ceilings (interposer/wafer utilization, reticle limits) are what actually cap throughput.

[diagram: where the bottleneck actually sits]

  LLM inference, by stage
  Prefill  ██████████████████  compute-bound  (AI ~1,024-1,215)
  Decode   ██                  memory-bound   (AI ~1.0)    only ~4-7 large GPUs  -> CoPoS panel workaround (2028-29)

Deep dive

1. Background — the scaling axis moved inward

Where Part 0 §2 planted the seed of “the memory wall,” this part deepens it. The confirmed anchors it inherits: (a) logic↔memory data movement costs 10–100× the energy of a logic operation (Horowitz-lineage data reaches 100–1000×); (b) HBM4’s 2,048-bit interface yields ~2 TB/s per stack; (c) digital CNM is commercialized (UPMEM, Samsung, SK Hynix) while analog CIM is research-only; (d) the true bottleneck of CIM/CNM is software and programmability. The through-line — “the bottleneck is somewhere else” (W28-CONV1) — recurs one layer deeper, inside memory and packaging.

2. What this deep dive newly establishes

Core answer: the scaling axis has moved from transistor density to bandwidth and packaging, and the genuine constraint is not the bandwidth number but thermals, yield and capacity.

• HBM4 doubles the interface — CONFIRMED (3-0): 2,048-bit (2× HBM3E’s 1,024-bit), official max ~2 TB/s per stack (JEDEC, 2025-04) vs. HBM3E’s ~1–1.2 TB/s. SK Hynix unveiled 16-Hi / 48 GB / 10 GT/s → effective ~2.56 TB/s per stack at CES 2026 (above baseline; CONFIRMED 3-0). Micron: 12-layer / 36 GB / 2,048-bit (CONFIRMED 2-0).
• The base die shifts from a memory node to a logic node — CONFIRMED (3-0): HBM4/4E build the base die on a foundry logic process (SK Hynix = TSMC 12FFC+/N5; Samsung = in-house foundry). The memory stack effectively becomes a co-processor; the memory/logic/packaging boundary dissolves.
• Timeline — CONFIRMED (3-0): HBM4 arrives in 2026 (with Nvidia Rubin and AMD MI400; partner samples 2025 Q3–Q4), with SK Hynix’s mass-production target at Q3 2026 (today it is “development complete,” not mass production — a vendor milestone). HBM4E lands end-2027 (Micron roadmap). Three suppliers (SK Hynix, Samsung, Micron) supply essentially the entire market — an oligopoly tied to the geopolitics of Part 0 §5.
• Advanced packaging is the new scaling axis — CONFIRMED (3-0): with Rubin-class reticles (~5.5×), a round 12-inch wafer holds only ~7 (sometimes 4) large GPUs, forcing a move to panel-level packaging (CoPoS, Chip-on-Panel-on-Substrate). TSMC’s CoPoS pilot line: equipment move-in 2026-02, completion 2026-06, mass-production ramp 2028–29 (CONFIRMED 3-0). Angular panels are a step-change in utilization and throughput.
• CoWoS is the gating bottleneck for AI-accelerator supply — CONFIRMED (3-0): as of late 2025, TSMC was still “narrowing” the supply gap, with no capacity-balance date given. The “130k CoWoS wafers/month target (end-2026, ~4×)” is a vendor-target claim [unadjudicated].

3. Strengths and limits of the methodology

Strengths: the core quantitative claims (HBM4 2,048-bit / ~2 TB/s, SK Hynix 16-Hi 48 GB, logic base die, 2026 arrival / Q3 2026 mass production, CoWoS gating, CoPoS 2028–29, decode memory-bound) all cleared adversarial verification at 3-0, and vendor claims are attributed as such. Limits: adversarial verification killed three vendor-timeline overstatements — (1) “HBM4 keeps HBM3E-class pin speed (~8 Gb/s) and gains bandwidth only via bus width” REFUTED (0-3; actual 10 GT/s, so pin speed rose too); (2) “HBM4 mass-production shipping starts 2025 Q4” REFUTED (0-3; actual mass production Q3 2026); (3) “6.40 GT/s (2024-07 JEDEC preliminary spec)” REFUTED (0-3; superseded by a final 10 GT/s). The real constraint is not the bandwidth number but yield, heat and supply — the taller the stack, the worse the known-good-die (KGD) economics. Six items remain unadjudicated (base-die logic-node detail, TSMC 130k CoWoS/month, a 3.3 TB/s advanced config) but overlap heavily with confirmed items, so the real gap is small. Synthesis was skipped only due to a session limit.

4. Neighbouring domains

No forced hook (in honesty). The memory wall and packaging connect weakly to CKM (cardio-renal-metabolic). The only non-forced link: AI-driven biology inference (protein and genome foundation models) is itself memory-bandwidth-bound, so the real compute cost and latency of bio-AI are also governed by HBM bandwidth, not FLOPs. On top of Part 0’s “the bio bottleneck is data and validation,” this adds a meta-consistency — the bio-AI inference bottleneck is also memory. It is not grafted onto the CKM axis.

5. Commercialization and market context (TRL, companies)

Company context (factual, neutral, no buy/sell implication): HBM is supplied by SK Hynix (000660.KS), Samsung (005930.KS) and Micron (MU); advanced packaging is concentrated in TSMC (TSM). Accelerators and interconnect/retimer roles involve Nvidia (NVDA), Broadcom (AVGO), Marvell (MRVL) and Astera Labs (ALAB), among others; digital CNM is commercialized at UPMEM, Samsung and SK Hynix, while analog CIM remains research-only. No single-winner conclusion is asserted; vendor roadmap dates, capacities and yield figures are all attributed as vendor/analyst claims.

6. The skeptic’s bottom line

• Adversarial verification complete — three vendor-timeline overstatements killed (§2): “HBM4 mass production 2025 Q4” (actual Q3 2026), “pin speed HBM3E-class 8 Gb/s” (actual 10 GT/s), “6.4 GT/s preliminary spec” (superseded by final 10) — all REFUTED. Vendor “development complete” is explicitly not mass production. The TSMC 130k CoWoS/month figure is retained as an unadjudicated vendor target.
• demo-gap: CXL pooling, CPO and analog CIM are largely pilot/roadmap, not large-scale deployed proof.
• Real bottleneck re-placed: behind “N TB/s / N PFLOPs” headlines, the actual rate-limiter is packaging capacity, heat, yield and the software ecosystem. Computing’s “the bottleneck is somewhere else” repeats within the memory and packaging layer.

7. What to watch (falsifiable predictions)

1. If most of the real-performance gain of next-generation large accelerators comes from HBM bandwidth and packaging rather than the logic node, this part’s thesis is reinforced; if node-shrink alone drives a real-performance jump, it is weakened.
2. If HBM stack height (16 → 20-Hi) stalls at a specific count due to heat and yield, “the bottleneck is physics (thermal/yield)” is confirmed.
3. If CoWoS capacity expansion actually sets the ceiling on 2026–27 AI-accelerator shipments, “packaging is the physical bottleneck” is confirmed.

References

• Chen, Tianqi, et al. 2024. “LLM inference roofline (prefill/decode arithmetic intensity).” arXiv:2402.16363. https://arxiv.org/abs/2402.16363
• Tom’s Hardware. 2026. “SK Hynix 16-Hi HBM4 48 GB; HBM4 development complete.” https://www.tomshardware.com/
• JEDEC. 2025. “High Bandwidth Memory (HBM4) DRAM standard (2,048-bit, ~2 TB/s/stack).” https://www.jedec.org/
• Wikipedia. “High Bandwidth Memory (HBM4 spec).” https://en.wikipedia.org/wiki/High_Bandwidth_Memory
• TrendForce. 2026. “TSMC CoPoS panel-level packaging pilot line.” https://www.trendforce.com/
• fusionww. “CoWoS / HBM as the AI supply bottleneck (2027).” (blog)

Source knowledge asset: knowledge-base/deep-dives/computing-power/part2-memory-wall-packaging.md (generated 2026-07-08, VERIFIED-partial: 16 confirmed / 3 refuted / 6 unadjudicated; synthesis skipped on session limit). This draft inherits the figures and source attributions of the original part and creates no new figures or sources.

Disclosure

This post is for information only and is not investment advice. The author holds no position in, and no financial interest in, the listed companies mentioned (SK Hynix 000660.KS, Samsung 005930.KS, Micron MU, TSMC TSM, Nvidia NVDA, Broadcom AVGO, Marvell MRVL, Astera Labs ALAB, and others).

COI note: This document describes multiple listed companies (SK Hynix, Samsung, Micron, TSMC, Nvidia, Broadcom, Marvell, Astera Labs and others) and private ones (UPMEM and others) in a factual, neutral technology/materials context. There is no buy/sell implication. Vendor roadmap dates, capacities and yield figures are all attributed as “vendor/analyst claims (projections),” with vendor “development complete ≠ mass production” noted and three vendor-timeline overstatements explicitly killed under adversarial verification.