All podcasts / Dwarkesh Podcast / Summary

How GPT, Claude, and Gemini are actually trained and served – Reiner Pope

2026-04-29 - 125 min - source - Read full transcript

Dwarkesh Patel (host)Reiner Pope

Key insights

Batch size is the single dominant lever behind API 'Fast Mode' pricing, and running unbatched can be 1000x more expensive.

Pope shows that weight-fetch time is fixed per forward pass regardless of how many users share it, so batching many requests together amortizes that fixed cost across far more tokens. Without batching, the cost and latency economics can be roughly a thousand times worse than with it, which is why providers push heavily toward large concurrent batches rather than serving users one at a time.

inference-economics

There is a hardware-derived optimal batch size of roughly 300 times the model's sparsity ratio, where compute time equals memory time.

Setting the weight-fetch memory time equal to the compute time and solving algebraically yields a dimensionless hardware constant near 300 (driven by FLOPs-per-byte on current GPUs), multiplied by the fraction of experts activated per token. For DeepSeek's roughly 1-in-8 activation ratio this lands near 2,000-3,000 tokens per batch, which matches real-world serving configurations.

inference-economics

The optimal batch size depends only on the model's sparsity ratio, not on overall model scale.

Because the balance-point derivation cancels out the number of active and total parameters except through their ratio, a much larger or smaller model with the same sparsity ratio needs essentially the same batch size to hit peak efficiency. This means centralization pressure from batching economies is weaker than it might seem, since the required batch (a few thousand concurrent sequences) is small relative to the traffic frontier labs actually see.

mixture-of-experts

Mixture-of-experts layers are sharded across GPUs within a single rack because the all-to-all communication pattern matches the scale-up network's full connectivity.

Every GPU can end up sending tokens to every other GPU depending on routing decisions, which is a perfect fit for a rack's internal switch-based full connectivity. Splitting experts across two racks forces roughly half the traffic onto the slower scale-out network, which Pope estimates as about 8x slower bandwidth than the in-rack scale-up network, making rack size a hard ceiling on how large an MoE layer can practically be.

mixture-of-experts

Pipeline parallelism shrinks the weight-memory footprint per GPU but does not reduce the KV cache footprint per GPU.

Because avoiding pipeline bubbles requires running as many concurrent micro-batches as there are pipeline stages, the reduction in stored KV cache per stage is exactly offset by needing that many more sequences in flight simultaneously. The savings on weights are real and useful, but pipelining is not a solution to the KV-cache-driven memory wall.

gpu-cluster-architecture

Larger GPU scale-up domains (Nvidia's jump from 8-GPU Hopper trays to 72-GPU Blackwell racks to ~500+ GPU Rubin) matter mainly for memory bandwidth, not memory capacity.

Pipelining already solves the memory-capacity problem for model weights by letting layers spread across racks, so the real driver of building bigger scale-up domains is aggregating more GPUs' memory bandwidth in parallel to load weights faster, which directly lowers decode latency and lets larger, sparser models run at usable speed.

gpu-cluster-architecture

Frontier models are likely being trained on roughly 100 times more tokens than Chinchilla-optimal once RL post-training and inference-serving costs are accounted for.

Pope argues the true optimization target isn't training compute alone but the sum of pre-training, RL, and inference cost, which tends to be minimized when those three costs are roughly equalized. Working backward from estimated global inference token throughput (hundreds of millions of tokens per second) and a model's typical two-month deployment lifetime yields a pre-training token count on the order of 100-200 trillion tokens, versus a Chinchilla-optimal estimate near 2 trillion for a ~100B active-parameter model.

scaling-laws

API pricing structure leaks real infrastructure detail, including implied KV cache size and whether decode is memory-bound.

Gemini's price jump above 200k-token contexts likely marks the crossover point where KV cache memory time overtakes weight-fetch time; working the roofline equations backward from that crossover yields an implied 1.5-2 kilobytes per token of KV cache, consistent with real dense-attention configurations. Separately, output tokens costing roughly 5x more than input tokens indicates decode is heavily memory-bandwidth-bound while prefill is compute-bound, since prefill amortizes memory time over many tokens processed in parallel.

inference-economics

Context length has plateaued around 100-200K tokens for the last year or two because KV cache memory-bandwidth cost, not compute cost, is the binding constraint.

The compute cost of attention grows only quadratically with a very shallow slope that only matters in the millions-of-tokens range, but memory-bandwidth cost to fetch the KV cache grows linearly and dominates well before that. Sparse attention offers a square-root-scale improvement but not an unlimited one, since attending to too small a fraction of tokens degrades model quality, which Pope argues is why nobody has pushed dramatically past current context lengths despite the technical possibility.

scaling-laws

Parallelism strategy in practice mirrors the model's own architecture rather than using generic decomposition techniques.

Engineers cut the model along the same axes it is already scaled along - experts go on different GPUs (expert parallelism), layers go on different racks (pipeline parallelism) - rather than using more generic approaches like tensor parallelism, which has become unprofitable now that individual experts are small. Pope frames this as 'the cutting matches the model architecture': any dimension a model is scaled along becomes a natural axis to shard along once it's large enough to be worth the communication cost.

gpu-cluster-architecture

Neural networks and cryptographic ciphers converged on similar mixing/scrambling architectures despite pursuing opposite goals, and a specific cipher construction was directly imported into deep learning.

Ciphers try to make structured input indistinguishable from randomness, while neural nets try to extract structure from apparently random data, yet both rely on mixing every input with every other input. The Feistel cipher construction, which makes a non-invertible function f invertible by carrying forward both inputs, was ported into 2017's RevNets to make entire transformer layers invertible, letting training rematerialize activations during the backward pass instead of storing them in HBM, trading more compute for less memory.

mixture-of-experts

Media referenced

Unified Scaling Laws for Routed Language Models - paper - Pulled up mid-conversation to show empirically how model quality changes as mixture-of-experts sparsity increases at fixed active-parameter count.
GShard - paper - Cited as an earlier mixture-of-experts technique that predates DeepSeek's approach.
Switch Transformer - paper - Cited alongside GShard as an older MoE technique whose results don't fully generalize to modern sparsity levels.
RevNets (reversible networks, 2017 paper) - paper - The paper that ported the cryptographic Feistel-cipher construction into neural networks to make transformer layers invertible and avoid storing activations for the backward pass.
Reiner Pope's blog post on cryptography and neural network convergent evolution - article - The starting point for the closing discussion of how cipher design and neural net design converge on similar mixing/scrambling architectures for opposite purposes.
Character AI blog post on alternating long/short context attention - article - Cited as a real-world example of dense attention with KV cache reuse across layers, used to sanity-check the bytes-per-token estimate derived from Gemini's pricing.
Dwarkesh's prior podcast interview with Dario Amodei - podcast - Referenced for Amodei's claim that in-context learning, not continual learning, may be sufficient for AGI - which Dwarkesh uses to ask whether infra could ever support 100-million-token context windows.

Companies

MatX - Reiner Pope is CEO of this AI chip startup; Dwarkesh discloses he is an angel investor in it.
Google - Pope previously worked on TPU architecture there; Google's scale-up domains are cited as historically larger than Nvidia's.
Nvidia - Its Blackwell NVL72 rack (72 GPUs) and upcoming Rubin generation (~500+ GPU scale-up) are the reference hardware for the whole roofline analysis.
DeepSeek - DeepSeek V3's 37B active / 700B total parameters and its published sparse-attention and MoE routing techniques are used as running examples throughout.
OpenAI - GPT-4's release timing and rumored >1 trillion parameter count anchor the discussion of why model sizes stalled and then jumped.
Google DeepMind / Gemini - Gemini's tiered API pricing (context length and cache-hit discounts) is reverse-engineered to infer KV cache byte size and memory-bandwidth bottlenecks.
Anthropic / Claude - Claude's Fast Mode/Slow Mode pricing motivates the opening question about why latency and cost trade off the way they do.
Character AI - Cited for its dense-attention architecture with shared KV cache across layers, also used in Gemma models.
Jane Street - Dwarkesh mentions a conversation with a GPU performance engineer there about why pipelining requires micro-batching.

Techniques and frameworks

Roofline analysis - The core analytical method used throughout: modeling inference time as the max of memory-fetch time and compute time to predict latency and cost.
Batching (continuous/dynamic batching) - Grouping concurrent user requests so weight-fetch cost is amortized across many tokens; the single biggest lever on cost and latency.
KV cache - The stored internal representation of past tokens that each new token attends to during decode; dominates memory bandwidth and capacity demands.
Expert parallelism - Sharding different MoE experts across GPUs within one rack, matched to the all-to-all communication pattern of routing.
Pipeline parallelism - Splitting model layers across racks; reduces per-GPU weight memory footprint but does not reduce KV cache footprint per GPU.
Tensor parallelism - Older technique of splitting computation within a single expert/layer; described as no longer profitable now that experts are small.
Sparse attention - Attending to only a subset of past tokens to reduce the memory-bandwidth scaling of context length; discussed as a partial but not complete fix for the memory wall.
Chinchilla scaling laws - The baseline for compute-optimal training token counts, used to estimate that frontier models are now trained on roughly 100x the Chinchilla-optimal token count once RL and inference costs are amortized in.
Feistel cipher / RevNets (reversible networks) - A cryptographic invertibility construction imported into neural nets so that transformer layers can be run backward and activations rematerialized instead of stored, saving training memory.
Rematerialization - Recomputing a KV cache from scratch from token IDs rather than storing it, one of several memory-tier tradeoffs (HBM vs. DDR vs. flash vs. disk) analyzed via API cache-pricing.

Summary

Dwarkesh Patel interviews Reiner Pope, CEO of chip startup MatX and formerly a TPU architect at Google, in a new "blackboard lecture" format built around first-principles derivation rather than free-form conversation. The episode opens with a concrete question - why does Claude's Fast Mode cost 6x more for 2.5x the speed - and uses it to motivate a full roofline analysis of transformer inference: modeling total time as the maximum of memory-fetch time (loading model weights plus the KV cache) and compute time (multiplying activations by active parameters). From this simple model, Pope derives that batch size is the dominant lever on both cost and latency, that there is a hardware-determined optimal batch size of roughly 300 times a model's sparsity ratio, and that running inference unbatched can be a thousand times more expensive than running it batched.

The conversation then moves to how mixture-of-experts models are physically laid out across GPU clusters. Expert parallelism (spreading different experts across GPUs) is the natural fit for a rack's fully-connected internal network, while pipeline parallelism (spreading layers across racks) trades a real reduction in weight-memory footprint for no reduction in KV cache footprint, because avoiding pipeline bubbles forces exactly enough concurrent micro-batches to cancel out the savings. This leads into a discussion of why Nvidia's scale-up domains have grown from 8 GPUs (Hopper) to 72 (Blackwell) to 500+ (Rubin): not primarily to solve memory capacity, which pipelining already handles for weights, but to aggregate more memory bandwidth in parallel and cut decode latency, which in turn is what unlocked training and serving much larger, sparser models than were feasible a few years ago.

A central analytical set-piece works out how much frontier labs may be over-training models relative to Chinchilla-optimal now that reinforcement learning post-training and inference serving costs both compete with pre-training for compute budget. By assuming the three cost buckets (pre-training, RL, inference) tend to be minimized when roughly equalized, and working backward from rough estimates of global inference token throughput and model deployment lifetimes, Pope and Patel arrive at pre-training token counts on the order of 100-200 trillion tokens - about 100x Chinchilla-optimal for a model with roughly 100 billion active parameters. They flag this as necessarily approximate, built on unreported industry numbers, but argue the method itself - solving for cost equalization - is a powerful way to sanity-check public claims.

The back half turns to reverse-engineering real infrastructure signal from public API pricing: Gemini's price jump above 200k-token contexts is interpreted as the crossover point where KV cache memory time exceeds weight-fetch time, which lets them estimate roughly 1.5-2 kilobytes of KV cache per token, matching plausible dense-attention configurations like those Character AI has described publicly. The gap between input and output token pricing (roughly 5x) is read as evidence that decode is heavily memory-bandwidth-bound while prefill is compute-bound, since prefill amortizes memory time over many parallel tokens. This section closes with an explanation of why context lengths have plateaued around 100-200K tokens for the past year or two: it is a memory-bandwidth ceiling, not a compute ceiling, and sparse attention only partially relieves it before quality degrades.

The episode ends with a shorter, less mathematical discussion prompted by a Pope blog post on convergent evolution between cryptography and neural networks - both rely on architectures that thoroughly mix and scramble their inputs, but for opposite purposes: ciphers destroy structure to produce apparent randomness, while neural nets extract structure from apparently random data. Pope traces a concrete lineage from cipher design into deep learning: the Feistel cipher construction, which makes non-invertible functions invertible by carrying forward both inputs, was directly imported into 2017's RevNets to make transformer layers invertible, allowing training to rematerialize activations during the backward pass instead of storing them, trading additional compute for reduced memory - the mirror image of the KV cache tradeoff (more memory to save compute) discussed throughout the rest of the episode.

Notable Quotes

"The big effect is batch size. What we're going to do now is quantify exactly what that looks like and what its implications are on latency and cost." - Reiner Pope

"If you do not batch together many users, the cost and the economics you get can be a thousand times worse than if you do batch many users together." - Reiner Pope

"The cutting matches the model architecture." - Reiner Pope

"The number of inference tokens should be about the same as the number of pre-training tokens, which should be about the same as the number of RL tokens, within factors that we're not able to reason about." - Reiner Pope

"It's kind of empowering to just set A equal to B and figure it out." - Reiner Pope