Multiply-accumulate is the fundamental primitive of AI chips because it is exactly what happens at every step of the matrix-multiply inner loop, and using a lower-precision multiply with a higher-precision accumulate controls rounding error.
Pope explains that a matrix multiply's triple for-loop performs a multiply-accumulate at each step, and because summing many products accumulates rounding error while a single multiplication does not, AI chips typically use lower-precision inputs (e.g. 4-bit) with a higher-precision accumulator (e.g. 8-bit).
chip-architecture
Multiplier circuit area scales as the product of the two operands' bit widths (p x q), so halving numeric precision should yield roughly a 4x area/throughput gain, not the 2x Nvidia's older marketing implied.
Using the Dadda-multiplier construction, Pope shows the gate count for a p-bit by q-bit multiply is p times q. Because that scaling is quadratic, going from FP8 to FP4 should be closer to a 4x speedup; Nvidia's B300-generation specs (3x for FP4 vs FP8) have started to acknowledge this, correcting earlier generations that only claimed a straight 2x per halving of precision.
ai-accelerator-design
In pre-Tensor-Core GPU and CPU designs, the cost of shuttling data between the register file and the ALU (via muxes) dwarfs the cost of the actual arithmetic logic.
Pope walks through the mux circuit needed to select an arbitrary register-file entry and shows its gate count is n x p (rows times bit width), then compares that to the p x q gates of the actual multiply-accumulate. Plugging in realistic numbers, the data-movement cost (muxing three operands in and out of an eight-entry register file) is roughly six times the size of the compute logic itself - motivating the shift to Tensor Cores.
compute-communication-tradeoff
Systolic arrays solve the register-file bottleneck by storing the weight matrix directly in local registers inside the array, so communication scales with the array's perimeter (O(x)) instead of its full size (O(xy)).
Instead of re-fetching the weight matrix from a shared register file every cycle, a systolic array daisy-chains weights into place once (slowly, to keep wiring cheap) and then streams input/output vectors through the array's edges. This is the architectural basis of both TPUs and Nvidia's Tensor Cores, and Pope frames it as the same underlying goal as inter-chip inference optimization: maximizing compute relative to communication.
compute-communication-tradeoff
A chip's clock cycle is set by its longest logic path between two registers, and designers trade area for clock speed by inserting pipeline registers - but past a point, nearly all the added area goes to synchronization rather than useful compute.
Because all circuitry on a chip synchronizes on a global clock (unlike software's expensive mutex-based synchronization), the clock period must exceed the slowest 'cloud of logic' between registers. Splitting logic with an extra pipeline register can double clock speed, but Pope shows a case where a circuit's area balloons roughly 8x for the same one unit of logic once you push clock speed to its practical limit - illustrating that very fast, low-latency chips can end up with lower actual throughput.
chip-architecture
FPGAs and ASICs implement the same conceptual logic model, but FPGAs pay roughly a 10x area and energy tax because every gate must be emulated via a generic 4-input lookup table (itself a 16-way mux) rather than hand-placed silicon.
A 4-input LUT costs about 32 gates to implement what a hand-built 3-AND-gate circuit would do directly, illustrating the general overhead of describing logic via a truth table instead of writing it out as gates. The business trade-off is a roughly $10,000 FPGA versus a roughly $30 million ASIC tape-out, so FPGAs make sense mainly when workloads change frequently (e.g. high-frequency trading) and deterministic latency plus fast iteration outweigh the efficiency loss.
semiconductor-economics
TPUs favor a software-managed 'scratchpad' over a hardware-managed CPU cache specifically to eliminate non-deterministic latency, since cache-hit variability - not raw clock speed - is the biggest source of unpredictable runtime on a CPU.
A CPU cache is roughly two orders of magnitude faster than DDR memory but whether an access hits the cache depends on unpredictable factors like what else is running. TPUs instead expose two distinct instructions - one for scratchpad, one for HBM - putting the locality decision explicitly in software's hands, trading some flexibility for deterministic timing.
chip-architecture
CPU cores are far larger than GPU cores mainly because of the branch predictor, a large structure that exists purely to guess future branch outcomes several cycles ahead so a fast, deep pipeline doesn't stall.
Pope notes that register files and caches have rough equivalents in both CPUs and GPUs, so they don't explain the size gap. The branch predictor, however, has no GPU analog: because evaluating a branch and updating the program counter can take multiple clock cycles, CPUs speculatively execute along a predicted path and must be able to correct course if wrong - an overhead GPUs mostly avoid, which is a major reason GPUs pack many more, much smaller cores per die.
chip-architecture
A GPU can be understood as a grid of many small TPU-like units - each Streaming Multiprocessor pairs a Tensor Core (analogous to a TPU's MXU) with local vector logic - trading a large shared systolic array for many small, flexible ones.
TPUs use a few very large, coarse-grained matrix units; GPUs tile many small, near-identical SMs across the die instead. The trade-off is real: TPUs move data through only two lines at each systolic array's boundary versus roughly 16 lines within a GPU's finer-grained SM structure, so GPUs pay less communication cost within an SM but need larger systolic arrays to amortize register-file overhead across bigger multiplies.
ai-accelerator-design
Running a chip at a much slower clock speed does not deliver a proportional energy-efficiency gain, because most chip energy comes from switching (charging/discharging capacitors) rather than idle-time draw - so the brain's efficiency advantage over silicon is not primarily explained by its slower 'clock.'
Pope walks through why dynamic/switching power dominates a chip's energy budget: a bit costs energy only when it toggles, so a circuit that idles between operations doesn't keep burning power. Clocking 1,000x slower cuts transitions and energy by roughly 1,000x per unit time, but throughput drops by the same factor, so it is not a free efficiency win in the way a naive brain-versus-chip comparison might suggest.
compute-communication-tradeoff
MatX's own architecture bet, the 'splittable systolic array,' is explicitly framed as trying to capture GPU-style small-unit flexibility while retaining the register-file amortization benefits of one large systolic array.
In response to Dwarkesh's question about whether MatX would want GPU-like small systolic arrays surrounded by SRAM but without the CUDA-architecture overhead of a full SM, Pope confirms the company has publicly discussed exactly this idea: big systolic arrays that can also behave as smaller, splittable ones.
ai-accelerator-design
Media referenced
Reiner Pope - The math behind how LLMs are trained and served - podcast - Dwarkesh's earlier conversation with Pope about data centers and inference math, referenced throughout as 'last time' and the backdrop for this chip-level follow-up.
Companies
MatX - Reiner Pope's AI chip startup; Dwarkesh discloses he is an angel investor. The episode's closing question about a 'splittable systolic array' is MatX's own design bet.
Nvidia - Used throughout as the reference GPU architecture (CUDA cores, Tensor Cores, B100/B200/B300 FP4 vs FP8 speedup claims).
Google - Pope's former employer, where he worked on software efficiency, compilers, and TPU architecture.
TSMC - Referenced as the foundry whose process design kit (PDK) sets the primitive logic gates and clock-cycle constraints chip designers work within.
Jane Street - Dwarkesh recalls prepping a prior episode with a Jane Street FPGA engineer, which motivates the FPGA vs ASIC discussion for high-frequency trading.
Groq - Cited as a company that has publicly advertised deterministic-latency chip design, contrasted with typical non-deterministic CPUs.
Techniques and frameworks
Dadda multiplier - The standard area-efficient method for building a multiply-accumulate circuit out of full adders, repeatedly compressing three input bits into two until one output number remains.
Systolic array - The core circuit behind TPUs and Nvidia Tensor Cores; stores a weight matrix locally in registers so data only needs to flow in and out at the array's boundary, turning O(xy) communication into O(x).
Pipeline register insertion - Splitting a long logic path with an extra register to raise clock speed, at the cost of more chip area spent on synchronization rather than computation.
Scratchpad memory - TPU-style software-managed local memory (as opposed to a hardware-managed CPU cache), used to get deterministic latency by letting instructions explicitly choose local vs off-chip memory.
Splittable systolic array - MatX's own architecture concept, aiming to combine the flexibility of many small GPU-like units with the register-file amortization benefits of one large systolic array.
Summary
This is Dwarkesh Patel's second conversation with Reiner Pope, CEO of the AI chip startup MatX (and a company Dwarkesh discloses he is an angel investor in). Where their first episode covered what happens inside a data center, this one is a ground-up, blackboard-style walkthrough of how a chip actually computes anything, starting from individual logic gates and building up to why GPUs, TPUs, FPGAs, and CPUs each end up looking the way they do.
Pope begins with the multiply-accumulate operation as the atomic primitive of AI chips, since it is exactly what a matrix multiply's inner loop does at every step. Using a Dadda-multiplier construction of full adders, he shows that circuit area for a p-bit by q-bit multiply scales as p times q - a quadratic relationship that explains why lower-precision arithmetic (FP4 vs FP8) delivers outsized gains, and why Nvidia's real speedup from halving precision should be closer to 4x than the 2x older generations claimed. From there the conversation moves to the register file and the hidden cost of a "mux": in classic CPU and pre-Tensor-Core GPU designs, simply routing data between a register file and an ALU costs many times more circuit area than the arithmetic itself. This sets up the episode's central architectural idea, the systolic array, which solves the data-movement problem by storing weights locally inside a grid of registers so data only needs to cross the array's boundary rather than being re-fetched every cycle - the shared design principle behind both TPUs and Nvidia's Tensor Cores.
The middle of the episode turns to clock cycles and pipelining: why a chip's clock speed is bounded by its longest logic path, how pipeline-register insertion trades area for speed, and how pushing that trade-off too far can leave a chip spending nearly all its area on synchronization rather than compute - directly hurting throughput even as latency improves. This leads into a comparison of FPGAs and ASICs, where Pope quantifies the roughly 10x area and cost penalty FPGAs pay for emulating arbitrary logic through generic lookup tables, and explains why that penalty is worth paying only when workloads change too often to justify a multi-million-dollar ASIC tape-out.
The back half covers why TPUs use software-managed "scratchpad" memory instead of a hardware-managed cache (to eliminate the non-determinism that comes from cache hit/miss variability), why CPU cores are so much larger than GPU cores (mostly the branch predictor, which has no real GPU equivalent), and a reframing of GPUs as grids of many small TPU-like units rather than a fundamentally different architecture. The episode closes with a brief comparison of brains and chips, where Pope pushes back on the intuition that a much slower clock speed alone explains the brain's energy efficiency, since chip energy is dominated by switching costs rather than idle draw. Throughout, Pope repeatedly returns to compute-versus-communication as the organizing lens for nearly every design decision in the episode, and the final exchange ties this back directly to MatX's own bet: a "splittable systolic array" meant to combine GPU-like flexibility with the amortization benefits of a large systolic array.
Notable Quotes
"Most of the decisions in chip design are sizing decisions." - Reiner Pope
"It's muxes all the way down." - Reiner Pope
"The throughput of your chip is the product of how much you get done per clock cycle - which is based on area efficiency - times how many clocks you get per second." - Reiner Pope
"From a very high-level point of view, the GPU has a lot of tiny TPUs tiled across the whole chip." - Reiner Pope