All podcasts / Dwarkesh Podcast / Summary

Adam Marblestone — AI is missing something fundamental about the brain

2025-12-30 - 110 min - source - Read full transcript

Dwarkesh Patel (host)Adam Marblestone

Key insights

The brain's advantage over neural nets may lie mostly in its reward and loss functions, not its architecture.

Marblestone argues machine learning favors mathematically simple objectives (next-token prediction, cross-entropy), while evolution built many distinct, developmentally-staged cost functions - effectively a curriculum encoded directly into the reward signal rather than into weights.

reward-functions-not-architecture

The genome's small size (3 gigabytes) is explainable if evolution mostly encodes compact reward functions and a repeating learning architecture, not detailed world knowledge.

If sample efficiency comes from well-designed loss functions rather than pretrained weights, a reward function can be a handful of lines of code; this resolves the puzzle of how so little genetic information could specify a system as capable as the human brain.

reward-functions-not-architecture

Byrnes's Learning Subsystem/Steering Subsystem split explains how evolution wires abstract, learned concepts to innate reward responses it never anticipated.

The Steering Subsystem (hypothalamus, brainstem, amygdala) has its own primitive sensors and hardcoded reflexes; the cortex learns to predict when those reflexes will fire, and whichever learned neurons succeed at that prediction get wired into the reward pathway - giving the reward function generalization it could not have innately.

steering-vs-learning-subsystem

Single-cell atlas data shows the brain's Steering Subsystem has far more distinct cell types than the cortex, consistent with it needing bespoke wiring for every hardcoded drive.

Cortical circuits reuse a repeating architecture across areas - not unlike stacking transformer layers - needing relatively little genomic 'code', while Steering Subsystem structures need many specialized cell types with distinct surface receptors to genetically pre-wire each reflex, from spider-flinch responses to social-status detection.

steering-vs-learning-subsystem

A capable general intelligence needs far less Steering Subsystem complexity than a human-aligned one does, so most of the brain's reward architecture is really about alignment, not raw capability.

Marblestone argues a 'paperclip maximizer' would still need drives like curiosity and exploration to learn well, but LLMs already show you can get far without most of the social-instinct machinery humans have - meaning the hard problem the brain solves is less 'how to be smart' and more 'how to want the right things.'

reward-functions-not-architecture

Test-time / inference-time compute in LLMs functions like Bayesian sampling, and capabilities that once needed that search get progressively distilled into the base model.

Marblestone frames chain-of-thought reasoning as literal sampling toward a posterior cause, paralleling older energy-based and probabilistic-AI methods; because digital minds can be copied, amortizing computation into weights is more valuable for AI than it evidently was for biological evolution, which may still rely on real-time stochastic sampling in neurons.

amortized-vs-inference-time-compute

The brain may do genuinely probabilistic, energy-based inference rather than fully amortized feedforward prediction, which would vindicate long-dismissed probabilistic-AI approaches.

Neurons are naturally stochastic, so the brain does not need an explicit random-number generator to sample from a distribution; this could mean cognition is closer to real-time energy minimization (as Yann LeCun and other 'old guard' researchers argued pre-2021) than to an LLM's largely feedforward, distilled predictions.

amortized-vs-inference-time-compute

A full mouse brain connectome could fall from several billion dollars to the tens-of-millions range via new optical, molecularly-annotated connectomics, echoing the Human Genome Project's cost collapse.

Just as genome sequencing costs fell roughly a millionfold once the field switched to massively parallel optical methods, Marblestone argues connectomics needs a technology-first strategy - E11 Bio started by building cheaper mapping tools rather than brute-forcing one expensive full-brain project.

connectomics-infrastructure

Marblestone doesn't expect brain-mapping research to matter for near-term AGI timelines, only for a longer, ~10-year scenario.

If transformative AI arrives within a few years, current neuroscience investment is largely irrelevant to that trajectory; the case for funding connectomics now rests on believing the single transformative event is still meaningfully more than five years out, in which case understanding the Steering Subsystem could matter for alignment.

connectomics-infrastructure

Formal verification (Lean) turns math proof-checking into a clean RLVR signal, which is why AI is rapidly automating the mechanical parts of proving but not yet the creative step of conjecturing new theorems.

Once a proof is expressed in Lean, correctness becomes mechanically checkable, making it as game-like as Go was for AlphaGo-style search; companies like Harmonic and AlphaProof already exploit this, but coming up with genuinely novel, interesting conjectures remains unsolved and possibly requires a separate loss function for 'good explanations.'

ai-for-math-verification

The same formal-verification approach could extend beyond math into provably secure software and a way for AI agents to verify each other's outputs rather than relying on opaque trust.

Marblestone points to davidad's ARIA-funded 'safeguarded AI' program, which uses provable specifications and explicit symbolic world models so that agents (or swarms of AI agents) could mathematically prove their outputs are safe or non-manipulative rather than requiring blind trust.

ai-for-math-verification

Books referenced

A Brief History of Intelligence - Max Bennett - Cited as a strong synthesis of the evolutionary path from simple RL in early brains to cortex-like structures in birds and mammals; Marblestone calls it widely respected among AI researchers.
The Brain from the Inside Out - György Buzsáki - Cited as the strongest counterargument to Marblestone's whole framing - Buzsáki argues neuroscience needs its own vocabulary rather than borrowing concepts (backprop, TD learning) invented for AI.

Media referenced

Intro to Brain-Like-AGI Safety - article - Steven Byrnes's blog series proposing the Learning Subsystem / Steering Subsystem theory that structures most of the episode's argument.
A Tutorial on Energy-Based Learning - paper - Yann LeCun paper referenced when discussing whether the brain does probabilistic, energy-based inference rather than amortized feedforward prediction.
What Does It Mean to Understand a Neural Network? - paper - Konrad Kording and Tim Lillicrap paper arguing interpretability should describe architecture, learning rules, and training data rather than trying to decode individual learned circuits.
Ilya Sutskever - Dwarkesh Podcast episode - podcast - Referenced repeatedly for Sutskever's remark that LLMs lack value functions and his open question of how the genome encodes abstract reward functions - the question this episode tries to answer.
Richard Sutton - Dwarkesh Podcast episode - podcast - Referenced in connection with Sutton's independent invention of temporal difference learning, which also shows up as dopamine reward-prediction-error signaling in the brain.
Andrej Karpathy - Dwarkesh Podcast episode - podcast - Referenced when discussing the range of AI timeline views among people Dwarkesh has interviewed.
Gwern's proposal on training models on the brain's hidden states - article - Early source for the idea of adding brain-activity prediction as an auxiliary loss function alongside standard labels, discussed as a form of brain-data-augmented training.

Companies

Convergent Research - Marblestone is CEO; incubates Focused Research Organizations (FROs) including E11 Bio and Lean-focused math infrastructure efforts.
E11 Bio - Convergent Research FRO building molecularly-annotated optical connectomics technology to cut the cost of mapping a full mouse brain from billions to tens of millions of dollars.
Astera Institute - Employs Steven Byrnes and just launched a neuroscience project based on Doris Tsao's work on vision-system architecture.
Harmonic - Billion-dollar-valuation company built on RLVR-driven formal math proving using Lean, cited as evidence the approach is already commercially validated.
ARIA (UK) - Funds davidad's 'safeguarded AI' program, an ARPA-style effort using provable safety properties and explicit symbolic world models.
Google DeepMind - Marblestone's former employer on the neuroscience team; also cited for originating temporal-difference-learning research inspired by dopamine neuroscience.

Techniques and frameworks

Learning Subsystem / Steering Subsystem framework - Steven Byrnes's model: a general-purpose Learning Subsystem (cortex) predicts the responses of an innately-wired Steering Subsystem (hypothalamus, brainstem, amygdala), letting evolution attach abstract learned concepts to hardcoded rewards it never anticipated.
Amortized inference - The idea that a feedforward network compresses an expensive iterative inference process (sampling toward a Bayesian posterior) into a single fast pass; discussed as the key trade-off against test-time/inference-time compute.
Temporal difference (TD) learning / reward prediction error - Neuroscience finding that dopamine encodes a reward-prediction-error signal rather than raw reward, which independently converges with Sutton's TD learning algorithm from reinforcement learning.
RLVR (reinforcement learning with verifiable rewards) - Applied to formal math proofs in Lean, where a proof's correctness is mechanically checkable, making it as clean a training signal as a game like Go.
Molecularly-annotated connectomics - E11 Bio's optical-microscopy approach to mapping brain wiring that also captures molecular identity at each synapse, not just which neurons connect to which.

Summary

Adam Marblestone, CEO of Convergent Research and a former DeepMind neuroscience researcher, joins Dwarkesh to answer what he calls "the quadrillion-dollar question": why do large language models need vastly more data than a human brain to reach a fraction of its capability. His central claim is that the AI field has fixated on architecture and learning algorithms while neglecting the third component of any learning system - the loss function. He argues evolution built many distinct, developmentally-staged reward functions rather than one simple objective like next-token prediction, and that this is the brain's real "secret sauce."

The episode's organizing framework, borrowed from AI-safety researcher Steven Byrnes, splits the brain into a general-purpose Learning Subsystem (the cortex, which does something like omnidirectional Bayesian inference) and an innately-wired Steering Subsystem (hypothalamus, brainstem, amygdala) that supplies hardcoded rewards evolution could specify without knowing anything about the modern world. The trick, per Byrnes, is that the Steering Subsystem trains predictors of its own responses; whichever cortical neurons succeed at predicting an innate reflex get wired into the reward pathway, letting abstract, learned concepts (the word "spider," the idea of embarrassing Yann LeCun) trigger primitive rewards evolution never explicitly anticipated. Marblestone cites single-cell atlas data showing far more distinct cell types in Steering Subsystem structures than in cortex, consistent with cortex reusing one repeating architecture while the Steering Subsystem needs bespoke, genetically pre-wired circuits for each reflex - which he offers as a partial explanation for why the human genome is so small relative to what it builds.

A second thread concerns inference itself: is the brain doing amortized, largely feedforward prediction (like an LLM), or something closer to real, sampling-based probabilistic inference of the kind Yann LeCun and other "old guard" AI researchers championed before 2021? Marblestone notes neurons are naturally stochastic, which would make sampling-based inference biologically cheap, and connects this to why LLMs still lack explicit value functions - something Ilya Sutskever flagged as strange in his own Dwarkesh appearance. He frames test-time compute and chain-of-thought as a form of that same sampling process, with capabilities that once required inference-time search gradually getting distilled ("amortized") into model weights over successive training generations.

The conversation turns practical when discussing Convergent Research's Focused Research Organizations. E11 Bio is trying to make brain connectomics dramatically cheaper using optical, molecularly-annotated mapping, aiming to bring a full mouse brain connectome down from a projected multi-billion-dollar cost to the tens-of-millions range - a cost trajectory Marblestone compares directly to the Human Genome Project's roughly millionfold price collapse once sequencing went massively parallel. He is candid that this science probably won't affect AI timelines under about five years, but argues it matters if the "single transformative event" is genuinely a decade or more away, in which case mapping the Steering Subsystem could inform alignment work more than capabilities work.

The final third of the episode pivots to Lean, the formal math-proving language Marblestone helps steer as a Convergent Research FRO. Because a Lean proof's correctness is mechanically verifiable, it becomes a clean reinforcement-learning-with-verifiable-rewards (RLVR) signal - explaining why companies like Harmonic and AlphaProof are making rapid progress automating the mechanical parts of proof-writing, even though generating genuinely novel conjectures remains unsolved. Marblestone extends this beyond pure math to provably secure software and hardware, and to davidad's ARIA-funded "safeguarded AI" program, which uses explicit, provable world models so that AI agents could eventually verify each other's outputs rather than relying on opaque trust - a possible answer to how a civilization of many AI agents might safely collaborate at scale.

Throughout, Marblestone repeatedly concedes uncertainty - on consciousness, on whether the brain represents concepts symbolically or geometrically, on continual learning - framing his own views as directional hunches from someone trying to build infrastructure (connectomes, Lean, single-cell atlases) that would let the field answer these questions empirically rather than by speculation.

Notable Quotes

"This might be the quadrillion-dollar question or something like that. You can make an argument that this is the most important question in science." - Adam Marblestone

"The fact that we don't have value functions at all in the LLMs is crazy. I think because Ilya said it, I can say it." - Adam Marblestone

"If every iPhone was also a brain scanner, you would not have this problem and we would be training AI with the brain signals. It's just the order in which technology has developed is that we got GPUs before we got portable brain scanners." - Adam Marblestone

"I think it's fair to say it's not super practical and relevant if you're in an AI 2027 scenario. And so what science I'm doing now is not going to affect the science of ten years from now." - Adam Marblestone

"The cortex doesn't know about spiders, it just knows about layers." - Adam Marblestone