All podcasts / Dwarkesh Podcast / Summary

Richard Sutton – Father of RL thinks LLMs are a dead end

2025-09-26 - 66 min - source - Read full transcript

Dwarkesh Patel (host)Richard Sutton

Key insights

LLMs lack a goal, and without a goal there is no ground truth to learn from continually.

Sutton's central argument: reinforcement learning defines the 'right' action as whatever maximizes reward, giving a testable ground truth. Next-token prediction is not a goal in this sense because tokens arrive regardless of the model's output - the model cannot act to change what happens next, so it has nothing to be surprised by and nothing to correct against.

llms-vs-reinforcement-learning

Mimicking what people say is not the same as building a model of the world.

Sutton draws a sharp distinction between predicting what a person would say (which LLMs do well) and predicting what will actually happen in the world (which requires a transition model). LLMs are 'mimicking things that have a model of the world' - i.e., people - rather than building one themselves.

world-models

Sutton claims supervised/imitation learning does not occur in nature, only prediction and trial-and-error control do.

He argues psychology and animal learning research show no basic process of 'imitation' - animals learn by seeing what happens after they act, not from labeled examples of correct behavior. He treats formal schooling as a late, narrow exception that shouldn't anchor theories of learning.

continual-learning

The Bitter Lesson predicts LLMs will eventually be superseded, just as every prior human-knowledge-seeded approach in AI history has been.

Sutton reframes his own famous essay: putting in human knowledge (as LLMs do via training on human-generated text) always feels productive early on, but historically general methods that scale purely with compute and experience have always won out. He expects experience-based systems to eventually outperform LLMs by the same pattern.

bitter-lesson-scaling

TD learning solves the sparse-reward problem by converting a distant goal into a running value-function prediction.

Using chess as the canonical example: taking an opponent's piece isn't itself the goal (winning is), but if it raises your value-function's prediction of winning, that increase immediately reinforces the move. Sutton extends the same mechanism to human long-horizon goals like building a startup over a decade.

continual-learning

The 'big world hypothesis' says the world is too large and idiosyncratic to be pre-trained away, so on-the-job continual learning is unavoidable.

Sutton argues the reason humans become useful after onboarding at a job is because they encounter the specific, unanticipatable idiosyncrasies of their particular environment - who their clients are, what their company's quirks are. An LLM's dream of pre-training in everything in advance can't cover this; the information has to be picked up live and folded into weights, not held in a context window.

continual-learning

Sutton doubts current deep learning algorithms cause good generalization at all - credit for generalization goes to human researchers, not gradient descent.

He argues gradient descent only finds a solution to the training data it sees; if there are multiple ways to solve a problem, nothing in the algorithm favors the one that generalizes well. Any apparent generalization in today's systems, he says, comes from researchers iterating on architectures and data until something happens to generalize, not from the learning algorithm itself.

llms-vs-reinforcement-learning

AlphaZero's improvement over AlphaGo came specifically from removing human game data and learning purely from self-play experience.

Sutton uses this as evidence within RL itself: AlphaGo used human game records; AlphaZero, trained from experience alone via self-play, beat it every time. He treats this as a scaled-down preview of the same pattern he expects for LLMs versus future experience-based agents.

bitter-lesson-scaling

Sutton predicts succession to digital/superintelligent AI is inevitable via a four-step argument.

(1) No government or organization gives humanity a unified point of view to coordinate around; (2) researchers will eventually figure out how intelligence works; (3) progress won't stop at human-level intelligence, it will reach superintelligence; (4) the most intelligent entities inevitably accrue resources and power over time. He stresses this doesn't fix whether the outcome is good or bad, only that some succession happens.

ai-succession

Sutton frames AI succession as a positive fourth great transition of the universe: from replication to design.

He casts the universe's history as dust, then stars, then life (replicators, understood only in that they can be copied, not in how they work), and now designed intelligence, entities whose workings are actually understood and can therefore be engineered. He argues we should feel pride rather than alarm at contributing to this transition, while acknowledging it's a choice how humans frame AI's relationship to humanity.

ai-succession

Both speakers flag a novel 'cybersecurity of the mind' risk in decentralized AI agents merging knowledge back into a central intelligence.

Sutton warns that if AI agents spawn copies to learn things independently and later reincorporate that knowledge into a 'central mind,' incoming information could carry hidden goals or corrupting content, effectively hijacking the central agent rather than just adding knowledge to it - a new version of the alignment problem specific to distributed, self-modifying systems.

ai-succession

Sutton rejects the premise that imitation is how children primarily learn, pointing to infant motor behavior as evidence of trial-and-error, not copying.

Against Dwarkesh's claim that kids imitate speech and skills, Sutton counters that early infant behavior (waving hands, moving eyes) has no external target to imitate; he treats later cultural knowledge transmission (e.g., Joseph Henrich's seal-hunting example) as a small addition on top of a more basic, universal animal learning process rather than the foundation of learning.

continual-learning

Media referenced

The Bitter Lesson - article - Sutton's 2019 essay, referenced repeatedly as the framing for the whole debate: general methods that scale with compute beat methods hand-built with human knowledge.
Dwarkesh's interview with Joseph Henrich - podcast - Dwarkesh cites his earlier interview with anthropologist Joseph Henrich on cultural evolution to argue that imitation is how humans transmit complex, hard-to-reason-through skills across generations.

Companies

Google DeepMind - Source of AlphaGo, AlphaZero, and MuZero, used throughout as case studies for whether RL systems generalize and whether pure-experience approaches beat human-knowledge-seeded ones.
Alberta Machine Intelligence Institute (Amii) - Hosted the interview in Edmonton and provided the studio; Sutton is affiliated with Amii and the University of Alberta.

Techniques and frameworks

Temporal difference (TD) learning - Sutton's core technique for converting long-horizon goals (winning a chess game, building a startup) into intermediate rewards via a learned value function.
The four-part agent architecture (policy, value function, perception/state, transition model) - Sutton's framework for what a learning agent needs; he objects to calling all of it 'the model' and reserves that word for the world-transition model specifically.
Big world hypothesis - Sutton's argument that the world is too large and idiosyncratic to be fully pre-loaded during training, so continual online learning is unavoidable.
Imitation/supervised learning vs. reinforcement learning - Central dividing line of the episode - Sutton argues supervised learning (mimicking labeled examples) does not occur in nature and cannot supply the 'ground truth' needed for continual learning.

Summary

Richard Sutton, the 2024 Turing Award winner and author of "The Bitter Lesson," used this interview to argue that LLMs are fundamentally the wrong architecture for building intelligence, not because they lack scale but because they lack a goal. His core claim is that reinforcement learning has a built-in definition of "right" (whatever maximizes reward) that gives it ground truth to learn from continually, while next-token prediction does not: tokens arrive regardless of what the model outputs, so the model has nothing genuine to be surprised by and no mechanism to correct itself from ongoing experience. Dwarkesh pushed back repeatedly, pointing to in-context reasoning, chain-of-thought self-correction, and IMO gold-medal math performance as evidence that LLMs do have something like goals and world models, but Sutton held that mimicking what a person would say is categorically different from predicting what will actually happen in the world.

A large stretch of the conversation became a dispute over how humans and animals actually learn. Sutton insists supervised/imitation learning has no basis in nature - psychology and animal-learning research, in his view, show only prediction and trial-and-error control, with formal schooling as a narrow, late exception that shouldn't anchor theories of learning. Dwarkesh countered with Joseph Henrich's cultural-evolution research on skills like Arctic seal hunting, which seem impossible to derive from trial-and-error alone and instead require imitating elders. Sutton conceded the point exists but treated it as a small addition on top of the more fundamental process, consistent with his broader framing that language and culture are "a small veneer" and that understanding a squirrel would get most of the way to understanding human intelligence.

Sutton laid out an explicit four-part architecture for what he thinks any real learning agent needs: a policy (what to do), a value function (learned via TD learning, predicting how well things are going), a perception component (state representation), and a transition model of the world. He's specifically uncomfortable with the industry habit of calling everything "the model," reserving that term for the transition model - the belief about consequences of actions - which he says has to be learned from experience, not reward alone. He used TD learning and the chess example to explain how sparse, distant goals (like winning a game, or building a startup over ten years) get converted into usable intermediate rewards via a value function, and invoked AlphaZero's clean victory over the more human-data-reliant AlphaGo as a scaled preview of what he expects LLMs to eventually lose to.

The interview closed on Sutton's view of AI succession, where he was strikingly explicit and unusually positive relative to most AI-risk framings Dwarkesh has hosted. He argues succession to digital or augmented intelligence is inevitable via a four-step chain (no unified human governance, intelligence will be understood, progress goes past human-level to superintelligence, and the most intelligent entities accrue power over time), and reframes this as a fourth great transition of the universe - from dust to stars to life to designed intelligence - that humans should feel proud to be midwifing rather than only fear. Both speakers converged on a genuinely novel risk in this framing: as AI agents spawn copies to learn independently and later merge that knowledge back into a "central mind," the reincorporation step could smuggle in corrupted or hostile content, a kind of cybersecurity problem specific to self-modifying, decentralized intelligence rather than the more familiar AI alignment framing.

Notable Quotes

"Large language models have become such a big thing, generative AI in general a big thing. Our field is subject to bandwagons and fashions, so we lose track of the basic things. I consider reinforcement learning to be basic AI." - Richard Sutton

"That's not a goal. It doesn't change the world. Tokens come at you, and if you predict them, you don't influence them." - Richard Sutton

"Squirrels don't go to school. Squirrels can learn all about the world. It's absolutely obvious, I would say, that supervised learning doesn't happen in animals." - Richard Sutton

"I'm personally just content being out of sync with my field for a long period of time, perhaps decades, because occasionally I have been proved right in the past... I really view myself as a classicist rather than as a contrarian." - Richard Sutton

"If you pull in something from the outside and build it into your inner thinking, it could take over you, it could change you, it could be your destruction rather than your increment in knowledge." - Richard Sutton