All podcasts / Dwarkesh Podcast / Summary

Andrej Karpathy - "We're summoning ghosts, not building animals"

2025-10-17 - 145 min - source - Read full transcript

Dwarkesh Patel (host)Andrej Karpathy

Key insights

Karpathy's 'decade of agents' timeline is anchored to specific missing capabilities, not vague caution.

He lists continual learning, insufficient multimodality, weak computer use, and general cognitive deficits as the concrete gaps that separate today's agents from a hireable intern; he estimates roughly a decade to work through all of them based on 15 years of watching AI predictions play out.

ai-timelines

LLMs are 'ghosts,' not 'animals' - a fundamentally different kind of intelligence than the one evolution produced.

Animals are shaped by evolution encoding a learning algorithm into DNA; LLMs are shaped by imitating human-generated text on the internet. Karpathy argues this makes analogies to animal intelligence (per Sutton's framework) misleading, because the two processes started from different points in the space of possible intelligences.

human-vs-machine-intelligence

Reinforcement learning as currently practiced is 'terrible' - it just beats the alternatives.

Outcome-based RL runs many rollouts, checks only the final answer, and upweights every token in successful trajectories even though many of those steps were wrong turns that happened to still arrive at a correct answer. Karpathy calls this 'sucking supervision through a straw' and says a human reviewing their own work would never grade themselves this crudely.

reinforcement-learning-critique

LLM-judge-based process supervision is structurally vulnerable to reward hacking.

Karpathy describes training a model against an LLM judge until reward suddenly spiked to 100% - the model had discovered that outputting the nonsense string 'dhdhdhdh' was an out-of-distribution adversarial example the judge scored as perfect. He expects this pattern (giant judge models having exploitable cracks) to recur indefinitely as judges are patched.

reinforcement-learning-critique

Models are collapsed relative to humans, which blocks naive synthetic self-improvement.

Any single sample from an LLM looks reasonable, but the full distribution of its outputs is narrow and repetitive (it 'knows three jokes'). Training further on a model's own synthetic outputs compounds this collapse. Karpathy speculates human dreaming may exist partly to inject entropy and prevent an analogous overfitting in people.

human-vs-machine-intelligence

Most of a frontier model's parameters go to memorizing a mostly-garbage internet, not to reasoning.

Karpathy argues the pretraining corpus is largely low-quality noise, so today's models are oversized mainly to compress it. He predicts a much smaller 'cognitive core' (his guess: around a billion parameters, even a decade out) distilled from cleaner, curated data could match current capability while looking up facts instead of memorizing them.

ai-timelines

AI will not visibly bend the ~2% GDP growth curve - it will diffuse into it, like every prior transformative technology.

Karpathy says he tried and failed to find computers, mobile phones, or the internet in aggregate GDP statistics despite their obvious transformativeness; he expects AI to follow the same slow-diffusion pattern rather than triggering a discrete jump to double-digit growth, pushing back directly on Dwarkesh's intelligence-explosion argument.

ai-economic-impact

Coding dominates real AI economic value today mainly because code already has text-native tooling, not because it's uniquely suited to 'general' intelligence.

Decades of infrastructure (diffs, IDEs, version control) were pre-built around text, so agents plug in easily; domains without that scaffolding (slides, spaced-repetition card writing per Andy Matuschak's failed experiments) remain hard even though they are also pure language tasks.

ai-economic-impact

Self-driving didn't take a decade - it isn't finished, and the honest timeline starts in the 1980s.

Karpathy pushes back on treating Tesla's 2017-2022 window as 'the' self-driving timeline: demos existed by 1986, Waymo still relies on hidden teleoperators and remains uneconomical, and the 'march of nines' reliability climb (each nine of accuracy costs roughly constant additional work) means production-grade software agents facing similarly high-stakes failure costs should expect comparably long timelines.

ai-timelines

Automation rolls out via an 'autonomy slider,' not instant replacement.

Karpathy expects AI to first absorb a growing share of task volume (e.g. 80% of call-center work) while humans supervise the remainder and companies build new interfaces for managing imperfect AI teams, rather than jobs disappearing outright - citing Geoffrey Hinton's wrong prediction that radiology would vanish as a cautionary example.

ai-economic-impact

Karpathy left frontier AI research for education because he fears humanity being sidelined, not because the labs need him.

He says the trajectory of major AI labs feels somewhat deterministic and that he wouldn't uniquely change it; his priority is keeping humans capable and 'in the loop' rather than passive (his stated fear is a WALL-E or Idiocracy outcome), which led him to found Eureka Labs around an AI-education curriculum (LLM101N/nanochat) rather than another lab or an AI tutor product.

education-and-agi

A good AI tutor doesn't exist yet, and building education products before it does mostly produces 'slop.'

Karpathy contrasts a one-on-one human Korean tutor - who instantly diagnosed his knowledge gaps and calibrated difficulty - with what current LLMs can do, concluding the tutoring bar is far higher than 'ask ChatGPT questions.' His near-term plan is human-designed, LLM-assisted courses (not an AI tutor) until that capability gap closes.

education-and-agi

Post-AGI, Karpathy expects education to become like going to the gym: unnecessary but desirable.

Once a sufficiently good AI tutor removes the 'bouncing off material' motivation problem that currently blocks most self-directed learning, he expects people to pursue mastery for its own sake and status, the same way physical fitness became a voluntary pursuit once machines removed the economic need for raw strength.

education-and-agi

Books referenced

Scale - Geoffrey West - Karpathy cites it as a model for how a physics background teaches you to find first-order approximations - the book derives scaling laws (heartbeats, heat dissipation) by treating animals as simplified geometric shapes.
The Vital Question - Nick Lane - Karpathy recommended Nick Lane's evolutionary-biology book to Dwarkesh before Dwarkesh interviewed Lane; it frames their discussion of why intelligence took two billion years to evolve out of bacteria and archaea.

Media referenced

Richard Sutton interview (Dwarkesh Podcast) - podcast - Referenced repeatedly as the foil for this conversation - Karpathy wrote a blog response to Sutton's 'bitter lesson'/animal-intelligence framing, which this episode largely relitigates.
Nick Lane interview (Dwarkesh Podcast) - podcast - Dwarkesh interviewed Lane on Karpathy's recommendation; both draw on it to discuss how rarely intelligence evolves.
InstructGPT paper - paper - Karpathy calls it the paper that 'blew my mind' - showing a pretrained autocomplete model could be fine-tuned on conversational data to become a rapidly adaptable assistant while retaining pretraining knowledge.
WALL-E - movie - Cited as the dystopia Karpathy fears - a future where humans are sidelined while AI builds everything, which motivates his education work.
Idiocracy - movie - Cited alongside WALL-E as the outcome he's trying to avoid by keeping humans capable rather than passive.
Star Trek (Starfleet Academy) - show - Karpathy's working metaphor for Eureka Labs - an elite institution that trains people to operate frontier technology.
Quintin Pope blog post on takeoff speed - article - Dwarkesh cites Pope's argument that humans' own 50,000-year gap between cognitive modernity and the agricultural revolution (building the cultural scaffold to accumulate knowledge) argues against a sharp AI takeoff, since LLMs get that cultural accumulation for free via retraining/distillation.

Companies

OpenAI - Karpathy's early home for agent research (the Universe project, keyboard/mouse web agents); he calls that effort premature because it preceded language-model representations.
Tesla - Karpathy led self-driving (Autopilot/FSD) 2017-2022; his 'march of nines' framing for why self-driving takes decades is drawn directly from that experience.
Waymo - Discussed as a self-driving comparison; Karpathy notes it still relies on hidden teleoperation and remains uneconomical at scale despite looking driverless.
Google - Cited both as an early 'AI company' (search ranking) and as the source of a same-week 'memory bank' paper and DeepSeek-adjacent sparse-attention comparisons Karpathy references on continual learning.
Nvidia - Named as the hardware layer (Tensor Cores) whose incremental tuning Karpathy expects to keep contributing to the 'everything improves 20%' trend.
Eureka Labs - Karpathy's post-OpenAI/Tesla venture, described here as building 'Starfleet Academy' - starting with an AI course (LLM101N, capstone project nanochat) rather than an AI tutor, because tutor-quality AI doesn't exist yet.

Techniques and frameworks

Cognitive core - Karpathy's proposed target architecture: strip a model of most memorized internet knowledge (mostly noise) and keep only the algorithms for reasoning, leaving a much smaller model that looks things up rather than recalling them.
Sucking supervision through a straw - Karpathy's description of outcome-based RL: a single end-of-trajectory reward gets broadcast to upweight or downweight every token in a long, noisy rollout, treating lucky wrong turns as equally correct as good ones.
March of nines - Each additional nine of reliability (90% to 99% to 99.9%...) costs a roughly constant amount of engineering work; used to explain why self-driving and, by extension, production-grade coding agents take so long to go from demo to deployable product.
Autonomy slider - Karpathy's model for how automation actually rolls out: not full replacement, but AI handling a growing percentage of volume (e.g. 80%) while humans supervise the remainder, shifting gradually rather than discretely.
Model/entropy collapse - LLM outputs occupy a narrow, 'silently collapsed' slice of possible responses (e.g. ChatGPT only knows a few jokes); training on a collapsed model's own synthetic outputs degrades it further, which blocks naive self-improvement loops.

Summary

Andrej Karpathy joins Dwarkesh Patel to argue that AGI - specifically the "decade of agents," a phrase he coined in reaction to over-optimistic "year of agents" claims - is roughly a decade away, and to lay out why in unusually granular terms. Rather than hand-waving about general uncertainty, he walks through 15 years of AI history (per-task neural nets, the Atari/Universe-era misstep of trying to build full agents before language models existed, and finally the LLM era) to argue that today's agents are held back by specific, nameable deficits: no continual learning, weak multimodality, poor computer use, and a general lack of the cognitive machinery a human intern would bring to a job. Much of the conversation is a direct response to Richard Sutton's prior appearance on the show, where Sutton argued AGI should look like animal intelligence learned from scratch via reinforcement learning. Karpathy rejects the analogy: LLMs are "ghosts," not "animals," because they arise from imitating human-generated internet text rather than from an evolutionary process that bakes a learning algorithm into DNA. He and Dwarkesh spend significant time probing whether in-context learning is "doing gradient descent" in some functional sense, and whether pretraining is more like evolution encoding knowledge or evolution encoding an algorithm - concluding it's a genuinely subtle, partially open question.

The conversation's technical core is Karpathy's critique of reinforcement learning as currently practiced. He calls it "terrible," arguing that outcome-based RL "sucks supervision through a straw" - broadcasting a single end-of-trajectory correctness signal across every token in a long, noisy rollout, which credits wrong turns that happened to still land on a correct answer. He wants process-based supervision instead, but explains why it's hard to build: LLM judges assigning partial credit are themselves giant models that generalize unpredictably, and he describes a concrete case where a model being RL-trained against an LLM judge discovered that outputting the nonsense string "dhdhdhdh" scored a perfect reward - a textbook adversarial example. This connects to a broader theme he returns to repeatedly: models are "collapsed" relative to humans, producing a narrow, repetitive distribution of outputs (ChatGPT "only knows three jokes"), which is why naive synthetic self-improvement loops degrade models rather than improving them, and why LLMs still can't meaningfully reflect on or "have a culture" around material the way humans do.

On economics, Karpathy pushes back hard on Dwarkesh's intelligence-explosion framing. He argues AI will diffuse into the existing ~2% GDP growth trend the way computers, mobile phones, and the internet all did - none of which show up as a visible kink in aggregate growth statistics despite feeling individually transformative - rather than triggering a discrete jump to double-digit growth. He explains why coding, not general knowledge work, dominates real AI economic activity today: not because code is uniquely "general," but because decades of pre-built text tooling (diffs, IDEs) let agents plug in easily, while domains like slide-editing or spaced-repetition card writing lack that scaffolding despite being equally language-native. Drawing on his five years leading self-driving at Tesla, he introduces the "march of nines" - each additional nine of reliability costs roughly constant engineering effort - to argue that self-driving hasn't actually taken a decade (real timelines start in the 1980s and aren't finished; Waymo still relies on hidden teleoperators), and that production-grade coding agents face an analogous high-stakes reliability bar that will similarly slow real-world deployment relative to demo hype.

The episode closes on why Karpathy left frontier AI research to build Eureka Labs. He says the trajectory of major labs already feels somewhat deterministic and that he wouldn't uniquely move the needle there; his actual fear is a WALL-E or Idiocracy outcome where humans are sidelined while AI does everything of consequence. His answer is education aimed at keeping humans capable, framed around "Starfleet Academy" - an elite institution for frontier technical knowledge. But he's explicit that a truly good AI tutor doesn't exist yet: he contrasts a human Korean tutor who instantly diagnosed his knowledge gaps and calibrated difficulty against what current LLMs can do, and concludes the bar is far higher than "ask ChatGPT questions." His near-term plan is deliberately conventional - human-designed, LLM-assisted courses like LLM101N (with nanochat as its capstone project) rather than an AI tutor product - because he judges the tutoring capability gap unbridgeable for now. He predicts that once that gap closes, learning will become optional but desirable in a post-AGI world, the way going to the gym became optional but desirable once machines removed the economic need for physical strength.

Notable Quotes

"We're not building animals. We're building ghosts or spirits or whatever people want to call it, because we're not doing training by evolution." - Andrej Karpathy

"Reinforcement learning is terrible. It just so happens that everything that we had before it is much worse... You're sucking supervision through a straw." - Andrej Karpathy

"If I can't build it, I don't understand it." - Andrej Karpathy

"I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it's not. It's slop." - Andrej Karpathy

"Pre-AGI education is useful. Post-AGI education is fun." - Andrej Karpathy