All podcasts / Invest Like the Best / Summary

Sergey Levine - Building LLMs for the Physical World

2026-03-31 - 69 min - source - Read full transcript

Patrick O'Shaughnessy (host)Sergey Levine

Key insights

Physical Intelligence's central bet mirrors the history of language models: full generality can be easier to achieve than narrow specialization.

Levine argues that just as machine translation and sentiment-analysis-specific systems lost out to general language models that could draw on much broader data, a robot trained across many tasks, environments, and even different physical bodies builds a foundation of physical understanding that makes each new narrow application far easier to bolt on than training a dishwashing or laundry-folding specialist from scratch.

robotic-foundation-models

The model reasons in language before acting, which is how it imports common sense it never directly experienced physically.

Physical Intelligence's vision-language-action model is trained like an LLM (text, then web images, then robot data) and uses chain-of-thought: given an instruction like 'clean up the kitchen,' it first reasons about what it should do before moving. This lets the intermediate inferences benefit from web-scale pretraining, which is how the system handles long-tail scenarios it has never practiced.

robotic-foundation-models

The bottleneck in robot performance shifted from physical dexterity to mid-level semantic reasoning, and coaching in language alone can fix it.

About six months before the interview, the team discovered that when a robot failed a task in a new environment, adding more teleoperation demonstrations wasn't necessary - simply labeling the robot's existing experience with high-level semantic commands (no new low-level action data) improved generalization. That means the system can now be improved by talking to it, not just by collecting more physical demonstrations.

robotic-foundation-models

Neuroscience showing tools become a physiological extension of the body motivates building one embodiment-agnostic model rather than per-robot systems.

Levine cites monkey studies where neurons tracking hand position instead track the tip of a held tool, evidence the brain treats a tool as part of the body. He extrapolates this to argue there is 'one problem, not many different problems' in robotics - a humanoid, a car, a bulldozer, and a robotic arm bolted to a table are all the same underlying control problem if solved at full generality.

robotic-foundation-models

Humanoid acrobatics and manipulation foundation models have converged on opposite data strategies, and the field has not resolved which will win.

Humanoid robots doing acrobatics rely almost entirely on simulation with often zero real-world data, while the manipulation foundation models Physical Intelligence builds rely on very little simulated data and large amounts of real-world data plus large foundation models. Levine calls this an unresolved and genuinely controversial question - it might be that one approach wins, or that some synthesis emerges.

bitter-lesson-and-generalization

End-to-end learning ('the bitter lesson') is broadly accepted in robotics for perception but still contested for control.

Levine says the field now largely accepts that learning has a place in robotics, but there is no universal acceptance that the machine should learn everything from data rather than being hand-programmed with physics knowledge. He steelmans the opposing view: in a complex open world where reliability matters, you may not be able to afford ignoring the textbooks' worth of known physics.

bitter-lesson-and-generalization

Levine places himself as more optimistic than veteran robotics researchers but more cautious than robotics entrepreneurs, calibrated against the field's thin track record.

He notes most robots doing useful work today still run 1980s-era technology because robotics AI has had 'precious few successes.' His optimism comes from recognizing puzzle pieces (chain-of-thought grounding, semantic-label fine-tuning) that address specific problems that stumped him earlier in his career, not from underestimating the field's difficulty.

bitter-lesson-and-generalization

Physical Intelligence deliberately runs minimal sensing - three cameras and no touch or force sensors - and lets learning compensate.

The robot's wrist-mounted cameras function as a de facto touch sensor because the model can detect local visual deformation when the gripper contacts an object. Levine argues a good learning method can compensate for deficient sensing, so the team stays agnostic about sensor choices (cameras, touch, force) rather than assuming more sensing is always better.

hardware-and-cost-curve

Robot arm hardware costs have fallen roughly 100x in a decade, which both enables and requires the shift to learning-based control.

Levine's first research robot (the PR2) cost about $400,000; his UC Berkeley lab later used a $30,000 robot; Physical Intelligence's current arms cost a fraction of that, with each arm at roughly a tenth of the $30,000 figure. Traditional precision-control methods can't function on such cheap, imprecise hardware - only learned control makes it viable, and the cost collapse is what makes broad experimentation with new robot form factors realistic.

hardware-and-cost-curve

Moravec's paradox is being reshaped by machine learning, and its sharpest edge falls on socially intimate tasks like elder and child care.

Humans wrongly assume tasks easy for us (picking up a cup) will be easy for machines and tasks hard for us (calculus) will be hard for machines - backwards, because our physical ease evolved for survival. Levine argues ML shifts this so that physically intricate tasks with easy-to-collect data get solved first, while tasks requiring direct physical interaction with vulnerable people (helping someone out of bed, changing a child's diaper) stay hardest longest - both because of that same evolved-ease bias and because the stakes of getting them wrong are unusually high.

physical-intelligence-limits

How fast robots get deployed will be gated by social trust thresholds, not just technical readiness, echoing the early self-driving car debate.

Levine points to imperfect early Tesla self-driving as a precedent: some domains will tolerate a robot that occasionally fails and improves from its mistakes (a robot that breaks a dish now and then), while others won't tolerate imperfection at all (a robot around small children), and figuring out which category a task falls into requires care independent of the underlying model's capability.

physical-intelligence-limits

Levine expects robotics labor impact to follow the coding-tools pattern: augmentation and role-shifting, not wholesale human replacement.

Drawing the direct parallel to how coding agents increased individual software engineer productivity rather than eliminating the role, he predicts a similar 'dance' in physical labor - some tasks fully automated, some done by a robot and person together, some where the human's job becomes making the robot more productive and vice versa - co-evolving as both the technology and the work itself change.

ai-labor-and-jobs

Books referenced

Prey - Michael Crichton - Patrick cites the novel's idea of a swarm that morphs into the optimal shape for a given problem when asking Levine how much Physical Intelligence thinks about innovating on robot form factor versus data and models.

Media referenced

Robot Olympics blog post - article - A blog post by Benji Holson (formerly of Everyday Robots) proposing an everyday-tasks alternative to China's acrobatic robot Olympics - opening doors, washing a greasy pan, picking up dog poop with a bag. Physical Intelligence used the list as an internal test of its task-onboarding pipeline and solved nearly all of them.
Interview with Richard Feynman - other - Cited for Feynman's discussion of teaching physics through analogies (e.g. describing particle 'spin' as not literally spinning), used to illustrate how physical analogies let people reason about domains far outside direct physical experience.

Companies

Physical Intelligence - The company being profiled - developing general-purpose robotic foundation models meant to control any embodied system to do any task; Patrick discloses he is an investor.
UC Berkeley - Where Levine did his postdoc under Peter Abbeel starting in 2014 after finishing his graduate degree, his entry point into robotics research.
Google - Where Levine worked on parallelizing robotic skill practice across many robots at once (the 'arm farm' project), an experience he cites for the sheer organizational leverage a young researcher could get there.
NVIDIA - Gave Levine an internship as a sophomore, an early bet on his potential that he names as one of the kindest things anyone has done for him professionally.
Boston Dynamics - Levine's favorite non-Physical-Intelligence robot company, especially the newest Atlas, and the company he says most inspires him for repeatedly showing the world things it didn't think were possible.
Tesla - Referenced twice: as the model for a data flywheel (a system useful enough on its own to keep generating more training data) and for the public trust debate that early imperfect self-driving triggered.
OpenAI - Cited as an organization that empowers individual researchers to pursue speculative pet projects, with ChatGPT described as originally John Schulman's personal experiment rather than planned corporate strategy.
Everyday Robots (Alphabet) - Former employer of Benji Holson, whose blog post proposing an everyday-tasks robot Olympics became an internal generality test at Physical Intelligence.
Positive Sum - Patrick O'Shaughnessy's investment firm; disclosed as an investor in Physical Intelligence at the top of the episode.

Techniques and frameworks

Vision-language-action (VLA) model - Physical Intelligence's base architecture - an LLM-style model first trained on text, then adapted with web image data, then further adapted with diverse robot data, giving it a foundation of world knowledge before it is taught to control robots.
Chain-of-thought grounding for common sense - Before acting, the robot reasons in language about the scene and task (e.g. 'clean up the kitchen' becomes 'I should pick up the plate'), which lets it draw on web-scale pretraining to handle unfamiliar situations rather than only what it directly practiced.
Semantic-label-only fine-tuning - Instead of adding more teleoperation data when a robot fails a task in a new setting, Physical Intelligence found that labeling the robot's existing experience with high-level language instructions alone - no new low-level action data - improved generalization, because the bottleneck had shifted from motor control to mid-level scene interpretation.
Task-onboarding pipeline stress test - Running a list of everyday, non-cherry-picked tasks (opening a door, washing a greasy pan, picking up dog poop) through the team's standard process for adding new tasks, as a way to validate the model's generality rather than building anything bespoke for the demo.

Summary

Patrick O'Shaughnessy talks with Sergey Levine, co-founder and researcher at Physical Intelligence (Patrick discloses he is an investor), about the company's bet that robotics should be solved the way language was: not with narrow, task-specific engineering, but with a single general foundation model that can control any embodied system to do any task. Levine traces his own path into the field - a 2014 postdoc under Peter Abbeel at UC Berkeley with zero prior robotics experience, a stint at Google parallelizing skill practice across many robots at once (the "arm farm"), and the realization that the field's two biggest wins, generative AI and deep reinforcement learning, are impressive for opposite reasons: one reproduces what humans already do, the other exceeds human performance in ways nobody anticipated. Physical Intelligence's technical strategy is an attempt to fuse both threads in a vision-language-action model trained like an LLM (text, then web images, then diverse robot data) that reasons about a task in language via chain-of-thought before acting, which is how it imports common sense it never directly practiced.

Much of the conversation works through what actually changed to make this tractable now. The team found that when their robot failed at a task in a new setting, they didn't need more teleoperation demonstrations - just labeling existing robot experience with high-level semantic instructions alone improved generalization, evidence that the bottleneck has shifted from raw motor control to mid-level scene interpretation. Levine also describes minimal, almost stubbornly agnostic hardware choices: three cameras, no touch or force sensors, with wrist cameras substituting for touch by detecting visual deformation on contact, and a bet that a good learning method compensates for weak sensing rather than requiring it to be solved separately. As proof of generality, the team ran a list of everyday tasks proposed in a blog post by Benji Holson (opening a door, washing a greasy pan, picking up dog poop) through their standard task-onboarding pipeline - not a bespoke demo - and solved nearly all of them, failing only at turning a dress shirt inside out (gripper too wide) and had to use a tool rather than fingers to peel an orange.

Levine is candid about what remains unresolved. Humanoid acrobatics and manipulation foundation models have converged on opposite data strategies (heavy simulation with near-zero real data versus heavy real-world data with almost no simulation), and he doesn't know which will win or whether some synthesis emerges. The "bitter lesson" - that learning from data beats hand-engineered physics knowledge - is now broadly accepted for perception in robotics but still contested for control, and he steelmans the counterargument that reliability in a genuinely open world may require leaning on textbooks of known physics rather than learning everything from scratch. He places his own calibration as more optimistic than veteran robotics researchers, precisely because robotics AI has "precious few successes" historically, but more cautious than robotics entrepreneurs.

The back half turns to what generality unlocks and what stays hard. Citing neuroscience showing tools become a physiological extension of the body (monkey neurons track a held tool's tip, not the hand), Levine argues there is fundamentally "one problem, not many different problems" across humanoids, cars, bulldozers, and fixed robotic arms - solve it at full generality and every form factor benefits. He reframes Moravec's paradox for the ML era: difficulty now tracks how easy data is to collect, not how intuitively simple a task feels, which means physically intricate tasks with easy data collection get solved while socially and physically intimate tasks - elder care, changing a child's diaper - stay hardest, both because human ease at them evolved for survival and because the stakes of failure are unusually high. Deployment pace, he argues, will be gated as much by social trust thresholds (are people comfortable with a robot that occasionally breaks a dish? around small children?) as by raw technical capability, echoing the early self-driving car trust debate. On labor, he predicts robotics will follow the pattern set by coding tools: augmentation and an evolving "dance" between people and robots rather than wholesale replacement.

The episode closes on personal reflection - Levine names three moments he now recognizes as people taking a bet on him (an NVIDIA internship as a sophomore, Peter Abbeel accepting him into a robotics postdoc with no robotics background, and Jeff Dean and Vincent Vanhoucke greenlighting the "arm farm" project when he was a junior research scientist) - and on what most inspires him going forward: Boston Dynamics, for repeatedly showing the world things it didn't think possible, and organizations like OpenAI that create space for individual researchers' pet projects (ChatGPT, he notes, began as John Schulman's personal experiment) to become world-changing.

Notable Quotes

"As you hear us discuss today, robotics has what I would call a scarecrow problem." - Patrick O'Shaughnessy

"The way to have a really exciting demo is to pick a really cool task, control everything else in the environment, like set it up so that it's perfectly clean, perfectly pristine, and just make it work in that one setting. That's the way you make a robot demo." - Sergey Levine

"There isn't like a humanoid problem and a car problem and a bulldozer problem and a robot bolted to the table problem. There's one problem." - Sergey Levine

"Are you comfortable with occasionally breaking your dishes?" - Sergey Levine

"Robotics is a very long history, which has precious few successes, especially when it comes to robotic AI." - Sergey Levine