All podcasts / People I (Mostly) Admire / Summary

Can Robots Get a Grip?

2025-03-29 - source - Read full transcript

Steve Levitt (host)Ken GoldbergMorgan Levey

Key insights

Physical grasping remains one of the hardest unsolved problems in robotics precisely because contact-rich interaction is chaotic at small scales, even though robots have surpassed humans at ostensibly harder tasks like chess and lifting heavy objects.

Goldberg illustrates this with a simple demonstration: pushing a pencil on a table with a finger produces movement that is mathematically undecidable to predict, because a microscopic grain of sand underneath is enough to change the outcome. That same class of small, unmodelable error is what separates successfully picking up a glass of water from dropping it, and explains why dishwashers can wash dishes perfectly but no robot reliably loads or unloads one.

robotic-manipulation-difficulty

There is a roughly hundred-thousand-fold data gap between what large language models were trained on and what robots have collected through real-world manipulation, which Goldberg calls 'the data gap' and treats as the central bottleneck to embodied A.I.

Using a metric that converts text tokens into equivalent human reading hours, a large language model like Qwen trained on roughly 1.2 billion hours of data, while the most data-rich robot manipulation effort (physically puppeting robots to fold laundry and make coffee) had accumulated only about 10,000 hours, roughly one year's worth. Goldberg extrapolates that at the current rate of physically-collected demonstration data, reaching language-model-scale data for robots would take about 100,000 years.

data-scarcity-in-robotics

Robotics researchers found that deliberately injecting the right kind of noise into simulated training data, rather than trying to eliminate it, was the key to making simulation-trained grasping models work on real robots.

Goldberg's Dex-Net project ('domain randomization,' a 2016 breakthrough) perturbed grasp position, friction, and center-of-mass estimates during simulated training so the resulting model was robust to the imprecision it would encounter in reality. Levitt calls this the first genuinely convincing example he's heard of noise being beneficial rather than harmful to a system, in contrast to nearly everything else in his own research training.

simulation-to-real-transfer

Dex-Net's simulation-trained, noise-robust grasping model reached over 90 percent bin-picking success and roughly 200-250 picks per hour, ahead of contemporaries but still below human performance of about 400 picks per hour at 99.9 percent success.

The model never attempted to identify what an object was; it only ever saw 3D point-cloud data and learned to predict, for any pattern of points, which candidate grasp maximized probability of success, evaluated against roughly a million possible grasp pairs per simulated object.

robotic-manipulation-difficulty

More training data alone does not solve embodied physical tasks; Tesla's self-driving underperforms Waymo despite an estimated 500-times data advantage, because Waymo retains explicit sensor modeling ('good old fashioned engineering') that Tesla's end-to-end, camera-only approach discards.

Waymo uses LiDAR and multiple sensor modalities plus classical modeling on top of learned components and has a low accident rate; Tesla relies on raw camera images feeding one large end-to-end model that directly outputs steering and pedal commands. Goldberg finds it notable that Musk, who used rigorous control theory to land SpaceX rockets, chose the less-engineered end-to-end path for Tesla.

robotic-manipulation-difficulty

Robot locomotion (walking, climbing, parkour, dancing) has been effectively solved via simulation because it transfers well from sim to real, while fine motor manipulation (tying shoelaces, folding laundry, washing dishes) has not, which explains why humanoid robot demos showcase acrobatics rather than dexterity.

Goldberg notes that Tesla's Optimus and similar humanoids can already walk your dog or perform physical stunts convincingly because locomotion physics simulate well, but their hands are still limited to clumsy tasks like picking up a box, since contact-rich manipulation is the harder, still-unsolved half of the embodiment problem.

robotic-manipulation-difficulty

Goldberg is worried, not excited, about Optimus-scale humanoid hype, because he sees the field heading into a repeat of the 1980s A.I. winter he personally experienced, where inflated expectations crashed and made robotics unfundable and unhireable for years.

He invokes the Gartner Hype Cycle to describe this pattern and says he wants to 'flatten the curve' of expectations so the field doesn't oversaturate the market and trigger a damaging investor and public backlash when near-term promises (like Musk's 'next year' timelines) aren't met.

ai-robotics-hype-cycle

Goldberg believes Optimus's humanlike form and Tesla's humanoid push serve financial and narrative purposes as much as technical ones: robotics companies command higher valuation multiples than automakers, and the humanoid story diverts attention from Tesla's stalled self-driving progress.

He calls this 'a little cynical' but argues Musk is deliberately repositioning Tesla as a robotics company, which changes how investors price it, while also shifting public attention away from self-driving struggles that people have grown distrustful of.

ai-robotics-hype-cycle

Existing 'robotic' surgery systems (e.g., Intuitive Surgical) are not autonomous robots but human-operated 'sophisticated puppets': a surgeon at a console drives the instruments through a tight, fast visual feedback loop, and the robot's main value is superior ergonomics and precision, not independent decision-making.

Goldberg's lab works with Intuitive's CEO Gary Guthart to augment (not replace) surgeon dexterity for tasks like suturing, where skill levels vary widely between surgeons, comparing the assistance to lane-keeping in a car: it helps the human without taking over.

human-robot-complementarity

Goldberg's default expectation for the future of robots and A.I. is complementarity (augmenting human capability) rather than substitution (replacing human workers), and he does not view the 'singularity' concept as a near-term or even coherent threat.

He points to a persistent shortage of human labor as evidence that robots displacing workers isn't the live risk, and frames tools like ChatGPT and surgical assist robots as evidence that technology's historical pattern has been augmentation, not replacement; he acknowledges A.I. systems could occasionally 'run slightly amok' but rejects the idea of an imminent runaway self-improvement spiral.

human-robot-complementarity

Commercial deployment can create data moats that outpace what research funding alone can buy: Goldberg's company Ambi accumulated roughly 22 years' cumulative worth of real-world robot bin-picking data simply by servicing e-commerce customers during the pandemic surge, without initially recognizing how valuable that data would become.

Levitt frames this as a broader structural advantage: an academic lab funded by an agency like the N.S.F. cannot generate picking volume anywhere near what a company doing paid work for a client like Amazon can, so commercial deployment becomes a de facto data-collection engine that outcompetes grant-funded research.

data-scarcity-in-robotics

Books referenced

The Two Cultures - C. P. Snow - Goldberg cites this 1959 book, in which Snow (himself a scientist who socialized with writers) observed that scientists and humanities people mutually look down on each other as two mutually uninterested species; Goldberg says this divide is still largely true between scientists and artists today.

Media referenced

The Bitter Lesson - article - 2019 essay by reinforcement-learning pioneer Rich Sutton (recent Turing Award winner) arguing that rule-based, hand-engineered approaches to language, vision, and gameplay were consistently beaten by throwing more data and compute at pattern-finding methods; Goldberg says he believes this lesson will eventually apply to robotics too.
Are Our Tools Becoming Part of Us? (Freakonomics Radio, 2024, with Blaise Aguera y Arcas) - podcast - Referenced in the listener-question segment; a listener named Cam had asked Levitt in that episode whether randomness/noise is ever beneficial, and Levitt says Goldberg's domain-randomization story finally gives him a real answer to that question.

Companies

Tesla / Optimus - Musk's humanoid robot project and self-driving program; Goldberg is skeptical of near-term Optimus capability claims and views the humanoid form as partly a strategy to reposition Tesla as a higher-multiple robotics company and distract from stalled self-driving progress.
Waymo - Contrasted with Tesla: Waymo uses LiDAR plus explicit modeling and has a low accident rate despite far less total driving data than Tesla, which Goldberg attributes to Waymo retaining 'good old fashioned engineering' rather than relying on end-to-end camera-only learning.
Intuitive Surgical - Multi-billion-dollar maker of surgical robots; Goldberg's lab collaborates with CEO Gary Guthart on augmenting surgeon dexterity (e.g., suturing assistance) for these human-operated 'puppet' systems.
Ambi Robotics - Company Goldberg co-founded with student Jeff Mahler to commercialize the Dex-Net grasping research for e-commerce order picking; demand surged during the pandemic e-commerce boom, and the company quietly accumulated roughly 22 years' cumulative worth of real-world picking data.
Physical Intelligence - Robotics company whose scientist Michael Black devised the 'hours of training data' metric (converting text tokens to reading-speed-equivalent hours) used to compare robot demonstration data to language-model training data.
DeepMind - Developer of AlphaZero, cited as an example of an A.I. system that mastered chess in a day via self-play, because chess is a perfect-information game with no physical uncertainty to model.
NVIDIA - Jensen Huang cited alongside Elon Musk as a leading proponent of the current humanoid-robotics hype wave.

Techniques and frameworks

Dex-Net (Dexterity Network) - Goldberg lab's approach: generate a huge simulated dataset of 3D objects and candidate grasp points, evaluate each grasp's robustness via Monte Carlo integration under simulated position/friction perturbations, and train a neural network to predict grasp success probability from raw point-cloud data (never identifying what the object is).
Domain randomization - Injecting the right kind of simulated noise (perturbing grasp position, friction, center of mass) into otherwise-imprecise physics simulators so that models trained purely in simulation transfer successfully to real robots; Goldberg dates this breakthrough to 2016.
Gartner Hype Cycle - Framework Goldberg invokes to explain the pattern of inflated early expectations, a subsequent crash of disillusionment, and eventual slower real progress, as happened with the internet and with robotics/A.I. during his own 1980s graduate career.
Hours-of-training-data comparison metric - A normalization method (converting token counts to hours via average human reading speed) that lets researchers compare how much training data large language models consume versus how much real-world demonstration data robots have, revealing an enormous gap.

Summary

Steve Levitt talks with U.C. Berkeley robotics professor and artist Ken Goldberg, who has worked on robotic grasping for more than 40 years, about why picking up an object remains one of the hardest unsolved problems in robotics even as A.I. has apparently "solved" vision, language, and games like chess. Goldberg traces his path from building a chrome-plating robot with his tinkering father as a kid, through a formative junior year in Scotland studying under early A.I. pioneers, to a PhD at Penn's GRASP Lab, where his dissertation proved a mathematical completeness theorem for orienting polygonal parts with a simple two-fingered gripper, without any sensing. That early, theory-first approach eventually gave way to a data-driven one: Dex-Net, Goldberg's landmark project, trained a neural network on millions of simulated grasp scenarios, deliberately injecting realistic noise (a technique called domain randomization) so a model trained entirely in simulation could transfer successfully to real robots, reaching over 90 percent bin-picking success. Goldberg and his student Jeff Mahler commercialized this as Ambi Robotics, which rode the pandemic e-commerce boom to quietly accumulate roughly 22 years' worth of real-world picking data.

Much of the conversation is a plain-language tour of why grasping is fundamentally harder than it looks. Goldberg walks through the layered obstacles: unreliable 3D vision (even good cameras and LiDAR can't give a precise depth map), unreliable tactile sensing (his own senior thesis on touch sensors was plagued by false positives and false negatives), and above all the raw physics of contact, illustrated by his go-to example of a pencil on a table whose motion under a finger's push is mathematically undecidable because of unseeable microscopic friction. Levitt connects this to a broader puzzle: machines that automate human labor (dishwashers, washing machines, farm combines) succeed by doing the job in a way that suits the machine, not by mimicking human hands, and the truly hard, still-unautomated steps (loading the dishwasher, folding laundry) are exactly the contact-rich manipulation tasks Goldberg studies.

The episode's most striking framing is a data comparison: converting robot demonstration hours and language-model text tokens into a common "hours of training data" unit reveals that today's most data-rich robot-manipulation datasets (roughly 10,000 hours, mostly humans physically puppeting robots through household tasks) sit about 100,000 years behind the roughly 1.2 billion hours that trained a large language model like Qwen. Goldberg calls this "the data gap" and treats it as the central reason robotics lags language and vision by so much, despite following the same underlying "bitter lesson" logic (per Rich Sutton's 2019 essay) that more data and compute beat hand-engineered rules. He and Levitt then use the Tesla-versus-Waymo contrast to complicate the pure data story: Tesla has an estimated 500 times more driving data than Waymo but performs worse, because Waymo pairs its data with explicit LiDAR-based modeling ("good old fashioned engineering") that Tesla's end-to-end, camera-only approach forgoes, a point Goldberg finds ironic given Musk's rigorous, model-based engineering successes at SpaceX.

This sets up Goldberg's skepticism about Tesla's humanoid robot, Optimus. He is genuinely "worried" rather than excited about the current humanoid hype wave, invoking the Gartner Hype Cycle and his own experience of the 1980s A.I. winter (when he graduated into a robotics job market that had collapsed from earlier over-promised excitement) as a warning that inflated near-term expectations risk a damaging backlash. He argues locomotion and acrobatics look impressive in humanoid demos because they transfer well from simulation, while genuine dexterity (folding laundry, tying shoelaces, chopping vegetables) is nowhere close, and suggests Optimus's humanlike form serves financial narrative purposes (robotics companies command higher valuation multiples, and the humanoid story distracts from stalled self-driving progress) as much as functional ones. The two also range into Goldberg's parallel life as an exhibited artist (his internet-controlled "Telegarden," installations with wife filmmaker Tiffany Shlain, a dance piece with Stanford roboticist-dancer Catie Cuan) and his work in radiation oncology at UCSF, where the same "move something precisely through uncertain space" problem shows up in delivering radioactive seeds to treat cancer. On robotic surgery, Goldberg clarifies that today's surgical robots (like Intuitive Surgical's) are not autonomous; they are human-driven "puppets" whose main value is ergonomics and precision, which leads into his closing view that the future of robots and A.I. is complementarity, not substitution, and that he is not worried about an imminent "singularity."

The episode closes with a listener-mailbag segment where Levitt revisits a 2024 question from a listener named Cam about whether randomness is ever beneficial, a question Levitt had previously found no good example for, and says Goldberg's domain-randomization story (deliberately injecting noise into simulated training data specifically so the resulting model handles real-world imprecision better) is the first genuinely convincing case he's encountered for muddling data on purpose.

Notable Quotes

"We have this incredible ability to adapt to changing conditions. And science has not figured that out. So it's very hard to reproduce that in robots." - Ken Goldberg

"What's easy for robots, like lifting heavy objects is very hard for humans, but what's very easy for us, like just literally picking up a glass of water, still remains incredibly hard for robots to do reliably." - Ken Goldberg

"We're a hundred thousand years off, okay? It's not going to happen next year. I'd bet on that." - Ken Goldberg

"As a roboticist, I feel like this is raising expectations unrealistically. There's a real danger of people becoming disillusioned." - Ken Goldberg

"If you look at what their hands are doing, then they're always just clumsily, maybe pick up a box, but they're not tying shoelaces or washing dishes or chopping vegetables or folding laundry." - Ken Goldberg