All podcasts / Lenny's Podcast / Summary

Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon

2026-01-11 - 86 min - source - Read full transcript

Lenny Rachitsky (host)Aishwarya Naresh RegantiKiriti Badam

Key insights

AI products break from traditional software in two fundamental ways: non-determinism and the agency-control tradeoff.

Traditional software maps a known intent to a predictable action (e.g. booking.com). AI products replace that with natural-language input (unpredictable user phrasing) and a probabilistic LLM output (sensitive to prompt phrasing, effectively a black box) - so both the input and output sides are unpredictable at once. Layered on top, every increase in agent autonomy is a decrease in human control, and that autonomy has to be earned through demonstrated reliability, not assumed.

ai-product-development

Build AI products by starting with high human control and low agency, then earn autonomy in stages.

The guests give a repeatable pattern: V1 has the AI suggest and a human approve (e.g. a support agent drafting an answer for a human to review), V2 lets the AI act with lighter review, V3 is full autonomy. Jumping straight to V3 (fully autonomous agents from day one) is the most common failure mode they see, because teams can't yet predict how users or the model will behave.

agency-and-control

"One-click agents" that promise value within days are marketing, not engineering.

Because enterprise data and workflows are messy - inconsistent taxonomies, undocumented tribal knowledge, dead or duplicated data fields - even a well-resourced team needs an agent to build up context over time. The guests estimate 4-6 months of iterative work to replace a critical workflow or produce real ROI, even with strong existing data and infrastructure.

ai-product-development

Reliability, not model capability, is the top blocker to shipping customer-facing AI products.

Aishwarya cites a UC Berkeley/Databricks survey where roughly 74-75% of enterprises named reliability as their biggest obstacle. She argues this is why most AI products today skew toward internal productivity tools rather than end-to-end customer-facing agents: teams aren't yet comfortable exposing users to the risk.

evals-and-monitoring

Evals and production monitoring are complementary, not competing, and neither alone is sufficient.

Evals are a curated dataset encoding what the team already believes matters and must not fail. Production monitoring catches emerging failure patterns nobody anticipated, via explicit signals (thumbs up/down) and implicit ones (a user regenerating an answer signals dissatisfaction even without a thumbs-down). New failure patterns found in monitoring feed back into building new evals - the two form a loop, not a binary choice.

evals-and-monitoring

The term "evals" has undergone semantic diffusion and now means different things to different practitioners.

Kiriti invokes Martin Fowler's term 'semantic diffusion': data-labeling companies mean error analysis by domain experts, some argue PMs should write LLM judges, others treat evals as the entire feedback loop. None of these usages are wrong in isolation, but conflating them causes real confusion about what to actually build.

evals-and-monitoring

AI product success rests on a three-part organizational triangle: leaders, culture, and technical prowess - not technology alone.

Leaders must rebuild intuitions built over a pre-AI career and stay personally hands-on (the Rackspace CEO's daily 4-6am AI catch-up block is the example given). Culture must empower subject matter experts to contribute rather than fear replacement, since their input is essential for calibrating agent behavior. Technical execution means matching the right tool - ML model, deterministic code, or LLM - to each piece of a workflow instead of defaulting to an agent for everything.

org-culture-for-ai

Multi-agent systems built as peer-to-peer 'gossip protocols' between specialized agents are largely misunderstood and unreliable.

Kiriti argues the common vision of decomposing a complex problem into many communicating peer agents doesn't match current model capabilities. A single supervisor agent orchestrating sub-agents (rather than agents coordinating directly with each other) is the pattern he's seen actually work, particularly in customer-facing use cases where uncontrolled agent-to-agent communication makes it very hard to govern what reaches the end user.

agency-and-control

"Pain is the new moat": durable advantage comes from accumulated organizational learning, not being first to market.

Kiriti's framing: successful AI companies aren't ahead because of a flashy feature or first-mover status, but because they went through the iterative, often painful process of learning which tradeoffs and non-negotiables matter for their specific problem. That lived, hard-won knowledge - not the model or the feature - is what's difficult for competitors to copy.

ai-product-development

As implementation gets cheaper via AI-assisted coding, design judgment and taste become the scarcer, more valuable skill.

Aishwarya argues that because building is now 'ridiculously cheap,' the differentiator shifts to knowing what's actually worth building - understanding the problem deeply enough to make good design and prioritization calls - rather than raw execution speed.

ai-product-development

User trust in an AI product changes user behavior in ways that can break the product's original design assumptions.

In an underwriting-assistant deployment, users were satisfied for three to four months, then - having built trust in the system - started asking much more open-ended questions ('what did previous underwriters do for a case like this?') that required fundamentally different retrieval and reasoning than the original design anticipated. The lesson: calibration isn't a one-time step; growing user trust itself is a trigger to recalibrate.

agency-and-control

Model or infrastructure changes (e.g. a model deprecation) reset your calibration and require re-running the feedback loop.

Aishwarya notes that events like GPT-4o being deprecated force a switch to a model with materially different properties, which invalidates prior calibration work. Teams need to treat these transitions as a trigger to re-run the observe-diagnose-fix loop, not assume prior tuning still holds.

evals-and-monitoring

Books referenced

When Breath Becomes Air - Paul Kalanithi - Aishwarya's most-recommended book; she cites Kalanithi's inversion of Socrates - 'if the unexamined life is not worth living, was the unlived life worth examining' - as a check against over-optimizing life the way you'd optimize a product.
The Three-Body Problem (series) - Liu Cixin - Kiriti's favorite sci-fi series; he ties its themes of abstract science quietly compounding into civilization-scale effects to how AI's value can be invisible day to day.
A Fire Upon the Deep - Vernor Vinge - Lenny recommends it back to the guests as a lesser-known AGI/superintelligence sci-fi novel he found via a Noah Smith newsletter recommendation.

Media referenced

Silicon Valley - show - Aishwarya is rewatching it and says its satire of tech culture and hype cycles feels timeless applied to the current AI wave.
Expedition 33 - other - Kiriti's recent favorite (video game), praised for gameplay, story, and music; unrelated to AI but cited as a rare moment of downtime.
UC Berkeley / Databricks enterprise AI reliability survey - paper - Aishwarya cites this survey (authors including Matei Zaharia and Ion Stoica) finding ~74-75% of enterprises name reliability as their top blocker to deploying customer-facing AI products.

Companies

OpenAI - Kiriti currently works on the Codex team; used as the reference point for how a coding-agent org balances evals against vibes and customer feedback.
Google - Kiriti spent roughly a decade building AI/ML infrastructure here before OpenAI.
Amazon - Aishwarya was an early AI researcher on Alexa; also one of the companies among the guests' 50+ AI product deployments.
Databricks - One of the companies among the guests' 50+ deployments; also co-authored the reliability survey Aishwarya cites.
Rackspace - Aishwarya worked with its CEO, who blocked 4-6am daily to catch up on AI news rather than take meetings - held up as a model of hands-on leadership.
Anthropic - Referenced via Claude Code's public stance (per its head, Boris Cherny) that it relies on 'vibes' over formal evals, which the hosts use to set up the evals-vs-monitoring discussion.
Air Canada - Cited as a cautionary real-world incident where an AI agent hallucinated a refund policy the airline was then legally bound to honor.
Kumo - Prior company where Kiriti built AI and ML infrastructure before OpenAI.

Techniques and frameworks

Continuous Calibration, Continuous Development (CCCD) - The guests' named framework (modeled on CI/CD) for building AI products: scope capability and curate a data set, deploy with evaluation metrics, monitor production for unexpected behavior, diagnose new error patterns, then either quick-fix or build new metrics and repeat.
Agency-control tradeoff - Core mental model: every unit of autonomy handed to an AI system is a unit of human control given up, so autonomy should only increase as the system earns demonstrated trust.
Problem-first approach - Discipline of defining the exact problem and constrained scope before reaching for agentic complexity, to avoid the 'slippery slope' of chasing solution sophistication.
Evals vs. production monitoring - Evals are a curated test set encoding known failure modes; production monitoring (explicit and implicit signals like regenerate-clicks) surfaces the emerging failure patterns evals can't anticipate. The guests argue both are necessary, neither is sufficient alone.
Success triangle (leaders, culture, technical prowess) - Framework the guests use to diagnose why AI initiatives succeed or fail at the org level, independent of the underlying technology.

Summary

Lenny Rachitsky hosts Aishwarya Naresh Reganti and Kiriti Badam, who together have worked on more than 50 AI product deployments across OpenAI (where Kiriti currently works on the Codex team), Google, Amazon, Databricks, and a range of startups and enterprises, and who teach a top-rated AI product course on Maven. The conversation is built around a single core claim: building AI products is fundamentally different from building traditional software, in two specific ways. First, AI products are non-deterministic on both sides - users express intent through unconstrained natural language, and the underlying LLM's output is sensitive to phrasing and effectively a black box - so teams can't map a fixed input to a fixed output the way they could with a traditional deterministic app. Second, every unit of autonomy an AI system is given is a unit of human control surrendered, and that autonomy has to be earned by demonstrated reliability rather than assumed.

From that framing, the guests lay out their central prescription: start any AI product with high human control and low agent agency, and only expand agency once the system has proven itself trustworthy in narrower, lower-risk versions. They walk through concrete progressions - a customer support agent moving from suggesting answers to a human, to drafting answers a human lightly edits, to fully autonomous resolution; a coding assistant moving from inline completions to larger auto-generated PRs the developer approves, to autonomous PRs. This staged approach is formalized as their "Continuous Calibration, Continuous Development" (CCCD) framework, modeled explicitly on CI/CD: scope the capability and curate a starting dataset, deploy with evaluation metrics, then continuously monitor production for behavior the team didn't anticipate, diagnose new failure patterns, and either quick-fix them or build new evaluation metrics before repeating the loop.

A substantial segment addresses the "evals" debate that dominated AI Twitter through 2025. The guests reject the framing that evals or production monitoring alone can solve reliability - evals are a curated dataset encoding known failure modes, while production monitoring (both explicit signals like thumbs-down and implicit ones like a user regenerating an answer) surfaces the emerging patterns nobody anticipated. Kiriti invokes Martin Fowler's "semantic diffusion" to explain why "evals" has become a confusing catch-all term meaning different things - error analysis, LLM judges, or the entire feedback loop - to different practitioners, none of whom are technically wrong. He also describes how the Codex team at OpenAI balances structured evals for guarding core product integrity against heavy reliance on direct customer feedback and "vibes," since a coding agent's use cases are too varied to fully specify in advance.

The guests frame organizational success around a "success triangle" of leaders, culture, and technical execution, arguing the technology is rarely the bottleneck. They cite a Rackspace CEO who blocked 4-6am daily to personally stay current on AI rather than delegate, and stress that leaders must be willing to relearn intuitions built over a pre-AI career. Culturally, they warn against fear-driven adoption where subject-matter experts withhold input because they feel threatened, versus an empowerment-driven culture where experts help calibrate agent behavior. Aishwarya also cites a UC Berkeley/Databricks survey finding roughly 74-75% of enterprises name reliability, not model capability, as their top blocker to shipping customer-facing AI products - which she says explains the current skew toward internal productivity tools over end-to-end customer-facing agents.

In a lightning round, the guests share recommendations spanning "When Breath Becomes Air" by Paul Kalanithi, the "Three-Body Problem" trilogy, and "A Fire Upon the Deep"; favorite recent products (Whisper Flow, Raycast, "caffeinate" for long-running local agent tasks); and closing philosophies. Kiriti's "pain is the new moat" framing closes out the substantive discussion: he argues durable competitive advantage in AI products comes not from being first to market or having a flashy feature, but from the accumulated, hard-won organizational knowledge of what doesn't work - knowledge competitors can't easily copy. Aishwarya extends this into a broader claim about the AI era: since implementation is becoming "ridiculously cheap," design judgment and taste - knowing what's actually worth building - become the scarcer and more valuable skill, for both companies and individual careers.

Notable Quotes

"It's about have you built the right flywheels in place so that you can improve over time." - Aishwarya Naresh Reganti

"If someone's selling you one-click agents, it's pure marketing. You don't want to buy into that." - Aishwarya Naresh Reganti

"Pain is the new moat." - Kiriti Badam

"Building is really cheap today. Design is more expensive." - Aishwarya Naresh Reganti