Inside OpenAI Enterprise: Forward Deployed Engineering, GPT-5, and More | BG2 Guest Interview

2025-09-11 - 69 min - source - Read full transcript

Apoorv Agarwal (host)Sherwin WuOlivier Godement

Key insights

Successful enterprise AI deployments require top-down buy-in paired with an empowered bottoms-up 'tiger team,' an evals-first culture, and patient iteration up the performance curve.

Godement said the clearest predictor of failure is skipping evals grounded in institutional knowledge, much of which lives only in employees' heads rather than documentation (even in supposedly well-documented functions like customer support). Climbing from a mediocre eval score toward 99% is, in his words, 'more art than science,' sometimes requiring OpenAI to fine-tune its own models to close the gap.

enterprise-ai-deployment

Forward deployed engineers (FDEs), a term borrowed from Palantir, embed directly with customers to wire models into systems that often lack clean APIs, since raw models know nothing about a company's CRM or internal tools.

Wu described FDEs standing up API gateways, connectors, and custom evals from scratch for customers like T-Mobile, and said OpenAI is actively growing that team given how effective it has been.

forward-deployed-engineering

T-Mobile's year-long deployment of OpenAI voice models into its customer support line fed real production learnings back into OpenAI's own models, shaping the GA real-time API snapshot released the week before this conversation.

Because OpenAI was so deeply embedded with T-Mobile, it could define what 'good' sounded like for a live voice support call and bring those lessons into the general model release, beyond T-Mobile-specific tuning.

forward-deployed-engineering

Los Alamos National Laboratory required a fully custom, air-gapped, on-prem deployment of OpenAI's o3 model onto its Venado supercomputer, including physically carrying the model weights into a facility that bans phones and other electronics.

Government security requirements meant researchers could not simply call OpenAI's API, so OpenAI's FDE and developer teams installed o3 directly on Los Alamos's own hardware and networking stack. The deployment is now shared remotely with Lawrence Livermore and Sandia labs, though OpenAI has almost no visibility into how it's actually used beyond limited customer-reported telemetry.

enterprise-ai-deployment

Physical autonomy (self-driving cars) has outpaced digital autonomy (AI agents) despite a much lower safety bar for the latter, mainly because roads and traffic laws are decades-old 'scaffolding' that AI agents mostly lack inside enterprises.

Wu argued agents are 'dropped in the middle of nowhere' without standardized interfaces, and that FDE work at successful customers is really about building that missing scaffolding: connectors, organized data, platform layers, so models have something structured to act on. Self-driving took roughly 10-15 years to mature, versus AI agents' roughly one year of real progress since reasoning models (o1) appeared.

enterprise-ai-deployment

GPT-5 was built to optimize model behavior (tone, instruction-following, willingness to say 'I don't know') as much as raw intelligence, using months of direct customer feedback rather than benchmark performance alone.

Godement called it the first major release built in that close a feedback loop with customers, and said the resulting improvement in usability, not just benchmark scores, is what builders responded to after launch.

model-capability-frontier

GPT-5's dramatically improved instruction-following created a 'monkey's paw' problem: customers' old prompts, tuned with repeated 'be concise' instructions to fight weaker models, made GPT-5 answers too terse.

OpenAI found that removing the extra conciseness instructions customers had layered on for older models restored GPT-5's answers to the expected length and quality, underscoring that prompt engineering still matters even as models improve.

model-capability-frontier

OpenAI is moving voice from a 'stitched' pipeline (speech-to-text, then LLM, then text-to-speech) to a unified real-time speech-to-speech model, to cut latency and stop losing emotional signal like tone and pauses at each stitch point.

The team chose to first make the voice experience production-ready with a modality-specific architecture and is now working backward to unify orchestration logic across text and voice. Most customers still use the stitched approach today because it's what worked with the last model generation.

model-capability-frontier

Reinforcement fine-tuning (RFT) layers reinforcement learning onto customer-provided gradable task data to push a model toward best-in-class performance on a narrow task, unlike supervised fine-tuning (SFT), which only steers behavior toward example completions.

RFT requires a set of clearly gradable tasks and an objective grader rather than prompt-completion pairs. Wu expects RFT to become the norm for teams pushing the frontier of capability in a specific domain, as base models get good enough at instruction-following that behavior steering alone no longer requires fine-tuning.

reinforcement-fine-tuning

Startups Rogo (financial-document parsing, staffed partly with ex-DeepMind researchers) and Accordance (tax, benchmarked against TaxBench) used RFT on their own data to reach state-of-the-art results in their verticals.

Godement noted that building the gradable datasets RFT needs requires deep, bottoms-up subject-matter expertise, the same pattern seen in successful enterprise deployments where domain knowledge lives with practitioners, not top-down mandates.

reinforcement-fine-tuning

Both guests are structurally bullish that AI will expand demand for software engineering rather than eliminate it, arguing rigid, hard-to-build software was never the ideal, only a byproduct of how expensive building used to be.

Godement said OpenAI now has product managers ship coded prototypes instead of written PRDs, using GPT-5 and Codex to build working demos in hours rather than pages of documentation, and expects that pattern of far more people coding, not fewer engineers, to spread broadly.

ai-future-of-work

Both guests describe themselves as 'AGI-pilled' from distinct reference points: Wu from realizing in 2023 he would never need to hand-write code again and from voice/multimodal progress, Godement from talking to GPT-4 immediately after joining OpenAI pre-ChatGPT and from watching Deep Research persistently work through multi-step queries he expected it to fail.

Both frame their conviction as a repeated personal test: throwing a task at the model that they didn't expect it to handle, and being surprised when it succeeded anyway.

ai-future-of-work

Media referenced

MIT report on AI deployment failure rates - paper - Referenced as the '95% of AI deployments don't work' finding that shook markets, used to frame the discussion of what separates successful enterprise deployments from failed ones.
Reddit post about a non-verbal brother - article - Wu cited a Reddit post shared internally at OpenAI about someone who used ChatGPT to build custom accessibility software for their non-verbal brother, as evidence that non-engineers can now build real software.
Andrej Karpathy tweet on GPT-5 Pro - other - Referenced in passing as praise for GPT-5 Pro one-shotting previously unsolved problems; noted that Sam Altman retweeted it.
Blog post chart comparing self-driving car and AI agent autonomy slopes - article - Apoorv references a chart from his own blog post showing tangible/felt autonomy over time for self-driving cars versus AI agents, used as the jumping-off point for the physical-vs-digital autonomy discussion.

Companies

OpenAI - Employer of both guests, who lead engineering and product for the OpenAI Platform (API, enterprise, public sector).
T-Mobile - Deployed OpenAI voice and text models to automate customer support, including live phone calls; case study for forward deployed engineering and the new real-time API.
Amgen - Healthcare/pharma customer using GPT-5 to speed up drug development R&D and the heavy admin/documentation work of bringing a drug to market.
Los Alamos National Laboratory - Government research lab given a custom, air-gapped, on-prem deployment of OpenAI's o3 model on its Venado supercomputer for national security research.
Lawrence Livermore National Laboratory - Shares remote access to the same Venado-based o3 deployment as Los Alamos.
Sandia National Laboratories - Also shares remote access to the same Venado-based o3 deployment.
Palantir - Origin of the 'forward deployed engineer' term that OpenAI borrowed; also Wu and Apoorv's shared employer background (2013 class).
Stripe - Olivier Godement's employer for nearly a decade before OpenAI, where he won an internal 'Golden Llama' award.
Quora - Prior employer of Sherwin Wu before OpenAI.
Opendoor - Prior employer of Sherwin Wu, through its IPO, before OpenAI.
Riot Games / League of Legends esports - Cited by Wu as his 'long' pick in the rapid-fire round, betting on continued growth of professional esports.
Waymo - Referenced as a comparison point for autonomous vehicle scale and revenue versus AI agent products.
Tesla (FSD) - Referenced for its billions of full self-driving test miles as a benchmark of physical autonomy scale.
Expo (cybersecurity company) - Cited as a customer with a major performance upgrade from switching to GPT-5.
Rogo - Financial services startup using OpenAI's reinforcement fine-tuning (RFT) to get best-in-class results parsing financial documents.
Accordance - Tax-focused startup using RFT to target state-of-the-art results on the TaxBench eval.
Parloa - Voice AI company in Altimeter's portfolio building customer support products on speech-to-speech models.
LiveKit - Voice infrastructure company in Altimeter's portfolio supporting real-time voice AI deployments.
Granola - Named by both guests as their favorite underrated AI tool, for meeting transcription/summarization and calendar integration.
Codex CLI (OpenAI) - Named by Wu as his favorite underrated AI tool, especially paired with GPT-5, for coding and for understanding unfamiliar codebases.

Techniques and frameworks

Forward Deployed Engineering (FDE) - OpenAI's model, borrowed from Palantir, of embedding engineers directly with enterprise customers to build custom integrations, connectors, and evals.
Reinforcement Fine-Tuning (RFT) - OpenAI's technique layering reinforcement learning onto customer-provided gradable tasks to push a model to best-in-class performance in a narrow domain, GA'd after being announced in OpenAI's '12 days of Christmas' event.
Supervised Fine-Tuning (SFT) - OpenAI's original fine-tuning approach using prompt-completion pairs to steer model behavior, contrasted with RFT's use of reinforcement learning.
Evals-first deployment methodology - Godement's framework for successful enterprise rollouts: define a clear, bottoms-up eval set before building, then iterate toward it, since 'moving target' goals without evals are the top failure pattern he's observed across hundreds of deployments.
Tiger team model - Combining top-down executive buy-in with a small cross-functional team mixing technical skill and institutional/domain knowledge, identified as a leading indicator of deployment success.
Real-time speech-to-speech API - Unified voice architecture replacing the 'stitched' pipeline of separate speech-to-text, LLM, and text-to-speech models, to cut latency and preserve tonal/emotional signal.

Summary

Apoorv Agarwal sits down with Sherwin Wu (Head of Engineering, OpenAI Platform) and Olivier Godement (Head of Product, OpenAI Platform) for a wide-ranging look at OpenAI's enterprise business, which predates ChatGPT and, in the guests' framing, is core to OpenAI's mission of distributing AGI's benefits broadly rather than only through the consumer chat product. They walk through three case studies at very different scales: T-Mobile, where a year-long forward deployed engineering (FDE) effort automated voice and text customer support and fed real-world learnings back into OpenAI's own real-time API; Amgen, where GPT-5 is used to accelerate both pharma R&D and the heavy regulatory/admin work of bringing a drug to market; and Los Alamos National Laboratory, which required a fully custom, air-gapped, on-prem install of the o3 model on its Venado supercomputer, including physically carrying model weights into a facility that bans phones.

Against the backdrop of the MIT report claiming 95% of AI deployments fail, Godement lays out a pattern he says predicts success across the "couple hundred" deployments he's seen: top-down executive buy-in combined with an empowered, cross-functional "tiger team," an evals-first culture (since institutional knowledge mostly lives in people's heads, not documentation), and patience climbing from a mediocre eval score toward the target. Wu connects this to a broader observation about autonomy: self-driving cars have outpaced AI agents in real-world deployment despite a far higher safety bar, mainly because roads and traffic laws are decades-old scaffolding that most enterprises simply don't have for AI agents yet. Much of what FDEs actually do, in his telling, is build that missing scaffolding, connectors, organized data, standardized interfaces, so a model has something structured to act on.

The conversation turns to GPT-5's development, where Godement emphasizes that behavior (tone, instruction-following, knowing when to say "I don't know") mattered as much as raw intelligence, shaped through an unusually close customer feedback loop. That same improved instruction-following produced a "monkey's paw" side effect: customers whose old prompts were stuffed with repeated "be concise" instructions to fight weaker models suddenly got overly terse answers from GPT-5, since it followed those instructions almost too literally. On model customization, Wu details reinforcement fine-tuning (RFT), OpenAI's newer technique that layers RL onto customer-supplied gradable tasks to push a model toward best-in-class performance in a narrow domain, citing Rogo (financial documents) and Accordance (tax, targeting TaxBench) as startups that used it to hit state-of-the-art results in their verticals.

A rapid-fire round covers lighter ground: Wu is long esports (specifically League of Legends' professional scene) and short the entire AI tooling/evals/RL-environments category, arguing the stack changes too fast for tooling companies to keep up; Godement is short memorization-heavy education and long healthcare/life sciences as the industry best positioned to benefit from AI, given how much of pharma work is admin rather than research. Both name Granola as their favorite underrated AI tool, with Wu adding Codex CLI paired with GPT-5. On the future of software engineering, both argue AI will expand demand for building software rather than shrink the profession, citing OpenAI's own product managers now shipping coded prototypes instead of written PRDs, and a Reddit post about someone using ChatGPT to build custom accessibility software for a non-verbal sibling as evidence anyone can build now. They close by naming personal highs (GPT-5's smooth, high-scale launch; OpenAI's first Dev Day) and lows (the November 2023 board upheaval, and a multi-hour December outage), and both confirm they're "AGI-pilled," each citing distinct moments, from never needing to hand-write code again, to first talking to GPT-4, to watching Deep Research persist through hard multi-step queries.

Notable Quotes

"Physical autonomy is ahead of digital autonomy in 2025... The bar for physical safety is so much higher... But yet, physical autonomy is ahead of digital autonomy, which seems counterintuitive." - Apoorv Agarwal

"AI agents are just kind of dropped in the middle of nowhere, and they kind of have to feel around for them." - Sherwin Wu

"Evals are much harder than what it looks to get done. And evals also oftentimes need to come up bottom up... it's actually very hard to have a top-down mandate." - Olivier Godement

"It's almost like the monkey paw where it's like developers and platform customers ask for better instruction following. They're like, yes, we'll give you really good instruction following, but it's like, you know, it follows it almost to a T." - Olivier Godement

"I actually think the reason why we have a platform and why we started with an API... comes back to the OpenAI mission... to distribute the benefits of it to everyone in the world, to all of humanity." - Sherwin Wu