OpenAI’s head of platform engineering on the next 12-24 months of AI | Sherwin Wu

00:00:0095% of engineers use Codex. 100% of ourPRs are reviewed by Codex for engineers.I don't know what job has changed morein the past couple years. Engineers arebecoming tech leads. They're managingfleets and fleets of agents. Itliterally feels like we're wizardscasting [music] all these spells. Andthese spells are kind of like going outand doing things for you. What do youthink people aren't pricing in yet? Thesecond or third order effects of theoneperson billion dollar startup toenable a one person billion dollarstartup. There might be a hundred othersmall startups building bespokesoftware. And so I think we mightactually enter into a golden age of B2BSAS.>> I've been hearing more and more there'sthis stress people feel when theiragents aren't working. There's a teamthat's actually doing an experimentright now within OpenAI where they aremaintaining a 100% codeex writtencodebase. They run into the exactproblems that [music] you're describing.And so usually you're like all rightI'll roll up my sleeves and figure itout. Team doesn't have that escapehatch.>> You've shared that listening tocustomers not always the right strategyin AI. The field and the modelsthemselves are just changing so soquickly. They tend to like disruptthemselves. The models will eat yourscaffolding for breakfast. What's youradvice to folks that are like, "Okay, Idon't want to miss the boat." Make sureyou're building for where the models aregoing and not where they are today.There's a quote from Kevin Whale, our VPof science here. He likes saying, "Thisis the worst the models will ever be."Today, my guest is Sherwin Woo, head ofengineering for OpenAI's

00:01:15API anddeveloper platform. Considering thatessentially every AI startup integrateswith OpenAI's APIs, Sherwin has anincredibly unique and broad view intowhat is going on and where things areheading. Let's get into it after a shortword from our wonderful sponsors.Today's episode is brought to you by DX,the developer intelligence platformdesigned by leading researchers. Tothrive in the AI era, organizations needto adapt quickly. But many organizationleaders struggle to answer pressingquestions [music] like which tools areworking? How are they being used? What'sactually driving value? DX [music]provides the data and insights thatleaders need to navigate this shift.With DX, companies like Dropbox,Booking.com, Adion, [music] and Intercomget a deep understanding of how AI isproviding value to their developers andwhat impact AI is having on engineeringproductivity. To learn more, visit DX'swebsite at getdx.com/lenny.That's getdx.com/lenny.

00:02:14Applications break in all kinds of ways.Crashes, slowdowns, regressions, and thestuff that you only see once real usersshow up. Sentry [music] catches it all.See what happened where, and why, downto the commit that introduced the error,the developer who shipped it, and the[music] exact line of code all in oneconnected view. I've definitely triedthe [music] five tabs and Slack threadapproach to debugging. This is better.Sentry shows you how the request moved,what ran, what slowed down, and whatusers [music] saw. Seir, Sentry's AIdebugging agent, takes it from there. Ituses all of that Sentry context to tellyou the root cause, suggest a fix, andeven opens a PR for you. It also reviewsyour [music] PR and flags any breakingchanges with fixes ready to go. TrySentry and SER for [music] free atcentry.io/lennyand use code Lenny for $100 in Sentrycredits. That's s nt r y.io/lenny.

00:03:15Sherwin, thank you so much for beinghere and welcome to the podcast. Thankyou. Thank you for having me.>> I want to start with what's feeling likea barometer of progress in AI,especially in engineering. Whatpercentage of your code, if you evenwrite code anymore, and your team's codeis written by AI at this point?>> I do write code occasionally now still.Uh and I actually say for managers likemyself, it's way easier to use these AItools uh than to manually code at thispoint. And so I know for myself and someof the other emuring managers at OpenAI,uh all of our code is written by bycodeex uh at this point. But morebroadly, there's just been this there'sjust so much energy. There's like atangible energy internally around justhow far these tools have gotten, howgood Codeex as a tool has gotten for us.And uh it's it's a little hard for us toexactly measure how much of the code isis written because the vast majority ofit I'd say like close to 100% is isusually generated by AI first. What wedo track though is is you know at thispoint the vast majority of engineers usecodeex on a daily basis. So 95%of engineers um use codeex. Um 100% ofour PRs are reviewed by codeex daily aswell. So basically any code that goesinto production that's merged in Kodaskind of has its eyes on and uh suggestsimprovements suggests

00:04:31changes uh uh inthe PRs. And so uh that's kind of whatwe're seeing internally. But by andlarge the most exciting is just theenergy that that there that there is. Umanother observation that we've had is uhengineers who tend to use codecs more uhopen way more PRs. So uh they'reactually opening 70% more PRs uh and uhthan than the engineers who aren't usingcodecs as much. Uh and the gap iswidening. So I feel like you know thepeople who are opening more PRs um arestarting to you know learn how to usethe tool more and more get moreefficient and that 70% gap keeps uh uhgrowing over time and so might haveactually increased since I last lookedat the at the number.>> Okay. So just to make sure we hear whatyou're saying you're saying all of thecode of these 95%uh engineers at at OpenAI is written byAI. It's written and then they reviewit.>> Yep. Yep.>> It's It's like crazy that that's almostlike not crazy anymore that we're justlike getting used to this. I thinkthere's still some getting used to to beclear. Uh there's also I think some youknow uh engineers who I think trust uhcodeex a little bit less but umbasically every day I talk to someonewho who uh is blown away by somethingthat I can do and and kind of like thetheir bar of of trust kind of uh or likehow much they trust the

00:05:46model to do onits own goes up over and over uh overtime. And there's a quote from KevinWhale our our um VP of of science hereand he likes saying this is the worstthe models will ever be. And so this isthe worst that the models ever be forsoftware engineering as well. And soover time you just see people trustingit more and more and then we'll see themodels get better and better as well.>> Yeah. Kevin Wheel, former podcast guest,uh he he said exactly that line on thispodcast and a few times.>> Yeah. Uh Peter the Claudebot/moldbotclawis what it's called now. uh developer uhrecently shared that he uses codecs forhis work and he feels like anytime itdoes things he just trusts that it hasdone the right job and he's just likealmost certain he could just commit itto master and it'll be great.>> Yeah. Yeah. He's a great um user ofcodeex. I know he's in close touch withthe team gives us great feedback. Um notsurprised that he uses it. I mean uhsorry it's called open claw.>> Open claw. Yeah.>> Open claw is a great is a great product.And then I saw that this I mean this isvery recent but this morning I thinkmold's book uh kind of like uh uh wasshared as well and seeing all the uh AIagents talk to each other is pretty uhpretty surreal. It's basically her ishappening in real life is what I'mhearing.>> Yeah. Yeah.>> So just like coming back to this crazymoment we are living through fourengineers in particular. We've

00:07:01gone fromyou write every line of code to now AIis writing all of your code. I don'tknow what job has changed more in thepast couple years like job that wedidn't expect to change this much wherejust like the job of an engineer is sodifferent in the entire lifespan of anengineer like in the past couple yearsit's now shifted to I don't write anymore code how do you imagine the role ofan engineer and the job of a softwareengineer looks in the next couple yearsjust like what is that job yeah it's Imean it's honestly being really cool tosee um uh and it's part of where theexcitement is because uh like the job islikely going to change prettysignificantly over the next one to twoyears. It kind of feels like we're stillfiguring things out though and sothere's like this excitement I knowespecially from some of the softwareengineers of like we're in this raremoment you know maybe over the next 12to 24 months where we'll kind of get tofigure things out ourselves and set ourstandards for ourselves in terms ofwhere I see uh I see this moving. So Ithink there's a common thing thateveryone's saying which is uh you knowpeople are generally like IC engineersare becoming tech leads. They'rebasically like managers now. They'remanaging fleets and fleets of agents. UmI know many of the engineers on my teambasically have like 10 to 20 uh threadskind of being pulled on at the sametime. Obviously not active runningcodeex

00:08:16uh jobs but uh just a lot ofparallel threads. They're checking in onwhat they're doing. They're steering theagents and codeex and and and and givingit feedback. And so their job has kindof really changed from just writing thecode itself into being almost like amanager. In terms of where I think thiswill go one to two years from now. Soone uh kind of metaphor that that I kindof always come back to here is actuallyfrom this uh is from this uh programmingtextbook uh that I read back in collegecalled sikp. I don't know if you'veheard of it. Uh structure and uhinterpretation of computer programs. Sosi sicp.um at at MIT it was really popular andand it was actually used as the uh uhintroductory it was the textbook for theintro programming course for a very longtime um and it kind of has this cultfollowing um it teaches you programminguh it teaches you a dialect of listcalled scheme uh and so it likeintroduces you to like functionalprogramming it's like very mindopeningin that way but the thing that wasmemorable for me about that book so I II kind of read it in college um the verybeginning of it kind of describesprogramming as a discipline and drawsthis metaphor to basically like sorcery.Like it says like software engineers arelike wizards and you're like likeprogramming languages are likeincantations

00:09:31and you're like you knowyou're you're saying you're issuingthese spells and these spells are kindof like going out and doing things foryou and and the challenge is like whatincantation do you have to say to makethe the program do what you want. Andthis book was written in 1980 so this isthis is a while ago and I think thatmetaphor is actually like kind ofpersisted over time. And I think it'sactually playing out as we move intothis uh new era of vibe coding or justlike what software engineering will looklike because programming languages werebasically these incantations. They'vechanged over time and the challenge hasalways and and the trend has been thatthese it's been easier and easier tokind of get them the the the computer todo what you want uh via programming. AndI think the current wave of AI is isprobably the next stage of thatevolution. it is now literallyincantations because you can tell youknow your uh you can tell codex you cantell cursor uh exactly what you want todo and then it'll all go do it for you.Uh, and I particularly like the wizardand like the the the sorcery analogybecause uh I think our current state isis starting to move towards kind of likethe the sorcerers apprentice uh you knowfrom Fantasia uh where Mickey Mouse islike you know he finds the sorcerer hatand he tries to do all these things andI actually think it's a really aptanalogy because one uh it's just it'sreally powerful now these incantationsyou can do can is is extremely highleverage but you kind of have to knowwhat you're doing

00:10:46right like inSorcerers Apprentice the whole plot islike Mickey goes wild the the broomslike go crazy and everything's flooding.I think he literally sets the like setsthe uh the brooms off on a task and thengoes asleep. Uh and and so, you know,it's like vi coding at it at its at itsgreatest and then eventually the the oldsorcerer comes back and like cleanseverything up. And um you know when Isee engineers kind of like doing thesethese these these 20 different uh codeexthreads at a time there there is someskill and there's some seniority andlike you know uh um a lot of thoughtthat needs to go into this because youwant to make sure that the the themodels aren't going off the rails. Uhyou definitely don't want to just likecompletely uh go away and and you knowlike ignore ignore the thing. But it'salso extremely high leverage like youknow a a very senior engineer who'swho's really prol uh proficient withthese tools uh can now just do way morethings via what they're doing. And and Ithink this is also what makes it fun.Like it literally feels like we'rewizards now. You know, it feels likewe're closer to to to to having uh uh toto making making it feel like this likemagical experience where we're, youknow, casting all these spells andhaving software do all these things foryou.>> I was thinking of the sorcerersapprentice exactly as the metaphor asyou were describing that. So I'm gladyou went there. Uh a previous podcastguest described it as you have a geniethat you can that grants you wishes andit's a useful

00:12:02frame because you have tobe very clear about the wish you wantlike if you want to be big like how bigit could be.>> Yeah. Or it might be like themonkeykey's paw type thing where youknow it's like you got what you want butwhat are the side effects?>> Um yeah. Yeah. I think that and theanalogy is great and um yeah the crazything for me is just the staying powerof that book. Sik be like it's calledthe wizard book. you know, people callit the wizard book because that is themetaphor that they kind of weavethroughout the the book. And um we'rewe've basically reached that point now,which is which is which is really cool.There's two kind of threads I want tofollow here. One is I've been hearingmore and more there's this like stressthat people feel when their agentsaren't working. You fire off all these,you know, codeex agents and then youhave to keep stay on top of them. Oh one's not working. I'm wastingtime. Uh do you do you feel that? Do youfeel that across your team at all?>> Yeah. Yeah. I mean, it happens all thetime. And I actually think like this iswhere the interesting part of all ofthis lies right now because these modelsaren't perfect. These tools aren'tperfect and we're still trying to figureout how to best interact with these uhwith with with codecs or with these AIagents to to get work done. We see thiscome up all the time. There's aparticularly interesting team that wehave internally. So there's a team thatthat's actually doing an experimentright now uh within OpenAI where theyare basically maintaining a 100% codeexwritten codebase. Uh so you know likeyou know uh uh some you

00:13:17know you'll havethe AI write code but you'll obviouslyend up like rewriting a lot of it andand you might need to like double checkand change things but this team is justfully codeex pill and just like leaningin entirely. Uh and they run into theexact problems that you're describingwhich is like you know their challengeis you know uh you know I want to getthis thing this feature built but Ican't get the agent to do it. And sousually there's an escape hatch whereyou know then you're like all right I'llroll up my sleeves and like figure itout and then instead of using codeex Imight use like tab complete and andcursor and things like that but thisteam uh uh for the experiment this teamdoesn't have that escape hatch. Uh andso then the challenge like how do I getthe the the agent to to to do this? Andum I actually think we're going to bepublishing a blog post from some of ourlearnings here. Um but a lot offascinating like paradigms and bestpractices are falling out of this. Um,one interesting thing that we'venoticed, I I don't know if this is whatyou you kind of feel, but we definitelyfeel it here, is a lot of the time, uh,when the coding agent is not doing whatyou want, it's usually a problem withcontext and just like information thatyou've given it. It's just you've eitherunderspecified or there's just notenough information around how to dosomething available to the agent,available to codeex. Uh and so uh whenwhen you have to solve it throughthrough that uh the challenge is then toto to add documentation and actually

00:14:32work around this this limitation andbasically encode more tribal knowledgethat's in your head somehow into thecodebase either via you know codecomments itself or code structure itselfor via text files like you know MD filesskills any type of additional resourceswithin the repository so that the modelcan um uh can better do its task.There's a whole bunch of other learningsfrom this uh this group which I think isfascinating uh to to explore. But yeah,kind of giving removing that escapehatch of of no longer using AI hasallowed them to start piecing together alot of the problems that we'll have tosolve if we really want to lean intoagents.>> Another uh issue people run into, youtalked about how people are shipping PRslike crazy, a lot more PRs if they'reworking with AI. Uh obviously codereview is becoming a bigger challenge.Is there anything you've figured out inyour team to help speed that up to makethat scale as and not just create thisterrible job for people where they'rejust sitting there reviewing PRs allday?>> Yeah, I mean one thing is Codex reviews100% of all of our PRs at this point.And so uh I actually think so one onereally interesting thing that's happenedis the things that tend to we hand wetend to hand to the models immediatelytend to be the things that annoy us orlike are the most boring parts of uhsoftware engineering. It's also why it'smore fun now because we

00:15:47get to do more,you know, more of the fun things. Um,for me, um, speaking more for myself, Ireally hated code reviews. It was likeone of the worst things for me. And thenI remember in my first job, uh, out ofcollege, uh, it was at it was at Quora.Um, I owned I was working on thenewsfeed and so I owned the code for thenewsfeed. And so I was a reviewer forNewsfeed and uh it was just like thecentral piece of code that everyonewould touch. And so I would just everymorning I'd log in and be like like 20to 30 code reviews. I just like oh mygoodness I got to like you know getthrough all these. Um I wouldprocrastinate and then it grows to like50. And so there's like a a lot of codereviews. Codeex is really good atreviewing code. Uh so actually one thingthat we've noticed that 52 in particularhas gotten extremely strongly adept atis reviewing code and especially whenyou kind of steer it in the rightdirection. And so uh for code reviewsyeah we create a lot of PRs but Codexreviews all of them and it makes youknow code reviews go from a you know Idon't know 10 15 minute task tosometimes even just like a two to threeminute task because you have a uh abunch of suggestions uh already alreadybaked in. Uh a lot of the times peoplewill uh especially for small PRs likeyou you actually don't even need peopleto review. We kind of trust codeex inthis way. Um the original author kind oflooks at Codex. It is you know thebenefit of code review is to have asecond pair of eyes to

00:17:02make sure thatyou're not doing anything dumb. Codex isa pretty smart second pair of eyes atthis point and so that's something thatthat we've heavily leaned into. Um thegeneral CI process and like the post uhkind of push and like deployment processhas also been heavily automated viacodeex internally at this point. If youtalk to a lot of engineers the thingthat annoys them the most is afteryou've written your beautiful code likehow do you get it into production? Youknow you got to you got to run throughall these tests you got to like you knowlint errors you code review. Um there'sa lot of automated stuff you can do withcodecs and so we've actually built sometools internally that that help automatethat process, automate the lint, youknow, if there's like a lint error, it'sa very easy codeex fix. Uh and then justit could just patch it and then kind ofrestart the CI process. Um so all ofthat is we're trying to collapse as asinto as as little work for an engineeras possible which and and the byproductof which is uh um uh they can they cannow merge and push out a lot more peers.Codec's writing the code, Codexreviewing its own code. I'm curious ifyou are open to using other models toreview your models work. Is that is thata path or is it just it's good enough,we don't need anything else?>> So, I will say there's there'sdefinitely a circular thing here andlike going back to sources apprenticelike you want to make sure you're notletting the brooms go crazy here. Um,and so, you know, we're very thoughtful,I'd say, around which PRs kind of arecompletely just codeex uh

00:18:17reviewed. Mostpeople still obviously take a look attheir PRs. Uh, and so it's not like it'sgoing to zero. It's more like going fromyou know 100% attention to like 30%attention which which just helps thingspush through. Uh in terms of likemultiple models uh so we we obviouslytest a lot of models internally and sowe have a lot of those. Um we useexternal models less. Um it's we wethink it's important to kind of dog foodour own models and kind of like getfeedback there. But uh you can also youknow there are a lot of like internalvariants of models that you can use togive you a different perspective um hereas well and and we found that to to workquite well.>> Okay. So just to just to make sure weget like a barometer of today's world atOpenAI in terms of AI and code uh justso I understand and then I want to moveon to a different topic. Uh 100% of codeacross OpenAI is written by codeex atthis point. Is that the way to frame it?I wouldn't make the statement that 100%of code running in production today wasis written by AI. Uh and and it's kindof hard to to to do attribution there.But the like almost every engineerheavily uses codecs in all of theirtasks at this point. And so I you knowif I were to guesstimate like the vastmajority of code at this point is it wasprobably authored by incredible. Okay.So there's a lot of talk and we've beentalking about kind of the

00:19:32IC role thework of an IC engineer. There's lesstalk about the changing role of amanager especially an engineeringmanager. How has your life as a managerchanged with the rise of AI and justwhat do you where do you think managerswhat's the role of a manager in thefuture?>> It's definitely changed less than anengineer. Uh there's no you know codeexfor managers just uh just yet. However,I use codeex quite a bit for for some ofthe um uh some of some of the like kindof more managery tasks that I do. I'dsay a couple things are are changing.There like some trends. So I don't thinkit's changed that much yet. Um, but Isee trends and I think if you play itout, you can kind of see where where alot of this is going. One thing thatthat's becoming increasingly clear iscodeex really empowers like topperformers to to get a lot like to be alot more productive. And so it reallylike and I think this is maybe true forAI more broadly like across societywhich is like the people who really leanin or like the people who have highagency or like will get get good atthese tools will kind of superchargethemselves. Uh and so I'm kind ofnoticing this now as well, which is likethe top performers kind of end up uh uhuh being a lot more a lot moreproductive. Uh and so you see a broaderspread

00:20:48uh in in team productivity inthis way. One so one thing that I'vealways done as as a managementphilosophy is to spend uh actually themajority of my time with top performersjust like make sure they're unblocked,make sure they're happy, make sure youknow they're they feel productive andthey feel heard. I think this is evenmore true uh in an AI world where youknow your top firmers are going to justlike really be shooting ahead uh usingthese tools. I think I think one exampleis is that the team that's you knowmaintaining a 100% codec generatedcodebase like just letting them kind ofrip and and and see what's happeningthere is something that's that's paiddividends. So I think that that's kindof one one trend that I'm seeing wherewhere where um spending even more timewith top performers for managers I thinkis is likely going to um uh continue.The other thing is I I so this is moreuh an observation but my sense is with alot of these AI tools available tomanagers. So le less like writing codebut just things like chat GBT withorganizational knowledge like being ableto do research and understandingorganizational context a lot better.Another good example is uh um we'redoing performance reviews right now andit's actually really easy to use chatGBT with internal knowledge hooked up toGitHub and like our notion docs andGoogle docs to give it get a really goodsense of what this person has done overthe last 12

00:22:0312 uh months uh and writinga little you know deep research reportfor it. My sense is I think managerswill be able to manage much larger teamsin this world kind of like how you knowlike software engineers are managing 20to 30 codeexes. Um my sense that thesetools will allow managers peoplemanagers to be higher leverage um and uhit will allow them to to to manage youknow teams of way more than than thecurrent best practice of I think it'slike six to eight right for softwareengineering. You kind of see thisapplied to you know like uh the non uhengineering domains like support or uhoperations where it's like you knowpreviously um uh where previously likethe size of support team might belimited but like as you can pass offmore things to agents you can actuallydo more work and also manage more peoplethis way. I think the same thing mighthappen for um people management as well,especially in tech companies. Um andwe're already seeing this. There's someteams uh where there are EM managing youknow quite a few people and they'redoing it pretty adeptly because of someof these tools where they can get higherleverage and understand what theirteam's doing, understand organizationalcontext a little bit better uh andoperate in that way. I love this advicethat the way you described it is you'vealways leaned into top performers andspent more time with them, unblock them,make sure they're happy. The way MarkAndre and he was just on the podcast,

00:23:18the way he phrased it is AI makes goodpeople better and it makes great peopleexceptional.>> Yeah. Yeah.>> And what you're saying here is just justdoing this more and more is probably theright move. Spending more time with thebest people on your team to unblockthem, make sure they have everythingthey need.>> Yeah. A very good example right now isuh there are I would say like a a groupof engineers internally who are reallycodeex and are thinking through what thebest practices are for interacting withthis model and that is just an extremelyhigh lever thing for them to do and sojust like as a manager I'm just likeyeah go explore this you know uhwhatever best practices come out of thisyou know we we have to share with theorg we'll we'll you know uh we'll we'lluh we do all these knowledge sharingsessions we'll we'll like sharedocuments and like best practiceseverywhere So things like that just uhyou know elevate everyone and uh and I Iview that as like you know anotherexample of this trend um uh that um thatwe're seeing where the top performersreally get exceptional.>> People just like have a sense this isbig. AI is changing so much. The worldis changing. Uh it's going to be a hugedeal. What do you think people aren'tpricing in yet into what will changeinto where things are heading? Just likewhat's an example of something you thinkare like okay we're not realizing thisyet. So, one of my favorite kind of uhuh

00:24:33like phrases or like things that havecome out of this whole AI wave is is theidea of the one person billion dollarstartup. I think I actually think Sammay have ke or like uh Sam Sam may havebeen the first one to say it, but it'sfascinating to think about, right? It'slike yeah, if you know if people are sohigh leverage, at some point there willlikely be um a oneperson billion dollarstartup. Um, and while I think that'sreally really cool, I think peoplearen't really pricing in the second orthird order effects of this. And andreally what you know because becausewhat the one person billion dollarstartup implies is that there's youknow, one person can just have so muchmore agency and so much more leverageusing one of these tools um that it isjust super easy for them to geteverything done that they need to forfor their business to, you know,ultimately create something that's abillion dollars. But I think there are acouple other implications of this. Oneof them is uh uh if it's easy for aperson to create a one person bill or ifit's possible for a person to create aone person billion dollar startup, italso means it's way easier for people tojust create startups in general. Like Iactually think this will like one secondorder effect to this is I think there'sgoing to be a huge like startup boom andlike small like SMB style boom um whereanyone can build software for anything,right? like uh uh one uh you're kind ofstarting see starting to see this playout in

00:25:49the AI startup scene wheresoftware's became a lot more verticaloriented where like these verticals uhlike creating some AI tool for somevertical tends to work quite wellbecause you know you really lean into uhthat particular domain you like reallyunderstand the use case for it and so ifyou play out AI there's no reason whyyou can't have like 100x more of thesethese startups uh and So I think I thinkone world that we might end up seeinghappen is in order to enable a oneperson billion dollar startup theremight be like a hundred other smallstartups building bespoke software thatworks extremely well to support uh othertypes of you know small small onepersonyou know billion dollar startups and soI think we might actually end uh enterinto a golden age of like B2B SAS uh andjust like software and startups ingeneral and so I think I think that'sthat's a really interesting trend to tokind of see because as it's as it's asit gets easier and easier to buildsoftware, um as it's easier and easierto uh you know uh uh run a company umyou might actually just end up seeingway more of these these these startups.So the way I I' I've been thinking aboutis like yeah there might be one uh oneperson billion dollar startup but theremight be like a hundred you know uhhund00 million startups there might betens of thousands

00:27:04of $10 millionstartups and as an individual it'sactually pretty great to have a $10million business like that's like enoughfor you're set for life at that pointand so you know we might really see seean explosion in that way and and I feellike people aren't aren't really youknow pressing that in. Um there'sanother kind of like third order effectto this you know and again all of theselike as you get to the further andfurther out predictions I think uh arethere's a lot of uncertainty I think ifwe end up moving to this world where youend up with these like kind of microcompanies building software that worksfor one or two people uh who own thecompany and and and are working there umI think the startup ecosystem willchange I think the VC ecosystem willchange you know might we might end up inuh in a world where there's just like ahandful of big players that are offeringplatforms and supporting all of thesestartups. But, you know, the types ofventure scale return startups that canreally 100 or thousandx your yourinvestment might actually end upshrinking if you end up having a bunchof these, you know, smaller 10 to$50million uh companies. Uh, which are notgreat for venture solid returns, but aregreat for the individuals, the highagency individuals who are now, youknow, really leaning into AI to to tobuild these businesses for themselves.>> I love how many uh order like uh ordereffects we've been through. I want tohear the fourth order effect now.Sherwin,

00:28:19I'm just joking. I I can't It'stoo fourth order is too too is toogigabrain for me. I can't I can't thinkthat far ahead.>> It's like inception where justeverything gets slower every time you godeeper into something every layer. Uh,okay. So, the billion-dollar startupI've been I think about this a lotbecause I I'm not going to be a billiondollar startup because what I'm doing isnot venture scale in any way and notsuper high leverage, but just seeing howmany support tickets I get from justlike the most ridiculous things. It'shard for me to imagine one person likeI'm bearish on this billion dollarstartup. I just want to share thisthought uh simply because of the supportcosts even if AI is helping you at abillion dollars just like unless yourACVs are you know very high and you havevery few customers it's just dealingwith support and people are like youknow like they can solve their ownproblems but they're like I'll emailsupport ask about this thing justdealing with that is hard to scale is inmy experience so unless you have in myopinion unless you have a bunch ofcontractors which I don't know does thatcount as a single person company I feellike it's very difficult to scale abillion dollar startup and not havesomeone helping you with at least thesupport work and AI I think will onlytake you so far. So I I I think that'strue. Uh, and actually I think my viewon

00:29:34it is is is slightly different, whichis I think that your, you know, Lenny'spodcast might end up becoming abillion-dollar startup. But um what Ithink might happen is uh instead of youkind of being the one person who has todispatch an AI to solve and fix thosesupport tickets, I think what might endup happening is there might be a wholesmattering of other startups that arebuilding software and super and likesuper tailored towards what you mightneed. And so, you know, uh there mightbe like 10 or 20 startups that buildsupport software for podcasts andnewsletters and uh that might be aoneperson startup. Like it doesn't needto be a big one. And uh it's it's andyou know they might be able to just codeup this product very very easily. Theyare able to kind of like build their ownthing and because it's so tailored andunique and hopefully you know useful foryou. It might be something that youpurchase um as the one person billiondollar startup.>> I would buy that. I would buy that.>> Yeah. there's like a question of likewhat you in-house and what you what youlike kind of uh outsource and what Ithink might happen is because the costof writing software and buildingproducts is is is is collapsing so muchyou might end up outsourcing a lot ofthis and in doing so reducing the sizeof your company uh and so that's kind ofthe world that I think might end uphappening again there's like highuncertainty [clears throat] in whatmight play out here but

00:30:49the end resultstill might be a one like one persondriving this like high high massiveleveraged company that might actuallyreach a billion dollars>> I could see that I also think aboutPeter at Clawbots/ /moldbot/openclawof just like how barrageed he is rightnow by all these asks and emails andpings and DMs and PRs just like I'mcurious to and he's not even making anymoney off this thing.>> Um>> yeah, I can't imagine what it's like tobe him right now. It's must be likeabsolutely insane. It it's probably likeum uh you know like the the months afterwe launched Hatchvt the craziness thatwas>> as one as one man.>> Uh he's coming out on the pod by the wayin in [clears throat] a week.>> Oh, that's exciting. Yeah. Uh maybe thefourth order effect is distributionbecomes increasingly important becausethere are so many freaking things tryingto get your attention. So people with anaudience and platform I think becomemore and more valuable which is goodgood stuff. Okay. Uh I wanted to comeback actually to your management stuff.So I really loved your insight aboutspending more time with top performershas been really successful to you. Justthinking about you as a manager of ateam that is building the platform thatpowers basically the entire AI economylike every AI startup is building onyour API. Uh clearly you're doing agreat job. What other kind of core

00:32:04management lessons have you learned?What do you find is really important andand and key to your success as a managerof engineers and just people?>> Yeah. Um, I I think a lot of the lessonsthat I've learned here, I don't know howspecific it is to the OpenA API or orsome of our enterprise products inparticular. I think my my managementphilosophy has obviously changed overtime, but I think it it's uh probablystayed the same more than it's changeduh over time. Uh, one of theseprinciples is is kind of what I talkedto you about before, which is, you know,spending a lot of time with with topperformers, like actually spending andlike to be very concrete, like it's likemore than 50% of your time with your topperformers with maybe your top like 10%uh performers and really really tryingyour best to empower them. The way thatI think about it is um is is is kind ofcome back to this analogy of softwareengineer as as as a surgeon um whichcomes from the the mythical man book.So, it's actually it's funny. So I Ipull it from the book, but in the bookthey actually described this world whereum I think they were like predicting thefuture cuz cuz I think the book waswritten like in the 70s or something. Umthey said that software engineeringmight end up moving into a world wherethat software engineers are likesurgeons or like in a surgery roomthere's like one person doing the work.Um

00:33:20and you know there's one person likecutting or whatever and like doing allthe surgery and everyone else in theroom is there to just support them,right? as like the nurse and like theand the resident and the fellow and thenthe surgeon's like I need a scalpel andthey give them scalpel and then uhthey're like I need you know this tooland this machine and they'll bring itover. Everyone's there to just like youknow support the one uh surgeon and sothe the the myth mammoth actuallypredicted that that is kind of thedirection that software engineers goingto go. I don't think that's exactlyplayed out where like you know it's muchmore collaborative and like it's notonly one person doing the work but I'vealways really liked that analogy and andand and uh that analogy is actually whatI strive to uh uh kind of like emulatein my own management philosophy which isum software engineering isn't reallylike surgery where it's not just oneperson doing work but the way in which Ilike treating the people on my team andthe way that I act as a manager is Iwant to uh empower them make them feellike they're a surgeon um and in in sofar at like as like making sure that I'msupporting them and making sure theyhave everything that they need to to dotheir work and it feels like they havean army of people kind of supportingthem um and looking around corners andgiving them everything that they needwhen it's really just me as the as themanager. And so like the example that Igive is is looking around corners andunblocking people especially from anorganizational perspective is extremelyextremely useful. And again going back

00:34:35to the AI conversations even moreimportant nowadays right like uh if ifpeople are just like cranking PR afterPR the main thing bottlenecking uhprogress and and you know shippingsomething tends to be organizational orlike processoriented and if you as amanager can kind of look around cornersand kind of unblock the team if you canyou know like if if the surgeon needsscalpel but you know the manager kind ofalready has a scalpel ready for themthat that's the best case scenario.That's kind of the the way that Iapproach uh u um management and andespecially uh engineering management.And so that's something that that'sreally really um stuck with me overtime. And even though you know softwareengineers aren't exactly surgeons, thatmetaphor has always kind of stayed in mymind as of as of uh uh for the rest ofmy career.>> I love that. And I I feel like I wonderif that's something AI can help with islook around corners and predict herethis engineer is going to be blocked bythis decision. We need to figure thisout. We need to get>> Yeah, that's actually a really good uhpoint. I haven't tried this yet, but Iwonder what would happen if I ask uhChad GBT hooked up to company knowledge,you know, like what are the activeblockers? Uh look through all the notiondocs, what are maybe Slack messages, youknow, it's probably in Slack somewhere.What are the active blockers on my teamand is there something I can do to tohelp? Um now very I have not thoughtabout that, but you're right.>> You just had an insight right here.>> Yeah. Yeah.

00:35:50Yeah.>> Uh and it's I think even moreinterestingly, what do you anticipatewill be a blocker for this engineer orthis team in the in the coming months or>> Yeah. You asked the you asked the model.Well, you asked the AI to do the secondand third order things. Anticipate that,man. Anticipate what the bloggers willbe next month, too. Uh,>> I think we've got a we've got a goodidea right here.>> Yeah. Yeah.>> This episode is brought to you by DataDog, now home to EPO, the leadingexperimentation and feature flaggingplatform. Product managers at theworld's best companies use Data Dog, thesame platform their engineers rely onevery day to connect product insights toproduct issues like bugs, UX friction,and business impact. [music]It starts with product analytics wherePMs can watch replays, review [music]funnels, dive into retention, andexplore their growth metrics. Whereother tools stop, data dog goes evenfurther. It helps you actually diagnosethe impact of funnel drop offs and bugsand UX friction. Once you know where tofocus, experiments prove what works. Isaw this firsthand when I was at Airbnb,where our experimentation platform wascritical for analyzing what worked andwhere things went wrong. And the sameteam that built experimentation atAirbnb built EPO beta do then lets yougo beyond the numbers with sessionreplay. Watch exactly how users interactwith heat maps and scroll maps to trulyunderstand

00:37:05their behavior. And all ofthis is powered by feature flags thatare tied to realtime data so that youcan roll out safely, target [music]precisely, and learn continuously. DataDog is more than engineering metrics.It's where great product teams learnfaster, fix smarter, and ship withconfidence. [music]Request a demo at dataq.com/lenny.That's data dogq.com/lenny.Okay, I'm going to shift to talkingabout the API and the platform that youall build. Some So, you work with a lotof companies implementing your API, yourplatform, building on on your on yourtools. You told me that you find that alot of companies actually have negativeROI on their AI deployments, which uh Ithink is what a lot of people read aboutand feel and think and it's interestingyou're actually seeing that. What what'sgoing on there? What are they doingwrong? What do you what what's happeningin the world of AI and deployments inROI?>> Yeah. So, so to be clear, I I I don'tlike explicitly see quantitative numbersaround this. uh you know uh it'sactually really hard to measure thesethings but especially from observingsome companies kind of trying to do AI Iwould not be surprised if a lot of AIdeployments are actually you knownegative ROI. I mean part of this too isI think there's also general sentiment

00:38:21um from uh folks uh around the country ulike basically outside of tech that AIis being forced onto them. Um, and Ithink part of this is is is uh uh uhprobably a symptom of some negative ROIuh AI deployments. A couple things I'veobserved around this. So one one thingis and I think I I come back to thisagain and again like I think we inSilicon Valley just forget that we livein a bubble. Like we are so like Twitteris a bubble sorry X is a bubble. UmSilicon Valley is a bubble. Softwareengineering is a bubble. most people uhin the world, most people in the US arenot software engineers, are not very AIpled um are not following every singlemodel release. And so uh uh and so we'rejust like highly out of the loop on howto use this technology and so you knowlike we um we always talk about allthese like best practices for codecs,all these like codeex build peoplewithin OpenAI. I'm sure everyone on Xwho posts are like crazy power users ofof these AI tools, you know, they theylean into skills, they lean intoagents.mmd,>> MCPS.>> Uh yes. Yeah. All all of that. And uhwhen I talk to some of these companiesand I and I talk to the the actualemployees using these, it's like themost basic thing that they're trying todo and they like have very littleunderstanding of exactly

00:39:36how thistechnology works. And so that that'sthat's kind of like one big observationfor me, which is like they're askingvery simple questions of these things.They're really not not pushing it justyet. And so that kind of goes back tothat kind of ties into to to what I whatI think um more companies do or likewhat could do or or what what a moreideal AI deployment setup looks like. Umand and this is kind of how we've runthings within OpenAI too. Um thecompanies where I think it's it startedto work really well have a combinationof both top down buyin. So it's like theseuite it's like you know we're we're wewant to become an AI AI first company.So there's buyin, they buy the tools,they have, you know, exact support, butit also has bottoms up adoption andbuyin. And so what I mean by that is ithas like actual employees doing the workwho are really excited about thetechnology and are willing to learn,evangelize, build best practices andkind of like knowledge share within theorganization. We've we've seen this alot internally. So like obviously OpenAIhas always wanted to be uh a veryAIcentric company but where when itreally started taking off was when waswith the introduction of codecs andthese tools where like people likeactual employees themselves could startapplying it to their work. Uh and Ithink you really need this because atthe end of the

00:40:51day everyone's work islike very different. It's like veryunique. Uh software engineering isdifferent than finance is different thanoperations different than go to marketand sales. Uh, and so there's like a lotof these like last mile intricacies ofwork that needs to really be done in abottoms up fashion. And so my sense is alot of these these AI deployments don'thave like don't have bottoms upadoption. Like it was like an exactmandate and it's extremely top down andis very divorced from what the actualwork looks like. And as an end result,you end up with a giant workforce thatdoesn't really understand thetechnology. is like, I know I'm supposedto use this and maybe it's like on myperformance review too, but um I'm notsure what to do. And they look around,no one else is doing it. There's no oneelse to learn from. Uh and so my my youknow my recommendation for companieskind of pushing this is is find or maybeeven staff a full-time team internallythat is this kind of tiger teaminternally that can um explore the fullextent of the capabilities apply tospecific workflows do the knowledgesharing uh create excitement uh withinfolks uh who might want to use thistechnology uh because in the absence ofthat it's very difficult to it'sactually very difficult to pick up>> and who who would you put on this tigerteam is it like engineerled do you findin your experience is it crossfunctional sort of team.>> Yeah, it's it's interesting. So, um also

00:42:06a lot of companies don't have softwareengineers. Uh and so the the patternI've seen is it tends to be these likesoftware engineering adjacent likebasically technical people but are notsoftware engineers. I think those arethe ones who get tend to get mostexcited uh around this. It's like, youknow, maybe the It's like maybe thelike, you know, support team operationslead who doesn't code but loves usingthese tools and, you know, is like anExcel wizard or something. And so it'slike technical adjacent or like codingadjacent and like, you know, prettytechnical. Those are the times likethose are the kinds of people I've seenin these companies who just like reallylight up and get excited around this.Um, and you can usually build a team uha team around that. But yeah, it's likeoftentimes not software engineers.Software engineers, I think, willunderstand this, but not every companyhas has software engineers. Um, isactually kind of a rarity. They'rethey're hard to find. They're expensive.Uh, and so it's it's these other othertypes of folks. What I'm hearing is theanti- pattern is top down. This is verythe CEO found exec team just like we aregoing to go AI first. We're going tolead into AI. Everyone's going to bejudged on their performance using AItools, how much your productivity isincreasing thanks to AI. And withoutwith that being just top down and notcreating a team that is bottom upspreading the the gospel,

00:43:22you find itdoesn't work.>> Yeah. Yeah. Exactly. Exactly.>> And the advice is find the people thatare most excited and instead of kind ofhaving them spread out through theorganization, you're what you find worksis create a little AI kind of evangelistteam that finds ways to use it and kindof spreads it across the work.>> Yeah. I mean another it's kind of likehearing you you play back to me. Anotherway to think about it, kind of tyingback to my own management philosophies,is find the high performers in AIadoption and empower them. You know, letthem build hackathons, let them, youknow, hold seminars, do knowledgesharing, kind of create the seeds of uhof excitement internally. Okay. Amazing.There's a couple hot takes I want tohear uh from you. Something that I'veseen you talk about and share. one is umyou've shared that talking to customersand listening to customers is not alwaysthe right strategy in AI and it mightoften lead you astray.>> I don't know if it's that hot of a take.I think the main thing here is soobviously you should talk to yourcustomers like it's it's like useful totalk to customers. I just think the AIfield um especially what I've seen overthe last kind of like three years um uhworking on the API and and seeing kindof all that evolve is the field and themodels themselves are just changing soso quickly they tend to

00:44:37like disruptthemselves especially around the liketooling and the scaffolding space. So uhthere there's this quote that I readactually earlier this week from a it'sfrom an ex article uh by this guy namedNicholas who's who's the founder of a astartup called finol uh where uh I thinkhe was he was sharing a lot of the bestpractices that he has learned throughbuilding AI agents for financialservices I think at a at a startupFinTool um and he had this phrase that Ithought was really good which is uh themodels will eat your scaffolding forbreakfast. Like if you look if yourewind back to 2022 right when Chad GBTlaunched um these models were pretty rawand there was like all this productscaffolding and and things especially inthe developer space to basically try andsteer the model and build a scaffoldingaround it to get it to do what you wantlike agent frameworks there's like likevector stores I think was like reallypopular back then uh and just like awhole smattering of tools here and asyou've kind of seen the field play outthat the models have just changed somuch uh that uh and and gotten so muchbetter that they ended up yeah literallyeating some of some of the scaffolding.Um and I think this is even true today.So I think the the article from Nicholasum actually you know the the currentscaffolding which is uh fashionable isskills files based context management. Icould see a world where at some point

00:45:52you know that's no longer useful uhwhere the model can actually you knowmanage all that themselves or like youknow uh uh or or or there might be youknow it's hard to predict but like mightmove on to some new paradigm where youknow need this file based like skillsskills type thing you have literallyseen this play out right like the agentframeworks I think are a little lessuseful now um there was a period of timelike 2023 where we thought vector storesand is is going to be like the main wayfor you to you know bring organizationalcontext into the models and you need toyou know uh vector and embed every bitof your corpuses and then you need to doall this work to like figure out thevector search to like optimize that tofill out the right information the righttime. All of that is scaffolding becausethe model you know was not good enoughand turns out you know in this case itturns out as the models get better abetter approach is actually to take outa lot of that logic and trust the modeland give it a set of tools for search.It doesn't need to be a vector store.You could actually just hook it up toany type of search. It could literallybe files on a file system like skills uhand agents MD uh to kind of steer it uhas well. Obviously, there's still aplace for vector stores. I know a lot ofcompanies are still using it, but thethat the entire scaffolding around thatand building an entire ecosystem aroundthat and assuming that's the onlyscaffolding that you need has has reallychanged. And so tying this back to likeyou

00:47:07know uh it's you know you don'talways have to listen to your customersbecause the field is changing so much atany point in time you know a lot ofpeople are kind of in this local localmaximum and if you just blindly listento your customers they'll be like yeah Iwant a better vector store like I want abetter uh I want a better you know agentframework for this and uh if you hadjust kind of only chased down that pathit actually would have led you to youknow build something that again is thelocal maxima whereas as the models getbetter we've had to reinvent event andkind of rethink the right right uh uhabstractions and the right tools andframeworks to to to to build aroundthese models. Um and the cool slashexciting slash kind of crazy annoyingpart is it's a moving target. And soyeah, like the current currentsmattering of of tools and frameworksright now will likely need to evolve andchange pretty significantly over time umas the models get smarter and better.But that is just the nature of buildingin the space. I think that's what makesit exciting. Uh but it also means whenyou talk to customers, you kind of needto balance the exact feedback that theywant uh with uh where you think themodels are going and where you thinkthings will uh trend over the next oneto two years. It's interesting how thisis um the bitter lesson is uh you knowthis big lesson that AI and ML folkslearned which is just like uh don't theless you over complicate the less logicyou add to

00:48:22to machine learning to AI themore it'll be able to scale and grow andjust like take it all away and let itjust just compute basically just give itmore power to to get smarter on its own.Yeah, there's literally a version of thebitter lesson applied to like buildingwith AI where you know we were trying toarchitect all this stuff around andturns out the models are just kind ofyou know eat it all away and and and andhonestly like OpenAI API team has likebeen guilty of this uh where we kind oflike took some you know left and rightturns when we shouldn't have um but uhyeah the models still end up models getbetter and uh we're all learning thebitter lesson day in and day out. Sowhat would be the the key takeaway forfolks building on say the API or justbuilding agents and you know having tobuild a little bit of this around fornow is it just yeah what would be theadvice?>> My general advice and I've been givingthis to people for a while and I thinkstill true today is make sure you'rebuilding for where the models are goingand not where they are today. um uh youknow the the it's it's clearly a movingtarget and I think a lot of thecompanies that I've seen startups thatI've seen really really do well is theybuild a product for an ideal like typeof capability that is like maybe 80% ofthe way there today and it like they endup you know having a product that likekind of works but it's like just

00:49:37almostthere but then as the models get betteryou know suddenly it might click andthen their product now is incrediblebecause it works you like uh uh likemaybe with like 03 at some point itsuddenly works with 5.1 5.2 suddenly itunlocks it but they're building theseproducts with the like the modelcapability improvements in mind and withthat you end up creating an experexperience that's way better than if youhad assumed that it's it's static in thefirst place. Um and so that would be mymy general uh advice which is you knowbuild for where where where the modelsare going and not not where they aretoday. You end up building a betterproduct. you may need to, you know, likewait a little bit, but like, you know,the models are getting so much better soquickly, you often don't need to wait umthat long.>> So to follow that thread, where are likein the next six to 12 months, where isthe API heading? Where's the platformheading? Where are the models heading?As much as you can share, I know there'sa lot of secrets here that maybe you'remost excited about or do you think thatpeople should start to prepare for andhowever much you can share?>> I mean, so the obvious one is um howlong of a task uh these models can docoherently. Um, so there's like the themeter benchmark that that I think trackssoftware engineering tasks and how long,you know, like how long of a task canthese models do uh 50% of the time, 80%of the time. Uh, I think we're atsomething like

00:50:53multi-hour tasks beingable to be done by uh softwareengineering tasks being able to be doneby um uh these frontier models uh 50% ofthe time and then I think 80% issomething like just under an hour. Butthe the the sobering thing about thatthat chart is they plot all the uhprevious models on this chart as well.So you can really see the trend of this.That's something that I'm really excitedabout which is you know I actually thinkproducts today really optimize for tasksthat the model can do for like minutesat a time. Like even codecs and like thecoding tools I'd say like you know it'sin the Cly you're kind of like seeing itbe interactive. It's really you knowquite optimized well for like maybe atmost 10-minute types. I have seen peoplepush codeex to the limit and do likemulti-hour long uh tasks. Uh but again II think that that's more of theexception. But I uh if you follow thistrend like I think like in the next 12to 18 months we could see models thatcould do multi-hour long tasks very verycoherently. At some point it might reachlike you know 6 hours a dayong taskwhere you kind of like dispatch it andhave it do you know do things on uh onits own for a while. The types ofproducts you build around that will lookvery different. you want to give themodel feedback. You obviously don't wantit to completely run wild for a day.Maybe you do, but but you probablydon't.

00:52:08Um and and then the the universeof things you can have the model doreally expand. So that's something thatI'm really um really excited aboutseeing. Another uh thing over the next12 to 18 months I think be really coolis improvements in our in the multimodalmodels. So uh and and actually by bymultimodality I'm mostly thinking aboutaudio here where uh the models arepretty good at audio. I think they'regoing to get a lot better um at audioover the next 6 to 12 months especiallythe like you know the um nativemultimodal models the speech to speechones I think there's also interestingwork uh being done around um new typesof models and architectures on themultimodal audio side uh as well but uhaudio especially in the enterprise andin a business setting I think is ahugely underrated uh domain still likeeveryone talks about coding it's alltext uh but uh we're talking in audioA lot of the world's business is donevia audio. Uh a lot of services andoperations are done via uh talking andaudio. And so uh I think that that areais going to look very exciting in thenext 12 to 18 months. And I think therewill be uh even more unlock for uh whatwe can do uh with with audio models uhthere as well.>> Amazing. So quick summary uh expectagents and uh AI tools to run

00:53:24longer tothat that trajectory to continue toincrease and then audio and speechbecoming a bigger deal more first partyand and native and better and core tothe experience.>> Yeah,>> extremely cool. Okay, I want to go backto one of your hot takes, another hottake that I've seen you discuss. You'rebig uh you're very bullish on businessprocess automation as an opportunity inthe world of AI. Talk about that. Yeah,this go this goes back to the thing thatI said previously which is um we we welive in a bubble in Silicon Valley andum a lot of the work that we do thatwe're used to software engineering youknow product management buildingproducts uh is very differently shapedthan the work that goes on um that runsour entire economy and I see this in andout when I talk to customers uh if youif you talk to any like you know companythat's not based in it's not a techcompany um there's a lot of businessprocesses. And so what what I mean bythis is is you know I generallydelineate it as you know there's like uhlike software engineering is kind oflike open-ended knowledge work rightit's and this is why I think tools likecodeex tend to be quite quite goodbecause it's exploring and and you'regiving it these like open-ended thingsbut software engineering isfundamentally like pretty open-ended uhand it's not very repeatable right solike you build

00:54:39a feature you're nottrying to build the exact same featureover and over again and a lot of liketech jobs are in the space I think likedata science is kind of in the space aswell. Even some of the like strategicfinance stuff. But as you move furtherand further away from softwareengineering and like what what is corein tech, a lot of jobs are just businessprocesses. They're like repeatablethings uh repeatable operations um thatyou know some manager at a company haskind of like iterated on. Um there'susually a standard operating procedurethat people want to do. Uh and you don'twant to deviate from it that much. youknow this like in software engineeringthe ingenuity is isn't isn't deviatingbut a lot of a lot of the the the workbeing done in the world is actually justum running through these procedures andoperations like if I you know if I callum a support line they're runningthrough one of these if I call myutility company there's a bunch ofprocesses and things that they can andcannot do um for me uh and so I'm I'mjust extremely bullish on this generalcategory of like and and and I thinkit's underrated because it's sodifferent from what we think about inSilicon Valley people tend to not thinkabout it. But how can we apply um AI uhand and some of the tools and frameworksthat we have towards this businessprocess automation towards automated

00:55:54automating and making easier umrepeatable business processes with highdeterminism um that is fully integratedwith business uh data and businessdecisions and and and different systemswithin an enterprise um and how can weactually make that that process betteruh because I actually think there's alot of opportunity and a lot of work tobe done uh in that area and we just wejust don't talk about it because it'sit's a little bit less uh uh in ourwheelhouse. So your take here just tomake sure I fully understand it is youthink there's a much uh biggeropportunity outside of engineering forAI to impact uh productivity ofcompanies and also jobs of these folksthat are doing these kind of repetitiveeasily automated tasks impact jobs andalso just impact how work is done likeso much of work is done in this way likeyou think about you know like what abasically we I I talk to customers allthe time big enterprises like like howhow will AI transform my company likehow will it run in in in a world uh withAI in like 20 years. Um and and youknow, software engineering is part ofthe story, but there's so much more onthe business process side. And Iactually think it might look even moredifferent on the business process side.And and the work there is is prettysubstantial. It's actually interesting.I don't know like from an absolutepercentage or absolute basis, I don'tknow if it's bigger or smaller thansoftware

00:57:09engineering. Like software ispretty huge and pretty extensive uh aswell. But it is pretty massive and it'sdefinitely bigger than you know uh uh uhit's bigger than you would think it isbased off of how how people talk aboutit or don't talk about it on X orTwitter. Okay. Uh going in a slightlydifferent direction uh having built theplatform building the API uh peoplebuilding on the API the biggest questionon people's minds is always just uh howdo I not have OpenAI squash my idea andbuild their own thing and then you knowdestroy this this market I created.What's the general policy? What's thegeneral philosophy of how startupsshould think about where open AI isunlikely to go? My my general answerhere is is um the market is so big andso massive like I actually think youknow startups should just not overlythink about where open AI or these labsare going. I've talked to a lot ofstartups you know that have you know notworked out, startups that are doingreally well. Every startup that I'veseen that has kind of fizzled out is notbecause open AI or you know big lab orGoogle or something has has come tosquash them. It's because they builtsomething and it like really didn'tresonate with with the customers.Whereas the ones that take off like evenin very competitive spaces like codinglike cursor is huge at this

00:58:24point andit's because they built something thatpeople really love. And so my generaladvice is like don't you know don'toverly stress about this. Just buildsomething that people like and you willyou will have a space in this. I can'toverstate how big of an opportunitythere is right now. Like the the theopportunity space of building with AI isso big. Like a good example of this isis like the space is so big that theoverton window of what is acceptable andnot acceptable for VCs to do hascompletely changed here. VCs are likeinvesting in like competitive companiesleft and right is just like the space isso big because the opportunity is is isis unlike anything that we've seenbefore. And while you know uh that thataffects how VCs operate from a startupperspective it's like the mostempowering thing in the world becausethe like even if you just buildsomething that that some people reallyreally love you will you will end upwith a massive massively valuablebusiness. Uh and so I that's why I tellpeople like don't don't overthink aboutit. The other thing like I also think isimportant to remember uh at least froman open AI perspective. One thing thatthat that we've always held very nearand dear which both Sam and Greg helpedyou know reinforce from the top as wellis we actually view ourselvesfundamentally as a like ecosystemplatform company. The API was our firstproduct. We think it's really importantfor us to foster this ecosystem andcontinue to you know uh support it and

00:59:39and not squash it. And so if you kind oflook at the decisions we make it this isall we've weave through it. Every singlemodel we've released in one of ourproducts gets released in the API. Likeeven, you know, we release these codeexmodels now that are a little bit moreoptimized for the Codex harness, butthey always find their way into the APIand like all of our, you know, uh,customers end up using those. We don'thold back on any of that. Uh, we thinkit's really important to keep ourplatform neutral and so, you know, wedon't block competitors. Um, we allowpeople to have access to our models. Umuh we also want you know like uh we'verecently been testing more of like thesignin with chatgbt you know uh productas well and so we we want to foster thisecosystem and I think it's reallyimportant that we do so. Uh the generallike thinking about this is like youknow a rising tide like lifts all boatsand you know we might be a aircraftcarrier we're like pretty big at thispoint but we think it's important toraise the tide uh because everyone kindof uh benefits and I think we'll benefitas well like our API itself has grownpretty significantly because we we actin this way and so I'd really encouragepeople not to view OpenAI as this kindof like you know thing that'll just uhuh shove people out of the way butinstead focus on on on buildingsomething valuable and we you knowremain committed to to to providing anopen ecosystem.>> Why why is that important to open AAI?Just this focus on building

01:00:54a platform,creating a way for people to buildbusinesses just like is that just that'sbeen the vision from the beginning. Wewant this to be a a platform. It's beenthe vision from the beginning. It comesgoes back to our charter actually likeour our mission. Um so the the open airmission has always been to one to buildAGI. So you know we're obviously doingthat but then the second thing is tolike spread the benefits of it to all ofhumanity and there's kind of like a lotof you know uh uh the main part there isall of humanity like uh and obviouslyChad GBD is trying to do this you knowwe're trying to reach however many youknow the whole world but very early onand this is why we we launched the APIyou know back in I think it was like2020 or something like really early wedon't think we as a company will be ableto reach all of humanity right likethere's I don't know every every cornerof the world is like like pretty prettypretty pretty deep. And so we actuallyfeel like in order for us to fulfill ourmission, we need to have some platformstyle thing here where we can empowerother people to build, you know, thecustomer support bot for podcasters andnewsletter hosts. Uh because we're notgoing to be able to do it ourselves. Uhand so we've largely seen this play outwith the API. Uh this is why we we youknow, we we we we talked to so many ofour customers and and and really, youknow, love seeing the diversity of ofthings built on. But yeah, it it's beenthere since day one because it's

01:02:10it'skind of we view it as an expression ofour mission.>> And you haven't even mentioned the uhthe app store that you guys arelaunching the chatbt app store.>> Yeah. Is is that under your umbrella bythe way or is that a different Oregonteam?>> It's a it's a different team. So it'sunder chatbt. We obviously collaboratevery closely with them and uh you knowthey built like an apps SDK uh which isbuilt in close collaboration with ourteam. Uh but that is more within thechat GBT umbrella. Uh but that is alsoanother like that's another example ofthis right. It's like Chad GBT is likewe we we we we kind of like have these800 million weekly active users who arejust coming over and over again. Likeit's a great asset to have as abusiness, but like man would it bebetter if we could somehow allow, youknow, uh other companies to come in andand and and uh take advantage of this aswell and and build for this thisaudience as well. And and thenultimately we think it'll help us expandthat that that group as well, right? Andso it's all it all kind of comes back tothe mission and uh we find that being aplatform being open tends to help here.Just that number 800 million I thinkit's majust like weekly weekly>> weekly act billion people using weeklyjust like it's absurd how many how thesenumbers we're just used to now butthat's in insane unprecedented>> yeah it's it's mindboggling for me tothink about from a scale perspective

01:03:25uhhonestly and the way I think about it islike 10% of the world uh and growing bythe way like it's just it's it'sshooting up um>> uh come to chat GPT uh um and and use itevery day or sorry every week.>> And this point I just want to doubledown on this point you're making. OpenAAI's mission was to make AI availableto all humanity. And I think some peoplediss that. They're like, oh, you know,it costs money and it's like uh like thefact that it it's there's a free versionof chat GPT that anybody can use that isnot so different from the most powerfulAI model that exists in the world forfree that's not gated that anyone coulduse. It's like if you have if you're abillionaire, there's only so much moreyou can get out of AI than what someoneyou know in a village in Africa can canget. And I know that's always beenreally important to open AI.>> Yeah. Yeah. I mean like uh that that'swhy I think we've leaned into the healthwork. We leaned into like like uheducation is going to be a veryinteresting here. Um the other in insanekind of trend here is is the free modelhas gotten so smart over time. like thefree model back in 2022 was, you know,like uh was good at the time, but it'slike nothing compared to what you gettoday because you get 2 GB 5 today. Uhand so the like, you know, raising thefloor across the world is kind of, youknow, something that we're really we'retrying to do and and we view it as aspart

01:04:40of our mission. The other flip sideof this, by the way, is like, you know,kind of talking about like thebillionaires or whatever. I know peoplehave saying like you're using the sameiPhone that like you know Steve or sorrylike Mark Zuckerberg's probably using orlike the billionaires are using but forlike $20 a month you're basically usingyou know like using the same AI that youknow the billionaires are using. Uh forlike $200 a month uh you get the samepro model that you know all thebillionaires are using but they'reprobably not using pro for everything.They're probably just using the the plustier ones uh for their day in and dayout. And so yeah, this kind of likedemocratization and just like spreadingof this this benefit like across all ofthe world is something that's reallymeaningful to us and something that umuh drives a lot of of of what we do. Onelast question just for folks that arethinking about building on the API orjust like oh wait I could do cool stuffwith open models and APIs. What whatdoes your API and and platform allowpeople to do? Like I know you can buildagents on top of the platform. Just talkabout what you allow. So fundamentallythe API offers a bunch of developerendpoints uh and and uh and thesedeveloper endpoints basically let yousample from our models. The most popularone that we have right now is one calledresponses API. Uh and so this is anendpoint and it's optimized for buildinglongunning agents. So agents that'llwork for a

01:05:55while. So what you canbasically you can at a very you know uhuh low level you're basically justgiving the model text. The model willwork for a while. you can kind of, youknow, pull it to see see what it'll doand then you'll get the model responseback at at some point. That's like thelowest level primitive that we have uhfor people and that's actually what alot of people use. That's the mostpopular way of building on top of APIwith that. It is like superunpopinionated and you can do basicallywhatever you want. It's like the lowestlevel thing. We've also started buildingmore and more kind of like layers ofabstraction on top to help people builduh some of these. Uh and so next layerup we have this thing called the agentsSDK which has also gotten extremelyextremely popular. Um this allows you touse you know the responses API or someother API endpoints that we have tobuild what you might more traditionallythink of as an agent like a you know anAI kind of working in an infinite loop.It might have sub agents that itdelegates to. It starts building allthis framework all this scaffoldingactually. You know we'll see where thisall goes. Um, but it makes it a loteasier for you to build these thesethese these kind of agents, giving itguard rails, allowing it to like farmout subtasks to other agents and andkind of like orchestrate a swarm ofagents. Uh, the agents SDK uh kind ofallows you to do that. And then abovethat, uh, we've now started

01:07:11buildingtools to help also with kind of like themeta level of deploying an agent. Uh sowe have this product called uh um agentkit uh uh uh and widgets uh which arebasically a bunch of UI components thatyou can use to very easily um build avery beautiful UI um on top of uh uheither our API or agents SDK um becauseyou know a lot of times these agentskind of look very similar from a UIperspective uh and so there's agent kitwe also have a smattering of like uheval products like an eval API where ifyou want to test and like you know seeif your models or your your agent oryour workflow was working. Uh you cantest it in a very quantitative way umusing our EDOLs product. And so yeah,that I I view it as like these thesevarious layers. They're all kind ofhelping you build um what you want umwith our AI uh with our models um andwith increasing levels of abstractionand and and and uh you know howopinionated it is. And so um you canstart you can do you can use the wholestack and and it it very quickly allowsyou to build an agent or you can go downdown the stack as low as you want tobasically responses API and buildwhatever you want uh because of how lowlevel it is.>> Sherwin, is there anything else that youwant to share? Anything else you want toleave listeners with? Anything wehaven't touched on that you think mightbe helpful before we get to our veryexciting lightning

01:08:26round? The only thingI' I'd leave folks with is yeah, I thinkum I think the next like two to threeyears are going to be some of the mostfun uh in tech and in the startup worlduh that that we'll have in a very longtime. And uh I would just encouragepeople to not uh not take it forgranted. Like I I entered the workforcein 2014. It was great for like a coupleyears. I felt like there was like aperiod of like five to six years whereit wasn't very exciting in tech. Uh andthen in the last three years has justbeen the most insanely excitingenergizing period uh of my career and Ithink the next two to three years aregonna be a continuation of that. And souh would encourage people not take itfor granted. I'm trying to not take itfor granted. At some point you know thiswave is going to play out and it's goingto be a lot more you know incremental.Uh but in the meantime we're going toget to explore a lot of really coolthings, invent a lot of new things andchange the world and change how we work.And so uh that's the main thing I' I'dleave folks with. I love this message. Iwant to spend a little more time on it.Um, when you say don't miss it, is itwhat do you recommend people do? Is itjust build, lean in, learn, join acompany building really interestingthings? Like what's what's your adviceto folks that are like, "Okay, I don'twant to miss the boat." Yeah, I wouldjust say engage with it. So, it'sbasically like what you said. Um, leanin. Um, building, uh, tools on top ofthis is is part of the, you know, it'spart of the story. Um, just

01:09:41using thetools like you don't, you know, youdon't need to be a software engineer toto lean into this. Um, all I think a lotof jobs are going to going to going tochange here. So just using the tools,understanding the limitations of what itcan and cannot do so that you can kindof watch the trend of what it can startto do um as the models improve and yeahand so it's basically like getting usedand getting getting used to thetechnology and getting familiar with itinstead of kind of like laying back anduh uh uh letting it letting it pass you.>> On the flip side of that, there's a lotof I think stress and just anxietyaround like there's so much happening.How do I keep up? I got to learncloudbot this week. Oh god. What isthere something you've learned about itjust not like you're at the center ofthis? How do you not get overly stressedand worried about missing things thatare going on and just stay on top ofnews? What what are some things you'vedone learned?>> Yeah, so I I think I'm personally a badexample of this because I am I'mbasically chronically online uh on X anduh our company Slack. So I I I actuallytry and absorb I end up absorbing a lotof it. What I will say though is justlike from observing other folks who areless, you know, addicted to this stufflike I am. Um, yeah, a lot of it isnoise. Like you don't need to you don'tneed to have like 110% of this kind ofpass your mind like like go into yourmind. Honestly,

01:10:56just leaning into likeone or two different tools startingsmall is already like you know more thanyou need here. I think just thecombination of like the frenetic pace ofthe industry X as a product just createslike this insane kind of like um uh uhlike yeah this insane like pace of ofnews which is honestly veryoverwhelming. Uh the main thing is likeyou don't need to be you don't need toknow all of that to to really engagewith what's happening right now and evensomething as simple as just like installthe codeex client and play around withit. install Chad GBHEN, connect it to acouple of your uh you know internal uhuh data sources, notion, slack, github,and see what it can and cannot do. Umall of that I think is a part of it.Amazing. Sherwin, with that we'vereached our very exciting lightninground. I've got five questions for you.Are you ready?>> Yeah. Yeah, absolutely.>> First question, what are two or threebooks that you find yourselfrecommending most to other people?>> Oh, I'll I'll talk about one non-fictionone and one fiction book. Uh the fictionbook was I just finished reading it. I II it was really I I really recommend it.It it's uh uh there is no anti-mimeticsdivision by Q&M. Uh it's a uh I thinkit's like an online author, but I saw itbeing shared on X. Uh this this uh it'slike a science fictiony kind of book. Umand it was I basically

01:12:11devoured it inlike two days. Um it was it's supersuper well written, super fascinating.It's about a government agency that'sfighting, you know, things that make youforget it. Um, and so it's just a verylike smart like creative book that thatand fresh uh honestly in terms of likesource material uh that that that Ireally like. So I'd recommend that one.Uh the book is also unintentionallyhilarious. So like it's like meant to belike this like sci-fi almost like horrorstyle book, but it was it was it was uhit made me laugh a couple times. So uhthat's the that's the um fiction booknon-fiction. So I'm going to cheat andI'm going to recommend two of them. Soin the last year I've been reading a lotmore about China and kind of like the USChina relations. And I think there aretwo books that came out in the last yearthat have been you know really reallyeye opening for me in in that regard.First one is the Dan Wang bookbreakneck. That one was really reallygood. I really liked his analogy of likethe lawyerly US is the lawyerly society.China is the engineering society uh andtheir pros and cons to each. I read itand I was like hm yeah does does seemlike we're run by lawyers uh in the US.So that's one. Uh and the other one isthe Patrick McGee book on Apple andChina was super super interesting. I'm ahuge Apple fanboy. Like if you could seemy uh desk right now, it's it's allApple stuff. But just like one, it wasjust super fascinating

01:13:26learning aboutApple's relationship to China. And thentwo, it just like had a lot of insideinformation about Apple as a companythat I found fascinating. So it was alsoquite a page turner and um also, youknow, very very timely a timely book aswell.>> The antiimetics book sounds amazing. I'mbuying it right now as you're talking.>> Yeah. Yeah. Yeah. It's it's like I thinkit's only like couple hundred pages. Iliterally finished in two days. It wasjust like so so good.>> Okay, great tip. Okay. Uh favoriterecent movie or TV show you have reallyenjoyed?>> Yeah, that one's tough cuz you know withI I have two kids and uh uh a busy joband so I really haven't had much time umto watch TV shows. Uh I will say in thelast couple weeks I watched a coupleepisodes. I'm actually a big anime guyand so uh I I watched a couple episodes.There's a new season of this animecalled Jujutsu Kaisen uh that's out. Uhso season 3 of JJK. uh was was wasreally good. Um, in general, uh I'm ahuge uh fan of uh Japanese anime. Ithink they create the most uh novel andunique uh plots uh and universes that uhwestern media has shied away from. Umand so uh generally a big fan of thatbut yeah it haven't really watched muchbut saw a couple episodes of JJKrecently.>> Extremely understandable in your role.

01:14:42>> Yeah.>> Favorite product you recently discoveredthat you really love?>> Yeah. Okay. So, so, uh, so I recently,uh, had to set up Wi-Fi and like homenetworking, and I went all in onUbiquiti, uh, routers, um, and Csecurity cameras. I had never heard ofit before. I had to do this. I alwaysjust had a very simple setup. Uh, and itis just such a well-built product. Uh, Idon't know if you used it before, butit's basically like the Apple of likehome networking. So, uh, beautifulproducts. Uh, but the thing thatactually makes it extremely good is itsoftware is good. Uh, and so they have areally great um, mobile app to helpmanage, you know, uh, all of the thehome networking. Um, and so basicallyUbiquity, you can use it to buy uh,wireless routers. Um, you need Ethernetuh, wiring throughout your house to useit. Um, but I actually think what makesit really good are security cameras. So,if you have security cameras that areplugged into the Ubiquiti ecosystem,they have an incredible mobile app. Uh,and Apple TV app and iPad app um to kindof see the live feed of your cameras andand so uh they're they're they're alittle pricey, but not that pricey. Uhbut it's been just an incredible productexperience.>> All right, I went EO so I made amistake. Good tip.>> EOS are pretty good, too. But, uh fullyconverted to Ubiquity at this point.>> Good tip. Okay, two more questions. Doyou have a favorite life motto that youfind

01:15:57yourself coming back to in work orin life?Yeah. Uh the one that I always, youknow, repeat to myself is, uh, uh, neverfeel sorry for yourself. There's a lotof things that are going to happen, youknow, uh, at work, uh, in life. Uh, andreminding yourself to never feel sorryand that you always have a sense ofagency to kind of pull yourselves up issomething that I've had to tell myself alot and, um, also something that Irepeat to to to to a lot of other folksas well. Last question. So, in yourprevious life, you worked at Open Doorwhere you led work on basically figuringout how much to uh pay for houses. Youbasically built a model that told thecompany, "Here's how much we'll pay forthis house." What's like a variable inthe price of a house that you didn'texpect is really important and impactsthe price of a house? There's a bunchthat were surprising. I'll I'll maybelist the the the the couple of mo mostuh uh interesting ones. um power linesand like uh high voltage power lineslike are super super uh actually impactyour price quite a lot. I didn't reallyfully internalize this until I went tolike Dallas and observe like when yourhouse sits next to one of these giantlike you know voltage lines is likebuzzing and most people have familiesyou don't want your kids kind of nearthere. Uh so I think that was one thatreally really uh kind of surprised me.>> That makes sense.>> Yeah. And then the

01:17:12other one which whichwas something that uh was alwayssomething really difficult for us to uhquantify uh was floor plans. Uh and soit is very important like yes of courseit's really important but just likequantifying what a good floor plan islike and what a really bad floor plan islike we were doing all these things oflike how wide is the kitchen and like isit a what style of kitchen is it andthen like where's the master bedroom andand so it was just really really hard toquantify but I remember floor plan was abig one because like we'd have a homethat like wouldn't sell and then our uhops team would go in and be like yeahit's a floor plan issue so like how doyou how could you tell it's like you goinside you just feel it it feels youknow the floor plan feelsUh so yeah, th those are ones that wereuh surprising. And then the last onethat was more impactful than I thoughtis um general like curb appeal and likeeven like the front door. Uh and so Iactually think there there's a Zillowbook on on this where the front doorreplacement tends to be the highest ROIuh for homes. Um but just like the feelof like as you walk up to the home as abuyer, what you're interacting with andthe first moments of the house, I thinkwas uh I'd underrated its importance.That is extremely interesting. Uh, and Ilove that you had to figure figure howto do all this in code and not walkfloor plans. I have a bunch of storiesaround like for floor plans

01:18:27there's likethere's like uh it's not digitized. Sothere's like a handful of people whohave like paper floor plans uh of likeall these homes in like Phoenix andDallas. Um yeah, a lot lot of fun funstories from the open door days.>> Okay, Sherwin, uh thank you so much fordoing this. This was incredible. Uhwhere can folks find you online and uhand how can listeners be useful to you?Yeah. So, I'm uh online on on Twitter onX. I'm just Sherwin Woo and uh yeah, Imostly just tweet about uh OpenAI andthe API and some of the products thatwe're launching. Uh and then how folkscan be uh can be useful to me. Uh I lovehearing about things that people arebuilding and so if you're working on astartup, if you're hacking on an idea,you know, would love to uh just reachout to me on X. Um I would love to hearabout uh what you're building and andlearn about how open I can help supportyou.>> Amazing. Sherwin, thank you so much forbeing here. Yeah. Thank you, Lenny.>> Bye, everyone. Thank you so much forlistening. If you found this valuable,you can subscribe to the show on ApplePodcasts, Spotify, or your favoritepodcast app. Also, please considergiving us a rating or leaving a reviewas that really helps other listenersfind the podcast. You can find all pastepisodes or learn more about the show atlennispodcast.com.See you in the next episode.