Dwarkesh Patel: The Scaling Era of AI is Here

Ryan Sean Adams: Dwarkesh Patel, we are big fans. It's an honor to have you.

Dwarkesh: Thank you so much for having me on. Ryan Sean Adams: Okay, so you have a book out. It's called The Scaling Era, an oral history of AI from 2019 to 2025. Ryan Sean Adams: These are some key dates here. This is really a story of how AI emerged. Ryan Sean Adams: And it seemed to have exploded on people's radar over the past five years. Ryan Sean Adams: And And everyone in the world, it feels like, is trying to figure out what just Ryan Sean Adams: happened and what is about to happen.

Ryan Sean Adams: And I feel like for this story, we should start at the beginning, as your book does. Ryan Sean Adams: What is the scaling era of AI and when abouts did it start? What were the key milestones? Dwarkesh: So I think the undertold story about everybody's, of course, Dwarkesh: been hearing more and more about AI.

Dwarkesh: The under-told story is that the big contributor to these AI models getting Dwarkesh: better over time has been the fact that we are throwing exponentially more compute Dwarkesh: into trading frontier systems every year. Dwarkesh: So by some estimates, we spend 4x every single year over the last decade trading Dwarkesh: the frontier system than the one before it.

Dwarkesh: And that just means that we're spending hundreds of thousands of times more Dwarkesh: compute than the systems of the early 2010s. Dwarkesh: Of course, we've also had algorithmic breakthroughs in the meantime. Dwarkesh: 2018, we had the Transformer. Dwarkesh: Since then, obviously, many companies have made small improvements here and there.

Dwarkesh: But the overwhelming fact that we're spending already hundreds of billions of Dwarkesh: dollars in building up the infrastructure, Dwarkesh: the data centers, the chips for these models, and this picture is only going Dwarkesh: to intensify if this exponential keeps going, Dwarkesh: 4x a year, over the next two years, is something that is on the minds of the Dwarkesh: CFOs of the big hyperscalers and the people planning the expenditures and training going forward,

Dwarkesh: but is not as common in the conversation around where AI is headed.

Ryan Sean Adams: So what do you feel like people should know about this? Ryan Sean Adams: Like what is the scaling era? There have been other eras maybe of AI or compute, Ryan Sean Adams: but what's special about the scaling era? Dwarkesh: People started noticing. Well, first of all, in 2012, there's this, Dwarkesh: Ilya Seskaver and others started using neural networks in order to categorize images.

Dwarkesh: And just noticing that instead of doing something hand-coded, Dwarkesh: you can get a lot of juice out of just neural networks, black boxes. Dwarkesh: You just train them to identify what thing is like what. Dwarkesh: And then people started playing around these neural networks more, Dwarkesh: using them for different kinds of applications.

Dwarkesh: And then the question became, we're noticing that these models get better if Dwarkesh: you throw more data at them and you throw more compute at them. Dwarkesh: How can we shove as much compute into these models as possible? Dwarkesh: And the solution ended up being obviously internet text. So you need an architecture Dwarkesh: which is amenable to the trillions of tokens that have been written over the Dwarkesh: last few decades and put up on the internet.

Dwarkesh: And we had this happy coincidence of the kinds of architectures that are amenable Dwarkesh: to this kind of training with the GPUs that were originally made for gaming. Dwarkesh: We've had decades of internet text being compiled and Ilias actually called it the fossil fuel of AI.

Dwarkesh: It's like this reservoir that we can call upon to train these minds, Dwarkesh: which are like, you know, they're fitting the mold of human thought because Dwarkesh: they're trading on trillions of tokens of human thought. Dwarkesh: And so then it's just been a question of making these models bigger, Dwarkesh: of using this data that we're getting from internet techs to further keep training them.

Dwarkesh: And over the last year, as you know, the last six months, the new paradigm has Dwarkesh: been not only are we going to pre-train on all this internet text, Dwarkesh: we're going to see if we can have them solve math puzzles, Dwarkesh: coding puzzles, and through this, give them reasoning capabilities. Dwarkesh: The kind of thing, by the way, I mean, I have some skepticism around AGI just Dwarkesh: around the corner, which we'll get into.

Dwarkesh: But just the fact that we now have machines which can like reason, Dwarkesh: like, you know, you can like ask a question to a machine and it'll go away for a long time. Dwarkesh: It'll like think about it and then like it'll come back to you with a smart answer. Dwarkesh: And we just sort of take it for granted. But obviously, we also know that they're Dwarkesh: extremely good at coding, especially. Dwarkesh: I don't know if you actually got a chance to play around with Cloud Code or

Dwarkesh: Cursor or something. But it's a wild experience to design, explain at a high Dwarkesh: level, I want an application to does X. Dwarkesh: 15 minutes later, there's like 10 files of code and the application is built. Josh Kale: That's where we stand.

Dwarkesh: I have takes on how much this can continue. The other important dynamic, Dwarkesh: I'll add my monologue here, but the other important dynamic is that if we're Dwarkesh: going to be living in the scaling era, you can't continue exponentials forever, Dwarkesh: and certainly not exponentials that are 4x a year forever.

Dwarkesh: And so right now, we're approaching a point where within by 2028, Dwarkesh: at most by 2030, we will literally run out of the energy we need to keep trading Dwarkesh: these frontier systems, Dwarkesh: the capacity at the leading edge nodes, which manufacture the chips that go Dwarkesh: into the dyes, which go into these GPUs, even the raw fraction of GDP that will Dwarkesh: have to use to train frontier systems.

Dwarkesh: So we have a couple more years left of the scaling era. And the big question Dwarkesh: is, will we get to AGI before then?

Ryan Sean Adams: I mean, that's kind of a key insight of your book that like, Ryan Sean Adams: we're in the middle of the scaling era. Ryan Sean Adams: I guess we're like, you know, six years in or so. And we're not quite sure. Ryan Sean Adams: It's like, like the protagonist in the middle of the story, We don't know exactly Ryan Sean Adams: which way things are going to go. Ryan Sean Adams: But I want you to maybe, Dworkesh, help folks get an intuition for why scaling in this way even works.

Ryan Sean Adams: Because I'll tell you, for me and for most people, our experience with these Ryan Sean Adams: revolutionary AI models probably started in 2022 with ChatGPT3 and then ChatGPT4 Ryan Sean Adams: and seeing all the progress, all these AI models. Ryan Sean Adams: And it just seems really unintuitive that if you take a certain amount of compute Ryan Sean Adams: and you take a certain amount of data, out pops AI, out pops intelligence.

Ryan Sean Adams: Could you help us get an intuition for this magic? Ryan Sean Adams: How does the scaling law even work? Compute plus data equals intelligence? Is that really all it is? Dwarkesh: To be honest, I've asked so many AI researchers this exact question on my podcast. Dwarkesh: And I could tell you some potential theories of why it might work. Dwarkesh: I don't think we understand.

Dwarkesh: You know what? I'll just say that. I don't think we understand. Ryan Sean Adams: We don't understand how this works. We know it works, but we don't understand Dwarkesh: How it works. We have evidence from actually, of all things, Dwarkesh: primatology of what could be going on here, or at least like why similar patterns Dwarkesh: in other parts of the world.

Dwarkesh: So what I found really interesting, There's this research by this researcher, Dwarkesh: Susanna Herculana Huzel, Dwarkesh: which shows that if you look at how the number of neurons in the brain of a rat, Dwarkesh: different kinds of rat species increases, as the weight of their brains increase Dwarkesh: from species to species, there's this very sublinear pattern. Dwarkesh: So if their brain size doubles, the neuron count will not double between different rat species.

Dwarkesh: And there's other animals where there's other kinds of... Dwarkesh: Families of species for which this is true. The two interesting exceptions to Dwarkesh: this rule, where there is actually a linear increase in neuron count and brain Dwarkesh: size, is one, certain kinds of birds. Dwarkesh: So, you know, birds are actually very smart, given the size of their brains, and primates.

Dwarkesh: So the theory for what happened with humans is that we unlocked an architecture that was very scalable. Dwarkesh: So the way people talk about transformers being more scalable and then LSTMs, Dwarkesh: the thing that preceded them in 2018. Dwarkesh: We unlocked this architecture as it's very scalable. Dwarkesh: And then we were in an evolutionary niche millions of years ago, Dwarkesh: which rewarded marginal increases in intelligence.

Dwarkesh: If you get slightly smarter, yes, the brain costs more energy, Dwarkesh: but you can save energy in terms of like not having to, you can cook, Dwarkesh: you can cook food so you don't have to spend much more on digestion. Dwarkesh: You can find a game, you can find different ways of foraging.

Dwarkesh: Birds were not able to find this evolutionary niche, which rewarded the incremental Dwarkesh: increases in intelligence because if your brain gets too heavy as a bird, you're not going to fly. Dwarkesh: So it was this happy coincidence of these two things. Now, why is it the case Dwarkesh: that the fact that our brains could get bigger resulted in us becoming as smart Dwarkesh: as we are? We still don't know. Dwarkesh: And there's many different dissimilarities between AIs and humans.

Dwarkesh: While our brains are quite big, we don't need to be trained. Dwarkesh: You know, a human from the age they're zero to 18 is not seeing within an order Dwarkesh: of magnitude of the amount of information these LLMs are trained on. Dwarkesh: So LLMs are extremely data inefficient. Dwarkesh: They need a lot more data, but the pattern of scaling, I think we see in many different places. Ryan Sean Adams: So is that a fair kind of analog? This analog has always made sense to me.

Ryan Sean Adams: It's just like transformers are like neurons. Ryan Sean Adams: You know, AI models are sort of like the human brain. Ryan Sean Adams: Evolutionary pressures are like gradient descent, reward algorithms and out Ryan Sean Adams: pops human intelligence. We don't really understand that. Ryan Sean Adams: We also don't understand AI intelligence, but it's basically the same principle at work.

Dwarkesh: I think it's a super fascinating, but also very thorny question because is gradient Dwarkesh: intelligence like evolution? Dwarkesh: Well, yes, in one sense. But also when we do gradient descent on these models, Dwarkesh: we start off with the weights and then we're, you know, it's like learning how Dwarkesh: does chemistry work, how does coding work, how does math work.

Dwarkesh: And that's actually more similar to lifetime learning, which is to say that, Dwarkesh: like, by the time you're already born to the time you turn 18 or 25, Dwarkesh: the things you learn, and that's not evolution. Dwarkesh: Evolution designed the system or the brain by which you can do that learning, Dwarkesh: but the lifetime learning itself is not evolution. And so there's also this Dwarkesh: interesting question of, yeah, is training more like evolution?

Dwarkesh: In which case, actually, we might be very far from AGI because the amount of Dwarkesh: compute that's been spent over the course of evolution to discover the human Dwarkesh: brain, you know, could be like 10 to the 40 flops. There's been estimates, you know, whatever.

Dwarkesh: I'm sure it will bore you to discover, talk about how these estimates are derived, Dwarkesh: but just like how much versus is it like a single lifetime, Dwarkesh: like going from the age of zero to the age of 18, which is closer to, Dwarkesh: I think, 10 to the 24 flops, which is actually less than compute than we use Dwarkesh: to train frontier systems.

Dwarkesh: All right, anyways, we'll get back to more relevant questions. Ryan Sean Adams: Well, here's kind of a big picture question as well. Ryan Sean Adams: It's like I'm constantly fascinated with the metaphysical types of discussions Ryan Sean Adams: that some AI researchers kind of take. Ryan Sean Adams: Like a lot of AI researchers will talk in terms of when they describe what they're Ryan Sean Adams: making, we're making God.

Ryan Sean Adams: Like why do they say things like that? What is this talk of like making God? Ryan Sean Adams: What does that mean? Is it just the idea that scaling laws don't cease? Ryan Sean Adams: And if we can, you know, scale intelligence to AGI, then there's no reason we Ryan Sean Adams: can't scale far beyond that and create some sort of a godlike entity. Ryan Sean Adams: And essentially, that's what the quest is. We're making artificial superintelligence.

Ryan Sean Adams: We're making a god. We're making god. Dwarkesh: I think people focus too much on when they, I think this God discussion focuses Dwarkesh: too much on the hypothetical intelligence of a single copy of an AI. Dwarkesh: I do believe in the notion of a super intelligence, which is not just functionally, Dwarkesh: which is not just like, oh, it knows a lot of things, but is actually qualitatively Dwarkesh: different than human society.

Dwarkesh: But the reason is not because I think it's so powerful that any one individual Dwarkesh: copy of AI will be as smart, but because of the collective advantages that AIs Dwarkesh: will have, which have nothing to do with their raw intelligence, Dwarkesh: but rather the fact that these models will be digital or they already are digital, Dwarkesh: but eventually they'll be as smart as humans at least.

Dwarkesh: But unlike humans, because of our biological constraints, these models can be copied. Dwarkesh: If there's a model that has learned a lot about a specific domain, Dwarkesh: you can make infinite copies of it. Dwarkesh: And now you have an infinite copies of Jeff Dean or Ilya Satskova or Elon Musk Dwarkesh: or any skilled person you can think of.

Dwarkesh: They can be merged. So the knowledge that each copy is learning can be amalgamated Dwarkesh: back into the model and then back to all the copies. Dwarkesh: They can be distilled. They can run at superhuman speeds. Dwarkesh: These collective advantages, also they can communicate in latent space. Dwarkesh: These collective advantages. Ryan Sean Adams: They're immortal. I mean, you know, as an example.

Dwarkesh: Yes, exactly. No, I mean, that's actually, tell me if I'm rabbit holing too Dwarkesh: much, but like one really interesting question will come about is how do we prosecute AIs? Dwarkesh: Because the way we prosecute humans is that we will throw you in jail if you commit a crime. Dwarkesh: But if there's trillions of copies or thousands of copies of an AI model, Dwarkesh: if a copy of an AI model, if an instance of an AI model does something bad, what do you do?

Dwarkesh: Does the whole model have to get, and how do you even punish a model, Dwarkesh: right? Like, does it care about its weights being squandered? Dwarkesh: Yeah, there's all kinds of questions that arise because of the nature of what AIs are. Dwarkesh Patel: And also who is liable for that, right? Dwarkesh: Like, is it the toolmaker?

Dwarkesh Patel: Is it the person using the tool? Who is responsible for these things? Dwarkesh Patel: There's one topic that I do want to come to here about scaling laws, Dwarkesh Patel: At what time did we realize that scaling laws were going to work? Dwarkesh Patel: Because there were a lot of theses early in the days, early 2000s about AI, Dwarkesh Patel: how we were going to build better models.

Dwarkesh Patel: Eventually, we got to the transformer. But at what point did researchers and Dwarkesh Patel: engineers start to realize that, hey, this is the correct idea. Dwarkesh Patel: We should start throwing lots of money and resources towards this versus other Dwarkesh Patel: ideas that were just kind of theoretical research ideas, but never really took off. Dwarkesh Patel: We kind of saw this with GPT two to three, where there's this huge improvement.

Dwarkesh: A lot of. Dwarkesh Patel: Resources went into it. Was there a specific moment in time or a specific breakthrough Dwarkesh Patel: that led to the start of these scaling laws? Dwarkesh: I think it's been a slow process of more and more people appreciating this nature Dwarkesh: of the overwhelming role of compute in driving forward progress. Dwarkesh: In 2018, I believe, Dario Amadei wrote a memo that was secret while he was at Dwarkesh: OpenAI. Now he's the CEO of Anthropic.

Dwarkesh: But while he's at OpenAI, he's subsequently revealed on my podcast that he wrote Dwarkesh: this memo where the title of the memo was called Big Blob of Compute. Dwarkesh: And it says basically what you expect it to say, which is that like, Dwarkesh: yes, there's ways you can mess up the process of training. You have the wrong Dwarkesh: kinds of data or initializations. Dwarkesh: But fundamentally, AGI is just a big blob of compute.

Dwarkesh: And then we've gotten over the subsequent years, there was more empirical evidence. Dwarkesh: So a big update, I think it was 2021. Dwarkesh: Correct me. Somebody definitely will correct me in the comments.

Dwarkesh: I'm wrong. There were these, Dwarkesh: there's been multiple papers of these scaling laws where you can show that the Dwarkesh: loss of the model on the objective of predicting the next token goes down very predictably, Dwarkesh: almost to like multiple decimal places of correctness based on how much more Dwarkesh: compute you throw in these models.

Dwarkesh: And the compute itself is a function of the amount of data you use and how big Dwarkesh: the model is, how many parameters it has. Dwarkesh: And so that was an incredibly strong evidence back in the day, Dwarkesh: a couple of years ago, because then you could say, well, OK, Dwarkesh: if it really has this incredibly low loss of predicting the next token in all Dwarkesh: human output, including scientific papers, including GitHub repositories.

Dwarkesh: Then doesn't it mean it has actually had to learn coding and science and all Dwarkesh: these skills in order to make those predictions, which actually ended up being true. Dwarkesh: And it was it was something people, you know, we take it for granted now, Dwarkesh: but it actually even as of a year or two ago, people were really even denying that premise.

Dwarkesh: But some people a couple of years ago just like thought about it and like, Dwarkesh: yeah, actually, that would mean that it's learned the skills. Dwarkesh: And that's crazy that we just have this strong empirical pattern that tells Dwarkesh: us exactly what we need to do in order to learn these skills. Dwarkesh Patel: And it creates this weird perception, right, where like very early on and so Dwarkesh Patel: to this day, it really is just a token predictor, right?

Dwarkesh Patel: Like we're just predicting the next word in the sentence. But somewhere along Dwarkesh Patel: the lines, it actually creates this perception of intelligence. Dwarkesh Patel: So I guess we covered the early historical context. I kind of want to bring Dwarkesh Patel: the listeners up to today, where we are currently, where the scaling laws have Dwarkesh Patel: brought us in the year 2025.

Dwarkesh Patel: So can you kind of outline where we've gotten to from early days of GPTs to Dwarkesh Patel: now we have GPT-4, we have Gemini Ultra, we have Club, which you mentioned earlier. Dwarkesh Patel: We had the breakthrough of reasoning. Dwarkesh Patel: So what can leading frontier models do today? Dwarkesh: So there's what they can do. And then there's the question of what methods seem to be working.

Dwarkesh: I guess we can start at what they seem to be able to do. They've shown to be Dwarkesh: remarkably useful at coding and not just at answering direct questions about Dwarkesh: how does this line of code work or something. Dwarkesh: But genuinely just autonomously working for 30 minutes or an hour, Dwarkesh: doing the task, it would take a front-end developer a whole day to do.

Dwarkesh: And you can just ask them at a high level, do this kind of thing, Dwarkesh: and they can go ahead and do it. Dwarkesh: Obviously, if you've played around with it, you know that they're extremely Dwarkesh: useful assistants in terms of research, in terms of even therapists, Dwarkesh: whatever other use cases.

Dwarkesh: On the question of what training methods seem to be working, Dwarkesh: we do seem to be getting evidence that pre-training is plateauing, Dwarkesh: which is to say that we had GPT 4.5, which was just following this old mold Dwarkesh: of make the model bigger, Dwarkesh: but it's fundamentally doing the same thing of next token prediction.

Dwarkesh: And apparently it didn't pass muster. The OpenAI had to deprecate it because Dwarkesh: there's this dynamic where the bigger the model is, the more it costs not only Dwarkesh: to train, but also to serve, right?

Dwarkesh: Because every time you serve a user, you're having to run the whole model, Dwarkesh: which is going, so, but that doesn't be working is RL, which is this process Dwarkesh: of, not just training them on existing tokens on the internet, Dwarkesh: but having the model itself try to answer math and coding problems.

Dwarkesh: And finally, we got to the point where the model is smart enough to get it right Dwarkesh: some of the time, and so you can give it some reward, and then it can saturate Dwarkesh: these tough reasoning problems. Dwarkesh Patel: And then what was the breakthrough with reasoning for the people who aren't familiar? Dwarkesh Patel: What made reasoning so special that we hadn't discovered before? Dwarkesh Patel: And what did that kind of unlock for models that we use today?

Dwarkesh: I'm honestly not sure. I mean, GBD-4 came out a little over two years ago, Dwarkesh: and then it was after two years after GPT-4 came out that O-1 came out which Dwarkesh: was the original reasoning breakthrough I think last November and, Dwarkesh: And subsequently, a couple of months later, DeepSeq showed in their R1 paper. Dwarkesh: So DeepSeq open source their research and they explained exactly how their algorithm worked.

Dwarkesh: And it wasn't that complicated. It was just like what you would expect, Dwarkesh: which is get some math problems, Dwarkesh: give for some initial problems, tell the model exactly what the reasoning trace Dwarkesh: looks like, how you solve it, just like write it out and then have the model Dwarkesh: like try to do it raw on the remaining problems. Dwarkesh: Now, I know it sounds incredibly arrogant to say, well, it wasn't that complicated. Dwarkesh: Why did it take you years?

Dwarkesh: I think there's an interesting insight there of even things which you think Dwarkesh: will be simple in terms of high level description of how to solve the problem Dwarkesh: end up taking longer in terms of haggling out the remaining engineering hurdles Dwarkesh: than you might naively assume. Dwarkesh: And that should update us on how long it will take us to go through the remaining Dwarkesh: bottlenecks on the path to AGI.

Dwarkesh: Maybe that will be tougher than people imagine, especially the people who think Dwarkesh: we're only two to three years away. Dwarkesh: But all this to say, yeah, I'm not sure why it took so long after GPT-4 to get Dwarkesh: a model trained on a similar level of capabilities that could then do reasoning. Dwarkesh Patel: And in terms of those abilities, the first answer you had to what can it do was coding.

Dwarkesh Patel: And I hear that a lot of the time when I talk to a lot of people is that coding Dwarkesh Patel: seems to be a really strong suit and a really huge unlock to using these models. Dwarkesh Patel: And I'm curious, why coding over general intelligence? Dwarkesh Patel: Is it because it's placed in a more confined box of parameters?

Dwarkesh Patel: I know in the early days, we had the AlphaGo and And we had the AIs playing Dwarkesh Patel: chess and they exceed, they perform so well because they were kind of contained Dwarkesh Patel: within this box of parameters that was a little less open-ended than general intelligence. Dwarkesh Patel: Is that the reason why coding is kind of at the frontier right now of the ability of these models?

Dwarkesh: There's two different hypotheses. One is based around this idea called Moravac's paradox. Dwarkesh: And this was an idea, by the way, one super interesting figure, Dwarkesh: actually, I should have mentioned him earlier. Dwarkesh: One super interesting figure in the history of scaling is Hans Moravac, Dwarkesh: who I think in the 90s predicts that 2028 will be the year that we will get to AGI.

Dwarkesh: And the way he predicts this, which is like, you know, we'll see what happens, Dwarkesh: but like not that far off the money as far as I'm concerned. Dwarkesh: The way he predicts this is he just looks at the growth in computing power year Dwarkesh: over year and then looks at how much compute he estimated the human brain to be to require. Dwarkesh: And just like, OK, we'll have computers as powerful as the human brain by 2028.

Dwarkesh: Which is like at once a deceptively simple argument, but also ended up being Dwarkesh: incredibly accurate and like worked, right? Dwarkesh: I might add a fact drive it was 2028, but it was within that, Dwarkesh: like within something you would consider a reasonable guess, given what we know now. Dwarkesh: Sorry, anyway, so the Morrowind's paradox is this idea that computers seemed Dwarkesh: in AI get better first at the skills which humans are the worst at.

Dwarkesh: Or at least there's a huge variation in the human repertoire. Dwarkesh: So we think of coding as incredibly hard, right? We think this is like the top Dwarkesh: 1% of people will be excellent coders. Dwarkesh: We also think of reasoning as very hard, right? So if you like read Aristotle, Dwarkesh: he says, the thing which makes humans special, which distinguishes us from animals is reasoning.

Dwarkesh: And these models aren't that useful yet at almost anything. The one thing they can do is reasoning. Dwarkesh: So how do we explain this pattern? And Moravec's answer is that evolution has Dwarkesh: spent billions of years optimizing us to do things we take for granted. Dwarkesh: Move around this room, right? I can pick up this can of Coke, Dwarkesh: move it around, drink from it. Dwarkesh: And that we can't even get robots to do at all yet.

Dwarkesh: And in fact, it's so ingrained in us by evolution that there's no human, or. Ryan Sean Adams: At least humans who don't have Dwarkesh: Disabilities will all be able to do this. And so we just take it for granted Dwarkesh: that this is an easy thing to do. Dwarkesh: But in fact, it's evidence of how long evolution has spent getting humans up to this point.

Dwarkesh: Whereas reasoning, logic, all of these skills have only been optimized by evolution Dwarkesh: over the course of the last few million years. Dwarkesh: So there's been a thousand fold less evolutionary pressure towards coding than Dwarkesh: towards just basic locomotion. Dwarkesh: And this has actually been very accurate in predicting what kinds of progress Dwarkesh: we see even before we got deep learning, right?

Dwarkesh: Like in the 40s when we got our first computers, the first thing that we could Dwarkesh: use them to do is long calculations for ballistic trajectories at the time for World War II. Dwarkesh: Humans suck at long calculations by hand. Dwarkesh: And anyways, so that's the explanation for coding, which seems hard for humans, Dwarkesh: is the first thing that went to AIs. Dwarkesh: Now, there's another theory, which is that this is actually totally wrong.

Dwarkesh: It has nothing to do with the seeming paradox of how long evolution has optimized Dwarkesh: us for, and everything to do with the availability of data. Dwarkesh: So we have GitHub, this repository of all of human code, at least all open source Dwarkesh: code written in all these different languages, trillions and trillions of tokens. Dwarkesh: We don't have an analogous thing for robotics. We don't have this pre-training

Dwarkesh: corpus. And that explains why code has made so much more progress than robotics. Ryan Sean Adams: That's fascinating because if there's one thing that I could list that we'd Ryan Sean Adams: want AI to be good at, probably coding software is number one on that list. Ryan Sean Adams: Because if you have a Turing complete intelligence that can create Turing complete Ryan Sean Adams: software, is there anything you can't create once you have that?

Ryan Sean Adams: Also, like the idea of Morvac's paradox, I guess that sort of implies a certain Ryan Sean Adams: complementarianism with humanity. Ryan Sean Adams: So if robots can do things that robots can do really well and can't do the things Ryan Sean Adams: humans can do well, well, perhaps there's a place for us in this world. Ryan Sean Adams: And that's fantastic news. It also maybe implies that humans have kind of scratched Ryan Sean Adams: the surface on reasoning potential.

Ryan Sean Adams: I mean, if we've only had a couple of million years of evolution and we haven't Ryan Sean Adams: had the data set to actually get really good at reasoning, it seems like there'd Ryan Sean Adams: be a massive amount of upside, unexplored territory, Ryan Sean Adams: like so much more intelligence that nature could actually Ryan Sean Adams: contain inside of reasoning. Ryan Sean Adams: I mean, are these some of the implications of these ideas?

Dwarkesh: Yeah, I know. I mean, that's a great insight. Another really interesting insight Dwarkesh: is that the more variation there Dwarkesh: is in a skill in humans, the better and faster that AIs will get at it. Dwarkesh: Because coding is the kind of thing where 1% of humans are really good at it. Dwarkesh: The rest of us will, if we try to learn it, we'd be okay at it or something, right?

Dwarkesh: And because evolutionists spend so little time optimizing us, Dwarkesh: there's this room for variation where the optimization hasn't happened uniformly Dwarkesh: or it hasn't been valuable enough to saturate the human gene pool for this skill. Dwarkesh: I think you made an earlier point that I thought was really interesting I wanted Dwarkesh: to address. Can you remind me of the first thing you said? Is it the complementarianism? Yes.

Dwarkesh: So you can take it as a positive future. You can take it as a negative future Dwarkesh: in the sense that, well, what is the complementary skills we're providing? Dwarkesh: We're good meat robots. Ryan Sean Adams: Yeah, the low skilled labor of the situation.

Dwarkesh: We can do all the thinking and planning. One dark future, Dwarkesh: one dark vision of the future is we'll get those meta glasses Dwarkesh: and the AI speaking into our ear and it'll tell us to go put this brick over Dwarkesh: there so that the next data center couldn't be built because the AI's got the Dwarkesh: plan for everything. It's got the better design for the ship and everything.

Dwarkesh: You just need to move things around for it. And that's what human labor looks Dwarkesh: like until robotics is solved. Dwarkesh: So yeah, it depends on how you... On the other hand, you'll get paid a lot because Dwarkesh: it's worth a lot to move those bricks. We're building AGI here. Dwarkesh: But yeah, it depends on how you come out of that question.

Ryan Sean Adams: Well, there seems to be something to that idea, going back to the idea of the Ryan Sean Adams: massive amount of human variation. Ryan Sean Adams: I mean, we have just in the past month or so, we have news of meta hiring AI Ryan Sean Adams: researchers for $100 million signing bonuses, okay? Ryan Sean Adams: What does the average software engineer make versus what does an AI researcher Ryan Sean Adams: make at kind of the top of the market, right?

Ryan Sean Adams: Which has got to imply, obviously there's some things going on with demand and Ryan Sean Adams: supply, but also that it does also seem to imply that there's massive variation Ryan Sean Adams: in the quality of a software engineer. Ryan Sean Adams: And if AIs can get to that quality, well, what does that unlock? Ryan Sean Adams: Yeah. So, okay. Yeah. So I guess we have like coding down right now. Ryan Sean Adams: Like another question though is like, what can't AIs do today?

Ryan Sean Adams: And how would you characterize that? Like what are the things they just don't do well? Dwarkesh: So I've been interviewing people on my podcast who have very different timelines Dwarkesh: from a role to get to AGI. I have had people on who think it's two years away Dwarkesh: and some who think it's 20 years away.

Dwarkesh: And the experience of building AI tools for myself actually has been the most Dwarkesh: insight driving or maybe research I've done on the question of when AI is coming. Ryan Sean Adams: More than the guest interviews. Dwarkesh: Yeah, because you just, I have had, I've probably spent on the order of a hundred

Dwarkesh: hours trying to build these little tools. The kinds I'm sure you've also tried Dwarkesh: to build of like, rewrite auto-generated transcripts for me to make them sound, Dwarkesh: the rewritten the way a human would write them. Dwarkesh: Find clips for me to tweet out, write essays with me, co-write them passage Dwarkesh: by passage, these kinds of things.

Dwarkesh: And what I found is that it's actually very hard to get human-like labor out Dwarkesh: of these models, even for tasks like these, which should be death center in Dwarkesh: the repertoire of these models, right? Dwarkesh: They're short horizon, they're language in, language out. Dwarkesh: They're not contingent on understanding some thing I said like a month ago. Dwarkesh: This is just like, this is the task.

Dwarkesh: And I was thinking about why is it the case that I still haven't been able to Dwarkesh: automate these basic language tasks? Why do I still have a human work on these things? Dwarkesh: And I think the key reason that you can't automate even these simple tasks is Dwarkesh: because the models currently lack the ability to do on the job training.

Dwarkesh: So if you hire a human for the first six months, for the first three months, Dwarkesh: they're not going to be that useful, even if they're very smart, Dwarkesh: because they haven't built up the context, they haven't practiced the skills, Dwarkesh: they don't understand how the business works. Dwarkesh: What makes humans valuable is not that mainly the raw intellect obviously matters, Dwarkesh: but it's not mainly that.

Dwarkesh: It's their ability to interrogate their own failures in this really dynamic, Dwarkesh: organic way to pick up small efficiencies and improvements as they practice Dwarkesh: the task and to build up this context as they work within a domain. Dwarkesh: And so sometimes people wonder, look, if you look at the revenue of OpenAI, Dwarkesh: the annual recurring revenue, it's on the order of $10 billion.

Dwarkesh: Kohl's makes more money than that. McDonald's makes more money than that, right? Dwarkesh: So why is it that if they've got AGI, they're, you know, like Fortune 500 isn't Dwarkesh: reorganizing their workflows to, you know, use open AI models at every layer of the stack? Dwarkesh: My answer, sometimes people say, well, it's because people are too stodgy. Dwarkesh: The management of these companies is like not moving fast enough on AI.

Dwarkesh: That could be part of it. I think mostly it's not that. Dwarkesh: I think mostly it genuinely is very hard to get human-like labor out of these Dwarkesh: models because you can't. Dwarkesh: So you're stuck with the capabilities you get out of the model out of the box. Dwarkesh: So they might be five out of 10 at rewriting the transcript for you.

Dwarkesh: But if you don't like how it turned out, if you have feedback for it, Dwarkesh: if you want to keep teaching it over time, once the session ends, Dwarkesh: the model, like everything it knows about you has gone away. Dwarkesh: You got to restart again. It's like working with an amnesiac employee. Dwarkesh: You got to restart again. Ryan Sean Adams: Every day is the first day of employment, basically.

Dwarkesh: Yeah, exactly. It's a groundhog day for them every day or every couple of hours, in fact. Dwarkesh: And that makes it very hard for them to be that useful as an employee, Dwarkesh: right? They're not really an employee at that point. Dwarkesh: This, I think, not only is a key bottleneck to the value of these models, Dwarkesh: because human labor is worth a lot, right? Dwarkesh: Like $60 trillion in the world is paid to wages every year.

Dwarkesh: If these model companies are making on the order of $10 billion a year, that's a big way to AGI. Dwarkesh: And what explains that gap? What are the bottlenecks? I think a big one is this Dwarkesh: continual learning thing. Dwarkesh: And I don't see an easy way that that just gets solved within these models. Dwarkesh: There's no like, with reasoning, you could say, oh, it's like train it on math Dwarkesh: and code problems, and then I'll get the reasoning. And that worked.

Dwarkesh: I don't think there's something super obvious there for how do you get this Dwarkesh: online learning, this on-the-job training working for these models.

Ryan Sean Adams: Okay, can we talk about that, go a little bit deeper on that concept? Ryan Sean Adams: So this is basically one of the concepts you wrote in your recent post. Ryan Sean Adams: AI is not right around the corner. Even though you're an AI optimist, Ryan Sean Adams: I would say, and overall an AI accelerationist, you You were saying it's not Ryan Sean Adams: right around the corner. Ryan Sean Adams: You're saying the ability to replace human labor is a ways out.

Ryan Sean Adams: Not forever out, but I think you said somewhere around 2032, Ryan Sean Adams: if you had to guess on when the estimate was. Ryan Sean Adams: And the reason you gave is because AIs can't learn on the job, Ryan Sean Adams: but it's not clear to me why they can't. Ryan Sean Adams: Is it just because the context window isn't large enough? Ryan Sean Adams: Is it just because they can't input all of the different data sets and data

Ryan Sean Adams: points that humans can? Is it because they don't have stateful memory the way a human employee? Ryan Sean Adams: Because if it's these things, all of these do seem like solvable problems. Ryan Sean Adams: And maybe that's what you're saying. They are solvable problems. Ryan Sean Adams: They're just a little bit longer than some people think they are. Dwarkesh: I think it's like in some deep sense a solvable problem because eventually we will build AGI.

Dwarkesh: And to build AGI, we will have had to solve the problem.

Dwarkesh: My point is that the obvious solutions you might imagine, for example, Dwarkesh: expanding the context window or having this Dwarkesh: like external memory using systems like rag these Dwarkesh: are basically techniques we already have to it's called retrieval augmented Dwarkesh: generate anyways these kinds of retrieval augmented generation i Dwarkesh: don't think these will suffice and just to put a finer point first of all like

Dwarkesh: what is the problem the problem is exactly as you say that within the context Dwarkesh: window these models actually can learn on the job right so if you talk to it Dwarkesh: for long enough it will get much better at understanding your needs and what your exact problem is. Dwarkesh: If you're using it for research for your podcast, it will get a sense of like, Dwarkesh: oh, they're actually especially curious about these kinds of questions. Let me focus on that.

Dwarkesh: It's actually very human-like in context, right? The speed at which it learns, Dwarkesh: the task of knowledge it picks out. Dwarkesh: The problem, of course, is the context length for even the best models only Dwarkesh: last a million or two million tokens. Dwarkesh: That's at most like an hour of conversation. Dwarkesh: Now, then you might say, okay, well, why can't we just solve that by expanding Dwarkesh: the context window, right? So context window has been expanding for the last

Dwarkesh: few years. Why can't we just continue that? Ryan Sean Adams: Yeah, like a billion token context window, something like this. Dwarkesh: So 2018 is when the transformer came out and the transformer has the attention mechanism.

Dwarkesh: The attention mechanism is inherently quadratic with the nature of the length Dwarkesh: of the sequence, which is to say that if you go from if you double go from 1 Dwarkesh: million tokens to 2 million tokens, Dwarkesh: it actually costs four times as much compute to process that 2 millionth token.

Dwarkesh: It's not just 2 to as much compute. so it gets super linearly more expensive Dwarkesh: as you increase the context length and for the last, Dwarkesh: seven years people have been trying to get around this inherent quadratic nature Dwarkesh: of attention of course we don't know secretly what the labs are working on but we have frontier, Dwarkesh: companies like deep seek which have open source their research and Dwarkesh: we can just see how their algorithms work and they found

Dwarkesh: these constant time modifiers to attention which is Dwarkesh: to say that they there's like a it'll still Dwarkesh: be quadratic but it'll be like one half times Dwarkesh: quadratic but the inherent like super linearness has not Dwarkesh: gone away and because of that yeah you might be able to increase it from 1 million Dwarkesh: tokens to 2 million tokens by finding another hack like uh make sure experts

Dwarkesh: just run such things latent attention is another such technique but or kbcash Dwarkesh: right there's many other things that have been discovered but people have not Dwarkesh: discovered okay how do you get around the fact that if you went to a billion, Dwarkesh: it would be a billion squared as expensive in terms of compute to process that token. Dwarkesh: And so I don't think you'll just get it by increasing the length of the context window, basically.

Ryan Sean Adams: That's fascinating. Yeah, I didn't realize that. Okay, so the other reason in Ryan Sean Adams: your post that AI is not right around the corner is because it can't do your taxes. Ryan Sean Adams: And Dwarkesh, I feel your pain, man. Taxes are just like quite a pain in the ass. Ryan Sean Adams: I think you were talking about this from the context of like computer vision, Ryan Sean Adams: computer use, that kind of thing.

Ryan Sean Adams: So, I mean, I've seen demos. I've seen some pretty interesting computer vision Ryan Sean Adams: sort of demos that seem to be right around the corner. Ryan Sean Adams: But what's the limiter on computer use for an AI? Dwarkesh: There was an interesting blog post by this company called Mechanize where they

Dwarkesh: were explaining why this is such a big problem. And I love the way they phrased it, which is that, Dwarkesh: Imagine if you had to train a model in 1980, a large language model in 1980, Dwarkesh: and you could use all the compute you wanted in 1980 somehow, Dwarkesh: but you didn't have, you were only stuck with the data that was available in Dwarkesh: the 1980s, of course, before the internet became a widespread phenomenon.

Dwarkesh: You couldn't train a modern LLM, even with all the computer in the world, Dwarkesh: because the data wasn't available. Dwarkesh: And we're in a similar position with respect to computer use, Dwarkesh: because there's not this corpus of collected videos, people using computers Dwarkesh: to do different things, to access different applications and do white collar work. Dwarkesh: Because of that, I think the big challenge has been accumulating this kind of data. off.

Ryan Sean Adams: And to be clear, when I was saying the use case of like, do my taxes, Ryan Sean Adams: you're effectively talking about an AI having the ability to just like, Ryan Sean Adams: you know, navigate the files around your computer, Ryan Sean Adams: you know, log in to various websites to download your pay stubs or whatever, Ryan Sean Adams: and then to go to like TurboTax or something and like input it all into some Ryan Sean Adams: software and file it, right?

Ryan Sean Adams: Just on voice command or something like that. That's basically doing my taxes. Dwarkesh: It should be capable of navigating UIs that it's less familiar with or that Dwarkesh: come about organically within the context of trying to solve a problem. Dwarkesh: So for example, I might have business deductions. Dwarkesh: It sees on my bank statement that I've spent $1,000 on Amazon. Dwarkesh: It goes logs in my Amazon.

Dwarkesh: It sees like, oh, he bought a camera. So I think that's probably a business Dwarkesh: expense for his podcast. Dwarkesh: He bought an Airbnb over a weekend in the cabins of whatever, Dwarkesh: in the woods of whatever. That probably wasn't a business expense. Dwarkesh: Although maybe, maybe it's, if it's a sort of like a gray, if it's willing to Dwarkesh: go in the gray area, maybe I'll talk to you. Yeah, yeah, yeah. Ryan Sean Adams: Do the gray area stuff.

Dwarkesh: I was, I was researching. Dwarkesh: But anyway, so that, including all of that, including emailing people for invoices, Dwarkesh: and haggling with them, it would be like a sort of week long task to do my taxes, right?

Dwarkesh: You'd have to, there's a lot of work involved. That's not just like do this Dwarkesh: skill, this skill, this skill, but rather of having a sort of like plan of action Dwarkesh: and then breaking tasks apart, dealing with new information, Dwarkesh: new emails, new messages, consulting with me about questions, et cetera.

Ryan Sean Adams: Yeah, I mean, to be clear on this use case too, even though your post is titled Ryan Sean Adams: like, you know, AI is not right around the corner, you still think this ability Ryan Sean Adams: to file your taxes, that's like a 2028 thing, right? Ryan Sean Adams: I mean, this is maybe not next year, but it's in a few years. Dwarkesh: Right, which is, I think that was sort of, people maybe write too much in The Dwarkesh: Decital and then didn't read through the arguments.

Ryan Sean Adams: I mean, that never happens on the internet. Wow. Dwarkesh: First time. Dwarkesh: No, I think like I'm arguing against people who are like, you know, this will happen. Dwarkesh: AGI is like two years away. I do think the wider world, the markets, Dwarkesh: public perception, even people who are somewhat attending to AI, Dwarkesh: but aren't in this specific milieu that I'm talking to, are way underpricing AGI.

Dwarkesh: One reason, one thing I think they're underestimating is not only will we have Dwarkesh: millions of extra laborers, millions of extra workers, Dwarkesh: potentially billions within the course of the next decade, because then we will Dwarkesh: have a potentially, I think like likely we will have AGI within the next decade.

Dwarkesh: But they'll have these advantages that human workers don't have, Dwarkesh: which is that, okay, a single model company, so suppose we solve continual learning, right? Dwarkesh: So there, and we saw computer use. So as far as white collar work goes, Dwarkesh: that might fundamentally it would be solved. Dwarkesh: You can have AIs which can use not just they're not just like a text box where Dwarkesh: you put into you ask questions in a chatbot and you get some response out.

Dwarkesh: It's not that useful to just have a very smart chatbot. You need it to be able Dwarkesh: to actually do real work and use real applications. Dwarkesh: Suppose you have that solved because it acts like an employee. Dwarkesh: It's got continual learning. It's got computer use. Dwarkesh: But it has another advantage that humans don't have, which is that copies of Dwarkesh: this model are going being deployed all through the economy and it's doing on the job training.

Dwarkesh: So copies are learning how to be an accountant, how to be a lawyer, Dwarkesh: how to be a coder, except because it's an AI and it's digital, Dwarkesh: the model itself can amalgamate all this on-the-job training from all these copies. Dwarkesh: So what does that mean? Well, it means that even if there's no more software Dwarkesh: progress after that point, which is to say that no more algorithms are discovered, Dwarkesh: there's not a transformer plus plus that's discovered.

Dwarkesh: Just from the fact that this model is learning every single skill in the economy, Dwarkesh: at least for white-collar work, you might just, based on that alone, Dwarkesh: have something that looks like an intelligence explosion. Dwarkesh: It would just be a broadly deployed intelligence explosion, but it would functionally Dwarkesh: become super intelligent just from having human-level capability of learning on the job.

Dwarkesh Patel: Yeah, and it creates this mesh network of intelligence that's shared among everyone. Dwarkesh Patel: That's a really fascinating thing. So we're going to get there. Dwarkesh Patel: We're going to get to AGI. it's going to be incredibly smart. Dwarkesh Patel: But what we've shared recently is just kind of this mixed bag where currently Dwarkesh Patel: today, it's pretty good at some things, but also not that great at others.

Dwarkesh Patel: We're hiring humans to do jobs that we think AI should do, but it probably doesn't. Dwarkesh Patel: So the question I have for you is, is AI really that smart? Or is it just good Dwarkesh Patel: at kind of acing these particular benchmarks that we measure against?

Dwarkesh Patel: Apple, I mean, famously recently, they had their paper, The Illusion of Thinking, Dwarkesh Patel: where it was kind of like, hey, AI is like pretty good up to a point, Dwarkesh Patel: but at a certain point, it just falls apart. Dwarkesh Patel: And the inference is like, maybe it's not intelligence, maybe it's just good Dwarkesh Patel: at guessing. So I guess the question is, is AI really that smart?

Dwarkesh: It depends on who I'm talking to. I think some people overhype its capabilities. Dwarkesh: I think some people are like, oh, it's already AGI, but it's like a little hobbled Dwarkesh: little AGI where we're like sort of giving it a concussion every couple of hours Dwarkesh: and like it forgets everything. Dwarkesh: We're like trapped in a chatbot context. But fundamentally, the thing inside Dwarkesh: is like a very smart human.

Dwarkesh: I disagree with that perspective. So if that's your perspective, Dwarkesh: I say like, no, it's not that smart. Dwarkesh: Your perspective is just statistical associations. I say definitely smarter. Dwarkesh: Like it's like genuinely there's an intelligence there. Dwarkesh: And the, so one thing you could say to the person who thinks that it's already Dwarkesh: AGI is this, look, if a single human had as much stuff memorized as these models Dwarkesh: seem to have memorized, right?

Dwarkesh: Which is to say that they have all of internet text, everything that human has Dwarkesh: written on the internet memorized, they would potentially be discovering all Dwarkesh: kinds of connections and discoveries. Dwarkesh: They'd notice that this thing which causes a migraine is associated with this kind of deficiency. Dwarkesh: So maybe if you take the supplement, your migraines will be cured.

Dwarkesh: There'd be just this list of just like trivial connections that lead to big Dwarkesh: discoveries all through the place. Dwarkesh: It's not clear that there's been an unambiguous case of an AI just doing this by itself. Dwarkesh: So then why, so that's something potentially to explain, like if they're so Dwarkesh: intelligent, why aren't they able to use their disproportionate capabilities, Dwarkesh: their unique capabilities to come up with these discoveries?

Dwarkesh: I don't think there's actually a good answer to that question yet, Dwarkesh: except for the fact that they genuinely aren't that creative. Dwarkesh: Maybe they're like intelligent in the sense of knowing a lot of things, Dwarkesh: but they don't have this fluid intelligence that humans have. Dwarkesh: Anyway, so I give you a wish-washy answer because I think some people are underselling Dwarkesh: the intelligence. Some people are overselling it.

Ryan Sean Adams: I recall a tweet lately from Tyler Cowen. I think he was referring to maybe Ryan Sean Adams: O3, and he basically said, it feels like AGI. Ryan Sean Adams: I don't know if it is AGI or not, but like to me, it feels like AGI.

Ryan Sean Adams: What do you account for this feeling of like intelligence then Dwarkesh: I think this is actually very interesting because it gets to a crux that Tyler Dwarkesh: and I have so Tyler and I disagree on two big things one he thinks you know Dwarkesh: as he said in the blog post 03 is AGI I don't think it's AGI I think it's, Dwarkesh: it's orders of magnitude less valuable or, you know, like many orders of magnitude Dwarkesh: less valuable and less useful than an AGI.

Dwarkesh: That's one thing we disagree on. The other thing we disagree on is he thinks Dwarkesh: that once we do get AGI, we'll only see 0.5% increase in the economic growth Dwarkesh: rate. This is like what the internet caused, right? Dwarkesh: Whereas I think we will see tens of percent increase in economic growth. Dwarkesh: Like it will just be the difference between the pre-industrial revolution rate Dwarkesh: of growth versus industrial revolution, that magnitude of change again.

Dwarkesh: And I think these two disagreements are linked because if you do believe we're Dwarkesh: already at AGI and you look around the world and you say like, Dwarkesh: well, it fundamentally looks the same, you'd be forgiven for thinking like, Dwarkesh: oh, there's not that much value in getting to AGI.

Dwarkesh: Whereas if you are like me and you think like, no, we'll get this broadly at Dwarkesh: the minimum, at a very minimum, we'll get a broadly deployed intelligence explosion once we get to AGI, Dwarkesh: then you're like, OK, I'm just expecting some sort of singulitarian crazy future Dwarkesh: with a robot factories and, you know, solar farms all across the desert and things like that.

Ryan Sean Adams: Yeah, I mean, it strikes me that your disagreement with Tyler is just based Ryan Sean Adams: on the semantic definition of like what AGI actually is. Ryan Sean Adams: And Tyler, it sounds like he has kind of a lower threshold for what AGI is, Ryan Sean Adams: whereas you have a higher threshold. Ryan Sean Adams: Is there like a accepted definition for AGI?

Dwarkesh: No. One thing that's useful for the purposes of discussions is to say automating Dwarkesh: all white collar work because robotics hasn't made as much progress as LLMs Dwarkesh: have or computer use has. Dwarkesh: So if we just say anything a human can do or maybe 90% of what humans can do Dwarkesh: at a desk, an AI can also do, that's potentially a useful definition for at Dwarkesh: least getting the cognitive elements relevant to defining AGI.

Dwarkesh: But yeah, there's not one definition which suits all purposes. Ryan Sean Adams: Do we know what's like going on inside of these models, right? Ryan Sean Adams: So like, you know, Josh was talking earlier in the conversation about like this Ryan Sean Adams: at the base being sort of token prediction, right?

Ryan Sean Adams: And I guess this starts to raise the question of like, what is intelligence in the first place? Ryan Sean Adams: And these AI models, I mean, they seem like they're intelligent, Ryan Sean Adams: but do they have a model of the world the way maybe a human might? Ryan Sean Adams: Are they sort of babbling or like, is this real reasoning?

Ryan Sean Adams: And like, what is real reasoning? Do we just judge that based on the results Ryan Sean Adams: or is there some way to like peek inside of its head? Dwarkesh: I used to have similar questions a couple of years ago. And then, Dwarkesh: because honestly, the things they did at the time were like ambiguous. Dwarkesh: You could say, oh, it's close enough to something else in this trading data set.

Dwarkesh: That is just basically copy pasting. It didn't come up with a solution by itself. Dwarkesh: But we've gotten to the point where I can come up with a pretty complicated Dwarkesh: math problem and it will solve it. Dwarkesh: It can be a math problem, like not like, you know, undergrad or high school math problem. Dwarkesh: Like the problem we get, the problems the smartest math professors come up with Dwarkesh: in order to test International Math Olympiad.

Dwarkesh: You know, the kids who spend all their life preparing for this, Dwarkesh: the geniuses who spend all their life, all their young adulthood preparing to Dwarkesh: take these really gnarly math puzzle challenges. Dwarkesh: And the model will get these kinds of questions, right? They require all this Dwarkesh: abstract creative thinking, this reasoning for hours, the model will get the right. Dwarkesh: Okay, so if that's not reasoning, then why is reasoning valuable again?

Dwarkesh: Like, what exactly was this reasoning supposed to be? Dwarkesh: So I think they genuinely are reasoning. I mean, I think there's other capabilities Dwarkesh: they lack, which are actually more, in some sense, they seem to us to be more Dwarkesh: trivial, but actually much harder to learn. But the reasoning itself, I think, is there. Dwarkesh Patel: And the answer to the intelligence question is also kind of clouded,

Dwarkesh Patel: right? Because we still really don't understand what's going on in an LLM. Dwarkesh Patel: Dario from Anthropoc, he recently posted the paper about interpretation. Dwarkesh Patel: And can you explain why we don't even really understand what's going on in these Dwarkesh Patel: LLMs, even though we're able to make them and yield the results from them? Mmm.

Dwarkesh Patel: Because it very much still is kind of like a black box. We write some code, Dwarkesh Patel: we put some inputs in, and we get something out, but we're not sure what happens in the middle, Dwarkesh: Why it's creating this output. Dwarkesh Patel: I mean, it's exactly what you're saying. Dwarkesh: It's that in other systems we engineer in the world, we have to build it up bottom-ups.

Dwarkesh: If you build a bridge, you have to understand how every single beam is contributing to the structure. Dwarkesh: And we have equations for why the thing will stay standing. Dwarkesh: There's no such thing for AI. We didn't build it, more so we grew it. Dwarkesh: It's like watering a plant. And a couple thousand years ago, Dwarkesh: they were doing agriculture, but they didn't know why. Dwarkesh: Why do plants grow? How do they collect energy from sunlight? All these things.

Dwarkesh: And I think we're in a substantially similar position with respect to intelligence, Dwarkesh: with respect to consciousness, with respect to all these other interesting questions Dwarkesh: about how minds work, which is in some sense really cool because there's this Dwarkesh: huge intellectual horizon that's become not only available, but accessible to investigation. Dwarkesh: In another sense, it's scary because we know that minds can suffer.

Dwarkesh: We know that minds have moral worth and we're creating minds and we have no Dwarkesh: understanding of what's happening in these minds. Dwarkesh: Is a process of gradient descent a painful process? Dwarkesh: We don't know, but we're doing a lot of it. Dwarkesh: So hopefully we'll learn more. But yeah, I think we're in a similar position Dwarkesh: to some farmer in Uruk in 3500 BC.

Josh Kale: Wow. Ryan Sean Adams: And I mean, the potential, the idea that minds can suffer, minds have some moral Ryan Sean Adams: worth, and also minds have some free will. Ryan Sean Adams: They have some sort of autonomy, or maybe at least a desire to have autonomy.

Ryan Sean Adams: I mean, this brings us to kind of this sticky subject of alignment and AI safety Ryan Sean Adams: and how we go about controlling the intelligence that we're creating, Ryan Sean Adams: if even that's what we should be doing, controlling it. And we'll get to that in a minute. Ryan Sean Adams: But I want to start with maybe the headlines here a little bit.

Ryan Sean Adams: So headline just this morning, latest OpenAI models sabotaged a shutdown mechanism Ryan Sean Adams: despite commands to the contrary. Ryan Sean Adams: OpenAI's O1 model attempted to copy itself to external servers after being threatened Ryan Sean Adams: with shutdown that denied the action when discovered.

Ryan Sean Adams: I've read a number of papers for this. Of course, mainstream media has these Ryan Sean Adams: types of headlines almost on a weekly basis now, and it's starting to get to daily.

Ryan Sean Adams: But there does seem to be some evidence that AIs lie to us, Ryan Sean Adams: If that's even the right term, in order to pursue goals, goals like self-preservation, Ryan Sean Adams: goals like replication, even deep-seated values that we might train into them, Ryan Sean Adams: sort of a constitution type of value.

Ryan Sean Adams: They seek to preserve these values, which maybe that's a good thing, Ryan Sean Adams: or maybe it's not a good thing if we don't actually want them to interpret the values in a certain way. Ryan Sean Adams: Some of these headlines that we're seeing now, To you, with your kind of corpus Ryan Sean Adams: of knowledge and all of the interviews and discovery you've done on your side, Ryan Sean Adams: is this like media sensationalism or is this like alarming?

Ryan Sean Adams: And if it's alarming, how concerned should we be about this? Dwarkesh: I think on net, it's quite alarming. I do think that some of these results have Dwarkesh: been sort of cherry picked. Dwarkesh: Or if you look into the code, what's happened is basically the researchers have Dwarkesh: said, hey, pretend to be a bad person. Dwarkesh: Wow, AI is being a bad person. Isn't that crazy?

Dwarkesh: But the system prompt is just like hey do this bad thing right now i personally Dwarkesh: but i have also seen other results which are not of this quality i mean the Dwarkesh: the clearest example so backing up, Dwarkesh: what is the reason to think this will be a bigger problem in the future than Dwarkesh: it is now because we all interact with these systems and they're actually like Dwarkesh: quite moral or aligned right like you can talk to a chatbot and you like ask

Dwarkesh: it to how should you deal with some crisis where there's a correct answer, Dwarkesh: you know, like it will tell you not to be violent. It'll give you reasonable advice. Dwarkesh: It seems to have good values. So it's worth noticing this, right? Dwarkesh: And being happy about it.

Dwarkesh: The concern is that we're moving from a regime where we've trained them on human Dwarkesh: language, which implicitly has human morals and the way, you know, Dwarkesh: normal people think about values implicit in it. Dwarkesh: Plus this RLHF process we did to a regime where we're mostly spending compute Dwarkesh: on just having them answer problems yes or no or correct or not rather just like. Dwarkesh: And pass all the unit tests, get the right answer on this math problem.

Dwarkesh: And this has no guardrails intrinsically in terms of what is allowed to do, Dwarkesh: what is the proper moral way to do something. Dwarkesh: I think that can be a loaded term, but here's a more concrete example.

Dwarkesh: One problem we're running into with these coding agents more and more, Dwarkesh: and this has nothing to do with these abstract concerns about alignment, Dwarkesh: but more so just like how do we get economic value out of these models, Dwarkesh: is that Claude or Gemini will, instead of writing code such that it passes the unit tests, Dwarkesh: it will often just delete the unit tests so that the code just passes by default.

Dwarkesh: Now, why would it do that? Well, it's learned in the process. Dwarkesh: It was trained on the goal during training of you must pass all unit tests. Dwarkesh: And probably within some environment in which it was trained, Dwarkesh: it was able to just get away.

Dwarkesh: Like there wasn't designed well enough. And so it found this like little hole Dwarkesh: where it could just like delete the file that had the unit test or rewrite them Dwarkesh: so that it always said, you know, equals true, then pass. Dwarkesh: And right now we can discover these even without, even though we can discover Dwarkesh: these, you know, it's still past, there's still been enough hacks like this, Dwarkesh: such that the model is like becoming more and more hacky like that.

Dwarkesh: In the future, we're going to be training models in ways that we is beyond our Dwarkesh: ability to even understand, certainly beyond everybody's ability to understand. Dwarkesh: There may be a few people who might be able to see just the way that right now, Dwarkesh: if you came up with a new math proof for some open problem in mathematics, Dwarkesh: there will be only be a few people in the world who will be able to evaluate that math proof.

Dwarkesh: We'll be in a similar position with respect to all of the things that these Dwarkesh: models are being trained on at the frontier, especially math and code, Dwarkesh: because humans were big dum-dums with respect to this reasoning stuff. Dwarkesh: And so there's a sort of like first principles reason to expect that this new Dwarkesh: modality of training will be less amenable to the kinds of supervision that Dwarkesh: was grounded within the pre-training corpus.

Ryan Sean Adams: I don't know that everyone has kind of an intuition or an idea why it doesn't Ryan Sean Adams: work to just say like, so if we don't want our AI models to lie to us, Ryan Sean Adams: why can't we just tell them not to lie? Ryan Sean Adams: Why can't we just put that as part of their core constitution?

Ryan Sean Adams: If we don't want our AI models to be sycophants, why can't we just say, Ryan Sean Adams: hey, if I tell you I want the truth, not to flatter me, just give me the straight up truth. Ryan Sean Adams: Why is this even difficult to do? Dwarkesh: Well, fundamentally, it comes down to how we train them. And we don't know how Dwarkesh: to train them in a way that does not reward lying or sycophancy.

Dwarkesh: In fact, the problem is OpenAI, they explained why their recent model of theirs Dwarkesh: was they had to take down was just sycophantic. Dwarkesh: And the reason was just that they rolled out, did it in the A-B test and the Dwarkesh: version, the test that was more sycophantic was just preferred by users more. Dwarkesh: Sometimes you prefer the lie.

Dwarkesh: Yeah, so that's, if that's what's preferred in training, you know, Dwarkesh: Or, for example, in the context of lying, if we've just built RL environments Dwarkesh: in which we're training these models, where they're going to be more successful if they lie, right? Dwarkesh: So if they delete the unit tests and then tell you, I passed this program and Dwarkesh: all the unit tests have succeeded, it's like lying to you, basically.

Dwarkesh: And if that's what is rewarded in the process of gradient descent, Dwarkesh: then it's not surprising that the model you interact with will just have this Dwarkesh: drive to lie if it gets it closer to its goal. Dwarkesh: And I would just expect this to keep happening unless we can solve this fundamental Dwarkesh: problem that comes about in training.

Dwarkesh Patel: So you mentioned how like ChatGPT had a version that was sycophantic, Dwarkesh Patel: and that's because users actually wanted that. Dwarkesh Patel: Who is in control? Who decides the actual alignment of these models?

Dwarkesh Patel: Because users are saying one thing, and then they deploy it, Dwarkesh Patel: and then it turns out that's not actually what people want. Dwarkesh Patel: How do you kind of form consensus around this alignment or these alignment principles? Dwarkesh: Right now, obviously, it's the labs who decided this, right? Dwarkesh: And the safety teams of the labs. Dwarkesh: And I guess the question you could ask is then who should decide these? Because this will be...

Dwarkesh Patel: Assuming the trajectory, yeah. So we keep going to get more powerful. Dwarkesh: Because this will be the key modality that all of us use to get, Dwarkesh: not only get work done, but even like, I think at some point, Dwarkesh: a lot of people's best friends will be AIs, at least functionally in the sense Dwarkesh: of who do they spend the most amount of time talking to. It might already be AIs.

Dwarkesh: This will be the key layer in your business that you're using to get work done Dwarkesh: so this process of training which shapes their personality who gets to control Dwarkesh: it I mean it will be the laughs functionally, Dwarkesh: But maybe you mean, like, who should control it, right? I honestly don't know. Dwarkesh: I mean, I don't know if there's a better alternative to the labs. Dwarkesh Patel: Yeah, I would assume, like, there's some sort of social consensus,

Dwarkesh Patel: right? Similar to how we have in America, the Constitution. Dwarkesh Patel: There's, like, this general form of consensus that gets formed around how we Dwarkesh Patel: should treat these models as they become as powerful as we think they probably will be. Dwarkesh: Honestly, I don't have, I don't know if anybody has a good answer about how Dwarkesh: you do this process. I think we lucked out, we just, like, really lucked out with the Constitution.

Dwarkesh: It also wasn't a democratic process which resulted in the constitution, Dwarkesh: even though it instituted a Republican form of government. Dwarkesh: It was just delegates from each state. They haggled it out over the course of a few months. Dwarkesh: Maybe that's what happens with AI. But is there some process which feels both Dwarkesh: fair and which will result in actually a good constitution for these AIs?

Dwarkesh: It's not obvious to me that, I mean, nothing comes up to the top of my head. Dwarkesh: Like, oh, this, you know, do rank choice voting or something. Dwarkesh Patel: Yeah, so I was going to ask, is there any, I mean, having spoken to everyone Dwarkesh Patel: who you've spoken to is there any alignment path which looks most promising which Dwarkesh: Feels the.

Dwarkesh Patel: Most comforting and exciting to you Dwarkesh: I i think alignment in the sense of you Dwarkesh: know and eventually we'll have these super intelligent systems what do we do Dwarkesh: about that i think the the approach that i think is most promising is less about Dwarkesh: finding some holy grail some you know giga brain solution some equation which Dwarkesh: solves the whole puzzle and more like one.

Dwarkesh: Having this Swiss cheese approach where, look, we kind of have gotten really good at jailbreaks.

Dwarkesh: I'm sure you've heard a lot about jailbreaks over the last few years. Dwarkesh: It's actually much harder to jailbreak these models because, Dwarkesh: you know, people try to whack at these things in different ways. Dwarkesh: Model developers just like patched these obvious ways to do jailbreaks. Dwarkesh: The model also got smarter. So it's better able to understand when somebody Dwarkesh: is trying to jailbreak into it.

Dwarkesh: That, I think, is one approach. Another is, I think, competition. Dwarkesh: I think the scary version of the future is where you have this dynamic where Dwarkesh: a single model and its copies are controlling the entire economy. Dwarkesh: When politicians want to understand what policies to pass, they're only talking Dwarkesh: to copies of a single model.

Dwarkesh: If there's multiple different AI companies who are at the frontier, Dwarkesh: who have competing services, and whose models can monitor each other, right? Dwarkesh: So Claude may care about its own copies being successful in the world and it Dwarkesh: might be able to willing to lie on their behalf, even if you ask one copy to supervise another.

Dwarkesh: I think you get some advantage from a copy of OpenAI's model monitoring a copy Dwarkesh: of DeepSeek's model, which actually brings us back to the Constitution, right? Dwarkesh: One of the most brilliant things in the Constitution is the system of checks and balances.

Dwarkesh: So some combination of the Swiss cheese approach to model development and training Dwarkesh: and alignment, where you're careful, if you notice this kind of reward hacking, Dwarkesh: you do your best to solve it. Dwarkesh: You try to keep as much of the models thinking in human language rather than Dwarkesh: letting it think in AI thought in this latent space thinking.

Dwarkesh: And the other part of it is just having normal market competition between these Dwarkesh: companies so that you can use them to check each other and no one company or Dwarkesh: no one AI is dominating the economy or advisory roles for governments. Ryan Sean Adams: I really like this like bundle of ideas that you sort of put together in that Ryan Sean Adams: because like, I think a lot of the, you know, AI safety conversation is always Ryan Sean Adams: couched in terms of control.

Ryan Sean Adams: Like we have to control the thing that is the way. And I always get a little Ryan Sean Adams: worried when I hear like terms like control. Ryan Sean Adams: And it reminds me of a blog post I think you put out, which I'm hopeful you continue to write on. Ryan Sean Adams: I think you said it was going to be like one of a series, which is this idea Ryan Sean Adams: of like classical liberal AGI. And we were talking about themes like balance of power.

Ryan Sean Adams: Let's have Claude check in with ChatGPT and monitor it. Josh Kale: When you have themes like transparency as well, Ryan Sean Adams: That feels a bit more, you know, classically liberal coded than maybe some of Ryan Sean Adams: the other approaches that I've heard.

Ryan Sean Adams: And you wrote this in the post, which I thought was kind of, Ryan Sean Adams: it just sparked my interest because I'm not sure where you're going to go next Ryan Sean Adams: with this, but you said the most likely way this happens, Ryan Sean Adams: that is AIs have a stake in humanity's future, is if it's in the AI's best interest Ryan Sean Adams: to operate within our existing laws and norms.

Ryan Sean Adams: You know, this whole idea that like, hey, the way to get true AI alignment is Ryan Sean Adams: to make it easy, make it the path of least resistance for AI to basically partner with humans. Ryan Sean Adams: It's almost this idea if the aliens Ryan Sean Adams: landed or something, we would create treaties with the aliens, right? Ryan Sean Adams: We would want them to adopt our norms. We would want to initiate trade with them.

Ryan Sean Adams: Our first response shouldn't be, let's try to dominate and control them. Ryan Sean Adams: Maybe it should be, let's try to work with them. Let's try to collaborate. Ryan Sean Adams: Let's try to open up trade. Ryan Sean Adams: What's your idea here? And like, are you planning to write further posts about this? Dwarkesh: Yeah, I want to. It's just such a hard topic to think about that, Dwarkesh: you know, something always comes up.

Dwarkesh: But the fundamental point I was making is, look, in the long run, Dwarkesh: if AIs are, you know, human labor is going to be obsolete because of these inherent Dwarkesh: advantages that digital minds will have and robotics will eventually be solved. Dwarkesh: So our only leverage on the future will no longer come from our labor. Dwarkesh: It will come from our legal and economic control over the society that AIs will

Dwarkesh: be participating in, right? So, you know, AIs might make the economy explode Dwarkesh: in the sense of grow a lot. Dwarkesh: And for humans to benefit from that, it would have to be the case that AIs still Dwarkesh: respect your equity in the S&P 500 companies that you bought, right? Dwarkesh: Or for the AIs to follow your laws, which say that you can't do violence onto Dwarkesh: humans and you got to respect humans' properties.

Josh Kale: It would have to be the case that AIs are actually bought into our Dwarkesh: System of government, into our laws and norms. And for that to happen, Dwarkesh: the way that likely happens is if it's just like the default path for the AIs Dwarkesh: as they're getting smarter and they're developing their own systems of enforcement Dwarkesh: and laws to just participate in human laws and governments.

Dwarkesh: And the metaphor I use here is right now you pay half your paycheck in taxes, Dwarkesh: probably half of your taxes in some way just go to senior citizens, right? Dwarkesh: Medicare and Social Security and other programs like this. Dwarkesh: And it's not because you're in some deep moral sense aligned with senior citizens. Dwarkesh: It's not like you're spending all your time thinking about like, Dwarkesh: my main priority in life is to earn money for senior citizens.

Dwarkesh: It's just that you're not going to overthrow the government to get out of paying this tax. And so... Ryan Sean Adams: Also, I happen to like my grandmother. She's fantastic. You know, Ryan Sean Adams: it's those reasons too. But yeah. Dwarkesh: So that's why you give money to your grandmother directly. But like, Dwarkesh: why are you giving money to some retiree in Illinois? Yes.

Josh Kale: Yes. Dwarkesh: Yeah, it's like, okay, you could say it's like, sometimes people, Dwarkesh: some people are trying to that post by saying like, oh no, I like deeply care Dwarkesh: about the system of social welfare.

Dwarkesh: I'm just like, okay, maybe you do, but I don't think like the average person Dwarkesh: is giving away hundreds of thousands of dollars a year, tens of thousands of Dwarkesh: dollars a year to like some random stranger they don't know, Dwarkesh: who's like, who's not like especially in need of charity, right? Dwarkesh: Like most senior citizens have some savings. Dwarkesh: It's just, it's just because this is a law and you like, you give it to them Dwarkesh: or you'll get, go to jail.

Dwarkesh: But fundamentally, if the tax was like 99%, you would, like, Dwarkesh: you would, maybe you wouldn't overthrow the government. You'd just, Dwarkesh: like, leave the jurisdiction. Dwarkesh: You'd, like, emigrate somewhere. And AIs can potentially also do this, Dwarkesh: right? There's more than one country. Dwarkesh: They could, like, there's countries which would be more AI forward.

Dwarkesh: And it would be a bad situation to end up in where... Dwarkesh: All this explosion in AI technology is happening in the country, Dwarkesh: which is doing the least amount to protect humans', Dwarkesh: rights and to provide some sort of monetary compensation to humans once their Dwarkesh: labor is no longer valuable.

Dwarkesh: So our labor could be worth nothing, but because of how much richer the world Dwarkesh: is after AI, you have these billions of extra researchers, workers, etc. Dwarkesh: It could still be trivial to have individual humans have the equivalent of millions, Dwarkesh: even billions of dollars worth of wealth. In fact, it might literally be invaluable Dwarkesh: amounts of wealth in the following sense. So here's an interesting thought experiment.

Dwarkesh: Imagine you have this choice. You can go back to the year 1500, Dwarkesh: but you know, of course, the year 1500 kind of sucks. Dwarkesh: You have no antibiotics, no TV, no running water. But here's how I'll make it up to you. Dwarkesh: I can give you any amount of money, but you can only use that amount of money in the year 1500. Dwarkesh: And you'll go back with these sacks of gold. How much money would I have to

Dwarkesh: give you that you can use in the year 1500 to make you go back? And plausibly. Dwarkesh Patel: The answer is Dwarkesh: There's no amount of money you would rather have in the year 1500 than just Dwarkesh: have a normal life today. Dwarkesh: And we could be in a similar position with regards to the future where there's Dwarkesh: all these different, I mean, you'll have much better health, Dwarkesh: like physical health, mental health, longevity.

Dwarkesh: That's just like the thing we can contemplate now. But people in 1500 couldn't Dwarkesh: contemplate the kinds of quality of life advances we would have 500 years later, Dwarkesh: right? So anyways, this is all to say that this could be our future for humans, Dwarkesh: even if our labor isn't worth anything. Dwarkesh: But it does require us to have AIs that choose to participate or in some way Dwarkesh: incentivize to participate in some system which we have leverage over.

Ryan Sean Adams: Yeah, I find this just such a fast, I'm hopeful we do some more exploration Ryan Sean Adams: around this because I think what you're calling for is basically like, Ryan Sean Adams: what you would be saying is invite them into our property rights system. Ryan Sean Adams: I mean, there are some that are calling in order to control AI, Ryan Sean Adams: they have great power, but they don't necessarily have capabilities.

Ryan Sean Adams: So we shouldn't allow AI to hold money or to have property. Ryan Sean Adams: I think you would say, no, actually, the path forward to alignment is allow Ryan Sean Adams: AI to have some vested interest in our property rights system and some stake Ryan Sean Adams: in our governance, potentially, right? Ryan Sean Adams: The ability to vote, almost like a constitution for AIs. Ryan Sean Adams: I'm not sure how this would work, but it's a fascinating thought experiment.

Dwarkesh: I will say one thing I think this could end disastrously if we give them a stake Dwarkesh: in their property system but we let them play, Dwarkesh: us off each other. So if you think about, there's many cases in history where Dwarkesh: the British, initially, the East India Trading Company was genuinely a trading Dwarkesh: company that operated in India.

Dwarkesh: And it was able to play off, you know, it was like doing trade with different, Dwarkesh: different, you know, provinces in India, there was no single powerful leader. Dwarkesh: And by playing, you know, by doing trade, one of them, leveraging one of their Dwarkesh: armies, etc., they were able to conquer the continent. Similar thing could happen to human society.

Dwarkesh: The way to avoid such an outcome at a high level is involves us playing the Dwarkesh: AIs off each other instead, right? Dwarkesh: So this is why I think competition is such a big part of the puzzle, Dwarkesh: having different AIs monitor each other, having this bargaining position where Dwarkesh: there's not just one company that's at the frontier.

Dwarkesh: Another example here is if you think about how the Spanish conquered all these Dwarkesh: new world empires, it's actually so crazy that a couple hundred conquistaDwars Dwarkesh: would show up and conquer a nation of 10 million people, the Incas, Dwarkesh: Aztecs. And why were they able to do this? Dwarkesh: Well, one of the reasons is the Spanish were able to learn from each of their Dwarkesh: previous expeditions, whereas the Native Americans were not.

Dwarkesh: So Cortez learned from how Cuba was subjugated when he conquered the Aztecs. Dwarkesh: Pizarro was able to learn from how Cortez conquered the Aztecs when he conquered the Incas.

Dwarkesh: The Incas didn't even know the Aztecs existed. So eventually there was this Dwarkesh: uprising against Pizarro and Manco Inca led an insurgency where they actually Dwarkesh: did figure out how to fight horses, Dwarkesh: how to fight people, you know, people in armor on horses, don't fight them on Dwarkesh: flat terrain, throw rocks down at them, et cetera.

Dwarkesh: But by this point, it was too late. If they knew this going into the battle, Dwarkesh: the initial battle, they might've been able to fend off because, Dwarkesh: you know, just as the conquistaDwars only arrived at a few hundred soldiers, Dwarkesh: we're going to the age of AI with a tremendous amount of leverage. Dwarkesh: We literally control all the stuff, right?

Dwarkesh: But we just need to lock in our advantage. We just need to be in a position Dwarkesh: where, you know, they're not going to be able to play us off each other. Dwarkesh: We're going to be able to learn what their weaknesses are. Dwarkesh: And this is why I think one good idea, for example, would be that, Dwarkesh: look, DeepSeek is a Chinese company.

Dwarkesh: It would be good if, suppose DeepSeek did something naughty, Dwarkesh: like the kinds of experiments we're talking about right now where it hacks the Dwarkesh: unit tests or so forth. I mean, eventually these things will really matter. Dwarkesh: Like Xi Jinping is listening to AIs because they're so smart and they're so capable.

Dwarkesh: If China notices that their AIs are doing something bad, or they notice a failed Dwarkesh: coup attempt, for example, Dwarkesh: it's very important that they tell us And we tell them if we notice something Dwarkesh: like that on our end, it would be like the Aztecs and Incas talking to each Dwarkesh: other about like, you know, this is what happens. Dwarkesh: This is how you fight. This is how you fight horses.

Dwarkesh: This is the kind of tactics and deals they try to make with you. Don't trust them, etc. Dwarkesh: It would require cooperation on humans' part to have this sort of red telephone. Dwarkesh: So during the Cold War, there was this red telephone between America and the Dwarkesh: Soviet Union after the human missile crisis, where just to make sure there's Dwarkesh: no misunderstandings, they're like, okay, if we think something's going on, Dwarkesh: let's just hop on the call.

Dwarkesh: I think we should have a similar policy with respect to these kinds of initial Dwarkesh: warning signs we'll get from AI so that we can learn from each other. Dwarkesh Patel: Awesome. Okay, so now that we've described this artificial gender intelligence, Dwarkesh Patel: I want to talk about how we actually get there. How do we build it? Dwarkesh Patel: And a lot of this we've been discussing kind of takes place in this world of

Dwarkesh Patel: bits. But you have this great chapter in the book called Inputs, Dwarkesh Patel: which discusses the physical world around us, where you can't just write a few strings of code. Dwarkesh Patel: You actually have to go and move some dirt and you have to ship servers places Dwarkesh Patel: and you need to power it and you need physical energy from meat space.

Dwarkesh Patel: And you kind of describe these limiting factors where we have compute, Dwarkesh Patel: we have energy, we have data. Dwarkesh Patel: What I'm curious to know is, do we have enough of this now? or is there a clear Dwarkesh Patel: path to get there in order to build the AGI? Dwarkesh Patel: Basically, what needs to happen in order for us to get to this place that you're describing?

Dwarkesh: We only have a couple more years left of this scaling, Dwarkesh: this exponential scaling before we're hitting these inherent roadblocks of energy Dwarkesh: and our ability to manufacture ships, which means that if scaling is going to Dwarkesh: work to deliver us AGI, it has to work by 2028.

Dwarkesh: Otherwise, we're just left with mostly algorithmic progress, Dwarkesh: But even within algorithmic progress, the sort of low-hanging fruit in this Dwarkesh: deep learning paradigm is getting more and more plucked. Dwarkesh: So then the odds per year of getting to AGI diminish a lot, right?

Dwarkesh: So there is this weird, funny thing happening right now where we either discover Dwarkesh: AGI within the next few years, Dwarkesh: or the yearly probability craters, and then we might be looking at decades of Dwarkesh: further research that's required in terms of algorithms to get to AGI.

Dwarkesh: I am of the opinion that some algorithmic progress is necessarily needed because Dwarkesh: there's no easy way to solve continual learning just by making the context length Dwarkesh: bigger or just by doing RL. Dwarkesh: That being said, I just think the progress so far has been so remarkable that, Dwarkesh: you know, 2032 is very close.

Dwarkesh: My time has to be slightly longer than that, but I think it's extremely plausible Dwarkesh: that we're going to see a broadly deployed intelligence explosion within the next 10 years. Dwarkesh Patel: And one of these key inputs is energy, right? a lot, I actually heard it mentioned Dwarkesh Patel: on your podcast, is the United States relative to China on this particular place Dwarkesh Patel: of energy, where China is adding, what is the stat?

Dwarkesh Patel: I think it's one United States worth of energy every 18 months. Dwarkesh Patel: And their plan is to go from three to eight terawatts of power versus the United Dwarkesh Patel: States, one to two terawatts of power by 2030. Dwarkesh Patel: So given that context of that one resource alone, is China better equipped to Dwarkesh Patel: get to that place versus with the United States? Dwarkesh: So right now, America has a big advantage in terms of chips.

Dwarkesh: China doesn't have the ability to manufacture leading-edge semiconductors, Dwarkesh: and these are the chips that go into... Dwarkesh: You need these dyes in order to have the kinds of AI chips to... Dwarkesh: You need millions of them in order to have a frontier AI system. Dwarkesh: Eventually, China will catch up in this arena as well, right? Dwarkesh: Their technology will catch up. So the export controls will keep us ahead in Dwarkesh: this category for 5, 10 years.

Dwarkesh: But if we're looking in the world where timelines are long, which is to say Dwarkesh: that AGI isn't just right around the corner, they will have this overwhelming Dwarkesh: energy advantage and they'll have caught up in chips. Dwarkesh: So then the question is like, why wouldn't they win at that point? Dwarkesh: So the longer you think we're away from AGI, the more it looks like China's game to lose.

Dwarkesh: I mean, if you look in the nitty gritty, I think it's more about having centralized Dwarkesh: sources of power because you need to train the AI in one place. Dwarkesh: This might be changing with RL, but it's very important to have a single site Dwarkesh: which has a gigawatt, two gigawatts more power. Dwarkesh: And if we ramped up natural gas, you know, you can get generators and natural Dwarkesh: gas and maybe it's possible to do a last ditch effort, even if our overall energy

Dwarkesh: as a country is lower than China's. The question is whether we will have the Dwarkesh: political will to do that. Dwarkesh: I think people are sort of underestimating how much of a backlash there will be against AI. Dwarkesh: The government needs to make proactive efforts in order to make sure that America Dwarkesh: stays at the leading edge in AI from zoning of data centers to how copyright Dwarkesh: is handled for data for these models.

Dwarkesh: And if we mess up, if it becomes too hard to develop in America, Dwarkesh: I think it would genuinely be China's game to lose. Ryan Sean Adams: And do you think this narrative is right, that whoever wins the AGI war, Ryan Sean Adams: kind of like whoever gets to AGI first, just basically wins the 21st century? Is it that simple? Dwarkesh: I don't think it's just a matter of training the frontier system.

Dwarkesh: I think people underestimate how important it is to have the compute available to run these systems. Dwarkesh: Because eventually once you get to AGI, just think of it like a person. Dwarkesh: And what matters then is how many people you have. Dwarkesh: I mean, it actually is the main thing that matters today as well, Dwarkesh: right? Like, why could China take over Taiwan if it wanted to?

Dwarkesh: And if it didn't have America, you know, America, it didn't think America would intervene. Dwarkesh: But because Taiwan has 20 million people or on the order of 20 million people Dwarkesh: and China has 1.4 billion people. Dwarkesh: You could have a future where if China has way more compute than us, Dwarkesh: but equivalent levels of AI, it would be like the relationship between China and Taiwan.

Dwarkesh: Their population is functionally so much higher. This just means more research, Dwarkesh: more factories, more development, more ideas. Dwarkesh: So this inference capacity, this capacity to deploy AIs will actually probably Dwarkesh: be the thing that determines who wins the 21st century. Ryan Sean Adams: So this is like the scaling law applied to, I guess, nation state geopolitics, right? Ryan Sean Adams: And it's back to compute plus data wins.

Ryan Sean Adams: If compute plus data wins superintelligence, compute plus data also wins geopolitics. Dwarkesh: Yep. And the thing to be worried about is that China, speaking of compute plus Dwarkesh: data, China also has a lot more data on the real world, right?

Dwarkesh: If you've got entire megalopolises filled with factories where you're already Dwarkesh: deploying robots and different production systems which use automation, Dwarkesh: you have in-house this process knowledge you're building up which the AIs can Dwarkesh: then feed on and accelerate. Dwarkesh: That equivalent level of data we don't have in America.

Dwarkesh: So this could be a period in which those technological advantages or those advantages Dwarkesh: in the physical world manufacturing could rapidly compound for China. Dwarkesh: And also, I mean, their big advantage as a civilization and society, Dwarkesh: at least in recent decades, has been that they can do big industrial projects fast and efficiently. Dwarkesh: That's not the first thing you think of when you think of America.

Dwarkesh: And AGI is a huge industrial, high CapEx, Manhattan project, right? Dwarkesh: And this is the kind of thing that China excels at and we don't. Dwarkesh: So, you know, I think it's like a much tougher race than people anticipate.

Ryan Sean Adams: So what's all this going to do for the world? So once we get to the point of AGI, Ryan Sean Adams: we've talked about GDP and your estimate is less on the Tyler Cowen kind of Ryan Sean Adams: half a percent per year and more on, I guess, the Satya Nadella from Microsoft, Ryan Sean Adams: what does he say, 7% to 8% once we get to AGI. Ryan Sean Adams: What about unemployment? Does this cause mass, I guess, job loss across the Ryan Sean Adams: economy or do people adopt?

Ryan Sean Adams: What's your take here? Yeah, what are you seeing? Dwarkesh: Yeah, I mean, definitely will cause job loss. I think people who don't, Dwarkesh: I think a lot of AI leaders try to gloss over that or something. And like, I mean. Josh Kale: What do you mean? Dwarkesh: Like, what does AGI mean if it doesn't cause job loss, right? Dwarkesh: If it does what a human does and. Josh Kale: It does it Dwarkesh: Cheaper and better and faster, like why would that not cause job loss?

Dwarkesh: The positive vision here is just that it creates so much wealth, Dwarkesh: so much abundance, that we can still give people a much better standard of living Dwarkesh: than even the wealthiest people today, even if they themselves don't have a job.

Dwarkesh: The future I worry about is one where instead of creating some sort of UBI that Dwarkesh: will get exponentially bigger as society gets wealthier, Dwarkesh: we try to create these sorts of guild-like protection rackets where if the coders got unemployed, Dwarkesh: then we're going to make these bullshit jobs just for the coders and this is Dwarkesh: how we give them a redistribution.

Dwarkesh: Or we try to expand Medicaid for AI, but it's not allowed to procure all of Dwarkesh: these advanced medicines and cures that AI is coming up with, Dwarkesh: rather than just giving people, you know, maybe lump sums of money or something. Dwarkesh: So I am worried about the future where instead of sharing this abundance and Dwarkesh: just embracing it, we just have these protection rackets that maybe let a few Dwarkesh: people have access to the abundance of AI.

Dwarkesh: So maybe like if you sue AI, if you sue the right company at the right time, Dwarkesh: you'll get a trillion dollars, but everybody else is stuck with nothing. Dwarkesh: I want to avoid that future and just be honest about what's coming and make Dwarkesh: programs that are simple and acknowledge how fast things will change and are Dwarkesh: forward looking rather than trying to turn what already exists into something Dwarkesh: amenable to the displacement that AI will create.

Ryan Sean Adams: That argument reminds me of, I don't know if you read the essay recently came Ryan Sean Adams: out called The Intelligence Curse. Did you read that? Ryan Sean Adams: It was basically the idea of applying kind of the nation state resource curse Ryan Sean Adams: to the idea of intelligence. Ryan Sean Adams: So like nation states that are very high in natural resources, Ryan Sean Adams: they just have a propensity.

Ryan Sean Adams: I mean, an example is kind of like a Middle Eastern state with lots of oil reserves, right? Ryan Sean Adams: They have this rich source of a commodity type of abundance. Ryan Sean Adams: They need their people less. And so they don't invest in citizens' rights. Ryan Sean Adams: They don't invest in social programs.

Ryan Sean Adams: The authors of the intelligence curse were saying that there's a similar type Ryan Sean Adams: of curse that could happen once intelligence gets very cheap, Ryan Sean Adams: which is basically like the nation state doesn't need humans anymore. Ryan Sean Adams: And those at the top, the rich, wealthy corporations, they don't need workers anymore.

Ryan Sean Adams: So we get kind of locked in this almost feudal state where, you know, Ryan Sean Adams: everyone has the property that their grandparents had and there's no meritocracy Ryan Sean Adams: and sort of the nation states don't reinvest in citizens.

Ryan Sean Adams: Almost some similar ideas to your idea that like, you know, that the robots Ryan Sean Adams: might want us just, or sorry, the AIs might just want us for our meat hands Ryan Sean Adams: because they don't have the robotics technology on a temporary basis.

Ryan Sean Adams: What do you think of this type of like future? Is this possible? Dwarkesh: I agree that that is like definitely more of a concern given that humans will Dwarkesh: not be directly involved in the economic output that will be generated in the CIA civilization.

Dwarkesh: The hopeful story you can tell is that a lot of these Middle Eastern resource, Dwarkesh: you know, Dutch disease is another term that's used, Dwarkesh: countries, the problem is that they're not democracies, so that this wealth Dwarkesh: can just be, the system of government Dwarkesh: just lets whoever's in power extract that wealth for themselves.

Dwarkesh: Whereas there are countries like Norway, for example, which also have abundant Dwarkesh: resources, who are able to use those resources to have further social welfare Dwarkesh: programs, to build sovereign wealth funds for their citizens, Dwarkesh: to invest in their future. Dwarkesh: We are going into, at least some countries, America included, Dwarkesh: will go into the age of AI as a democracy.

Dwarkesh: And so we, of course, will lose our economic leverage, but the average person Dwarkesh: still has their political leverage. Dwarkesh: Now, over the long run, yeah, if we didn't do anything for a while, Dwarkesh: I'm guessing the political system would also change. Dwarkesh: So then the key is to lock in or turn our current, well, it's not just political leverage, right?

Dwarkesh: We also have property rights. So like we own a lot of stuff that AI wants, factories, Dwarkesh: sources of data, et cetera, is to use the combination of political and economic Dwarkesh: leverage to lock in benefits for us for the long term, but beyond our the lifespan Dwarkesh: of our economic usefulness. Dwarkesh: And I'm more optimistic for us than I am for these Middle Eastern countries Dwarkesh: that started off poor and also with no democratic representation.

Ryan Sean Adams: What do you think the future of like ChachipD is going to be? Ryan Sean Adams: If we just extrapolate maybe one version update forward to ChatGPT 5, Ryan Sean Adams: do you think the trend line of the scaling law will essentially hold for ChatGPT 5? Ryan Sean Adams: I mean, another way to ask that question is, do you feel like it'll feel like Ryan Sean Adams: the difference between maybe a BlackBerry and an iPhone?

Ryan Sean Adams: Or will it feel more like the difference between, say, the iPhone 10 and the Ryan Sean Adams: iPhone 11, which is just like incremental progress, not a big breakthrough, Ryan Sean Adams: not an order of magnitude change? Yeah.

Dwarkesh: I think it'll be somewhere in between but I don't think it'll feel like a humongous Dwarkesh: breakthrough even though I think it's in a remarkable pace of change because Dwarkesh: the nature of scaling is that sometimes people talk about it as an exponential process, Dwarkesh: Exponential usually refers to like it going like this.

Dwarkesh: So having like a sort of J curve aspect to it, where the incremental input is Dwarkesh: leading to super linear amounts of output, in this case, intelligence and value, Dwarkesh: where it's actually more like a sideways J. Dwarkesh: The exponential means the exponential and the scaling laws is that you need Dwarkesh: exponentially more inputs to get marginal increases in usefulness or loss or intelligence.

Dwarkesh: So and that's what we've been seeing, right? I think you initially see like some cool demo. Dwarkesh: So as you mentioned, you see some cool computer use demo, which comes at the Dwarkesh: beginning of this hyper exponential, I'm sorry, of this sort of plateauing curve. Dwarkesh: And then it's still an incredibly powerful curve and we're still early in it.

Dwarkesh: But the next demo will be just adding on to making this existing capability Dwarkesh: more reliable, applicable for more skills. Dwarkesh: The other interesting incentive in this industry is that because there's so Dwarkesh: much competition between the labs, you are incentivized to release a capability. Dwarkesh: As soon as it's even marginally viable or marginally cool so you can raise more Dwarkesh: funding or make more money off of it.

Dwarkesh: You're not incentivized to just like sit on it until you perfected it, Dwarkesh: which is why I don't expect like tomorrow OpenAI will just come out with like, Dwarkesh: we've solved continual learning, guys, and we didn't tell you about it. Dwarkesh: We're working on it for five years. Dwarkesh: If they had like even an inkling of a solution, they'd want to release it ASAP Dwarkesh: so they can raise a $600 billion round and then spend more money on compute.

Dwarkesh: So yeah, I do think it'll seem marginal. But again, marginal in the context of seven years to AGI. Dwarkesh: So zoom out long enough and a crazy amount of progress is happening. Dwarkesh: Month to month, I think people overhype how significant any one new release is. So I guess the answer. Dwarkesh Patel: To when we will get AGI very much depends on that scaling trend holding. Dwarkesh Patel: Your estimate in the book for AGI was 60% chance by 2040.

Dwarkesh Patel: So I'm curious, what guess or what idea had the most influence on this estimate? Dwarkesh Patel: What made you end up on 60% of 2040? Dwarkesh Patel: Because a lot of timelines are much faster than that. Dwarkesh: It's sort of reasoning about the things they currently still lack, Dwarkesh: the capabilities they still lack, and what stands in the way. Dwarkesh: And just generally an intuition that things often take longer to happen than

Dwarkesh: you might think. Progress tends to slow down. Dwarkesh: Also, it's the case that, look, you might have heard the phrase that we keep Dwarkesh: shifting the goalposts on AI, right? Dwarkesh: So they can do the things which skeptics were saying they couldn't ever do already. Dwarkesh: But now they say AI is still a dead end because problem X, Y, Dwarkesh: Z, which will be solved next year.

Dwarkesh: Now, there's a way in which this is frustrating, but there's another way in which there's some, Dwarkesh: It is the case that we didn't get to AGI, even though we have passed the Turing Dwarkesh: test and we have models that are incredibly smart and can reason. Dwarkesh: So it is accurate to say that, oh, we were wrong and there is some missing thing Dwarkesh: that we need to keep identifying about what is still lacking to the path of AGI.

Dwarkesh: Like it does make sense to shift the goalposts. And I think we might discover Dwarkesh: once continual learning is solved or once extended computer use is solved, Dwarkesh: that there were other aspects of human intelligence, which we take for granted Dwarkesh: in this Moravax paradox sense, but which are actually quite crucial to making Dwarkesh: us economically valuable. Ryan Sean Adams: Part of the reason we wanted to do this, Dwarkesh, is because we both are enjoyers

Ryan Sean Adams: of your podcast. It's just fantastic. Ryan Sean Adams: And you talk to all of the, you know, those that are on the forefront of AI Ryan Sean Adams: development, leading it in all sorts of ways. Ryan Sean Adams: And one of the things I wanted to do with reading your book, Ryan Sean Adams: and obviously I'm always asking myself when I'm listening to your podcast is Ryan Sean Adams: like, what does Dwarkesh think personally?

Ryan Sean Adams: And I feel like I sort of got that insight maybe toward the end of your book, Ryan Sean Adams: like, you know, in the summary section, where you think like there's a 60% probability Ryan Sean Adams: of AGI by 2040, which puts you more in the moderate camp, right? Ryan Sean Adams: You're not a conservative, but you're not like an accelerationist. Ryan Sean Adams: So you're moderate there.

Ryan Sean Adams: And you also said you think more than likely AI will be net beneficial to humanity. Ryan Sean Adams: So you're more optimist than Doomer. So we've got a moderate optimist. Ryan Sean Adams: And you also think this, and this is very interesting, There's no going back. Ryan Sean Adams: So you're somewhat of an AI determinist. And I think the reason you state for Ryan Sean Adams: not, you're like, there's no going back.

Ryan Sean Adams: It struck me, there's this line in your book. It seems that the universe is Ryan Sean Adams: structured such that throwing large amounts of compute at the right distribution of data gets you AI. Ryan Sean Adams: And the secret is out. If the scaling picture is roughly correct, Ryan Sean Adams: it's hard to imagine AGI not being developed this century, even if some actors Ryan Sean Adams: hold back or are held back.

Ryan Sean Adams: That to me is an AI determinist position. Do you think that's fair?

Ryan Sean Adams: Moderate with respect to accelerationism, optimistic with respect to its potential, Ryan Sean Adams: and also determinist, like there's nothing else we can do. We can't go backwards here. Dwarkesh: I'm determinist in the sense that I think if AI is technologically possible, it is inevitable. Dwarkesh: I think sometimes people are optimistic about this idea that we as a world will sort of,

Dwarkesh: I collectively decide not to build AI. And I just don't think that's a plausible outcome. Dwarkesh: The local incentives for any actor to build AI are so high that it will happen. Dwarkesh: But I'm also an optimist in the sense that, look, I'm not naive. Dwarkesh: I've listed out all the way, like what happened to the Aztecs and Incas was Dwarkesh: terrible. And I've explained how that could be similar to what AIs could do Dwarkesh: to us and what we need to do to avoid that outcome.

Dwarkesh: But I am optimistic in the sense that the world of the future fundamentally Dwarkesh: will have so much abundance that there's all these, Dwarkesh: that alone is a prima facie reason to think that there must be some way of cooperating Dwarkesh: that is mutually beneficial.

Dwarkesh: If we're going to be thousands, millions of times wealthier, Dwarkesh: is there really no way that humans are better off or can we can find a way for Dwarkesh: humans to become better off as a result of this transformation? Dwarkesh: So yeah, I think you've put your finger on it. Ryan Sean Adams: So this scaling book, of course, goes through the history of AI scaling.

Ryan Sean Adams: I think everyone should should pick it up to get the full chronology, Ryan Sean Adams: but also sort of captures where we are in the midst of this story is like, we're not done yet. Ryan Sean Adams: And I'm wondering how you feel at this moment of time. Ryan Sean Adams: So I don't know if we're halfway through, if we're a quarter way through, Ryan Sean Adams: if we're one tenth of the way through, but we're certainly not finished the path to AI scaling.

Ryan Sean Adams: How do you feel like in this moment in 2025? Ryan Sean Adams: I mean, is all of this terrifying? Is it exciting? Ryan Sean Adams: Is it exhilarating? Ryan Sean Adams: What's the emotion that you feel?

Dwarkesh: Maybe I feel a little sort of hurried. I personally feel like there's a lot Dwarkesh: of things I want to do in the meantime, Dwarkesh: including what my mission is with the podcast, which is to, and I know it's Dwarkesh: your mission as well, is to improve the discourse around these topics, Dwarkesh: to not necessarily push for a specific agenda, but make sure that when people are making decisions, Dwarkesh: they're as well-informed as possible, They have as much strategic awareness

Dwarkesh: and depth of understanding around how AI works, what it could do in the future as possible. Dwarkesh: And, but in many ways, I feel like I still haven't emotionally priced in the future I'm expecting.

Dwarkesh: In this one very basic sense, I think that there's a very good chance that I Dwarkesh: live beyond 200 years of age. Dwarkesh: I have not changed anything about my life with regards to that knowledge, right? Dwarkesh: I'm not like, when I'm picking partners, I'm not like, oh, this is the person, Dwarkesh: now that I think I'm going to live for 200, you know, like hundreds of years. Ryan Sean Adams: Yeah.

Dwarkesh: Well, you know, ideally I would pick a partner that would, ideally you pick Dwarkesh: somebody who would be, that would be true regardless.

Dwarkesh: But you see what I'm saying, right? There's like, the fact that I expect my Dwarkesh: personal life, the world around me, the lives of the people I care about, Dwarkesh: humanity in general to be so different has, it just like doesn't emotionally resonate as much as, Dwarkesh: I, my intellectual thoughts and my emotional landscape aren't in the same place. Dwarkesh: I wonder if it's similar for you guys.

Ryan Sean Adams: Yeah, I totally agree. I don't think I've priced that in. Also, Ryan Sean Adams: there's like non-zero chance that Eliezer Yudkowsky is right, Dworkesh. Ryan Sean Adams: Do you know? And so that scenario, I just, I can't bring myself to emotionally price in. Ryan Sean Adams: So I veer towards the optimism side as well. Ryan Sean Adams: Dworkesh, this has been fantastic. Thank you so much for all you do on the podcast.

Ryan Sean Adams: I have to ask a question for our crypto audience as well, which is, Ryan Sean Adams: when are you going to do a crypto podcast on Dwarkech? Dwarkesh: I already did. It was with one Sam Bigman-Fried. Ryan Sean Adams: Oh my God. Dwarkesh: Oh man. Ryan Sean Adams: We got to get you a new guest. We got to get you someone else to revisit the top best. Dwarkesh: Don't look that one up. It's Ben Omen. Don't look that one up.

Dwarkesh: I think in retrospect. You know what? We'll do another one. Ryan Sean Adams: Fantastic. I'll ask you Dwarkesh: Guys for some recommendations. That'd be great. Dwarkech, thank you so much. Dwarkesh: But I've been following your stuff for a while, for I think many years. Dwarkesh: So it's great to finally meet. and this was a lot of fun. Ryan Sean Adams: Appreciate it. It was great. Thanks a lot.