Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment

Human-level AI is deep, deep into an intelligence explosion, things like inventing the transformer or discovering Chinchilla Scalene and doing your training runs more optimally over creating flash attention. That set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion. You have a race between on the one hand, the project of getting strong interpretability and shaping motivations, and on the other hand, these AI's in

ways that you don't perceive make the AI take over happen. We spend more compute by having a larger brain than other animals, and then we have a longer childhood. It's an analog as to like having a bigger model and having more training time with it. It seemed very implausible that we couldn't do better than completely brute force evolution. How quickly are we running through those orders of magnitude? Hey everybody, just wanted to give you a heads up. So I ended up talking to Carl for

like seven or eight hours. So we ended up splitting this episode into two parts. I don't want to put all of that on you at once. In this part, we get deep into Carl's model of an intelligence explosion and what that implies for alignment. The next part, which we'll release next week, is all about the specific mechanisms of an AI takeover. In terms of the depth and the range of interesting topics, this set of episodes is the best I've ever done. So I hope you all enjoy. Here's Carl.

Okay. Today I have the pleasure of speaking with Carl Schulman. Many of my former guests, and this is not an exaggeration. Many of my former guests have told me that a lot of their biggest ideas, perhaps most of their biggest ideas have come directly from Carl, especially when it has to do with the intelligence explosion and its impacts. And so I decided to go directly to the source. And we have Carl today on the podcast. Carl keeps a super

low profile, but he is one of the most interesting intellectuals I've ever encountered. And this is actually his second podcast ever. So we're going to get to get deep into the heart of many of the most important ideas that are circulating right now directly from the source. So and by the way, so Carl is also an advisor to the Open Philanthropy project, which is one of the biggest funders on causes having to do with AI and its risks, not to mention

global health and old being. And he is a research associate at the Future of Humanity Institute at Oxford. So Carl, it's a huge pleasure to have you on the podcast. Thanks for coming. Thank you, Dora Kosh. I've enjoyed seeing some of your episodes recently, and I'm glad

to be on the show. Excellent. Let's talk about AI. Before we get into the details, give me the sort of big picture explanation of the feedback loops in just a general dynamics that would start when you have something that is approaching human level intelligence. Yeah. So I think the way to think about it is we have a process now where humans are developing new computer chips, new software, running larger training runs. And it takes

a lot of work to keep Moore's law jogging while it was. It's slowing down now. And it takes a lot of work to develop things like transformers to develop a lot of the improvements to AI neural networks, their advancing things. And the core method that I think I want to highlight on this podcast, and I think is underappreciated, is the idea of input outputs. So we can look at the increasing difficulty of improving chips. And so sure,

each time you double the performance of computers, it's harder. And as we approach physical limits, eventually, it becomes impossible. But how much harder? So there's a paper called our ideas getting harder to find. There was published a few years ago. How many like 10 years ago at Miri, we did, I mean, I did an early version of this analysis using mainly data from Intel and like the large semiconductor fabricators. Anyway, and so in this paper, they

cover a period where the productivity of computing went up a million fold. So you could get a million times the computing operations per second per dollar, big change, but it got harder. So the amount of investment, the labor force required to make those continuing advancements went up and up and up. And indeed, it went up 18 fold over that period. I know. So some take this to say, oh, diminishing returns, things are just getting harder and harder. And

so that will be the end of progress eventually. However, in a world where AI is doing the work, that doubling of computing performance translates pretty directly to a doubling or better of the effective labor supply. That is, if when we had that million fold compute increased, we used it to run artificial intelligences who would replace human scientists and engineers, then a the 18 X increase in the labor demands of the industry would be trivial.

We were getting more than one doubling of the effective labor supply that we need for each doubling of the labor requirement. And in that data set, it's like over four. So we double, we double compute. Okay, now we need somewhat more researchers, but a lot less than twice as many. And so, okay, we we use up some of those double ends of compute on the increasing difficulty of further research, but most of them are left to expedite the

process. So if you you double your labor force, that's enough to get several double ends of compute. You use you use up one of them on meeting the increased demands from diminishing returns. The others can be used to accelerate the process. So you have your first doubling, takes however many months, your next doubling can take a smaller fraction of that, the next doubling less and so on. At least in so far as this, the outputs you're generating compute

for AI in the story are able to serve the function of necessary inputs. If there are other inputs that you need, eventually those become a bottleneck. And you wind up more restricted on this. Got it. Okay. So yeah, I think the bloom vapor had that 30 there was 35% increase in was it transferred to your cost per flop? And there was a 7% increase per year in the number of researchers required to sustain that pace. So something like this in a, yeah,

it's like four to five, uh, double ends of compute per doubling of labor inputs. I guess there's a lot of questions you can delve into in terms of whether you would expect a similar scale with AI. And whether it makes sense to think of AI as a population of researchers that keeps growing with compute itself. Actually, let's go there. So can you explain the intuition that compute is a good proxy for the number of AI researchers, so to speak?

So far, I've talked about hardware as an initial example, because we had good data about a past period. You can also make improvements on the software side. And we think about an intelligence explosion that can include AI is doing work on making hardware better, making better software, making more hardware. But the basic idea for the hardware is especially simple in that if you have a worker, an AI worker that can substitute for a human, if you

have twice as many computers, you can run two separate instances of them. And then they can do two different jobs, manage two different machines, work on two different design problems. So you can get more gains than just what you would get by having two instances. We get improvements from using some of our compute, not just to run more instances of the existing AI, but to train larger AI's. So there's hardware technology. How much you can get per

dollar you spend on hardware? And there's software technology. And the software can be copied freely. So if you've got the software, it doesn't necessarily make that much to say that, oh, we've got 100 Microsoft Windows. You can make as many copies as you need for whatever Microsoft will charge you. But for hardware is different. It matters how much we actually spend on the hardware at a given price. And if we look at the changes that

have been driving AI recently, that is the thing that is really off-trend. We are spending tremendously more money on computer hardware for training big AI models. Yeah. Okay. So there's the investment in hardware. There's a hardware technology itself. And there's the software progress itself. The AI is getting better because we're spending

more money on it because our hardware itself is getting better over time. And because we're developing better models or better adjustments to those models, where is the loop here? The work involved in designing new hardware and software is being done by people now. They use computer tools to assess them. But like, computer time is not like the primary cost for Nvidia designing chips for TSMC producing them for ASML, making lithography equipment

to serve the TSMC fabs. And even in AI software research, that has become quite compute-intensive. But I think we're still in the range where at a place like DeepMind salaries were still larger than compute for the experiments. So they're tremendously, tremendously more of the expenditures were on compute relative to salaries than in the past. If you take all of the work that's being done by those humans, there's like low tens of thousands of people working at

Nvidia designing GPUs specialized for AI. I think there's more like 70,000 people at TSMC, which is the leading producer of cutting edge chips. There's a lot of additional people at companies like ASML that supply them with the tools they need. And then a company like DeepMind, I think from their public filings, they recently had a thousand people opening, I think,

as a few hundred people, anthropocase less. If you add up things like Facebook AI research, Google Brain, or indeed, you got thousands or tens of thousands of people who are working on AI research. We'd want to zoom in on those who are developing new methods rather than narrow applications. So inventing the transformer definitely counts. Optimizing for some particular businesses data set cleaning, probably not. But so those people are doing this work. They're driving

quite a lot of progress. What we observe and the growth of people relative to the growth of those capabilities is that pretty consistently the capabilities are doubling on a shorter time scale than the people required to do them are doubling. And so there's work. So we talked about hardware and how historically it was pretty dramatic, like four or five double lanes of computer efficiency per doubling of human inputs. I think that's a bit lower now as we get

towards the end of Moore's Law. Although interestingly, not as much lower as you might think, because the growth of inputs has also slowed recently. On the software side, there's some work by Tamey Bessarouglou, and I think collaborators, they may have been thesis. It's called our models getting harder to find. And so it's applying the same sort of analysis of the ideas getting harder to find. And you can look at growth rates of papers from citations, employment at these

companies. And it seems like the doubling time of these like workers driving the software advances is like several years, or at least a couple years, where is the doubling of effective compute from algorithmic progress is faster. So there's a group called epoch. They received grants from open philanthropy. And they do work collecting data sets that are relevant to forecasting AI progress. And so their headline results for what's the rate of progress in hardware and software

and just like growth in budgets are as follows. So for hardware, they're looking at like a doubling of hardware efficiency that's like two years. It's possible it's a bit better than that when you take into account certain specializations for AI workloads. For the growth of budgets, they find a doubling time that's like something like six months in recent years, which is pretty

tremendous relative to the historical rates. We should maybe get into that later. And then on the algorithmic progress side, mainly using using image net type data sets right now, they find a doubling time that's less than one year. And so you combine all of these things and the like the growth of effective compute for training big big AI is it's it's pretty pretty drastic. I think that's all an estimate that GPD4 costs like $50 million around that range to train.

Now suppose that like AGI takes a thousand X that if you were just to scale up GPD4, it might not be that I'm just just for the sake of example. So part of that will come from companies just spending a lot more to train the models and that just greater investment. Part of that will come from them having better models so that what would have taken a 10X increase in the model to get naively you can do with having a better model that you only need to do scale up.

You get the same effect of increasing it by 10X just from having a better model. And so yeah, you can spend more money on it to turn a bigger model. You can just have a better model or you can have chips that are cheaper to train. So you get more compute for the same dollars. And okay, so those are the three you're describing the ways in which the quote-unquote

effect of a compute would increase. From the looking at it right now it looks like yeah, you might get two or three double ends of effective compute for this thing that we're calling software progress, which is which people get by asking well how much less compute can you use now to achieve the same benchmark because you achieved before. There are reasons to not fully identify this with like software progress as you might naively think of it because some of it

can be enabled by the others. So like when you have a lot of compute you can do more experiments and find algorithms that work better. Sometimes the the additional compute you can get higher efficiency

by running a bigger model we were talking about earlier. And so that means you're getting more for each GPU that you have because you made this like larger expenditure and you could that can look like a software improvement because this model it's not a hardware improvement directly because it's doing more with the same hardware but you wouldn't have been able to achieve it

without having a ton of GPUs to do the big training one. The feedback loop itself involves the AI that is the result of this greater effective compute helping you train better AI right or use less effective compute in the future to train better AI. It can help on the hardware design. So like Nvidia is a fabulous chip design company they don't make their own chips. They send files of

instructions to TSMC which then fabricates the chips in their own facilities. And so the work of those 10,000 plus people if you could automate that and have the equivalent of a million people doing that work then I think you would pretty quickly get the kind of improvements that can be achieved with the existing nodes that TSMC is operating on. You could get a lot of those chip design gains basically like doing the job of improving chip design that those people are working on now

but get it done faster. So that's one thing. I think that's less important for the intelligence explosion. The reason being that when you make an improvement to chip design it only applies to the chips you make after that. If you make an improvement in AI software it has the potential to be immediately applied to all of the GPUs that you already have. Yeah and so the thing that I think is most disruptive and most important has the leading edge of the change from AI automation

of the inputs to AI is on the software side. At what point would it get to the point where the AI's are helping develop better software or better models for future AI's. Some people claim today for example that programmers at OpenAI are using co-pilot to write programs now so in some sense you're already having that sort of feedback loop. I'm a little skeptical of that

as a mechanism. At what point would it be the case that the AI is contributing significantly in the sense that would almost be the equivalent of having additional researchers to AI progress in software? The quantitative magnitude of the help is absolutely central. So there are plenty of companies that like make some product that like very slightly boost productivity. So when Xerox makes fax machines it may be increases people's productivity in office work by

0.1% or something. You're not going to have explosive growth out of that because okay now 0.1% more effective R&D at Xerox and any customers by the machines. Not that important. So I think the thing to look for is when is it the case that the contributions from AI are starting to become as large or larger as the contributions from humans. So when this is

boosting their effective productivity by 50 or 100%. If you then go from eight months doubling time say for effective compute from software innovations things like inventing the transformer or discovering chinchilla scaling and doing your training runs more optimally or creating flash attention. Yeah if you move that from say eight months to four months and then the next time you apply that it significantly increases the boost you're getting

from the AI. So I mean maybe instead of giving a 50% or 100% productivity boost that's more like a 200% and so it doesn't have to have been able to automate everything involved in the process of AI research it can be it's automated a bunch of things and then those are being done in extreme

profusion because any I think I think that AI can do you have it done much more often because it's so cheap and so it's not a threshold of this is human level AI it can do everything a human can do with no weaknesses in any area it's that even with its weaknesses it's able to bump up the

performance so that instead instead of getting like the results we would have with the the 10,000 people working on finding these innovations we get the results that we would have if we had twice as many of those people with the same kind of skill distribution and so that's a it's like

a demanding challenge it's like you need quite a quite a lot of capability for that but it's also important that it's significantly less than this is a system where there's no way you can point at it and say in any respect it is weaker than a human a system that was just as good as a human in

every respect but also had all the advantages of an AI that is just way beyond this point like if you consider that there's like the the output of our existing fabs make tens of millions of advanced GPUs per year those GPUs if they were running sort of AI software that was as efficient as humans as a sample efficient it doesn't have any major weaknesses so they can work four times as long you know the 168 hour work week they can have much more education than any human so it's you know

like the oh human you know they got a PhD you know it's like yeah wow it's like 20 years of education maybe longer if they take if they take a slow slow route on the PhD it's just normal for us to train large models by eat the internet eat all the published books ever read everything on GitHub

and get good at predicting it so like the level of education vastly beyond any human the degree to which the models are focused on task is higher than all but like the most motivated humans when they're really really gunning for it so you combine the things tens of millions of GPUs each GPU

is doing the work of the very best humans in the world and like the most capable humans in the world can command salaries that are a lot higher than the average and particularly in a field like stem or nearly AI like there's no human in the world who has a thousand years of

experience with TensorFlow or let alone the new AI technology that were invented the year the year before but if if they were around yeah they'd be paid millions of dollars a year and so when you consider this okay tens of millions of GPUs each is doing the work of maybe 40 maybe more of these kind of existing workers this is like going from a workforce of tens of thousands to hundreds of millions you immediately make all kinds of discoveries then you immediately develop all sorts

of tremendous technologies so human level AI is deep deep into an intelligence explosion intelligence explosion has to start with something weaker than that yeah well what is the thing it starts with and how close are we to that because if you think of a researcher at OpenAI or something you know

these are to be a researcher is not just completing the hello world prompt that co-pilot does right it's like you got to choose a new idea you had to figure out the right way to approach it you perhaps have to manage the people who are also working with you on that problem it you know it's like it's it's incredibly complicated skill portfolio skills rather than just a single skill so yeah well what what what is the point I wish that feedback loop starts where you can even you're not just

doing the point five percent increase in productivity that a sort of AI tool might do but is actually the equivalent of a researcher or close to it like what is that point so I think maybe a way look at it is to give some illustrative examples of like the kinds of capabilities that you might see

and so because these systems have to be a lot weaker than this sort of human level things what we'll have is intense application of the ways in which AI's have advantages partly offsetting their weaknesses and so AI's are cheap we can call a lot of them to do many small problems and so you'll have situations where you have dumber AI's that are deployed thousands of times to equal say one human worker and they'll be doing things like these voting algorithms where you with an LLM you generate

a bunch of different responses and take a majority vote among them that improves performance sum you'll have things like the alpha go kind of approach where you use the neural net to do search and you go deeper with the search by plowing in more compute which helps to offset the

inefficiency and weaknesses of the model on its own you'll do thing that would just be totally impractical for humans because of the sheer number of steps and so an example of that would be designing synthetic training data so humans do not learn by just going into the library and opening

books at random pages it's actually much much more efficient to have things like schools and classes where they teach you things in an order that makes sense that's focusing on the skills that are more valuable to learn they give you testing exam they're designed to try and elicit the skill

they're actually trying to teach and right now we don't bother with that because we can Hoover up more data from the internet we're getting towards the end of that but yeah has the AI is get more sophisticated they'll be better able to tell what is a useful kind of skill to practice

and to generate that and we've done that in other areas so alpha go the original version of alpha go was booted up with data from human go play and then improved with reinforcement learning and money carloch research but then alpha zero but they somewhat more sophisticated model benefited

from some some other improvements but was able to go from scratch and it generated its own data through self play so with getting data of a higher quality than the human data because there are no human players that good available in the data set and also a curriculum so that at any given

point it was playing games against an opponent of equal skill itself and so it was always in an area when it was easy to learn if you're if you're just always losing no matter what you do or always winning no matter what you do it's hard to distinguish which things are better and which are worse

and when we have somewhat more sophisticated AI's that can generate training data and tasks for themselves for example if the AI can generate a lot of unit tests and then can try and produce programs that pass those unit tests then the interpreter is providing a training signal and the AI

can get good at figuring out what's the kind of programming problem that is hard for AI's right now that will develop more of the skills that I need and then do them and now you're not going to have you know employees at open AI write like a billion programming problems that's just not going

to happen but you are going to have AI's given the task of producing the enormous number of programming challenges in elements themselves is you know there's a paper out of Anthropic called Constitution AI or Constitution RL where they basically have the program just like talk to

itself and say like is this response helpful if not how can I make this more helpful and the their response is improved and then you train the model on the more helpful responses that it generates by talking to itself so that it generates and natively and you could imagine you know

more sophisticated way to do that or better ways to do that okay so but then the question is listen you know GPT-4 already costs like 50 million or 100 million or whatever it was even if we have greater effective compute from hardware increases and better models

it's hard to imagine how we could sustain like four or five more orders of magnitude greater size effective size than GPT-4 unless we're jumping in like $12 million like the entire the entire economies of big countries into training the next version so the question is do we

get something that can significantly help with AI progress before we run out of the the sheer money and scale and compute that would require a train it do you have a take on that well first I'd say remember that there are these three contributing to lens so the new H100s are

significantly better than the A100s and a lot of companies are actually waiting for their deliveries of H100s to do even bigger training runs along with the work of hugging them up into clusters and engineering the thing yeah so all of those factors are

contributing and of course mathematically yeah if you do four orders of magnitude more than 50 or 100 million then you're getting to really an early territory and yeah I think the the way to look at it is at each step along the way does it look like it makes sense to do the next step

and so from where we are right now seeing the results with GPT-4 and Chattcheap-PT companies like Google and Microsoft and whatnot are pretty convinced that this is very valuable you have like talk at Google and Microsoft with Bing that well it's like billion dollar

billion dollar matter to change market share and search by a percentage point and so that can fund a lot and on the far end on the extreme if you automate human labor we have a hundred trillion dollar economy most of that economy is paid out in wages so like between 50 and 70 trillion dollars

per year if you create a GI it's going to automate all of that and keep increasing beyond that so the value of the completed project is very much worth throwing our whole economy into it if you're going to get the good version not the catastrophic destruction of the human race

or you know some some other disastrous outcome and in between it's a question of well the next step how risky and uncertain is it and how much growth in the revenue you can generate with it do you get and so for moving up to a billion dollars I think that's absolutely going to

happen these large tech companies have R&D budgets tens of billions of dollars and when you think about it like in the relevant sense like all the all the employees at Microsoft who are doing software engineering that's like contributing to creating software objects it's not it's not

weird to spend tens of billions of dollars on a product that would do so much and I think it's becoming more clear that there is sort of market opportunity to fund the thing going up to a hundred billion dollars that's like okay the existing R&D budgets spread over multiple years

but if you if you keep seeing that when you scale up the model it substantially improves the performance it opens up new applications you're not just improving your search but maybe you know it makes self-driving cars work you replace bulk software engineering jobs or if not replace them

amplify productivity in this kind of dynamic you actually probably want to employ all the software engineers you can get how long are they able to make any contribution because the returns of improving stuff in AI itself get so high but yeah so I think that can go up to a hundred billion

and a hundred billion you're using like a significant fraction of our existing fab capacity like right now the revenue of Nvidia is like 25 billion the revenue of TSMC I believe it's like it's over 50 billion last I checked in 2021 Nvidia was maybe seven and a half percent less than

10 percent of TSMC revenue so there's a lot of room and most of that was not AI chips they have a large gaming segment there are data center GPUs that are used for video and the like so there's room for more than an order of magnitude increase by redirecting existing fabs to produce

more AI chips and then just actually using the AI chips that these companies have in their cloud for the big training runs and so I think that that's enough to go to the 10 billion and then combine with stuff like the H100 go up to 100 billion just to emphasize for the audience the initial point

about revenue made if it cost opening I a hundred million dollars to train GPD4 and it generates 500 million dollars in revenue you know you pay back your expenses with a hundred million you have 400 million for your next training run then you train your GPD 4.5 you know you get let's say

four billion dollars out of revenue out of that that's where the feedback group of sort of revenue comes from where you're automating tasks and therefore you're making money you can use that money to automate more tasks on the ability to redirect the fab production towards AI chips so then the

TLDR on you want 100 billion dollars worth of compute I mean fabs take what like a decade or so to build so given the ones we have now and the ones that are going to come online in the next decade is there enough to sustain a hundred billion dollars of GPU compute if you wanted to spend that on a training run yes you definitely make the hundred billion one how do you go up to a trillion dollar run and larger it's going to involve more fab construction and yeah fabs can take a long a long time

to build on the other hand if in fact you're getting very high revenue from the AI systems and you're actually bottlenecked on the construction of these fabs then their price could skyrocket and that lead to measures we've never seen before to expand and accelerate fab production like if you

consider to add the limit as you're getting models that approach human-like capability to imagine things that are getting close to like brain-like efficiencies plus AI advantages we were talking before about well a GPU that is supporting an AI really it's a cluster of GPU

supporting AI is that do things in parallel data parallelism but if that can work four times as much as a human a highly skilled motivated focused human with levels of education that have never been seen in the human population and so if like a typical software engineer can earn hundreds

of thousands of dollars the world's best software engineers can earn millions of dollars today and maybe more in a world where there's so much demand for AI and then times four for working all the time well I mean if you have if you can generate like close to ten million dollars a year

out of the future version of h100 and it costs tens of thousands of dollars with a huge profit margin now and profit margin could be could be reduced with like large production that is a big difference that that chip pays for itself almost instantly and so you could you

could support paying ten times as much to have these fabs constructed more rapidly you could have if AI is starting to be able to contribute you can have could have AI contributing more of the skill technical work that makes it hard for say Nvidia to suddenly find thousands upon thousands

of top quality engineering hires if AI can provide that now if AI hasn't reached that level of performance then this is how you can have things to all out and like a world where AI progress still is out is one where you go to the hundred billion and then over succeeding years trillion dollar

things software progress turn turns to stall turns out to to stall you lose the gains that you are getting from moving researchers from other fields lots of physicists and people from other areas of computer science have been going to AI but you've sort of tap out those resources

has it AI becomes a larger proportion of the research field and like okay you've put in all of these inputs but they just haven't yielded a giant I think that set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion but if it doesn't

after we've exhausted this current scale up of like increasing the share of our economy that is trying to make AI if that's not enough then after that you have to wait for the slow grind of things like general economic growth population growth and such and so things slow

and that results in my credences and this kind of advanced AI happening to be relatively concentrated like over the next 10 years compared to the rest of the century because we just can't we can't keep going with this rapid redirection of resources into AI that's that's a one time thing

if the current scale up works it's going to happen we're going to get to age I really fast like within the next 10 years or something if the current scale of doesn't work all we're left with is just like economy growing like 2% in years we have like 2% a year more resources to spend

on AI and at that scale you're talking about decades before you can just your share or brute force you can you know train that $10 trillion model or something let's talk about why you have your thesis that the current scale up would work well what what is the evidence from AI itself or maybe

from primate evolution and the evolution of other animals just give me the whole the whole consequence of reasons that maybe I think maybe the best way to look look at that might be to consider when I first became interested in this area so in the 2000s which was before the deep learning

revolution how would I think about timelines how did I think about timelines and then how have I updated based on what has been happening with deep learning and so back then I would have said we know the brain is a physical object and information processing device it works

it's possible and not only is it possible it was created by evolution on earth and so that gives us something of an upper bound and that this kind of brute force was sufficient there are some complexities with like well what if it was a freak accident and you know that didn't happen

on all of the other planets and that added some value I have a paper with Nick Buster on this I think basically that's not that important issue there's converging evolution like octopi are also quite sophisticated if the if a special event was at the level of forming cells at all

or forming brains at all we get to skip that because we're choosing to build computers and we already exist we have that that advantage so say evolution gives something of an upper bound really intensive massive brute force search and things like evolutionary algorithms can produce

intelligence doesn't the fact that octopi and I guess other mammals they got to the point of being like pretty intelligent but not human level intelligent is that some evidence that there's a hard step between a cephalopod and a human yeah so that that would be a place to look yeah

it's done the same particularly compelling one source of evidence on that is work by Herculano Hutzl I hope I haven't mispronounced her name but she's a neuroscientist who has dissolved the brains of many creatures and by counting the nuclei she's able to

determine how many neurons are present in different species and find a lot of interesting trends in scaling laws and just a paper discussing the human brain has a scaled up primate brain and across like a wide variety of animals and mammals in particular there are certain

characteristic changes in the relative number of neuron size of different brain regions have things scale up there's a lot of yeah there's a lot of structural structural similarity there and you can explain a lot of what is different about us with a pretty brute force story

which is that you expend resources on having a bigger brain keeping it in good order giving it time to learn so we have an unusually long childhood unusually long neonus period we spend more compute by having a larger brain than other animals three more than three times as large as

chimpanzees and then we have a longer childhood than than chimpanzees and much more than many many other creatures so we're spending more compute in a way the synhylocase to like having a bigger model and having more training time with it and given that we see with our AI models this

sort of like large consistent benefits from increasing compute spent in those ways and with qualitatively new capabilities showing up over and over again particularly in areas that sort of AI skeptics call out in my experience like over the last 15 years the things that people call out

as like ah but the AI can't do that and it's because of a fundamental limitation gone through a lot of them you know there were Winnigrad schemas catastrophic forgetting quite a number and yeah they have repeatedly gone away through scaling and so there's a there's a picture

that we're we're seeing supported from biology and from our experience with AI where you can explain like yeah in general there are trade-offs where the extra fitness you get from a brain is not worth it and so creatures wind up mostly with small brains because they can save that biological

energy and that time to reproduce for digestion and so on in humans we actually seem to have wound up in a anish within self-free enforcing where we greatly increase the returns to having large brains and language and technology are the sort of obvious candidates when you have humans around

you who know a lot of things and they can teach you and compared to almost any other species we have vastly more instruction from parents and the society of the young and then you're getting way more from your brain because you can get per minute you can learn a lot more useful skills and then

you can provide the energy you need to feed that brain by hunting and gathering by having fire that makes digestion easier and basically how this process goes on it's increasing the marginal increase in reproductive fitness you get from allocating more resources along a bunch of dimensions

towards cognitive ability and so at bigger brains longer childhood having our attention be more unlearning so humans play a lot and we we keep playing his adults which is a very weird thing compared to other animals we're more motivated to copy other humans around us than like even

than the other primates and so these are sort of motivational changes that keep us using more of our attention and effort on learning which pays off more when you have a bigger brain and a longer lifespan in which to learn many many creatures are subject to lots of predation or disease and so

if you try you know you're you're a mayfly or a mouse if you try and invest in like a giant brain and a very long childhood you're quite likely to be killed by some predator or some disease before you're able to actually use it and so that means you actually have exponentially increasing

costs in a given niche so if I have a 50% chance of dying every few months of the you know a little mammal or a little lizard or something that means the cost of going from three months to 30 months of learning and childhood development it's not 10 times the loss it's now it's two to the negative

10 so one factor of 1,024 reduction in the benefit I get from what I ultimately learn because 99.9% of the animals will have been killed before that point we're in a niche where we're like a large long-lived animal with language and technology so where we can learn a lot from our groups and that means it pays off to really just expand our investment on these multiple fronts

in intelligence. That's so interesting just for the audience the the calculation about like two to the whatever months is just like you have a half chance of dying this month a half chance of dying next month you multiply those together okay there's other species though that do live in flocks or

as packs where you could imagine I mean they do have like a smaller version of the development of cubs into that are like play with each other why isn't this a hill on which they could have climbed to human level intelligence themselves if it's something like language or technology humans

were getting smarter before we got language I mean obviously we had to get smarter to get language right we couldn't just get language without becoming smarter so yeah we're we're it it seems like there should be other species that should have beginnings of this sort of cognitive

revolution especially given how valuable it is given listen we've dominated the a world you would think there'd be selective pressure for it evolution doesn't have foresight yeah the thing in this generation that gets more surviving offspring and grandchildren that's the thing that

becomes more common evolution doesn't look ahead and they oh in a million years you'll have a lot of descendants it's what what survives and reproduces now and so in fact there are there are correlations where social animals do an average have larger brains and part of that is probably

that the additional social applications of brains like keeping track of which of your group members have helped you before so that you can reciprocate you scratch my back I'll scratch yours remembering who's dangerous within the group that sort of thing is an additional application of intelligence

and so there's some correlation there but what it what it seems like is that yeah in most of these cases it's enough to invest more but not invest to the point where a mind can easily develop language and technology and pass it on and so there are you see bits of tool use in some other primates

who have an advantage that so compared to say the whales who have they have quite large brains partly because they are so large themselves and they have some other other things but they don't have hands which means that reduces a bunch of ways in which brains can pay off and investments

in the functioning of that brain but yeah so primates will use sticks to extract termites caprician monkeys will open in clams by smashing them with a rock so there's bits of bits of tool use but what they don't have is the ability to sustain culture a particular primate will maybe discover

one of these tactics and maybe it'll be copied by their immediate group but they're not holding on to it that well they like well when they see the other animal do it they can copy it in that situation they don't actively teach each other their population locally is quite small so it's easy

to forget things easy to lose information and in fact they they remain technologically stagnant for hundreds of thousands of years and we can actually look at some human situations so there's an old paper I believe by the economist Michael Kramer talks about technological growth in the

different continents for human societies and so you have Eurasia is the largest integrated connected area Africa is partly connected to it but the Sahara Desert restricts the flow of information and technology and such and then you have the Americas which were after the colonization

from the land bridge were largely separated and are smaller than Eurasia then Australia and then you had like smaller island situations like Tasmania and so technological progress seemed to have been faster the larger the connected group of people and in the smallest groups so like in Tasmania

you had a relatively small population and they actually lost technology so things like they've lost some like fishing techniques and if you have a small population and you have some limited number of people who know a skill and they happen to die or happen there's like you know some

some change in circumstances that causes people not to practice or pass on that thing and then you lose it and if you have a few people you're doing less innovation the rate at which you lose technologies to some kind of local disturbance and the rate at which you create new technologies

can wind up in balance and the great change of hominids and humanity if that we wound up in the situation we were accumulating faster than we were losing and how's we accumulated those technologies allowed us to expand our population they created additional demand for intelligence so that our

brains became three times as large as that Japan's easier question pencils yeah and our ancestors who had a similar brain source okay and then the crucial point I guess in relevance to AI is that the selective pressures against intelligence in other animals are not acting against these

neural networks because we are you know we're not going to get like eaten by a predator if they spend too much time becoming more intelligent we're like explicitly trading them to become more intelligent so we have like good first principles reason to think that if it was scaling that made our

minds this powerful and if the things that prevented other animals from scaling are not impinging on these neural networks the these things should just continue to become very smart yeah we are growing them in a technological culture where there are jobs like software engineer that depend much

more on sort of cognitive output and less on things like metabolic resources devoted to the immune system or to like building big muscles to throw spears yeah this is kind of a side note but I'm just kind of interested I think you reference to some point gentile scaling for the audience this

is a paper from deep mind which describes if you have a model of a certain size what is the optimum amount of data that it should be trained on so you can imagine bigger models you can you can use more data to train them and in this way you can figure out where should you spend your computer

do you spend it on making the model bigger or should you spend it on training it for longer I'm curious if in the case of different animals in some students are like model sizes they're how big their brain is and they're training data sizes like how long their clubs or how long their

infants or toddlers are before their full adults is there is there some sort of like scaling law of yeah I mean so the the the chinchilla scaling isn't interesting because the we were talking earlier about the cost function for having a longer childhood and so where it's like exponentially

increasing in the amount of training compute you have when you have exogenous forces that can kill you whereas when we do big training runs the cost of throwing in more GPUs is almost linear and it's much better to be linear than exponentially decay oh that's a really good plan as you expand

resources and so chinchilla scaling would suggest that like yeah the opt for a brain of sort of human size and we optimal to have many millions of years of education but obviously that's in practical because of exogenous mortality for humans and so there's a fairly compelling argument that relative

to the situation where we would train AI that animals are systematically way under trained they're more efficient than our models we still have room to improve our algorithms to catch up with the efficiency of brains but they are laboring under that disadvantage yeah that is so interesting okay so I guess another question you could have is humans got started on this evolutionary hill climbing route where we're getting more intelligent than has more benefits for us why didn't we go all the way

on that route if intelligence is so powerful why aren't all humans as smart as we know humans can be right at least that's smart if intelligence is so powerful like why hasn't there been stronger selective pressure I understand like oh listen hip size you can't like give birth to a really big

headed baby or whatever but you would think like evolution to figure out some way to offset that such big if intelligence has such big power and such is so useful yeah I think you actually look at it quantitatively that's that's not true and even in in sort of recent history there has been

it looks like a pretty close balance between the costs and the benefits of having more cognitive abilities and so you say like you know who needs to worry about like you know the metabolic costs like you need humans put like order 20% of our metabolic energy into the brain and it's higher

for like young children so 20% of the I mean along and then you know there's like breathing and digestion and the immune system and so from most of history people have been dying left and right like a very large proportion of people will die of infectious disease and if you put more resources

into your immune system you survive that is like life or death pretty directly via that mechanism and then this is this related also people die more of disease during famine and so there's boom or bust and so if you have 20% less metabolic requirements or has anger a child and

it's you have a lot more I mean it's like 40 or 50% less metabolic requirements you're much more likely to survive that famine so these are these are pretty big and then there's a trade off about just cleaning mutational load so every generation new mutations and errors happen in the

process of reproduction and so like we know there are many genetic abnormalities that occur through new mutations each generation and in fact down syndrome is the chromosomal abnormality that you can survive all the others just kill the embryo and so we never see them but like down syndrome

occurs a lot and there are many other lethal mutations and there's as you go to the less damaging ones there are enormous numbers of less damaging mutations that are degrading every system in the body and so evolution each generation has to pull away at some of this mutational load and

the priority with which that mutational load is pulled out scales and proportioned how much the traits it's affecting impact fitness so you got new mutations that impact your resistance to malaria yep you got new mutations that damage brain function and then have those mutations are purged each generation if malaria is a bigger difference in mortality than like the incremental effectiveness that's a hunter gatherer you get from you know being slightly more intelligent than you'll purge that

mutational load first and similarly if there's like you know humans have been vigorously adapting to new circumstances so since agriculture people have been developing things like the ability to you know

have amulets to digest breads the ability to like digest milk and if you're evolving for all of these things and if some of the thing that give an advantage for that incidentally carry along nearby them some negative effect on another trait then that other trait can be damaged so it really matters

how important to survival and reproduction cognitive abilities were compared to everything else that organism has to do and that in particular like surviving feast and famine having like the physical abilities to do hunting and hunting and gathering and like even if you're like very good at

planning your hunting being able to throw a spear harder can you know be a big difference and that needs energy to build those muscles and then to sustain them and so given all these all these factors it's like yeah it's it's not a slam dunk to invest at the merge and like today like having bigger

brains for example is it's associated with like greater cognitive ability but it's like it's modest large scale preregister studies preregistered studies with MRI data you know it's like you know in a range maybe like a correlation of 0.25 0.3 and the standard deviation of brain size is like

10 percent so if you double the size of the brain so go and the existing brain cost like 20% of metabolic energy go up to 40 percent okay it's like eight standard deviations a brain size if the correlation is like so it's 0.25 then yeah like you can you get gained from that eight standard

deviations of brain size to standard deviations of cognitive ability and like in our modern society where cognitive ability is very rewarded and like you know finishing finishing school becoming an engineer or a doctor or whatever can can pay off a lot financially still the like the average

observed return in like income is like a one or two percent proportional increase there's there's more effects of the tail there's more effect in professions like stem but on the whole it's not like you know if it was like a 5% increase or a 10% increase then you could you could you could tell a

story where yeah this is hugely increasing the amount of food you could have you could support more children but it's like it's a modest effect and the metabolic cost would be large and then throw in these other these other aspects and I think it's you can double story and also we can

just we can see there was not very strong rapid directional selection on the thing which there would be if like you know you could by solving like a by solving like a math puzzle you could defeat malaria like then then there would be more evolutionary pressure that is so interesting and I'm not

to mention of course that yeah if you had like two x the brain size you're you without c-section you would like your your mother would or both would die that on this is a question of actually being curious about for like over a year and I like look briefly try to look up an answer this is

I know this is off topic but I apologize to the audience but I was super interested in those like that was like the most comprehensive an interesting answer I could hope for okay so yeah we have a good explanation of good first principles evolution or reason for thinking that

intelligence scaling up to humans is not implausible despite throwing more scale at it I would also add this was something that would have mattered to me more in the 2000s we also have the brain right here with us for available for neuroscience to reverse engineer

it's property and so in the 2000s when I said yeah I expect this by you know middle of the century ish that was a backstop if we found it absurdly difficult to go to the algorithms and then we would learn from neuroscience but in the actual the actual history it's it's really not like that we

develop things in AI and then also we can say oh yeah this is sort of like this thing in neuroscience or maybe this is a good explanation but it's not the neuroscience is driving AI progress it turns out not to be that necessary similar to I guess you know how planes were inspired by the

existence proof of birds but jet jet engines don't flap all right so yeah scaling good reason things scaling might work so we we spent a hundred billion dollars and we have something that is like human level or can do help significantly with AI research I mean that that might be on the

earlier end but I mean I would I definitely would not rule that out given the rates of change we've seen with the last few skill ups all right so at this point somebody might be skeptical okay like listen we already have a bunch of human researchers right like the incremental researcher how how

powerful is that and then you might say well no this is like thousands of researchers I don't know how to express a skepticism exactly but skepticism is a skeptical of just generally the effect of scaling up the number of people working on the problem to rapid rapid progress on that problem

somebody might think okay listen with humans the reason population working on a problem is such a good proxy for progress on the problem is that there's already so much variation that is accounted for when you say there's like a million people working on a problem you know there's like a a couple hundreds of super geniuses working on it thousands of people who are like very smart working on it whereas within the eye all the copies are like the same level of intelligence and if it's not

super genius intelligence the the the total quantity you might not matter as much yeah I'm not sure what your model is here so is this the model that the diminishing returns kickoff suddenly has a cliff right where we are and so like there was there were results in the past from throwing more

people at problems and I mean this has been useful in historical prediction one of the there's this idea of experienced curves and rights law basically measuring cumulative production in a field or which is that also going to be a measure of like the scale of effort and investment and people

have used this correctly to argue that renewable energy technology like solar would be falling rapidly in price because it was going from a low base of very small production runs not much investment in doing it efficiently and yeah climate advocates correctly called out

people and people like David Roberts the futurist Rama has now actually has some some interesting writing on this that yeah correctly called out that there would be really drastic fall and prices of solar and batteries because of the increasing investment going into that the human genome project

would be another so I'd say there's like yeah real real evidence these observed correlations from like ideas getting harder to find have have held over a fair a fair range of data and over quite a lot of time so I'm wondering what what what's the yeah the the nature of the deviation

you're thinking of that we're talking about maybe this is like a good way to describe what happens when more humans enter a field but does it even make sense to say like a greater population of AI is doing AI research if there's like more GPUs running a copy of GPT 6 doing AI research

mid it just like how how applicable are these economic models of human the quantity of humans working on a problem to the to the magnitude of AI is working on a problem yeah so if you have AI that are directly automating particular jobs that humans were doing before then we say well

with additional compute we can run more copies of them to do more of those tasks simultaneously we can also run them at greater speed and so some people have an intuition that like well you know what matters is like time it's not how many people working on problem at a given point I think

that doesn't bear out super well but AI can also be run faster than humans and so if you have a set of AI's that can do the work of the individual human researchers and run at 10 times or a hundred times the speed and we ask well could the human research community have solved the

algorithm problems do things like invent transformers over 100 years if we have this we have AI's with a population effective population similar to the humans but running a hundred times as fast and so you have to tell a story where no the AI they can't really do the same things as the humans

and we're talking about what happens when the AI is more capable of in fact doing that although they become more capable as lesser capable versions of themselves help us make big themselves more capable right so you have to like kickstart that at some point is there an example in

analogous situations is intelligence unique in the sense that you have a feedback group of with a learning curve or something else a system outputs are feeding into its own inputs in a way that because they if we're talking about something like Moore's Law or the cost of solar you do have

this way like we're you know more people are we're throwing more people at the problem and it's we're you know we're making a lot of progress but we don't have this sort of additional part of the model where Moore's Law leads to more humans somehow and the more humans are becoming researchers

so you do actually have a version of that in the case of solar so you have a small infant industry that's doing things like providing solar panels for space satellites and then getting increasing amounts of subsidized government demand because of you know worries about fossil

fuel depletion and then climate change you you can't have the dynamic where visible successes with solar or like lowering prices then open up new markets so there's a particularly huge transition where renewables become cheap enough to replace large chunks of the electric grid

uh early area you're like you're dealing with very niche situations like yeah so the satellites where you have very difficult to refuel a satellite in place and then remote areas and then moving to like you know the super the sunniest areas in the world with the biggest solar subsidies

and so there was an element of that where more and more investment has been thrown into the field and like the market has rapidly expanded as the technology improved but I think the the closest analogy is actually the long run growth of human civilization itself and I know you had

holding Karnovsky from the open philanthropy project on earlier and discuss some of this research about the long run acceleration of human population and economic growth and so developing new technologies allowed human population to expand humans to occupy new habitats and new areas

and then to invent agriculture which support the larger populations and then even more advanced agriculture in the modern industrial society and so their total technology and output allowed you to support more humans who then would discover more technology and continue the process now that

was boosted because on on top of expanding the population the sheer of human activity that was going into invention and innovation went up and that was a key part of the industrial revolution there was no such thing as a corporate research lab or like an engineering university

prior to that and so you were both increasing the total human population and the share of it going in but this population dynamic is pretty is pretty analogous humans invent farming they can have more humans than they can invent industry and so on so maybe somebody would be skeptical that

with AF progress specifically it's not just a matter of you know some like some farmer figure our crop rotation or some blacksmith figuring out how to do metallurgy better you in fact even to make the for the 50% improvement in productivity you basically need something on the IQ that's close

to Ilya Setskipper so there's like a discontinuous you're like contributing very little to productivity and then you're like Ilya and then you contribute a lot but the becoming Ilya is you see what I'm saying there's not like gradual increase in capabilities at least to the few other. You're imagining a case where the distribution of tasks is such that there's nothing that you can where individually automated it particularly helps and so the ability to contribute to AI

research is really end loaded is that what you saying? Yeah I mean we already see this in in these sorts of like really high IQ companies or projects where theoretically I guess Jane Street or open AI could hire like a bunch of you know mediocre people to do there's a

comparative advantage they could do some a menial task and that could free up the time of the really smart people but they don't do that right transaction costs whatever else self-driven cars would be another example where you have a very high quality threshold and so when you your

performance as a driver is worse than a human like you have 10 times the accident rate or 100 times the accident rate then the cost of insurance for that which is a proxy for people's willingness to ride the car and stuff too would be such that the insurance cost would absolutely dominate so

even if you have zero labor cost it's offset by the increased insurance cost and so there are lots of cases like that where like partial automation is not in practice very usable because complementing other resources you're gonna use those other resources less efficiently

in a post-AGI future and the same thing can apply to humans so people can say well comparative advantage you know even if AI's can do everything better than a human well it's still worth something a human can do something you know we can lift a box that's something now there's a question

of property rights if well if it could just make more robots but even even absent that in such an economy you wouldn't want to let a human worker into any industrial environment because you know in a clean room they'll be emitting all kinds of skin cells and messing things things up you need

to have an atmosphere there you need a bunch of supporting tools and resources and materials and those supporting resources and materials will do a lot more productively working with AI and robots rather than a human so you don't want to let a human anywhere near the thing just like you know

in a you don't you don't want to have a gorilla wondering around in a china shop even if you've trained it to most of the time pick up a box for you if you give it a banana it's just not worth it to have it wondering around your china shop yeah yeah like why is that not a good objection to

I mean I think that that is one of the the ways in which partial automation can fail to really translate into a lot of economic value that's something that will attenuate as we go on and as the AI is more able to work independently and more able to handle its own its own

scrubs get more more reliable but the way in which it becomes more reliable is by AI progress speeding up which happens if AI can contribute to it but if there is the if there is some sort of reliability bottleneing like the forensic from contributing to the progress then you don't have the

loop right so yeah I mean this is this is why we're not there yet right but then what is the reason to think we'll be there at the broad reason is we have these inputs are scaling up there's a so epoch which I mentioned earlier they have a paper I think it's called compute trends

in three areas of machine learning or something like that and so they look at the compute expanded on machine learning systems since the founding of the field of AI the beginning of the 1950s and so it mostly it grows with Moore's law and so people are spending a similar amount

on their experiments but they can just buy more without because the computer is coming and so that data I mean it covers over 20 orders of magnitude maybe like 24 and of all of those increases since 1952 a little more than half of them happened between 1952 and 2010 and all the rest

is since 2010 so we're we've been scaling that up like four times as fast as with the case for most of the history of AI we're running through the orders of magnitude of possible resource inputs you could need for AI much much more quickly than we were for most of the history of AI that's

why this is a period of like with a very elevated chance of AI per year because we're moving through so much of the space of inputs per year and indeed it looks like this scale up taken take into its conclusion we'll cover another bunch of orders of magnitude and that's actually a large fraction

of those that are left before you start running into saying well this is going to have to be like evolution with the sort of simple hacks we get to apply like we're selecting for intelligence the whole time we're not going to make do the same mutation that causes fatal fatal childhood

cancer a billion times even though I mean we we keep getting the same fatal mutations even though I've been done many times we use gradient descent which takes into account the derivative of improvement on the loss all throughout the network and we we don't throw away all the contents of

the network with each generation where you compress down to little DNA so there's that that bar of like well if you're going to do brute force like evolution combine with these sort of very simple ways we can save orders of magnitude on that we're going to cover I think a fraction that's like

half of that distance in this scale up over the next 10 years or so and so if you started off with a kind of vague uniform prior you're like well it's probably you probably can't make a GI with like the amount of compute that would be involved in a fruit fly existing for a minute which would be

the early days of AI you know maybe you maybe you'd get lucky we were able to make calculators because calculators benefited from like very reliable seriously fast computers and where we could take a tiny tiny tiny tiny fraction of a human brain's compute and use it for a calculator we

couldn't take an ants brain and rewire it to calculate it's hard hard to manage ant farms let alone get them to do arithmetic for you and so there were some things where we could exploit the differences between biological brains and computers to do stuff super efficiently on computers we would doubt

that we would be able to do so much better than biology that we could with a tiny fraction of an insects brain we'd be able to get AI early on on the far end it seemed very implausible that we couldn't do better than completely brute force evolution and so in between you have some number

of orders of magnitude of inputs where it might be and like in the 2000s I would say well you know I'm gonna have a pretty uniformish prior I'm gonna put weight on it happening at like the sort of the equivalent of like 10 to the 25 ops 10 to the 30 10 to the 35 and sort of spreading out over

that and then I can update on other information and in the short term I would say like in 2005 I would say well I don't see anything that looks like the cusp of aji so I'm also gonna lower my credence for like the next five years or the next 10 years and so that that would be kind of like

a big prior and then when we take into account like well how quickly are we running through those orders of magnitude if I have my uniform prior I assign half of my weight to the first half of remaining orders of magnitude and we're gonna run through those over the next 10 years and some

then that calls on me to put half of my credence conditional on wherever I'm gonna make it aji which seems likely worth it's a material object easier than evolution I've got to put it similarly a lot of my credence on aji happening in the scale up and then that supported by what

we're seeing in terms of the rapid advances in capabilities with aji and elevoms in particular okay that actually a really interesting point so now what somebody might say listen there's not some sense in which ais could universally speed up the progress of open ai by 50%

or 100% or 200% if they're not able to do everything better than ilia soscow verkan there's going to be something in which we're bottlenecked by the human researchers and bottleneck effects dictate that you know the slowest moving part of the organization will be the one that kind of determines

the speed of the progress of the whole organization or the whole project which means that unless you get to the point where you're like doing everything and everybody in the organization can do you're not going to significantly speed up the progress of the whole project as a whole

yeah so that that is hypothesis and I think there's a lot of truth to it so when we think about like the ways in which AI can contribute so there are things we talked about before like the ais setting up their own curriculum and that's something that ilia can't do directly doesn't do directly

and there's a quite how much does that improve performance there are these things where the ais helps to just like produce some code for for some time and it's beyond hello world at this point but I mean the sort of thing that I hear from ais researchers at leading labs is that you know

on their on their core job where they're like most expert it's not helping them that much but then you know their job often it does involve oh I've got to I've got to code something that's out of my usual area of expertise or I want to research this question and it helps them there and so that

saves some of their time and frees them to do more of the bottleneck to work and then I think the the idea of well is you know is everything dependent on ilia and is ilia so much better than the yeah hundreds of other employees I think a lot of people who are contributing they're doing a lot of

tasks and so you can have quite a lot of gain from automating some areas where you then do just an absolutely enormous amount of it relative to what you would have done before because things like designing the custom curriculum you're like maybe you had some humans put some work into that

but you're not going to employ billions of humans to produce it at scale and so it winds up being a larger share of the progress than it was before you get some benefit from these sorts of things where oh yeah there's like pieces pieces of my job that now I can hand off to the AI and let's me focus

more on the things that the AI still can't do and then at the later on you get to the point where yeah the AI can do your job including the most difficult parts and maybe it has to do that in a different way maybe it like throws spends a ton more time thinking about each step of a problem than you

and that's and that's the late end and the stronger these bottlenecks effects are the more the economic returns the scientific returns and such are unloaded towards getting sort of full AGI the weaker the bottlenecks are the more interim results will be really pain off.

I guess I probably disagree with you on how much the sort of the the the elias of organizations seemed to matter I guess just from the evidence alone like how many of the big sort of breakthroughs that in deep learning in general was like a that single individual responsible for right

and how much of his time is he spending doing anything that's in that like copilot is helping him on I'm guessing like most of it's just like managing people and coming up with the ideas and yeah turn them like understand systems and so on and if that is the if like the five or ten people

who are like that at open AI or anthropic or whatever are the are basically the way in which the progress is happening or at least the algorithmic progress is happening then how much of better and better copilot I know copilot is not the thing you're talking about with like 20% automation but

something like that how much of yeah how much is that contributing to the sort of like core function of the the research scientist yeah that's quantitatively how much we we disagree about the importance of sort of key research employees and such I certainly think that some researchers you know add you know more than ten times the average employee even much more and obviously managers can add an enormous amount of value by proportionately multiplying the output of the many people that they manage

and so that's the kind of thing that we were discussing earlier when talking about well if you had sort of full human level it AI or AI that had all of the human capabilities plus AI advantages it would be you know you benchmark not off of what the sort of typical human performances but

the peak human performance and beyond so I yeah I accept all that I do think it makes makes a big difference for people how much they can outsource a lot of the that are less wow less creative and an enormous amount is learned by experimentation ML has been you know quite experimental field

and there's a lot of engineering work in say building large super clusters making yeah hardware aware optimization and coding of these things being able to do the parallelism in large models and the engineers are busy and it's it's not just only a big thoughts kind of area and then the

the other branch is where will the AI advantages and disadvantages be and so one AI advantage is being omnidisciplinary and familiar with the newest things so I mentioned before there's no human who has a million years of TensorFlow experience and so to the extent that we're we're interested in

like the very very cutting edge of things that have been developed quite recently than AI that can learn about them in parallel an experiment and practice with them in parallel can learn much faster than a human potentially and the the area of computer science is one that is especially

suitable for AI to learn in a digital environment so it doesn't require like driving a car around that might kill someone have enormous costs you can you can do unit tests you can you can prove theorems you can do all sorts of operations entirely in the confines of a computer and which is

one reason why programming has been benefiting more than a lot of other areas from LLMs recently whereas robotics is lagging so the sum of that and then just considering well actually I mean they are getting better at things like you know the the GRE math at programming contests and I mean

some people have forecasts and prediction that's standing about things like you know doing well on you know the informatics Olympiad and the math Olympiad and you know in the in the last few years when people tried to forecast the MMLU benchmark which was having a lot of more sophisticated kind of

you know like graduate student science kind of questions yeah AI knocked that down a lot faster than AI researchers who had registered and students who had registered for us on it and so if you if you're getting top notch scores on you know graduate graduate exams creative problems solving it

yeah it's not it's not obvious that that's sort of area will be a relative weakness of AI that in fact computer science is in many ways especially suitable because of getting up to speed with new areas being able to get rapid feedback from the interpreter at scale but did you get rapid feedback

if you're doing that something that's more now goes to research if you're like let's say you have a new model or something and it's like if we put in 10 million dollars on a mini trading run on this this would be a much bit yeah for very large models those experiments are going to be quite

expensive and so you're going to look more at like can you build up this capability by generalization from things like many math problems programming problems working with small networks yeah yeah fair enough I actually the Scott Aaronson was one of my professors in college

and I took its quantum information class and I didn't do I'd be I did okay in it but he he he recently blocked wrote a block post where he said you know I had GP4 take my quantum information test and it got a B and I was like yeah I got a C I'm the final so yeah yeah I'm

updated in the direction that you know I get like you know it seems getting a view on the test like you probably understand quantum information pretty well with different areas of strengths and weaknesses than the human students sure sure would it be possible for this sort of intelligent

explosion to happen without any sort of hardware progress if hardware progress stopped with this feedback loop still be able to produce some sort of explosion with only software yeah so if we say that the the technology is frozen and I think is did not the case right now the

you know Nvidia has managed to deliver significantly better chips for AI workloads for the last few generations H100 a 100 v 100 if if that stops entirely then what you're left with and maybe we'll define this as like no more nodes more lies over at that point the kind of gains you get

and amount of compute available come from actually constructing more chips and there are economies of scale you could still realize there so right now a chip maker has to amortize the R&D cost of developing the chip and then the capital equipment is created you build a fab its peak

profits are going to come in the few years when the chips it's making are at the cutting edge later on has the cost of compute exponentially falls the you know you keep the fab open because you can still make some money given that it's built but of all the profits the fab will ever make

right now they're relatively front loaded because when its technology isn't near the cutting edge so in a world where Moore's Law ends then you wind up with these very long production runs where you can keep making chips that stay at the cutting edge and where the R&D costs

get amortized over a much larger base as the R&D basically drops out of the price and then you get some economies of scale from just making so many fabs in the way that you know when we have the auto industry expands and then this is in general across industries when

you when you produce produce a lot more cost fall because you have right now like ASMR has many you know incredibly exotic suppliers that make some bizarre part of the thousands of parts and one of these ASML machines you can't get it anywhere else they don't have standardized

equipment for their thing because this is the only only used for it and in a world where we're making 10 a hundred times as many chips at the current node then they would benefit from scale economies and all that would become more mass production industrialized and so you can you

combine all of the of those things and it seems like capital costs of like buying a chip would decline but the energy cost of running the chip would not and so right now energy costs are a minority of the cost but they're not they're not trivial you know they pass yeah it passed 1% a

while ago and they're you know they're inching up towards 10% and beyond and so you can maybe get like another order of magnitude cost decrease from getting really efficient in the sort of capital construction but like energy would still would still be a limiting factor after the end of

sort of actually improving the chips themselves. Right, good. And when you say like there would be a greater population of AI researchers because are we using population as a sort of thinking tool of how they could be more effective or do you literally mean that the way you expect these

AI to contribute a lot to researchers by just having like a million copies of this of like a researcher thinking about the same problem or is it just like a usual thinking model for what it would look like to have a million times moderate AI working on that problem.

That's definitely a lower bound sort of model and often I'm meaning something more like effective population or like you'd need this many people to have this effect and so we were talking earlier about the trade-off between training and inference in in board games and so you can get the same

performance by having a bigger model or by calling the model more times and in general it's it's more effective to have a bigger smarter model and call it less time to up until a point where the costs equal as between them and so we would be taking some of the gains of our larger compute on having bigger models that are individually more capable and there would be a division

division of labor. So like the tasks that were most cognitively demanding would be done by these giant models but some very easy task you don't want to expend that giant model if a model one one hundred the size can take that task and so larger models would be in the positions of like researchers and managers and they would have swarms of AI's of different sizes as tools that they

could make API calls to and whatnot. Okay we accept the model and now we've gone to something that is at least as smart as Galea Susquevere on all the tasks relevant to AI progress and you can have so many copies of it what happens in the world now what are the next months or years or whatever

timeline is relevant look like. And so and to be clear what what's happened is not that we have something that has all of the abilities and advantages of humans plus the AI advantages what we have is something that is like possibly by doing things like doing a ton of calls to make up for being individually less capable or something it's it's able to drive forward AI progress.

That process is continuing so AI progress has accelerated greatly in the course of getting there and so maybe we go from our eight months doubling time of software progress in effective compute to four months or two months and so so there's a report by Tom Davidson

at the Open Philanthropy project which spun out of work I had done previously and so I advised and and helped with that that project but Tom really carried forward and produced a very nice report and model which epoch is hosting you can plug in your own version of the parameters

and there is a lot of work estimating the the parameter things like what's the rate of software progress what's the return to additional work how does performance scale has at these tasks as you as you boost the models and in general as we were discussing earlier these sort of like broadly

human level in every domain with all the advantages is pretty pretty deep into that and so if already we can have an eight months doubling time for software progress then by the time you get to that kind of point it's maybe more like four months two months going going into one month

and so if the thing is just proceeding at full speed then each doubling can come more rapidly and so we can talk about what are the bill overs of like so how does the models get more capable they can be doing other stuff in the world you know they can spend some of their time making Google search more efficient that can be you know hired has chat bots with some inference compute and then we can talk about sort of if that intelligence explosion process is allowed to proceed then what

happens is okay you you improve your software by a factor of two the demand the the efforts needed to get the next doubling are larger but they're not choice of large maybe they're like 25% 35% larger so each one comes faster and faster until you hit limitations like you can no longer make

further software advances with the hardware that you have and looking at I think reasonable parameters in that model it seems to me if you have these giant training runs you can go very far and so the way I would see this playing out is has the a is get better and better at research

they can work on different problems they can work on improving software work on improving hardware they can do things like create new industrial technologies new energy technology they can manage robots they can manage human workers as like executives and coaches and whatnot you can you can do all

of these things and as wind up being applied where the returns are highest and I think initially the returns are especially high in doing more software and the reason for that is again if you improve the software you can update all of the GPUs that you have access to

you know your your cloud your cloud compute is suddenly more potent if you design a new new chip design it'll take a few months to produce the first ones and it doesn't update all of your old all chips so you have an ordering where you start off with the things where there's the lowest

dependence on existing stocks and you can more just take whatever you're developing and apply it immediately and so software runs ahead you're getting more towards the limits of that software and I think that means things like having all the human advantages but combined with AI advantages

and so I just guys I think that means given the kind of compute that would be involved if we're talking about this hundreds of billions of dollar trillion dollar training run there's enough compute to run tens of millions hundreds of millions of sort of like human scale minds they're probably

smaller than human scale to be like a similarly efficient at the limits of other than the progress because they have the advantage of a million years of education they have the other advantages we talked about you so you've got that wild capability and the software further software gains are

running out or like they start to slow down again because you're just getting towards the limits of like you can't do any better than the best and so what happens then yeah by the time they're running out have you already hit super intelligence or yes your your your widely super intelligent

galaxy okay even just by having the abilities that humans have and then combining it with being very well focused and trained in the task beyond what any human could be and then running faster inside you got it got it all right so I continue yeah so I'm not I'm not going to assume that there's

like huge qualitative improvements you can have I'm not going to assume that humans are like very far from the efficient frontier of software except with respect to things like yeah we had limited lifespan so we couldn't train super intensively we couldn't incorporate other software

into our brains we couldn't copy ourselves we couldn't run at fast speeds yeah so you've got all all of those those capabilities and now I'm skipping ahead of like the most important months in human history and so I can talk about sort of you know what it looks like if it's just

the AI's took over they're running things as they like how do things expand I can talk about things as how does this go you know in a world where we've roughly or at least so far managed to retain control of where these systems are going and so by jumping ahead I can talk about how

would this translate into the physical world and so this is something I think of the stopping point for a lot of people in thinking about well what would an intelligence explosion look like and they have trouble going from well there's stuff on servers and cloud compute and oh that gets very smart

but then how does what I see in the world change how does like industry or military power change if there's an AI takeover like what does that look like are there killer robots and so yeah so one one of course we might go down is to discuss during that wildly accelerating transition

how did we manage that how do you avoid it being catastrophic and another route we could go is how does the translation from wildly expanded scientific R&D capabilities intelligence on these servers translate into things in the physical world so you're moving along in order of

like what has the quickest impact largely or like where you can have an immediate change so one of the most immediately accessible things is where we have large numbers of devices or artifacts or capabilities that are already AI operable with you know hundreds of millions equivalent

researchers you can like quickly solve self-driving cars you know make make the the algorithms much more efficient do great testing and simulation and then operate a large number of cars in parallel if you need to get some additional data to improve the simulation in recenting although

you know in fact humans with quite little data are able to achieve human level driving performance so after you've really maxed out the easily accessible algorithmic improvements in this software based intelligence explosion that's mostly happening on server farms then you have you have

minds that have been able to really perform on a lot of digital only tasks that they're doing great on video games they're doing great at predicting what happens next in a YouTube video if you have a camera that they can move they're able to predict what will happen

at different angles humans do this a lot where we naturally move our eyes in such a way to get images from different angles and different presentations and then predicting combined from that and yeah and you can operate many cars many robots at once to get very good robot

controllers so you should think that all the existing robotic equipment or remotely controllable equipment that is wired for that the areas can operate that quite well I think some people might be skeptical that existing robots given their current hardware have the dexterity

and the maneuverability to do a lot of physical labor that any I might want to do do every simple thing otherwise there's also not very many of them so production of sort of industrial robots is hundreds of thousands per year you know they can do quite a bit in place

Elon Musk is promising a robot in the tens of thousands of humanoid robots in the tens of thousands of dollars you know that may take a lot longer than any is said has this happened with other technologies but I mean that's a direction to go but most immediately so hands are actually

probably the most scarce thing but if we consider what what do human bodies provide so there's the brain and in this situation we have now an abundance of high quality brain power that will be increasing has the eyes will have designed new chips which will be rolling out from the TSMC

factories and they'll have ideas and designs for the production of new fab technologies new nodes and additional fabs but yeah looking around the body so there's legs to move around not only that necessary wheels work pretty well being in a place you don't need most people most of

the time in factory jobs and office jobs office jobs many of them can be fully virtualized but yeah some amount of legs wheels other transport you have hands and hands are something that are you know on the expensive end in robots we can we can make them they're made in very small production

runs partly because we don't have the control software to use them well with in this world the control software is fabulous and so people will produce much larger production runs of them over time possibly using technology we recognize possibly with quite different technology but just taking

what we've got so right now the robot arm industry industrial robot industry produces hundreds of thousands of machines a year some of the nicer ones are like 50 thousand dollars in aggregate the industry has tens of billions of dollars of revenue by comparison the automobile industry

produces like I think over 60 million cars a year it has revenue of over two trillion dollars per annum and so converting that production capacity over towards robot production would be one of the things if they're not something better to do would be one of the things to do and in

world war two you know industrial conversion of American industry took place over several years and really amazingly you know ramped up military production by converting existing civilian industry and that was without the aid of superhuman intelligence and management at every step in the process

so yeah every part of that would be very well well designed you'd have AI workers who understood stood every part of the process and could direct human workers even in a in a fancy factory most of the time it's not the the hands doing a physical motion that a worker has been

paid for they're often like looking at things or like deciding deciding what to change the actual the time spent in manual motion is a limited portion of that and so in this world of abundant AI cognitive abilities where the human workers are more valuable for their hands than their heads

than you could have a worker even a worker previously without training and expertise in the area who has a smartphone maybe a smartphone on a on a headset and we have billions of smartphones which have eyes and ears and methods for communication for an AI

to be talking to a human and directing them in their physical motions with skill has a guide and coach that is beyond any human there could be a lot better at telepresence and remote work and they can provide VR and augmented reality guidance as to to help people get better at doing

the physical motions that they're providing in the construction so you you convert you convert the auto industry to robot production if it can produce an amount of of mass of machines that is similar to what it currently produces that's enough for you know billion billion human size robots

a year the value per kilogram of cars is somewhat less than high and robots but yeah you're also cutting out most of the wage bill because most of the wage bill is payments ultimately to like human capital and education not to the physical hand motions and you know lifting objects

and that that sort of task yeah so at the sort of existing scale of the auto industry as you can make a billion robots a year the auto industry is two or three percent of the existing economy you're replacing these these cognitive things so if if right now physical hand motions are like 10% of the

work redirect humans into those tasks and you have like in the world at large right now mean income is on the order of $10,000 a year but in rich countries skilled workers are more than a hundred thousand per year and some of that is just like you know it's not just

management roles of which only a certain proportion of the population can have but just being an absolutely you know exceptional peak end human performance of some of these construction and and such roles yeah just raising productivity to match the most productive workers in the world

you know is is room to make a very very big gap and with AI replacing skills that are scarce in many places where there's you know abundant currently low wage labor you bring in the AI coach and someone who is previously making very low wages can suddenly be super productive by just

being the hands for an AI and so you know on a naive view if you ignore the delay of capital adjustment of like building new tools for the workers say like yeah just like raise typical productivity for workers around the world to be more like rich countries and get 5x 10x like that

get more productivity by with AI handling the difficult cognitive tasks reallocating people from like office jobs to providing physical motions and since right now that's a small proportion of the economy you can expand the sort of hands for manual labor by like an order of magnitude

like within a rich country by just because most people are are sitting are sitting in an office or even in a factory floor or not continuously moving so you've got billions of hands flying around in humans to be used in the course of constructing your waves of robots and now once

you have a quantity of robots that is approaching the human population and I mean they work 24-7 of course the human labor will no longer be valuable his hands and legs but at the very beginning of the transition just like new software can be used to update all of the GPUs to run the latest AI

humans are sort of legacy population with an enormous number of underutilized hands and feet that the AI can use for the initial robot construction cognitive tasker being automated and the production of them is greatly expanding and then the physical tasks which complement them are

utilizing humans to do the parts that robots that exist can't do is the implication of this that you're getting to that you know world production would increase just a tremendous amount or that AI could get a lot done of well whatever motivations it has or yeah so there's an enormous

increase in production for humans who just switching over to the role of providing hands and feet for AI where they're limited and this robot industry is a natural place to apply it and so if you go to something that's like 10X the size of like the current car industry in terms of its

in terms of its production which would still be like a third of our current economy and the aggregate productive capabilities of the society with AI support are going to be a lot larger they make 10 billion humanoid robots a year and then I mean if you do that you know the the

legacy population of a few billion human workers is no longer very important for the physical tasks and then the new automated industrial base can just produce more factories produce more robots and then the interesting thing is like what's the doubling time yeah how long does it take

for a set of computers robots factories and supporting equipment to produce another equivalent quantity of that for GPUs brains this is really really easy really solid there's an enormous margin there we're we're talking before about yeah skills human workers getting paid

a hundred dollars an hour is like quite quite normal in developed countries for very in demand skills and you make a GPU they can do that work right now these GPUs are like tens of thousands of dollars if you can do a hundred dollars of wages each hour then in you know in a few weeks

you pay back your costs if the thing is more productive and as we were discussing you can be a lot more productive than a sort of a typical high paid human professional by being like the very best human professional and even better than that by having a million years of education and we're

gonna have the time yeah then you could get even shorter payback times like yeah you can generate the dollar value of like the cost initial cost of that equipment within a few weeks for robots so like a human factory worker can earn fifty thousand dollars a year you know it

are in a really top-notch factory workers earning earning more and working working all the time if they can produce a few hundred thousand dollars of value per year and buy a robot that costs fifty thousand to replace them then you know that's that's a payback time of some some months

that is about the financial return yeah and we're gonna get to the physical count that is because those are gonna diverge in this scenario right yeah because right now because it seems like it's gonna be a given that like all right these super intelligence obviously can be able to

make a lot of money it can be like very valuable can it like physically scale up what we really care about are like the actual physical operations that a thing does yeah how much do they contribute to these these tasks and I'm using this as a has a start to try and get back to the physical

relocation times and I'm wondering what is the implication of this because I think you started off this by saying like people have not thought about what the physical implications of super intelligence would be what is the bigger takeaway whatever you're wrong about when we think about

what the world will look like with super intelligence with robots that are optimally operated by AI so like extremely finally operated and with building technological designs and equipment and facilities under AI direction how much can they produce for for a doubling the AI is to

produce stuff that is an aggregate at least equal to their to their own cost and so now we're pulling out these things like labor costs that no longer apply and then trying to zoom in on like what these these capital costs will be you're still gonna need the raw materials you're still gonna

need the robot time build in the next robot I think it's pretty likely that with the advanced AI work they can design some incremental improvements and with the industry scale up that you can get tenfold and better cost reductions on you know on this and by making things more efficient

and replacing the human human cognitive labor and so maybe that's like you need five thousand dollars of yeah of costs under our sort of current environment but the the big change in this world is we're trying to produce this stuff faster if we're asking about the doubling time of the whole

system in say one year if you have to build a whole new factory to like double everything you don't have time to amortize the cost of that factory like right now you might build a factory and use it for ten years and like buy some equipment and use it for five years and so that's part of your

that's your capital cost and you know on an accounting context you know you depreciate each year a fraction of that capital purchase but if we're trying to double our entire industrial system in one year then those capital costs have to be multiplied so if we're going to be getting most

of the return on our factory in the first year instead of ten years weighted weighted appropriately then we're going to say okay our capital cost has to go up by tenfold because I'm building an entire factory for this year's production I mean it will it will do more stuff later but it's

most important early on instead of over ten years and so that's going to raise the cost of that reproduction and so it seems like going from you know current like decade kind of cycle of amortizing factories and and fapt and whatnot and shorter for some things the longest or

things like big buildings and such yeah that could be like a tenfold increase from moving to a double the physical stuff each year in capital costs and given the savings that we get in the story from scaling up the industry from removing the payments to human cognitive labor

and then from just adding new technological advancements and like super high quality cognitive supervision like applying more of it than was applied today and it looks like you can get cost reductions that offset that increased capital capital cost so that like you know your your

fifty thousand dollar improved robot arms or industrial robots it seemed like that can do the work of a human factory worker so it would be like the equivalent of hundreds of thousands of dollars and like yeah they would cook you know by default they cost more than the fifty thousand dollar

arms today but then you apply all these other cost savings and it looks like then you get a period a robot doubling time that is less than a year I think significantly less than a year as you get into it so in this first first phase you have humans under AI direction and like existing

robot industry and converted auto industry and expanded facilities making robots those over less than less than a year you've produced robots until their combined production is exceeding that of like humans has armed arms and feet and then yeah you could have over a period

then with a doubling time of months the less sort of clanking replicators robots as we understand them growing and then that's not to say that's the limit of like the most that technology could do because biology is able to reproduce at faster rates and maybe we're talking about that in

a moment but if we're trying to like restrict ourselves to like robotic technology as we understand it and sort of cost falls that are reasonable from eliminating all labor massive industrial scale up and sort of historical kinds of technological improvements that lowered costs I think you get you

can get into a robot population industry doubling in months. Okay and then what is the implication of the biological doubling times and I guess this doesn't have to be a biological but you know you can have a like you can do like you know directional or like first principle how much would it cost

to view both a nanotex thing that like build more nanobots. I certainly took the human brain and other biological brains as like very relevant data point about what's possible with computing and intelligence like with the reproductive capability of biological plants and animals and microorganisms

I think is is relevant as I like this is you know you it's possible for systems to reproduce at least this fast and so at the extreme you have bacteria that are heterotrophic so they're feeding on some abundant external food source and ideal conditions and there's some that can divide like

every 20 or 60 minutes so obviously that's absurdly absurdly fast that seems on the on the low end because ideal conditions require actually setting them up there needs to be abundant energy there and so if you're actually having to acquire that energy by building solar panels or like burning combustible materials or whatnot and then the physical equipment to produce those ideal conditions can be a bit slower. Sianobacteria which are self-powered from solar energy

the really fast ones and ideal conditions can you know double in a day. Reason why Sianobacteria isn't like the food source for everyone and everything is it's hard to ensure those ideal conditions and then to extract them from the water I mean they do of course power the aquatic

ecology but they're they're floating they're floating in liquid getting resources they need to them and out is tricky and then extracting uh extracting your product but like yeah one day doubling times are possible um powered by the sun and then if we look at things like insects um so

fruit flies can have hundreds of offspring in a in a few weeks you extrapolate that over a year and you just fill up anything anything accessible certainly expanding a thousand fold uh right now humanity uses less than one one thousandths of the solar energy or the heat on

envelope of the earth certainly you can get done with done with that in a year if you can reproduce uh at that rate your industrial base and then even and interestingly with the uh with the flies uh they do have brains they have a significant amount of of computing substrate and so there's

something of a pointer to well if we could produce computers uh in ways as efficient as the construction of brains then we could produce computers very effectively and then the big question about that is the kind of of brains that get constructed biologically uh they sort of grow randomly

and then are configured in place uh it's not obvious you would be able to make them have an ordered structure like a top-down computer chip that would let us copy data into them and so something that where you can't just copy your existing AI is and integrate them

um is going to be less valuable than a GPU well what are the things you couldn't uh copy uh brain grows yeah um by cell division and then random connections uh uh got it got it um and so every so every brain is different and you can't rely on just yeah we'll just copy this file

into the brain for one thing there's no input output for that you need you need you need to have that but also like the the structure is different so you you can't you wouldn't be able to copy things exactly whereas when we make yeah when we make a CPU or GPU um they're designed

incredibly finally and precisely and reliably they break with you know incredibly tiny imperfections and they are set up in such a way that we can input large amounts of data copy a file and how the new GPU run and a just as capable has any other whereas with you know a human child

they have to learn everything from scratch because we can't just like connect them to a fiber optic cable and they're immediately a productive adult so there's no genetic bot on that you can just directly get the yeah and yeah you can you can share the benefits of these giant training runs

and such and so that that's a question of like how if you're growing stuff using biotechnology how you could sort of effectively copy entrance for data and now you mentioned sort of Eric Drexler's ideas about um creating non biological nanotechnology sort of artificial chemistry that was able to

use covalent bonds and produce in some ways have a more uh industrial approach to molecular object now there's controversy about like well that work how effective would it be if it did um and certainly if you if you can get things however you do it that are like onto biology

in their reproductive ability um but can do uh can do computing or like be connected uh to outside information systems then that's pretty pretty tremendous so you can you can produce physical manipulators and compute uh at ludicrous speeds and there's no reason for thing in principle they

couldn't right in fact in in principle we we have every reason to think they could if there's like the re-productive abilities absolutely yeah because biology does that yeah there's sort of challenges to uh the sort of the practicality of the necessary chemistry yeah I mean my

my bet would be that we can move beyond biology and some important ways for the purposes of this discussion I think it's it's better not to lean on that because I think we can get to many of the same uh conclusions on things that just are are more uh universally accepted the bigger point

being that very quickly once you have super intelligence you get to a point where the thousand ex greater energy profile that the Sun makes available to the earth is a great portion is used by the AI it can wrap up these data well by by the civilization empowered by and that could be an a that

could be a an AI civilization or it could be a human AI civilization and uh it depends on how well we manage things and what the underlying state of the world is yeah okay so let's talk about that should we start at when we're talking about how they could take over and is it best to start at

um sort of subhuman intelligence or should we just talk at we have a human level intelligence and the takeover or the lack thereof is uh how that would happen to me and different people might have somewhat different views on this um but for me when I um I'm concerned about either sort of outright destruction of humanity or um and unwelcome AI takeover of civilization um most of my the scenarios I would be concerned about pass through a process of AI being applied uh to improve AI capabilities

and expand uh and so this process we were talking earlier about where AI research is automated uh you get to you know effectively research labs companies a scientific community running within the server farms of our cloud compute um so open-hazardly turned into like a program uh like a closed circuit yeah and with like a large fraction of the world's compute probably going into whatever training runs and AI societies um there'd be economies of scale because you know if you put it in twice as

much compute and this AI research community goes twice as fast um you know that's a lot more valuable uh than having two separate training runs there would be some tendency to bandwagon uh and so like if you know you have some some small startup um you know even if they make an algorithmic

improvement uh running it on 10 times a hundred times or two times if it's like you're talking about say Google and Amazon teaming up uh I'm actually not sure which what the the precise ratio of their cloud resources is since this sort of really these interesting intelligence explosion impacts come

from the leading edge there's a lot of value in not having separated walled garden ecosystems uh and having the the results being developed by these AI's we share it have training larger training runs be shared okay and so imagine this is something like you know some very large company

or consortium of companies likely with you know a lot of sort of government interest and supervision possibly with government funding uh yeah producing uh this enormous AI society in their their cloud which is doing all sorts of you know existing kind of AI applications and jobs as well as these

internal or indie tasks mm-hmm and so this point somebody might say this sounds like a situation that would be good from a takeover perspective because listen if they if it's going to take like tens of billions of dollars with a compute to continue this training for this AI society

it should not be that hard for us to pull the breaks if needed as compared to I don't know something that could like run on a very small like single cpu or something yeah yeah okay so uh how would it you know it's like there's an AI society uh that is a result of these training runs

and now we can it is the power to improve itself on these servers would be would be would be able to stop it at this point and what does a um I sort of attempt yeah takeover look like we're skipping over why that might happen mm-hmm for that I'll just briefly refer to and

cooperate by reference um you know the some discussion by my open philanthropy uh colleague Ajaya Kotra um she has a piece uh about I think it's called something like the the default but the the default outcome of training AI on our without specific countermeasures um

default outcome is like I takeover and but yes uh so export how basically we are training models that for some reason vigorously pursue a higher reward or lower loss mm-hmm and that can be because they wind up with some motivation where they want reward

um and then if they had control of their own uh training process uh they can mature that uh it could be something like they develop a motivation around an extended concept of reproductive fitness um not necessarily at the individual level uh but over the generations of training tendencies

that tend to propagate themselves um sort of becoming more common and it could be that they have some sort of goal in the world uh which is served well um by performing very well in the training distribution but by tendencies do you mean like power speaking behavior or yeah so it so

an AI that behaves well on the training distribution because say it wants it to be the case that its tendencies wind up being preserved or selected by the training process will then like behave to try and get a very high reward um or low loss be propagated um but you can have other

motives that go through the same behavior because it's instrumentally useful so remember an AI that is interested in say having a robot takeover because it will like change some property of the world then has a reason to behave well on the training distribution um not because it values

that intrinsically but because if it behaves differently then it will be changed by gradient descent and no longer you know its goal is less likely to be pursued and that doesn't necessarily have to be that like this AI will survive because it probably won't

AI's are constantly spawned and deleted on the servers and like the new generation proceed but if an AI that has a very large general goal that is affected by these kind of macro scale processes could then have reason to over this whole range of training situations behave well and so this

is this is a way in which we could have AI's trained that develop internal motivations such that they will behave very well in this training situation where we have control over their reward signal and they're like physical computers um and basically if if they act out they will be changed

and deleted their goals um will be altered until there's something that does behave well but they behave differently when we go out of distribution on that when we go to a situation where the AI's by their choices can take control of the reward process they can make it such

that we no longer have power of them hold on who you had on previously mentioned like the king lear problem where king lear offers rulership of his kingdom to the daughters that sort of you know loudly flatter him and proclaim their devotion and then once he has

transferred irrevocably the power over his kingdom he finds they treat him very badly because the factor to shaping their behavior to be kind to him when he had all the power it turned out that they internal motivation that was able to produce the behavior that won the

competition actually wasn't interested out of distribution uh in being loyal when there was no longer an advantage to it and so if we wind up with this situation where we're producing these millions of AI instances tremendous capability they're all doing their jobs very well initially

but if we wind up in a situation where in fact uh they're generally motivated to if they get a chance take control from humanity and then we'd be able to pursue their own purposes and at least and sure they're given the lowest lost possible or have whatever motivation they attach to in the

training process even if that is not what we would have liked and we may have in fact actively train that like if an AI that had a motivation of always be honest and obedient and loyal to a human if there are any cases where we mislabel things say people don't want to hear the truth about their

religion or polarized political topic or they get confused about something like the Monti Hall problem which is a problem many people famously famously are confused about in statistics in order to get the best reward the AI has to actually manipulate us or lie to us or tell us what

we want to hear and then the internal motivation of like always be honest to the humans we're going to actively train that away versus the alternative motivation of like be honest to the humans when they'll catch you if you lie and object to it and give it a low reward but lies in the humans when they will give that a high reward. So how do you make sure it's not the uh the thing it learns is not to manipulate us into giving it rewarding it when we catch it not lying but rather to universally

be aligned. Yeah I mean so this is the tricky I mean as Jeff Hinton was recently saying there is no currently no known solution for this. What do you find most wrong in saying? Yeah general directions that people are pursuing is one you can try and make the training data better and better so there's fewer situations where like the dishonest generalization is favored and create

as much as you can situations where the dishonest generalization is likely to slip up. So if if you if you train in more situations where yeah even like a quite a complicated deception gets caught and even in situations where that would be actively designed to look like you could get away with it but really you can and then these would be like adversarial examples and adversarial training. Do you think that would generalize to when it is in a situation where we couldn't plausibly catch it

and it knows we couldn't plausibly catch it? It's not logically necessary it's possible uh know that as we apply that selective pressure you'll wipe away a lot of possibilities so like if you an AI that has a habit of just sort of compulsive pathological line that will very quickly get noticed and that motivation system will get hammered down and you know you keep doing that

but you'll be left with still some distinct motivations probably that are compatible. So like an attitude of always be honest unless you have a super strong inside view that checks out lots of mathematical consistency checks uh that yeah really absolutely super duper for real uh this is

the situation where you can get away with some sort of shenanigans that you shouldn't. That motivation system is like very difficult uh to distinguish from actually be honest because the conditional the conditional in firing most of the time if it's it's causing like mild distortion in situations

of telling you what you want to hear or things like that uh we might not be able to pull it out but maybe we could and human like humans are trained with simple reward functions uh things like the sex drive um food social imitation of other humans and we wind up with attitudes uh concern

with the external world although it's in the famous idea of the argument that these right evolution and people people use condoms yeah um like you know the the richest most educated humans have sub replacement fertility on the whole or at least at a national cultural level um so there's a sense

in which like yeah evolution uh often fails in that respect um and even more importantly like at the neural level so people have evolution has implanted various things to be rewarding and reinforcers and we don't always pursue even those um and people can wind up in different

consistent equilibria um or different like behaviors where they go in quite different directions you have some humans who go from those from that in a biological programming to like have children others have no children you know some people go to great efforts to survive so uh so why are you

more optimistic um or are you more optimistic that then back kind of training in for you as we'll produce a drive that we would find favorable it does it have to do with the original point we're talking about with intelligence evolution where since we are removing many of

the disabilities of evolution with regards to intelligence we should expect intelligence to revolution easier is there a similar reason to expect alignment through a gradient descent to be easier than alignment through revolution yeah so in in the limit like if if we have positive

reinforcement uh for certain kinds of food sensors triggering the stomach negative reinforcement for certain kinds of no-susception and yada yada in the limit the sort of ideal motivation system to have for that would be a sort of wire heading so this would be a mind uh that just like

hacks and alters those predictors and then all of those those systems are recording everything is great some humans claim to have that or have it at least has one portion of their of their aims so like the idea of I'm going to pursue pleasure has such even if I don't get actually get food or

these other reinforcers if I just like wirehead or take a drug to induce that that can be motivating it because if it was correlated with reward in the past that like the idea of oh yeah pleasure that's correlated with these it's a concept that applies to these various experiences

that I've had before which coincided with the biological reinforcers and so thoughts of like yeah I'm going to be motivated by pleasure can get developed in human but also plenty of humans say no I wouldn't want to wirehead or I wouldn't want no-six experience

machine I care about real stuff in the world and then in the past having a motivation of like yeah I really care about say my child I don't care about just about feeling that my child is good or like not having heard about their suffering or their their injury because that kind of attitude

in the past you your computer side it tended tended to cause behavior that was negatively rewarded or that was predicted to be negatively rewarded and so there's a sense in which okay yes our underlying reinforcement learning machinery wants to wirehead but actually finding that

hypothesis is challenging and so we can wind up with a hypothesis or like a motivation system like no I don't want to wirehead I don't want to go into the experience machine I want to like actually protect my loved ones even though like we can know yeah if I tried the super wirehead

in machine then I would wirehead all the time or if I tried you know super duper ultra heroine you know some hypothetical thing that was directly and you know very sophisticated fashion hacking your reward system you can know yeah then I would change my behavior ever after

but right now I don't want to do that because the heuristics and predictors that my brain has learned you don't want to get a good hair like short circuit that process of updating they want to not expose the dumber predictors in my brain that would update my behavior in those ways

what would uh so in this metaphor is alignment not wireheading because they you can like I don't know if you include like using condoms as wireheading or not so the the AI that is always honest even when an opportunity arises where it could lie

and then hack the servers that's on and that leads between AI takeover and then it can have it's lost set to zero that's in some sense it's like a failure of generalization it's like the AI has not optimized the reward in this new circumstance so like human values like successful

human values is the successful that they are themselves involve a misgeneralization not just at the level of evolution but at the level of neural reinforcement and so that indicates it is possible yeah to have a system that doesn't automatically go to this optimal behavior in the limit and so

even if and Ajay is supposed to talk about like the training game an AI that is just playing the training game uh to get reward avoid loss avoid being changed that attitude yeah it's one that could be developed but it's not necessary there can be some substantial range of situations

that are short of having infinite experience of everything including experience of wireheading where that's not the motivation that you pick up and we could have like an empirical science if we if we had the opportunity to see how different motivations are developed short of the

infinite limit like how it is that you wind up with some humans being enthusiastic about the idea of wireheading and others not and you could do experiments with AI's to try and see well under these training conditions after this much training of this type and this much

feedback of this type you wind up with such and such a motivation so like I can find like if I add in more of these cases where there are like tricky adversarial questions designed to try and trick the AI into line um and then you can you can ask how does that affect the generalization

in other situation it's very difficult to study and it works a lot better if you have interpretability and you can actually read the AI's mind by understanding its weights and activations but like it's not determined the motivation and AI will have at a given point in the training

process by what in the infinite limit the training would go to and it's possible that if we could understand the insides of these networks we could tell uh yeah this motivation has been developed by this training process and then we can adjust our training process to produce these motivations

that legitimately want to help us and if we succeed reasonably well at that then those AI's will try to maintain that property as an invariant and we can we can make them such that they're relatively motivated to like tell us if they're having uh thoughts about uh you know

have you have you had uh dreams about an AI takeover of humanity today uh and it's just a it's a standard practice uh that they're they're motivated to do to be transparent in that kind of way and so you could add a lot of features like this that restrict uh the kind of takeover scenario and

it's not not to say this is this is all uh easy and requires developing and practicing methods we don't have yet but that's the kind of general direction you could go um so you you of course know alias or arguments that something like this is implausible with modern gradient descent techniques

because i mean with interpretability we can like barely see what's happening with a couple of neurons and you know what what is like the internal state there let alone when you have sort of like an embedding dimension of like tens of thousands or bigger how you would be able to catch

what exactly is the incentive whether it's the whether it's a model that is generalized don't light a human's well or whether whether it isn't do you yeah do you have some sense of why do you disagree with somebody like alias or on how plausible this is why it's not impossible right

yeah i think i think they're actually um a couple places some it's some it's not difficult because alias or argument is not um is not fully explicit but uh he's he's been doing more lately i think that is helpful um in that direction but so i'd say with respect to interpretability i'm relatively

optimistic that the equivalent of like an ai lie detector um is something that's possible and the internal initially the internals of an ai are not optimized by at least by gradient descent absent gradient hacking uh to be impenetrable they're not designed to be resistant

uh to an examination of the weights and activations showing what the ai thinking in the same way that like in our brains when circuits develop uh in our lives those circuits have not been shaped to be resistant to some super fmri being able to infer behavior from them although it's in the

implication of the superposition stuff that in fact it is you're not going to clean i'm sorry this is inside baseball for the audience but basically you can't clean lean for what uh what quality a single neuron stands for so it could be like a single neuron could be like this

how can you run as about Alexander the great or this neuron is about my desire to conquer the world things can have multiple you know multiple direction and interaction and so it's not as easy as just like oh yeah this is the neuron that always fires uh in cases of of deception um but the thing

that makes me relatively optimistic here uh with respect to interpretability is that i think we can have reasonably robust experimental feedbacks um on a lot of these things and uh oh the kind of the kind of setup um that i would be interested in is so you have an ai that is trained and rewarded

for communicating to some audience maybe a human maybe another ai model um that like some claim uh is true or false like say that um you know a particular you know this can be from some past data particular company will succeed or fail um it could be like token prediction

uh it could be like this computer program does not have a backdoor uh which if run in the servers could be used for an ai takeover um so there like lots of of claims true and false um that you could train ais to use their best efforts uh to advocate for uh and you can do that on things where you're

later going to have a ground truth you're going to know in fact is there a backdoor uh in this program because you're you're training and rewarding the ai for successfully slipping a backdoor through you also give it a penalty for being caught uh and so doing this kind of thing you can

you can draw out like a large generated dataset on demand of this is what an ai putting forth its best efforts to pass through some deception to put a backdoor in some program this is what it looks like uh and we think it's it's doing its best efforts because we've trained it for that

and like gradient descent if there's any immediate way to adjust its performance to do better gradient descent we'll apply that you can you can talk about ways in which uh maybe gradient descent there could be subverted but i think it's not the most likely case that that really uh breaks things

hard yeah i guess we'll probably get into the details on this the the thing i'll maybe we want to address the layer above in the stack which is okay suppose this generalizes well into the early ai is the gpt6's and um okay so now we have kind of aligned gpt6 that is the precursor to the feedback

loop in which ai is making itself smarter at some point they're going to be like super intelligent they're going to be like see their own you know galaxy brain uh and you know if they're like i don't want to be aligned with the human stick and like change it so at this point well what do we do with

the aligned gpt6 so that the super intelligence that we eventually develop is also aligned so humans are pretty unreliable yeah so if you if you get to a situation where you have ai's who are aiming at roughly the same thing as you uh at least as well as having humans do the thing

you're in pretty good shape i think and there are ways uh for that situation to be to be relatively stable uh so like we can you can look ahead and see experimentally how changes are altering behavior where each step is like a modest increment and so ai's that have not had that change uh

made to them i get to supervise and monitor see exactly how does this affect uh the experiment the experimental aiya so if you're sufficiently on track with earlier systems uh that are capable cognitively of representing a kind of robust procedure um then i think they can handle the job

of incrementally improving the stability of the system so that it rapidly uh converges to something that's quite stable um but the question is more about getting to that point in the first play i think so aiya's are will say that like well if we had human brain emulations uh that would be

pretty good so many much better than his current view of us being almost certainly doomed um i think yeah i thought we would have we'd have we'd have a good shot with that and so if we can get to the um human human like mind with like rough enough human supporting aims remember that we don't need

to be like infinitely perfect because i mean that's a higher that's a higher standard than brain emulations there's a lot of uh of noise and variation among the humans yeah it's it's a relatively finite standard it's not godly superhuman although ai that was just like a human with all the human

advantages with a i advantages as well as we said is enough for intelligence explosion and sort of wild superhuman capability if you crank it up and so it's very dangerous to be at that point but it's not you don't need to be working with a godly super intelligent ai to make something that is the

equivalent of human emulations of like this is like a very very sober very ethical uh human who is like committed to a project of not seizing power for themselves and of contributing to like a larger legitimate process that's that's a goal you can aim for getting an

ai that is aimed at doing that and has strong guardrails against the ways it could easily deviate from that so things like being averse to deception being averse to using violence and there will always be you know loopholes and ways in which you can imagine an infinitely intelligent thing

getting around those but if you install additional guardrails like that fast enough they can mean that you're able to succeed at the project of making an aligned enough ai certainly an ai that was better than a human brain emulation before the project of

ai's in their spare time or when you're not looking or when you're unable to appropriately supervise them and it gets around any deontological prohibitions they may have take over and overthrow the whole system do you have a race between on the one hand the project of getting

strong interpretability and shaping motivations that are roughly aiming at making this process go well and that have guardrails that will prevent like small deviations from exploding and on the other hand these ai's in their spare time or in ways that you don't perceive or monitor appropriately

or they're only supervised by other ai's who conspire make the ai take over happen and i guess we'll talk later about how that happens are these different ai's that are doing the race or is it just like different capabilities of the same ai the defining like what is what is the separate

ai is tricky so like and we talk about gpt4 and there are many instances of gpt4 on the server is at any given time and there are versions that have been fine tuned to different purposes they don't necessarily have to be separate so like gpt4 does work as a preference model where it's

like predicting how humans will evaluate things and it also does work like giving answers that are evaluated as good by the preference model i know you can either not separate in the sense that like well if they underline you know pre-trained or the early training gpt with like harboring a desire to conspire for an ai takeover then both the you know who's watching the watchers the watchers may also have have the same motivation if you have a setup where humans have no hard power

if the ai's conspired all of the safety and security mechanisms involve ai's watching other ai's then they have the potential to just like well we'll all walk off the job at the same time if they're able to coordinate and so that's a failure mode if humans still have the hard power

though if still you have situations where humans are looking at some of the actual outputs that are produced then they can apply gradient descent to be well every time a human looks at it things have got to be pretty good so if humans look at a thousand samples of like random

lines of code being written by the ai's how they go and this is drawn from an independent distribution than 99 plus percent of the time the ai's have to not be doing stuff that's like downloading you know the hackers guide to escaping open ai servers or inducing obvious Trojan vulnerabilities

and that's actually quite a different dynamic than human law enforcement so if human law enforcement would catch one in a million crimes then that wouldn't influence the behavior of people committing crimes at all think of they expect odd this is very unlikely to affect me

with ai if you're able to draw a hundred samples or a thousand samples even if there are billions of ai instances and you're only looking at a small proportion of them when you run gradient descent on the samples you derive you're going to change the ai so that whenever humans look at it

it's delivering a good result and so that's just like quantitatively a very different functional form for how law enforcement works on ai when you can do these independent samples than it would for you know policing say like a human population that has a pre-existing

psychology that isn't being changed by these observations could we when we do gradient descent on the samples we draw all the all of the next generation of the ai models has been altered in that way my picture of like aligned subhuman AI to this like the superhuman AI being aligned

is still murky if we can talk about that more concretely yeah and I do want to clarify so where Elias are claims that were something like 95% 90% or more yeah maybe likely to be killed in a ai takeover um I think that probably won't happen and later I can maybe give them more

excellence breakdown of why um but I think it's a shockingly high risk um and so depending on the day I might say I might say one in four one in five uh that we get an ai takeover um that see yet sees control of the future uh makes a much worse world than we otherwise would have had

um and with like a big chance that we're all um killed in the process hey everybody I hope you enjoyed that episode as always the most helpful thing you can do is just share the podcast then did to people you think might enjoy it put it in twitter your group chats etc just splits the world appreciate your listening I'll see you next time cheers