But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions. And also pass human performance. If you're based and you're only smart enough, you just ask it like, what would it person be like? Great insight and wisdom and capability to do. Okay, today I have the pleasure of interviewing Ilya Sutskever, who is the co-founder and chief scientist of OpenAI. Ilya, welcome to the Lunar Society.
Thank you, happy to be here. First question and no humility allowed. There's many scientists, or maybe not that many scientists, who will make a big breakthrough in their field. There's far fewer scientists who will make multiple independent breakthroughs that define their field through their career. What is the difference? What distinguishes you from other researchers? Why have you been able to make multiple breakthroughs in their field?
Well, thank you for the kind words. It's hard to answer that question. I mean, I try really hard. I gave it everything you got. And that worked so far. I think that's all there is to it. Got it. What's the explanations for why there aren't more illicit uses of GPT? Why aren't more foreign governments using it to spread propaganda or scam grandmothers or something?
I mean, maybe they've really gotten to do it a lot. But it also wouldn't surprise me if Summer Week was going on right now. Certainly I imagine that we'd be taking some of the open source models and trying to use them for that purpose. Like I sure I would expect this would be something that would be interested in the future.
It's like technically possible that you just haven't thought about it enough or haven't like done it at scale using their technology. Or maybe it's happening with you as an elite. Would you be able to track it if it was happening? I think large scale track is possible. Yes. I mean, it's requires of all special operation is possible. Now, there's some window in which AI is very economically valuable on the scale of airplanes. Let's say what we haven't reached agi yet. How big is that window?
I mean, I think this window is hard to give you a precise answer, but it's definitely going to be like a good multi-year window. It's also a question of definition because AI before it becomes agi is going to be increasingly more valuable year after year.
I'd say in an exponential way. So it's some but in some sense it may feel like, especially in hindsight, it may feel like there was only one year or two years because those two years were larger than the previous years. But I would say that already. Last year, they've been a fair amount of economic value produced by AI next year is going to be larger and larger after that. So I think that this is going to be a good multi multi year chunk of time.
What that's going to be true. I would say from now to the agi pretty much. Okay, well, because I'm curious if there's a startup that's using your models, right. At some point if you have a GI, there's only one business in the world, right. It's open AI. How much window do they have? Is any business have where they're actually producing something that agi can produce? Yeah, well, I mean, it's the same. It's the same question is asking how long until agi.
Yeah, I think it's a hard question to answer. I mean, I hesitate to give you a number also because there is this thing where effect where people who are optimistic people who are working on the technology tend to underestimate the time it takes to get there. But I think that the way I ground myself is by thinking about a self driving car in particular, there is an analogy where if you look at the.
So if a Tesla and if you look at the self driving behavior of it, it's like it looks like it does everything. It does everything. But it's also clear that there is still a long way to go in terms of reliability. And we might be in a similar place with respect to our models where it also looks like we can do everything.
And at the same time, it will be we'll need to do some more work until we really iron out all the issues and make it really good and really reliable and robust and well behaved by 2030. What percent of GDP is AI? Oh gosh, hard to answer that question. Very hard to answer the question. Give me an over under. Like the problem is with my error bars and lock scale. So I could imagine like I could imagine like a huge percentage. I could imagine a little disappointed small percent. The same.
Okay, so let's take the counterfactual where it is a small percentage. Let's say it's 2030 and you know, not that much economic value has been created by these elements. My best explanation. So I really don't think that's a likely possibility. Yeah. So that's that's the preface to the end to the comment. But if I were to take the premise of your question, well, like why were things disappointing in terms of real world impact?
My answer would be reliability. If somehow it ends up being the case that you really want them to be reliable and then it have not been reliable or reliability now to be harder than we expect. I really don't think that will be the case. But if I had to pick one. If I had to pick one and you tell me like, hey, like why didn't things work out? It would be reliability that you still have to look over the answer is and double check everything.
That's just really what's the damper on the economic value that can be used by those systems. They'll be taking logically matured. It's just a question of whether it will be reliable enough. Yeah, well, in some sense, not reliable means not technological maturity. See what I mean. Yeah, fair enough. What's after generative models, right? So before you're working on reinforcement learning. Is this is this basically it? Is this a paradigm that gets us to a GI or is there something after this?
I mean, I think this paradigm is going to go really, really far that would not underestimate. I think it's quite likely that this exact paradigm is not going to be the quiet age. I form factor. I mean, I hesitate to say precisely what the next paradigm will be. But I think it will probably involve integration of all the different ideas that came with came in the past. Is there some specific one you're referring to or?
I mean, it's hard to be specific. So you could argue that next token prediction can only help us match human performance. And maybe not surpass it. What would it take to surpass human performance? So I challenge the claim that next token prediction cannot surpass human performance. It looks like on the surface it cannot. It looks on the surface if you just learn to imitate.
To predict what people do, it means that you can only copy people. But the here is a controversial argument for white and it might not be quite so if your neural net is if you're basically on it is smart enough. You just ask it like, what would it what would a person with great insight and wisdom and capability do? Maybe such person doesn't exist. But there's a pretty good chance of the neural net will be able to extrapolate.
How such a person would behave. Do you see what I mean? Yes, although where would it get the sort of insight about what that person would do if not from. From the data of regular people because like if you think about it, what does it mean to predict the next token well enough? What does it mean actually? It's actually it's a much it's a deeper question than it seems predicting the next token well means that you understand. The underlying reality that led to the creation of the token.
It's not statistics like it is statistics, but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this statistics. And so then you say, okay, well, I have all those people. What is it about people that creates their behaviors? Well, they have, you know, they have thoughts and they have feelings and ideas and they do things in certain ways. All of those would be deduced from next token prediction.
And I'd argue that this should make it possible not indefinitely, but to a pretty decent degree to say, well, can you guess what you do if you took a person with like this characteristic and that characteristic? Like such a person doesn't exist. But because you're so good at predicting the next token, you should still be able to guess what that person would do this hypothetical imaginary person. These are great their mental ability. Then the rest of us.
When we're doing reinforcement learning on these models, how long before most of the data for the reinforcement learning is coming from AI's and not humans? I mean already most of the different reinforcement learning is coming from the eyes. Yeah, well, it's like the humans are being used to train the reward function. But then the but then the reward function.
Inter and its interaction with the model is automatic and all the data that's generated in the during the process of reinforcement learning is created by AI. Like if you look at the current. I would say the Nick paradigm, which is in getting some significant attention because of chat GPT reinforcement learning from human feedback. But with human feedback, the human feedback is being used to train the reward function.
And then the reward function is being used to create the data which trains them all. Got it. And is there any hope of just removing the human from the loop and have it improve itself and some sort of alpha go away? Yeah, definitely. I mean, I feel like in some sense our hopes for like our plan like very much so the thing you really want is for the human teachers that tell you that teach the AI for them to collaborate with an AI.
You might want to think about it. And you might want to think of it as being in a world where the human teachers do 1% of the world and the work and the AI do 99% of the work. You don't want it to be 100% AI, but you do want it to be a human machine collaboration which teaches the next machine. Currently, I mean, I have a chance to play around these models. They seem bad at multi-step reasoning and they have been getting better. But what does it take to really surpass that barrier?
I mean, I think dedicated training will get us there more improvements to the base models who get us there. But like fundamentally, I also don't feel like they're that bad at multi-step reasoning. I actually think that they're bad at mental multi-step reasoning, but they're not allowed to think out loud. But when they are allowed to think out loud, they're quite good. And I expect this to improve significantly both with better models and be special training.
Are you running out of reasoning tokens? Are there enough of them? I mean, you know, it's okay. So for context on this question, there are claims that indeed at some point we'll run out of tokens in general, the trained those models.
And yeah, I think this will happen one day and we'll, by the time that happens, we need to have other ways of training models, other ways of productively improving their capabilities and sharpening their behavior, making sure they're doing exactly precisely what we want without more data. You haven't run out of data yet. There's more. Yeah, I would say the data situation is still quite good. There are still lots to go. But at some point, yeah, at some point data will run.
Okay, what is the most valuable source of data? Is it read it to the books? What would you train many other tokens about the varieties for? Generally speaking, you'd like tokens which are speaking about smarter things, don't come to charge like more interesting. Yeah. So I mean, all this all the sources which you mentioned available. Okay, so maybe not Twitter, but do we need to go multi models to get more tokens or do we still have enough text tokens left?
I mean, I think that you can still go very far in text only, but going multi models seems like a very good direction. If you're comfortable talking about this, like, where is the place where we haven't scraped the tokens yet? Oh, I mean, yeah, obviously, I mean, I can't answer that question for us, but I'm sure I'm sure that for everyone, there's a different answer to that question.
How many orders of magnitude improvement can we get just not from scale or not from data, but just from algorithm improvements? Hard to answer, but I'm sure there is some. It is some a lot or is so a little. But only one way to find out. Okay. Let me get to your like quick fire opinions about these different research directions retrieval transformers. So just like somehow storing the data outside of the model itself and retrieving it somehow.
Seems promising. What do you see that as a path forwarder? I think it seems promising robotics was the right step for opening eye to leave that behind. Yeah, it was back then. It really wasn't possible to continue working on robotics because there was so little data like back then, if you wanted to do on robot, if you wanted to work on robotics, you needed to become a robotics company, you needed to really have a giant group of people working on building robots and maintaining them and having.
And even then, like if you only if you want to have 100 robots, it's a giant operation is already, but you're not going to get that much data. So in a world where most of the progress comes from the combination of compute and data, right, that's where we've been where it was the combination of compute and data that drove the progress. There was no path to data from robotics. So back in the day, then you made a decision to stop working on robotics. There was no path forward. Is there one now?
So I'd say that now it is possible to create a path forward, but one needs to really commit to the task of robotics. You really need to say, I'm going to build like many thousands, tens of thousands, hundreds of thousands of robots and somehow collect data from them and find a gradual path where the robots are doing something slightly more useful and then the data that they get from these rope and then the data that is obtained and used to train the models need to something slightly more useful.
Because you could imagine is kind of gradual path of improvement, where you build more robots, they do more things, you collect more data and so on, but you really need to be committed to this path.
If you say, I want to make robotics happen, that's what you need to do. I believe that there are companies who are thinking about such doing exactly that, but I think that you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all. I think one could make progress in robotics today with enough motivation.
What ideas are you excited to try, but you can't because they don't work well on current hardware. I don't think current hardware is limitation. I think it's just not the case. Got it. But anything you want to try, you can just spin it up. Of course. The thing you might say, well, I wish current hardware was cheaper or maybe it had higher, maybe it would be better if it was higher memory process bandwidth, let's say. But by and large hardware is just a limitation.
Let's talk about alignment. Do you think we'll ever have a mathematical definition of alignment with medical definition of things unlikely? I do think that we will instead have multiple, but rather than achieving one mathematical definition, I think we'll achieve multiple definitions that look at alignment from different aspects. And I think that this is how we will get the assurance that we want.
And by which I mean you can look at the behavior. You can look at the behavior in various tests, the control aims in various adversarial stress situations. You can look at how the neural net operates from the inside. I think you have to look at all several of these factors at the same time. And how short do you have to be before you release a model in the wild? 100% 95%. But it depends on how capable the model is.
The more capable the model is, the more the higher the Z, the more confident to be. Okay. So just say it's something that's almost AGI. Where is AGI? Well, depends what you're a jack and do keep in mind that AGI is an ambiguous term also like your average college undergrad is an AGI, right? You see what I mean? There is significantly bigger in terms of what is meant by AGI. So depending on where you put this mark, you need to be more or less confident.
Well, you mentioned a few of the paths towards alignment earlier. What is the one you think is most promising at this point? Like I think that it will be a combination really think that you will not want to have just one approach. I think people want to have a combination of approaches where we you spend a lot of compute, but we're certainly probably to find any mismatch between the behavior you want it to teach and the behavior that exhibits.
We look inside into the neural net using another neural to understand how it operates on the inside. I think all of them will be necessary every approach like this reduces the probability of misalignment. And you also want to be in a world where you're. The degree of alignment keeps of increasing faster than the capability of the models. I would say that right now our understanding of our models is still quite rudimentary.
We made some progress, but much more progress is possible. And so I would expect that ultimately the thing that we'll really succeed is when we will have a small neural net that is well understood. That's given the task to study the behavior of a large neural net that is not understood to verify. By what point is mostly I research being done by I mean so today when you use copilot right what fraction how do you do the how do you divide it up.
So I expect at some point you ask your you know the senate of Chad G. Peter you say hey like I'm thinking about this and this can you suggest. Foodful ideas I should try and you would actually get foodful ideas. I don't think that's we will make it possible for you to solve problems you couldn't solve before. Got it, but it's somehow just telling the human skipping them ideas faster or something.
It's not itself interacting with the one example. I mean you could you could slice it in a variety of ways. But I think the bottom of the air is what idea is good insights and that's something but the neural net could help with these. If you design some like a billion dollar prize for some sort of alignment research result or product what is the concrete creature in your set for that billion dollar price. There's something that makes sense for such a price.
It's funny that you asked this I was actually thinking about this exact question. I haven't I haven't come up with an exact criterion yet maybe something that be the benefit maybe a prize were. We could say that two years later or three or five years later we look back and say like that was the main result. So rather than say that there is a price committed that decides right away you wait for five years and then award it retractively.
But there's no concrete thing we can identify yet as it like you solve this particular problem and you're you made a lot of progress. I think a lot of progress yet so I wouldn't say that this would be the. The full thing. Do you think end to end training is the right architecture for bigger and bigger models or do we need better ways of just connecting things together. I think end to end train is very promising a thing connected mixed together is a promise. Everything is promising.
The opening is projecting revenues of a billion dollars in 2024 that might very well be correct but I'm just curious when you're talking about a new general for psychology. Do you estimate how they went follow it will be like that but why that particular number. I mean you look at the current you look at the cut you know we've already had a beef so we've had a product.
For quite a while now from back from the GPT three days from two years ago through the API and we've seen how it grew we've seen how. The response to Dali has grown as well and so you see how the response to chat GPT's and I think all of this gives us information that allows us to make. And so you can see how the response to the relatively sensible extrapolations of 2024.
Maybe that would be that would be one answer like you need to have a data you can't come up with those things out of thin air because otherwise your error bars will be like. By your earbuds are going to be like 100x in each direction. I mean the most exponentials don't stay exponential especially when they get into bigger and their quantities right so I how do you determine in this case that. Like would you bet against the eye. I'm not after talking with you.
Let's talk about what like a post age I future looks like. Are people like you you know I'm guessing you're working like 80 hour weeks towards this grand goal that you really assess what are you going to be satisfied in a world where you're basically living in an AI retirement home or like what is a word what is your what are you.
What are you doing after a G I come so very tricky question you know I think where where will people find meaning but I think I think that that's something that AI could help us be. Like. One thing I imagine is that we'll all be able to become more enlightened because we interact with an AGI this will help us see the world more correctly become better on the inside as a result of interact like imagine talking to the best meditation teacher in history.
I think that would be a helpful thing but I also think that because the world will change a lot it will be very hard for people to understand. What is happening precisely and how to go and how to really contribute one thing that I think some people will choose to do is to become part AI in order to really expand their minds and understanding to really be able to solve the hardest problems that society will face then are you going to come party. Very tempting it is tempting you.
You think they'll be physically embodied humans and 3000 3000 how do I know it's going to have an 3000 like what what is it look like are there so like humans walking around on earth or every guest I can't really about what you actually want to look like 3000 well I mean that that that the thing is here's the thing like let me describe to you what I think is not quite right about the question like it implies like oh like we get to decide how we want the world to look like.
I don't think that picture is correct I think change these don't be constant and so of course even after a G.I. built it doesn't mean that the world will be static the world will continue to change the world will continue to evolve. And it will go through all kinds of transformations and I really have no I don't think anyone has any idea of how the world will look like in 3000 but.
I do hope that there will be a lot of descendants of human beings who believe happy fulfilled lives for their free to do as their wish as they see fit where they are the ones who are. Solving their own problems like one of the things which I would not want one one one one world which I would find very exciting is one where you know you will this powerful tool and then the government said okay so.
The A.G.I. said that society should be running such a way and now we should run society in such a way at much rather have a world where people are still free to make their own mistakes and suffer their consequences and gradually evolve morally and progress forward on their own strength see what I mean with the A.G.I. providing more like a base safety net how much time you spend hanging out with this kinds of things versus just doing the research that.
I do think about those things a fair bit yeah things are very just questions. So it in what ways have the capabilities we have today in what ways have these are passed were expected them to be in 2015 and what ways are they still not where you're going to start the movie.
By this point I mean in fairness that it's so that expect to be in many things in 2015 I my thinking was a lot more I just don't want to bet against deep learning I want to make the biggest possible bet on deep learning don't know how but it will figure it out. Is there any specific way in which it's been more than expected or less than expected. I think concrete prediction you added 2015 that's been.
Frounced. You know unfortunately I don't remember concrete predictions I made in 2015 but I definitely but I definitely think that overall in 2015 I just want to move to make the biggest bet possible on deep learning but I didn't know exactly didn't have a specific idea of how far things will go. In seven years well I mean 2015 I did have all these best with people into any 16 maybe 2017 that things will go really far.
But specifics so it's like it's both it's both the case that it surprised me and I was making these aggressive predictions but I think maybe I believe them only only only 50% on the inside. Well what do you believe now that even most people at open AI would find farfetched. I mean I think that at this because we communicate a lot of the open AI people have a pretty good sense of what I think and so yeah we've reached the point of open AI with the thing we see I do and all these questions.
So Google has you know it's custom TPU hardware it has all this data from all the users you know Gmail what and so on. Does it given an advantage in terms of training bigger models and better models than you. So I think like when the first first when the TPU came out I was really impressed and I thought wow this is amazing but that's because I didn't quite understand hardware back then. What really turned out to be the case is that TPUs and GPUs are almost the same thing.
They are very very similar it's like I think a GPU chip is a little bit bigger I think a TPU chip is a little bit smaller it may be a little bit cheaper but then they make more GPUs than TPUs. So I think the GPUs might be cheaper after all but fundamentally you have a big processor and you have a lot of memory and there is a bottleneck between those two.
And the problem that both the TPU and the GPU are trying to solve is that by the amount of time it takes you to move one floating point from the memory to the processor you can do several hundred floating point operations on the processor which means that you have to do some kind of batch processing. And in this sense both of these architectures are the same so I really feel like hardware like in some sense the only thing that matters about hardware is cost cost per flop overall systems cost.
Okay and there is a much that much different. Well actually don't know I mean I don't know how much what the GPU costs are but I would suspect that probably not if anything probably views are more expensive because there is less of them. So I'm doing your work how much of the time it's been you know configuring the right in socializations making sure the training run goes well and getting the right hyper parameters and how much is it just coming up with whole new ideas.
I would say it's a combination but I think that coming up with it's a combination but coming up with whole new ideas is actually not it's it's like a modest part of the work certainly coming up in new ideas is important but I think even more important is to understand the results to understand the existing ideas to understand what's going on.
Normally you have these you know neural is a very complicated system right and you ran it and you get some behavior which is hard to understand what's going. Understanding the results figuring out what next experiment to run a lot of the time spent on that understanding what could be wrong what could have caused the system is the neural net produce a result which was not expected.
I'd say a lot of time is spent as well of course coming up in new ideas but not new ideas I think like I don't I don't like this this and. Raining as much it's not that it's false but I think the main activity is actually understanding. How do you see the difference between the two so at least in my mind when you say come up with new ideas I'm like oh like what happened if it is such and such.
Where is understanding it's more like like what is this whole thing you like what are the real underlying. The nominant that are going on what are the underlying effects like why why are we doing things this way not another way and of course this is very adjacent to what can be described as coming up with ideas but I think the understanding part is where the real action takes place.
Does that describe your entire career like everything back on like image net or something was that more new idea or was I more understanding i was definitely understand. It was a new understanding of very old things. What is the experience of training on Azure been like using as a fantastic I mean yeah Microsoft has been a very very good partner for us and they've really. Helped take Azure and make it bring it to a point where it's really good for a male.
And they're super happy with it how how vulnerable is a whole ecosystem do something that might happen in Taiwan so let's say there's like a tsunami in in Taiwan or something what what happens to you in general like it's definitely going to be a significant set back.
It's not going to like it might be something equivalent to like no one will be able to get more more compute for a few years but expect computers will spring up like for example I believe that Intel has fabs just of the previous year of like a few generations ago. That means that if Intel wanted to they could produce something GPU like from like four years ago.
But yeah it's not the best let's say i'm actually not sure about if if if if my statement about Intel is correct but I do know that there are fabs outside of Taiwan that is not as good. But you can still use them and still go very far with them it's just. It just cost it's just a setback what inference could cost for have it as these models get bigger and bigger so I have a different way of looking at this question yeah it's not that inference will become cost prohibitive.
Inference of better models will indeed become more expensive. What is it prohibitive well it depends on how useful is it like if it is more useful than it is expensive than it is not prohibited like to give you an analogy like suppose you want to talk to a lawyer you have some case you or need some advice or something you're perfectly happy to spend for $100 an hour.
So if your neural net could give you like really reliable legal advice you'd say i'm happy to spend $400 for that advice and suddenly inference becomes very much non prohibitive. The question is can can neural net produce an answer good enough at this cost. Yes and you will just have different like price discrimination different yeah different models of the problem is already the case today so on our product.
The API is sort of multiple neural nets of different sizes and different customers use different neural nets of different sizes depending on their use case. Like if someone can take a small model and fine tune it and get something that satisfactory for them they'll use that yeah but if someone wants to do something more complicated and more interesting there is the biggest model.
How do you prevent these models from just becoming commodities where these different companies just they just pay the shutters prices down until it's basically the cost of the GPU run. I think there is without question a force that's trying to create that and the answer is you got to keep on making progress you got to keep improving the models you got to keep on coming up with new ideas and making models better and more reliable more trustworthy so you can trust their answers all those things.
I think it's like 2025 and the model from 2024 somebody just offering it a cost and it's like so pretty good why would people use a new one from 2025 if the one from just a year old there is you know even better. So there are several answers there for some use cases that may be true there will be a new model for 25 which would be driving the more interesting use cases there's also going to be a question of inference cost like you can.
So you have to try to serve the same model at less cost so that will be different the same model will be served will cost different some different amounts to serve. For different companies I can also imagine some degree of specialization to where some companies may try to specialize in some area and be stronger in an error area compared to other companies and I think that too may. That may be a response to commoditization to some degree.
So over time do these different companies do their research directions converge with their diverge are they doing similar and similar things over time are they doing are they going up branching off in the different areas. So that's in the near term it looks like these conversions in the like I expect this going to be a convergence a divergence convergence behavior where there is a lot of convergence on the near term work.
There's going to be some divergence on the longer term work but then once the longer term work starts to yield through that I think they will be conversions again. Got it. One of them finds the most promising area they everybody just that's right now there is obviously less less publishing now so it will take longer before this promising direction gets to discovered that's how I imagine the thing is going to be convergence the versions convergence.
Yeah we talked about this a little bit at the beginning but you know as foreign governments learn about how capable these models are how do you are you worried about spies or some sort of attack to get your weights or you know somehow abuse these models and learn about them. Yeah it's definitely something that you absolutely can discount that. And yeah something that we might guard against the best of our ability but it's going to be a problem for everyone who is building this.
How do you prevent your weights from leaking or what I mean you have like really good security people and like how many people have the if they wanted to just like ask the agent to the weights. How many people could do that. I mean like what I can say is that the security people that we have the built to they've done a really good job so that I'm really not worried about the way to be leaked. What kinds of emerging properties are expecting from these models at this scale.
Is there something that just comes about day novo. I'm sure things will come I'm sure really new surprising properties would come up I would not be surprised the thing which I'm really excited about or the thing we should like to see is reliability and controllability. I think that this will be very very important class of emerging properties.
If you have reliability and controllability I think that helps you solve a lot of problems reliability means you can trust the models out with controllability means you can control it. And we'll see what it will be very cool if those emergent properties did exist. Is there somewhere you can predict it advance. Like what will happen in this for American world have an ever America.
It's possible to make some predictions about specific specific capabilities though it's definitely not simple and you can do it in a super fine grain way at least today. But I think getting better at that is really important than anyone who is interested in who has research ideas on how to do that I think that can be a valuable contribution. How seriously do you take these scaling laws if like there's a paper that says like oh you just increase.
You need this many orders and I need to get more to get all the reasoning out like do you take that seriously or do you think it breaks down at some point. Well, the thing is that the scaling not tells you what happens as you what happens to your look to your next word prediction accuracy right. There is a whole separate challenge of linking next word prediction accuracy to reasoning capability.
I do believe that indeed the reason link but this link is completely and we may find that there are other things that can give us more reasoning pre unit effort. Like for example some special look you know you mentioned reasoning tokens and I think they can be helpful. It can be there can be probably some things. Is this is something you're considering just hiring humans to generate tokens for you or is it all going to come from that already exists out there.
I mean I think that relying on people to teach our models to do things. Especially you know to make sure that they are well behaved and they don't produce false things I think is an extremely sensible thing to do. Isn't it odd that we have the data we need exactly the same time as we have the transformer at the exact same time that we have these GPUs like is it odd to you that all these things happen at the same time or do not see that way. I mean it is definitely an interesting.
It is an interesting situation that is the case I will say that it is odd and it is less odd on some level here is why it's less odd. What is the driving force behind the fact that the data exists that the GPUs exist the different former exists so.
The data exists because computers became better and cheaper we've got smaller and smaller transistors and suddenly at some point it became economical for every person to have a personal computer once everyone has a personal computer you really want to connect the internet.
Once you have the internet you have suddenly data appearing in great quantities the GPUs were improving concurrently because you have more small and smaller transistors and you are looking for things to do with them gaming turned out to be the thing that you could do and then at some point the gaming GPU and video said wait a second Ryan.
It turned it into a general purpose GPU computer maybe someone will find it will find it useful turns out it's good for neural nets so it could it could have been the case that maybe the GPU would have arrived five years later or 10 years later if what let's suppose gaming wasn't the same it's kind of hard to imagine what does it mean if gaming isn't the same.
But it could maybe there was a counterfactual world where GPUs arrived five years after the data five years before the data in which case maybe things would move a little bit worse things would have been as ready to go as they are now. But that's the picture which I imagine all this progress in all these dimensions is very intertwined it's not a coincidence that.
Like you don't get to pick and choose which dimension in which dimensions things improve if you see what I mean how inevitable is this kind of progress so if like let's say you and Jeffry and then a few other pioneers if they were never born this is the deep learning revolution happen around the same time how much does it delay.
I think maybe there would have been some delay maybe like your delays it's really hard to tell it's really hard to tell I mean I hesitate to give a lot a lot a longer answer because okay well then you'd have GPU. You'd have GPUs would keep on improving right then at some point I cannot see how someone would not have discovered it was here's the other thing is it if okay so let's suppose no one is done computer skip getting faster and better becomes easy and easy to train these neural nets.
Because you have bigger GPUs so it takes less engineering effort train one you don't need to optimize your code as much you know when the when the image net data set came out it was huge and it was very very difficult to use.
Imagine wait for a few years and it becomes very download and people can just just think her so I would imagine that like a modest number of years maxing this would be my guess I hesitate I hesitate to give it to give a lot a longer answer though you know you can't you can't run you can't rerun the world you don't know let's go back to alignment for a second as somebody who deeply understands these models what is your intuition of how hard alignment will be like I think we the cause so here's what I would say.
I think we the current level of capabilities I think we have a pretty good set of ideas of how to align them but I would not underestimate the difficulty of alignment of models that are actually smarter than us of models that are capable of misrepresenting their intentions.
Like I think I think it's something to to think to think about a lot into research I think this is one area also by the way you know like oftentimes academic researchers asked me ask me where what's the best place where they can contribute. And I think alignment research is one place where I think academic researchers can make very many contributions.
I believe that do you think academia will come up with an insight about actual capabilities or is that going to be just the companies at this point. The companies will realize the capabilities I think it's very possible for academic research to come up with those insights I think it's just it doesn't seem to happen that much for some reason but I don't I don't think there's anything fundamental about academia.
Like it's not like academia can't I think maybe they're just not thinking about the right problems or something because maybe it's just easier to see what needs to be done inside these companies. I see but there's a possibility that somebody could just realize yeah I don't know these things like why would I possibly rule this out. What are the concrete steps by which these language models are actually impacting the world of atoms and not just the world of bits.
Well, you see I don't think that there is a distinction clean distinction between the world of bits and the world of atoms. Suppose the neural net tells you that hey like here is like something that you should do and it's going to improve your life but you need to like rearrange your apartment in a certain way. You go and rearrange your apartment as a result. The neural net impact the world of atoms just fair enough fair enough.
Do you think it'll take a couple of additional breakters as important as a transformer they get to super human AI or do you think we basically got the insights in the books somewhere and we just need to implement them and connect them. So I don't really see such a big distinction in those two cases and let me explain why. Like I think what's what one of the ways in which progress is taken place in the past is that we've understood that something had a property.
A desirable property all along but you didn't realize. So is that a breakthrough you can say yes it is is that an implementation of something on the books also yes. My feeling is that a few of those are quite likely to happen but that in hindsight it will not feel like a breakthrough. Everybody is going to say oh well of course like it's totally obvious that such and such thing can.
And work you see with a transformer the reason it's been brought up as a big as a specific advance is because it's the kind of thing that was not obvious or almost anyone. So we look and say yeah like it's not something which they knew about. But if an advance comes from something like let's consider that is the most fundamental advance of deep learning that the big neural network trained with back propagation and do a lot of things like where is the novelty. It's not in the neural network.
It's not in the back propagation. But then somehow it's the kind of but it was it is most definitely a giant conceptual breakthrough because for the longest time people just didn't see that. But then now that everyone sees that I was going to say well of course like it's totally obvious big neural net. Everyone knows that they can do it. What is your opinion of your former advisors? New forward forward algorithm? I think that it's an attempt to brain a neural network without back propagation.
And I think that this is especially interesting if you are motivated to try to understand how the brain might be learning its connections. The reason for that is that as far as I know neuroscientists are really convinced that the brain cannot implement back propagation because the signals in the synopsis only moving one direction. And so if you have a neuroscience motivation and you want to say okay.
How can I come up with something that tries to approximate the good properties of back propagation without doing back propagation. That's what the forward forward algorithm is trying to do. But if you are trying to just engineer a good system, there is no reason to not use back propagation. Like it's the only algorithm. I guess I've heard you in different contexts talk about the like using humans as the existing example case that you know, AGI exists right.
At what point do you take the metaphor less seriously and feel they don't feel the need to pursue it in terms of research. Because it is important to you as a sort of existence case. Like at what point does stop caring about humans as an existence case of intelligence. Or as the sort of as an example in the model you want to follow in terms of pursuing intelligence in models. I see. I mean, like you got a I think it's good to be inspired by humans. I think it's good to be inspired by the brain.
I think there is an art into being inspired by humans in the brain correctly. Because it's very easy to latch on to an non essential quality of humans or of the brain. And I think many people who wants who many people whose research is trying to be inspired by humans and by the brain often gets a little bit specific. People get a little bit too. But cognitive science model should follow at the same time consider the idea of the neural network itself the idea of their artificial neuron.
This too is inspired by the brain, but it turned out to be extremely fruitful. So how do you do this? What behaviors of human beings are essential that you say like this is something that proves to us that it's possible. What is in essential? No, actually this is like some emergent phenomena of something more basic. And we just need to focus on our own on doing getting our own basics right.
I would say I would say that it's like I think one should one can and should be inspired by human intelligence with care. Final question. Why is there in your case such a strong correlation between being first to the deep learning revolution and still being one of the top researchers? You would think that these two things wouldn't be that correlated. Why is that the correlation?
I don't think those things are super correlated indeed. I feel like in my case, I mean honestly it's hard to answer the question. You know, I just kept on. I kept trying really hard and it turned out to have suffice thus far. So it's a perseverance. I think it's a necessary but not a sufficient condition. Like you know many things need to come together in order to really figure something out. Like you need to really go for it and also need to have the right way of looking at things.
So it's hard to give them like a really meaningful answer to this question. All right. Ilya, it has been a true pleasure. Thank you so much for coming out of the lunar society. I appreciate you bringing us to the offices. Thank you. Yeah, I really enjoyed it. Thank you very much. Hey, everybody. I hope you enjoyed that episode. Just wanted to let you know that in order to help pay for the bills associated with this podcast. I'm turning on pay subscriptions on my set stack at warcashbattell.com.
No important content on this podcast will ever be pay-walled. So please don't donate if you have to think twice before buying a cup of coffee. But if you have the means and you've enjoyed this podcast or grind some kind of value out of it, I would really appreciate your support. As always, the most helpful thing you can do is just share the podcast. Then to people you think might enjoy it, put it in Twitter, your group chats, etc. Just blitz the world. Appreciate your listening.
I'll see you next time. Cheers.