The following is a conversation with Dario Amodei, CEO of Anthropic, the company that created Claude, that is currently and often at the top of most LLM benchmark leader boards. On top of that, Dario and the Anthropic team have been outspoken advocates for taking the topic of AI safety very seriously, and they have continued to publish a lot of fascinating AI research on this and other topics.
I'm also joined afterwards by two other brilliant people from Anthropic. First, Amanda Asco, who is a researcher working on alignment and fine tuning of Claude, including the design of Claude's character and personality. A few folks told me she has probably talked with Claude more than any human at Anthropic. So she was definitely a fascinating person to talk to about prompt engineering and practical advice on how to get the best out of Claude. After that,
Chris Ola, stopped by a fur chat. He's one of the pioneers of the field of Mechanistic Interpretyability, which is an exciting set of efforts that aims to reverse engineer neural networks to figure out what's going on inside, inferring behaviors from neural activation patterns inside the network. This is a very promising approach for keeping future super intelligent AI systems safe. For example, by detecting from the activations when the model is trying to deceive the human,
it is talking to. Now a quick few second mention of the sponsor. Check them out in the description. It's the best way to support this podcast. We got on-cord for machine learning, notion for machine learning, power, note taking and team collaboration. Shopify for selling stuff online, better help for your mind and element for your health. Choose lies in my friends. Also, if you want to work with our amazing team, we just want to get in touch with me for whatever reason,
go to lexfrieben.com slash contact. Now, onto the full ad reads. I try to make these interesting, but if you skip them, please still check out our sponsors. I enjoy their stuff. Maybe you will too. This episode is brought to you by on-cord. A platform that provides data-focused AI tooling for data annotation, curation and management and for model evaluation. We talk a little bit about public benchmarks in this podcast. I think mostly focused on software engineering, sweet bench.
There's a lot of exciting developments about how do you have a benchmark that you can't cheat on. But if it's not public, then you can use it the right way, which is to evaluate how well is the annotation, the data curation, the training, the pre-training, the post-training, all of that. How's that working? Anyway, a lot of the fascinating conversation with the anthropic folks was focused on the language side. There's a lot of really incredible work that on-cord is doing about
annotating and organizing visual data. They make it accessible for searching, for visualizing, for granular curation and all that kind of stuff. I'm a big fan of data. It continues to be the most important thing. The nature of data, what it means to be good data, whether it's human-generated or synthetic data, keeps changing, but it continues to be the most important component of what makes for a generally intelligent system, I think, and also for specialized intelligent systems as well.
Go try out on-cord to curate, annotate, and manage your AI data at encord.com slash Lex. That's encord.com slash Lex. This episode is brought to you by the thing that keeps getting better and better and better notion. It used to be an awesome note-taking tool. Then it started being a great team collaboration. Note-taking for many people and management of all kinds of other project stuff across large teams. Now, more and more and more is becoming a AI super-powered note-taking and
team collaboration tool. Really integrating AI probably better than any note-taking tool I've used, not even close, honestly. Notion is truly incredible. I haven't gotten a chance to use an ocean on a large team. I imagine that that's really when it begins to shine, but on a small
team, it's just really, really, really amazing. The integration of the AI assistant inside a particular file for summarization, for generation, all that kind of stuff, but also the integration of an AI assistant to be able to ask questions about across docs, across Wikis, across projects, across multiple files to be able to summarize everything, maybe investigate project progress based on all the different stuff going on in different files. So, really, really nice integration of AI.
Try Notion AI for free when you go to Notion.com slash Lex. That's all lowercase Notion.com slash Lex. To try the power of Notion AI today. This episode is also brought to you by Shopify, a platform designed for anyone to sell anywhere with a great looking online store. I keep wanting to mention Shopify's CEO Toby, who's brilliant, and I'm not sure why he hasn't been on the podcast yet. I need to figure that out. Every time I'm in San Francisco, I want to talk to him.
So, he's brilliant on all kinds of domains, not just entrepreneurship or tech, just philosophy and life, just his way of being. Plus, an accent adds to the flavor profile of the conversations. I've been watching a cooking show for a little bit. Really, I think my first cooking show, it's called Class Wars. It's a South Korean show where chefs with Michelin stars compete against
chefs with all Michelin stars. And there's something about one of the judges that just the charisma and the way they describe every single detail, a flavor, a texture of what makes for a good dish. Yeah, so it's contagious. I don't really even care. I'm not a foodie. I don't care about food in that way, but he makes me want to care. So, anyway, that's why he's the term flavor profile. Referring to Toby, which has nothing to do with what I should probably be saying. And that is
that you should use Shopify. I've used Shopify. Super easy, create a store, lexroman.com slash store through the cell of few shirts. Anyway, sign up for a $1 per month trial period at Shopify.com slash lex. That's all lowercase. Go to Shopify.com slash lex to take your business to the next level today. This episode is also brought to you by BetterHelp, spelled H-E-L-P-H-H-H-H-L-P. They figure out what
you need and match it with a licensed therapist in under 48 hours. It's for individuals. It's for couples. It's easy to create affordable, available worldwide. I saw a few books by a young in psychologist. And I was like in a delirious state of sleepiness and I forgot the right of his name down, but I need to do some research. I need to go back. I need to go back to my younger self when I dream of being a psychiatrist and reading, I'm going to Freud and reading Carl Young,
reading it the way young kids maybe read comic books. They were my superheroes of sorts. Kamu as well, Kafka, Nietzsche, Hesse, Dostoyevsky, the sort of 19th and 20th century literary philosophers of sorts. Anyway, I need to go back to that. Maybe have a few conversations about Freud. Anyway, those folks, even if in part wrong or true revolutionaries, were truly brave to explore the mind and the way they did, they showed the power of talking and delving deep
into the human mind, into the shadow through the use of words. So highly recommend and that better help is a super easy way to start. Check them out at betterhelp.com slashlex and saving you first month. That's better help.com slashlex. This episode is also brought to you by Element, my daily zero sugar and delicious electrolyte mix that I'm going to take a sip of now. It's been so long that I've been drinking element that I don't even remember life before element.
I guess I used to take salt pills because it's such a big component of my exercise routine to make sure I get enough water and get enough electrolytes. Yeah. So combined with the fast thing that I've explored a lot and continued to do to this day and combined with low carb diets that I'm a little bit off the wagon on that one. I'm consuming probably like 60, 70, 80, maybe 100 some days grams of
carbohydrates. Not good, not good. My happiest is when I'm below 20 grams or 10 grams of carbohydrates. I'm not like measuring it out. I'm just using numbers to sound smart. But I don't take dieting seriously, but I do take the signals that my body sends quite seriously. So without question,
making sure I get enough magnesium and sodium and get enough water is priceless. A lot of times when I have headaches, which is felt off or whatever, were fixed nearly immediately and sometimes after 30 minutes, which is drinking water with the electrolytes is beautiful and is delicious. Watermelon salt, the greatest flavor of all time. Get a sample pack for free with any purchase. Try it at drinkelement.com slash Lex. This is a Lex Freeman podcast to support it.
Please check out our sponsors in the description. And now dear friends, here's Dario, Amade. Let's start with a big idea of scaling laws and the scaling hypothesis. What is it? What is it's history and what was that today? So I can only describe it as it relates to kind of my own experience, but I've been in the AI field for about 10 years. And it was something I noticed very early on. So I first joined the AI world when I was working at Baidu with Andrew Ng in late 2014,
which is almost exactly 10 years ago now. And the first thing we worked on was speech recognition systems. And in those days, I think deep learning was a new thing. It had made lots of progress. But everyone was always saying we don't have the algorithms we need to succeed. We're not, we're only matching a tiny, tiny fraction. There's so much we need to kind of discover algorithmically. We haven't found the picture of how to match the human brain. And when, you know, in some ways,
it was fortunate I was kind of, you know, you can have almost beginners luck, right? I was like a new comer to the field. And you know, I looked at the neural net that we were using for speech, the recurrent neural networks. And I said, I don't know, what if you make them bigger and give them more layers? And what if you scale up the data along with this, right? I just saw these as like
independent dials that you could turn. And I noticed that the model started to do better and better as you gave them more data as you as you made the models larger as you trained them for longer. And I didn't measure things precisely in those days. But, but along with, with colleagues, we very much got the informal sense that the more data and the more compute and the more training you put into these models, the better they perform. And so initially my thinking was, hey,
maybe that is just true for speech recognition systems, right? Maybe, maybe that's just one particular quirk one particular area. I think it wasn't until 2017 when I first saw the results from GPT one that it clicked for me that language is probably the area in which we can do this. We can get trillions of words of language data. We can train on them. And the models we were trained in those days were tiny. You could train them on one to eight GPUs. Whereas, you know, now we train
jobs on tens of thousands soon going to hundreds of thousands of GPUs. And so when I when I saw those two things together, and you know, there were a few people like Ilya Sutskiver who you've interviewed who had somewhat similar views, right? He might have been the first one though. I think a few people came to came to similar views around the same time, right? There was, you know,
Rich Sutton's bitter lesson. There was Goren wrote about the scaling hypothesis. But I think somewhere between 2014 and 2017 was when it really clicked for me when I really got conviction that, hey, we're going to be able to do these incredibly wide cognitive tasks if we just if we just scale up the models. And at every stage of scaling, they're always arguments. And you know, when I first heard them, honestly, I thought, probably I'm the one who's wrong. And you know, all these all
these experts in the field are right. They know the situation better better than I do right. There's you know, the Chomsky argument about like you can get syntactics, but you can't get semantics. There was this idea, you can make a sentence, make sense, but you can't make a paragraph, make sense. The latest one we have today is, you know, we're going to run out of data or the data isn't high quality enough for models can't reason. And each time every time we manage to we manage to either
find a way around or scaling just is the way around. Sometimes it's one, sometimes it's the other. And so I'm now at this point, I still think, you know, it's it's it's always quite uncertain. We have nothing but inductive inference to tell us that the next few years are going to be like the next the last 10 years. But but I've seen I've seen the movie enough times. I've seen the story happen for enough times to really believe that probably the scaling is going to continue.
And that there's some magic to it that we haven't really explained on a theoretical basis yet. And of course, the scaling here is bigger networks, bigger data, bigger compute. Yes, in particular linear scaling up of bigger networks, bigger training times, and more and more data. So all of these things, almost like a chemical reaction, you know, you have three ingredients in the chemical reaction and you need to linearly scale up the three ingredients. If you scale up one
not the others, you run out of the other reagents and the and the reaction stops. But if you scale up everything everything in series, then then the reaction can proceed. And of course, now that you have this kind of empirical science slash art, you can apply to other more nuanced things like scaling laws applied to interpretability or scaling laws applied to
post training or just seeing how does this thing scale? But the big scaling law, I guess the underlying scaling hypothesis has to do with big networks, big data leads to intelligence. Yeah, we've documented scaling laws in lots of domains other than language, right? So initially, the paper we did that first showed it was in early 2020, where we first showed it for language. There was then some work late in 2020, where we showed the same thing for other modalities,
like images, video, text image, image to text, math, they all had the same pattern. And you're right. Now, there are other stages like post training or there are new types of reasoning models. And in all of those cases that we've measured, we see similar types of scaling laws. A bit of a philosophical question, but what's your intuition about why bigger is better in terms of network size and data size? Why does it lead to more intelligent models?
So in my previous career as a biophysicist, so I did physics undergrad and then biophysics in grad school. So I think back to what I know as a physicist, which is actually much less than what some of my colleagues at Anthropic have in terms of expertise in physics, there's this concept called the one over ethnoise and one over x distributions, where often, just like if you add up a bunch of
natural processes, you get a Gaussian, if you add up a bunch of kind of differently distributed natural processes, if you like, if you like, take a probe and hook it up to a resistor, the distribution of the thermal noise in the resistor goes as one over the frequency. It's some kind of natural convergent distribution. And I think what it amounts to is that if you look at a lot of things that are produced by some natural process that has a lot of different scales,
right? Not a Gaussian, which is kind of narrowly distributed, but if I look at kind of like large and small fluctuations that lead to electrical noise, they have this decaying one over x distribution.
And so now I think of like patterns in the physical world, right? If I, or in language, if I think about the patterns in language, there are some really simple patterns, some words are much more common than others, like the, then there's basic noun verb structure, then there's the fact that, you know, nouns and verbs have to agree, they have to coordinate, and there's the higher level set in structure, then there's the thematic structure of paragraphs.
And so the fact that there's this regressing structure, you can imagine that as you make the networks larger, first they capture the really simple correlations, the really simple patterns, and there's this long tail of other patterns. And if that long tail of other patterns is really smooth, like it is with the one over f noise in, you know, physical processes like, like, like, like resistors, then you can imagine as you make the network larger, it's kind of capturing more and more of that
distribution. And so that smoothness gets reflected in how well the models are at predicting that how well they perform. Language isn't evolved process, right? We've developed language, we have common words and less common words, we have common expressions and less common expressions, we have ideas who shays that are expressed frequently, and we have novel ideas. And that process
has developed, has evolved with humans over millions of years. And so the, the, the guess, and this is pure speculation would be, would be that there is, there's some kind of long tail of distribution of, of the distribution of these ideas. So there's the long tail, but also there's the height of the hierarchy of concepts that you're building up. So the bigger the network, presumably you have a higher capacity to, exactly. If you have a small network, you only get the
common stuff, right? If I take a tiny neural network, it's very good at understanding that, you know, a sentence has to have, you know, a verb adjective noun, right? But it's, it's terrible at deciding what those verb adjective and noun should be and whether they should make sense. If I make it just a little bigger, it gets good at that. Then suddenly it's good at the sentences, but it's not good at the paragraphs. And so these, these, these rare and more complex patterns
get picked up as I add, as I add more capacity to the network. Well, the natural question then is, what's the ceiling of this? Yeah. How complicated and complex is the real world? How much does this stuff is there to learn? I don't think any of us knows the answer to that question. My strong instinct would be that there's no ceiling below the level of humans, right? We humans are able to
understand these various patterns. And so that, that makes me think that if we continue to, you know, scale up these, these, these models to kind of develop new methods for training them and scaling them up, that will at least get to the level that we've gotten to with humans. There's then a question of, you know, how much more is it possible to understand than humans do? How much, how much is it possible to be smarter and more perceptive than humans? I, I would guess the answer has, has got
to be domain dependent. If I look at an area like biology and, you know, I wrote this essay, Machines of Loving Grace, it seems to me that humans are struggling to understand the complexity of biology, right? If you go to Stanford or Harvard or to Berkeley, you have whole departments of, you know, folks trying to study, you know, like the immune system or metabolic pathways. And, each person understands only a tiny bit part of it, specializes, and they're struggling to
combine their knowledge with that of, with that of other humans. And so I have an instinct that there's, there's a lot of room at the top for AIs to get smarter. If I think of something like materials in the, in the physical world or, you know, like addressing, you know, conflicts between humans or something like that, I mean, you know, it may be there's only, some of these problems are not intractable but much harder. And, and it may be that there's only, there's only so well you can
do at some of these things, right? Just like with speech recognition, there's only so clear I can hear your speech. So I think in some areas, there may be ceilings and, you know, that are very close to what humans have done in other areas, those ceilings may be very far away. And I think we'll only find out when we build these systems. There's, it's very hard to know in advance. We can speculate but we can't be sure. And in some domains, the ceiling might have to do with
human bureaucracies and things like this as you write about. Yes. So humans fundamentally have to be part of the loop. That's the cause of the ceiling, not maybe the limits of the intelligence. Yeah. I think in many cases, you know, in theory, technology could change very fast, for example, all the things that we might invent with respect to biology. But remember, there's, there's a, you know, there's a clinical trial system that we have to go through to actually administer
these things to humans. I think that's a mixture of things that are unnecessary and bureaucratic and things that kind of protect the integrity of society and the whole challenge is that it's hard to tell, it's hard to tell what's going on. It's hard to tell which is which, right? My, my view is definitely, I think in terms of drug development, my view is that we're too slow and we're too conservative. But certainly, if you get these things wrong, you know, it's, it's possible to, to,
to risk people's lives by, by being, by being, by being too reckless. And so at least, at least, some of these human institutions are in fact, protecting people. So it's, it's all about finding the balance. I strongly suspect that balance is kind of more on the side of pushing to make things happen faster, but there is a balance. If we do hit a limit, if we do hit a slowdown in the scaling laws, what do you think would be the reason? Is it compute limited, data limited?
Is it something else? I deal limited. So a few things now we're talking about hitting the limit before we get to the level of, of humans and the skill of humans. So, so I think one that's, you know, one that's popular today, and I think, you know, could be a limit that we run into. I, like most of the limits, I would bet against it, but it's definitely possible. Is we simply run out of data? There's only so much data on the internet. And there's issues with the quality of
the data, right? You can get hundreds of trillions of words on the internet, but a lot of it is, is repetitive or it's search engine, you know, search engine optimization, drivel, or maybe in the future it'll even be text generated by AIs itself. And so I think there are limits to what to what can be produced in this way. That said, we, and I would guess other companies, are working on ways to make data synthetic, where you can, you know, you can use the model to generate more
data of the type that you have that you have already, or even generate data from scratch. If you think about what was done with deep mines, AlphaGo Zero, they managed to get a bot all the way from, you know, no ability to play Go whatsoever to above human level just by playing against itself. There was no example data from humans required in the AlphaGo Zero version of it. The other direction, of course, is these reasoning models that do chain of thought and stop to think
and reflect on their own thinking. In a way, that's another kind of synthetic data coupled with reinforcement learning. So my, my guess is with one of those methods, we'll get around the data limitation, or there may be other sources of data that are, that are available. We could just observe that even if there's no problem with data, as we start to scale models up, they just stop getting better. It's, it's seen to be our, our reliable observation that they've gotten better.
That could just stop at some point for reason we don't understand. The answer could be that we need to, you know, we need to invent some new architecture. It's been, there have been problems in the past with, say, numerical stability of models where it looked like things were, were leveling off, but, but actually, you know, when we, when we found the right on block or they didn't end up doing so. So perhaps there's new, some new optimization method or some new technique we need to,
on block things. I've seen no evidence of that so far, but if things were to, to slow down, that perhaps could be one reason. What about the limits of compute, meaning the expensive nature of building bigger and bigger data centers? So right now, I think, you know, most of the frontier model companies I would guess are operating, you know, roughly, you know, one billion dollar scale plus or minus a factor of three, right? Those are the models that exist now or are being
trained now. I think next year we're going to go to a few billion and then 2026, we may go to, you know, above 10, 10 billion and probably by 2027, their ambitions to build 100, 100 billion dollar, 100 billion dollar clusters. And I think all of that actually will happen. There's a lot of determination to build the compute to do it within this country. And I would guess that it actually
does happen. Now, if we get to 100 billion, that's still not enough compute, that's still not enough scale, then either we need even more scale or we need to develop some way of doing it more efficiently, of shifting the curve. I think between all of these, one of the reasons I'm bullish about powerfully, I happening so fast is just that if you extrapolate the next few points on the curve,
we're very quickly getting towards human level ability, right? Some of the new models that, that we developed, some some reasoning models that have come from other companies, they're starting to get to what I would call the PhD or professional level, right? If you look at their their coding ability, the latest model we released, Sonnet 3.5, the new updated version, it gets something like 50% on sweet bench and sweet bench is an example of a bunch of professional real world software
engineering tasks at the beginning of the year. I think the state of the art was three or four percent. So in 10 months, we've gone from 3% to 50% on this task and I think in another year, we'll probably be at 90%, I mean, I don't know, but might even be, might even be less than that. We've seen similar things in graduate level math, physics and biology from models like OpenAI's 01. So if we, if we just
continue to extrapolate this, right? In terms of skill, skill that we have, I think if we extrapolate the straight curve within a few years, we will get to these models being above the highest professional level in terms of humans. Now, will that curve continue? You pointed to and I've pointed to a lot of reasons, you know, possible reasons why that might not happen. But if the extrapolation curve continues, that is the trajectory we're on. So Anthropic has several competitors.
It'd be interesting to get your sort of view of it all. OpenAI, Google, XAI, Meta, what does it take to win in the broad sense of win in the space? Yeah. So I want to separate out a couple things, right? So, you know, andthropics, andthropics mission is to kind of try to make this all go well, right? And, you know, we have a theory of change called Race to the Top, right? Race to the Top is about trying to push the other players to do the right thing by setting an example. It's not
about being the good guy. It's about setting things up so that all of us can be the good guy. I'll give a few examples of this. Early in the history of Anthropic, one of our co-founders, Chris Ola, who I believe you're interviewing soon, you know, he's the co-founder of the field of Mechanistic Interpretyability, which is an attempt to understand what's going on inside AI models. So we had him and one of our early teams focus on this area of interpretability, which we think
is good for making models safe and transparent. For three or four years, that had no commercial application whatsoever. It still doesn't today. We're doing some early betas with it, and probably it will eventually, but, you know, this is a very, very long research bet in one in which we've built in public and shared our results publicly. And we did this because, you know, we think it's a way to make models safer. An interesting thing is that as we've done this, other companies
have started doing it as well. In some cases, because they've been inspired by it, in some cases, because they're worried that, you know, if other companies are doing this that look more responsible, they want to look more responsible, too. No one wants to look like the irresponsible actor. And so they adopt this, they adopt this as well, when folks come to Anthropic, interpretability is often a draw, and I tell them, the other places you didn't go, tell them why
you came here. And then you see soon that there's interpretability teams elsewhere as well. And in a way, that takes away our competitive advantage because it's like, oh, now others are doing it as well, but it's good for the broader system. And so we have to invent some new thing that we're doing that others aren't doing as well. And the hope is to basically bid up the importance of doing the right thing. And it's not about us in particular, right? It's not about having
one particular good guy. Other companies can do this as well. If they join the race to do this, that's the best news ever, right? It's just it's about kind of shaping the incentives to point upward instead of shaping the incentives to point to point downward. And we should say this example of the field of mechanistic interpretability is just a rigorous nonhand wavy way of doing AI safety. Yeah. Or it's tending that way. Trying to. I mean, I think we're still early in terms of
our ability to see things. But I've been surprised at how much we've been able to look inside these systems and understand what we see, right? Unlike with the scaling laws where it feels like there's some law that's driving these models to perform better. On the inside, the models aren't, you know, there's no reason why they should be designed for us to understand them, right? They're designed to operate. They're designed to work just like the human brain or human biochemistry.
They're not designed for a human to open up the hatch, look inside and understand them. But we have found, and you know, you can talk in much more detail about this to Chris, that when we open them up, when we do look inside them, we find things that are surprisingly interesting. And as a side effect, you also get to see the beauty of these models, you get to explore the sort of beautiful nature of large neural networks through the mech and turb kind of methodology. I'm amazed at
how clean it's been. I'm amazed at things like induction heads. I'm amazed at things like, you know, that we can, you know, use sparse auto encoders to find these directions within the networks and that the directions correspond to these very clear concepts. We demonstrated this a bit with the Golden Gate Bridge Quad. So this was an experiment where we found a direction inside one of the neural networks layers that corresponded to the Golden Gate Bridge. And we just turned that
way up. And so we released this model as a demo. It was kind of half a joke for a couple days, but it was illustrative of the method we developed. And you could take the Golden Gate, you could take the model, you could ask it about anything, you know, it would be like how you could say, how is your day? Anything you asked because this feature was activated, it would connect to the
Golden Gate Bridge. So it would say, you know, I'm feeling relaxed and expansive, much like the arches of the Golden Gate Bridge or, you know, it would masterfully change topic to the Golden Gate Bridge and it integrated. There's also a sadness to it to the focus ahead on the Golden Gate Bridge. I think people quickly found love with it. I think so people already miss it because it was
taken down I think after a day. Somehow these interventions on the model, where you kind of adjusted behavior somehow emotionally made it seem more human than any other version of the model. Strong personality, strong identity. It has strong personality. It has these kind of like obsessive interests. You know, we can all think of someone who's like obsessed with something. So it does make it feel somehow a bit more human. Let's talk about the present. Let's talk
about Claude. So this year a lot has happened in March, Claude III, Opus, Sonnet, Hikou, were released then Claude III, five, Sonnet in July with an updated version just now released, and then also Claude III, five, Hikou was released. Okay. Can you explain the difference between Opus, Sonnet, and Hikou and how we should think about the different versions?
Yeah. So let's go back to March when we first released these three models. So, you know, our thinking was different companies produce kind of large and small models, better and worse models. We felt that there was demand both for a really powerful model. You know, and that might be a little bit slower that you'd have to pay more for. And also for fast cheap
models that are as smart as they can be for how fast and cheap. Whenever you want to do some kind of like difficult analysis, like if I want to write code for instance, or I want to bring store my ideas or I want to do creative writing, I want the really powerful model. But then there's a lot of practical applications in a business sense where it's like I'm interacting with a website.
I'm like doing my taxes or talking to a legal advisor and I want to analyze a contract or we have plenty of companies that are just like, I want to do auto-complete on my IDE or something. For all of those things, you want to act fast and you want to use the model very broadly. So we wanted to serve that whole spectrum of needs. So we ended up with this kind of poetry theme
and so what's a really short poem? It's a Haiku. And so Haiku is the small, fast cheap model that is, you know, was at the time was really surprisingly, surprisingly intelligent for how fast and cheap it was. Sonnet is a medium-sized poem, right? A couple paragraphs. And so Sonnet was the middle model. It is smarter, but also a little bit slower, a little bit more expensive. And an opus, like a magnum,
opus is a large work. Opus was the largest, smartest model at the time. So that was the original kind of thinking behind it. And our thinking then was, well, each new generation of models should shift that trade-off curve. So when we release Sonnet 3.5, it has the same, roughly the same, you know, cost and speed as the Sonnet 3 model. But it increased its intelligence to the point where it was smarter than the original opus 3 model, especially for code, but also just in general.
And so now, you know, we've shown results for Haiku 3.5. And I believe Haiku 3.5, the smallest new model, is about as good as opus 3, the largest old model. So basically, the aim here is to shift the curve. And then at some point, there's going to be an opus 3.5. Now, every new generation of models has its own thing. They use new data. Their personality changes in ways that we kind of, you know, try to steer, but are not fully able to steer. And so there's never quite that exact
equivalence where the only thing you're changing is intelligence. We always try and improve other things and some things change without us, without us knowing or measuring. So it's very much in exact science. In many ways, the manner and personality of these models is more in art than it is the science. So what is sort of the reason for the span of time between say, cloud opus 3.0 and 3.5? What takes that time, if you can speak to? Yeah. So there's different processes.
There's pre-training, which is just kind of the normal language model training. And that takes a very long time. That uses these days tens of thousands, sometimes many tens of thousands of GPUs or TPUs or training, or we use different platforms, but accelerator chips. Often, training for months. There's then a kind of post-training phase where we do reinforcement learning from human feedback, as well as other kinds of reinforcement learning that that phase is getting
larger and larger now. And, you know, often that's less of an exact science. It often takes effort to get it right. Models are then tested with some of our early partners to see how good they are. And they're then tested both internally and externally for their safety, particularly for catastrophic and autonomy risks. So we do internal testing, according to our responsible scaling
policy, which I, you know, could talk more about that in detail. And then we have an agreement with the US and the UK AI Safety Institute, as well as other third-party testers in specific domains to test the models for what are called CBRN risks, chemical, biological, radiological, and nuclear, which are, you know, we don't think that models pose these risks seriously yet, but every new model we want to evaluate to see if we're starting to get close to some of these more dangerous
these more dangerous capabilities. So those are the phases. And then, you know, then it just takes some time to get the model working in terms of inference and launching it in the API. So there's just a lot of steps to actually to actually making a model work. And of course, you know, we're always
trying to make the processes as streamlined as possible, right? We want our safety testing to be rigorous, but we want it to be rigorous and to be, you know, to be automatic to happen as fast as it can without compromising on rigor, same with our pre-training process and our post-training process. So, you know, it's just like building anything else. It's just like building airplanes. You want to make them, you know, you want to make them safe, but you want to make the process
streamlined. And I think the creative tension between those is, you know, is an important thing in making the models work. Yeah, rumor on the street, I forget who was saying that, Anthropic is really good tooling. So I probably a lot of the challenge here is on the software engineer side is to build the tooling to have like a efficient low friction interaction with the
infrastructure. You would be surprised how much of the challenges of, you know, building these models comes down to, you know, software engineering, performance engineering, you know, you, you know, from the outside, you might think, oh man, we had this eureka breakthrough, right? You know, this movie with the science, we discovered it. We figured it out, but, but I think, I think all things, even, even, you know, incredible discoveries like they almost always come down to
the details. And often super, super boring details. I can't speak to whether we have better tooling than other companies. I mean, you know, haven't been at those other companies at least, at least, not recently. But it's certainly something we give a lot of attention to. I don't know if you can say, but from three, from cloud three to cloud three, five, is there any extra pre training going on as they mostly focus on the post training? There's been leaps in performance. Yeah, I think,
I think at any given stage, we're focused on improving everything at once. Just, just naturally, like they're different teams, each team makes progress in a particular area in, in, in making a particular, you know, their particular segment of the relay race better. And it's just natural that when we make a new model, we put, we put all of these things in at once. So the data you have, like the preference data you get from RLHF, is that applicable? Is there a waste to apply it
to newer models as you get trained up? Yeah, preference data from old models sometimes gets used for new models. Although, of course, it, it performs somewhat better when it's, you know, trained on, it's trained on the new models. Note that we have this, you know, constitutional AI method such that we don't only use preference data, we kind of, there's also a post training process where we train the model against itself. And there's, you know, new types of post training, the model against
itself that are used every day. So it's not just RLHF, it's a bunch of other methods as well. Post training, I think, you know, is becoming more and more sophisticated. Well, what explains the big leap in performance for the new Sonnet 35? I mean, at least in the programming side. And maybe this is a good place to talk about benchmarks. What doesn't mean to get better? Just the number one up. But, you know, I, I, I program, but I also love programming and I
clawed 35 through cursors, what I use to assist me in programming. And there was, at least experientially and, anecdotally, it's gotten smarter at programming. So what, like, what, what does it take to get it to get it smarter? We observe that as well, by the way. There were a couple of very strong engineers here at Anthropic who all previous code models, both produced by us and produced by all the other companies hadn't really been useful to, hadn't really been
useful to them. You know, they said, you know, maybe, maybe this is useful to beginner, it's not useful to me. But Sonnet 3.5, the original one for the first time, they said, oh my god, this helped me with something that, you know, that it would have taken me hours to do. This is the first model that's actually saved me time. So again, the water line is rising. And, and then I think, you know, the new Sonnet has been, has been even better. In terms of what it, what it takes, I mean,
I'll just say it's been across the board. It's in the pre training, it's in the post training, it's in various evaluations that we do. We've observed this as well. And if we go into the details of the benchmarks, so, sweet benches basically, you know, since, since, you know, since you are a programmer, you know, you'll be familiar with like poll requests and, you know, just, just poll requests are like, you know, they like a sort of a sort of atomic unit of work, you know,
you could say, I'm, you know, I'm implementing one, I'm implementing one thing. And, and so, sweet bench actually gives you kind of a real world situation where the code basis is in the current state. And I'm trying to implement something that's, you know, that's described in, described in language. We have internal benchmarks where we measure the same thing. And you say, just give the model free reign to like, you know, do anything run, run, run anything, edit anything.
How, how well is it able to complete these tasks? And it's that benchmark that's gone from it, can do it 3% of the time to it can do it about 50% of the time. So I actually do believe that if we, you know, you can gain benchmarks, but I think if we get to 100% of that benchmark in a way that isn't kind of like over trained or, or, or game for that particular benchmark, probably represents a real and serious increase in kind of, in kind of programming, programming ability.
And I would suspect that if we can get to, you know, 90, 90, 95% that that, that, that, you know, it will, it will represent ability to autonomously do a significant fraction of software engineering tasks. Well, ridiculous timeline question. When is Gladopus 3.5 coming out? Not giving an exact date, but, you know, they're, they're, you know, as far as we know, the plan is still to have a clawed 3.5 opus. Are we going to get it before GTA 6 or no?
Like Duke Nukem forever? So, was that game that there was some game that was delayed 15 years? Was that Duke Nukem forever? Yeah. And I think GTA is not just releasing trailers. It, you know, it's only been three months since we released the first sonnet. Yeah, it's incredible, the incredible pace of release. It just, it just tells you about the pace. Yeah. The expectations for one thing is you're going to come out. So, what about 4.0? So,
how do you think about sort of as these models get bigger and bigger about versioning? And also, just versioning in general, why sonnet 3.5 updated with the date? Why not sonnet 3.6? It's actually naming is actually an interesting challenge here, right? Because I think a year ago, most of the model was pre-training. And so, you could start from the beginning and just say,
okay, we're going to have models of different sizes. We're going to train them all together and, you know, we'll have a family of naming schemes and then we'll put some new magic into them and then, you know, we'll have the next, the next generation. The trouble starts already when some of them take a lot longer than others to train, right? That already messes up your time, time a
little bit. But as you make big improvements in as you make big improvements in pre-training, then you suddenly notice, oh, I can make better pre-trained model and that doesn't take very long to do. And, but, you know, clearly it has the same, you know, size and shape of previous models. So, I think those two together as well as the timing, timing issues, any kind of scheme you come up with, you know, the reality tends to kind of frustrate that scheme, right? It tends to kind of
break out of the, break out of the scheme. It's not like software, you can say, oh, this is like, you know, 3.7, this is 3.8. No, you have models with different, different trade-offs. You can change some things in your models, you can train, you can change other things, some are faster and slower at inference, some have to be more expensive, some have to be less expensive. And so, I think all the companies have struggled with this. I think we did very, you know, I think we were in a good,
good position in terms of naming when we had Haiku, Sonnet and Oks. Great start. We're trying to maintain it, but it's not, it's not, it's not perfect. So, we'll, we'll try and get back to the simplicity, but it, it, it, just the nature of the field, I feel like no one's figured out naming. It's somehow a different paradigm from like normal software. And so, we just, none of the
companies have been perfect at it. I mean, something we struggle with, surprisingly much relative to, you know, how, relative to how trivial it is, to, you know, for the, the, the, the, the grand science of training the models. So, from the user side, the user experience of the updated Sonnet 3.5 is just different than the previous June 2024, Sonnet 3.5. It would be nice to come up with some kind of labeling that embodies that because people talk about Sonnet 3.5, but now there's a
different one. And so, how do you refer to the previous one and the new one? And it, it, uh, when there's a distinct improvement, it just makes conversation about it, uh, just challenging. Yeah. Yeah. I, I definitely think this question of there are lots of properties of the models that are not reflected in the benchmarks. Um, I, I think, I think that's, that's definitely the case and everyone agrees. And not all of them are capabilities. Some of them are, you know,
models can be polite or brusque. They can be, uh, you know, uh, very reactive or they can ask you questions. Um, they can have what, what feels like a warm personality or a cold personality. They can be boring or they can be very distinctive like Golden Gate Claude was. Um, and we have a whole, you know, we have a whole team kind of focused on, I think we call it Claude character, uh, Amanda leads that team and we'll, we'll talk to you about that. But it's still a very inexact science.
And often we find that models have properties that we're not aware of. The, the fact of the matter is that you can, you know, talk to a model 10,000 times and there are some behaviors you might not see, uh, just like, just like with a human, right? I can know someone for a few months and, you know, not know that they have a certain skill or not know that there's a certain side to
them. And so I think, I think we just have to get used to this idea and we're always looking for better ways of testing our models to, to demonstrate these capabilities and, and also to decide, which are, which are the, which are the personality properties we want models to have and which we don't want to have that itself. The normative question is also super interesting. I got to ask a
question from Reddit. From Reddit. Oh boy. You know, there, there's just a fascinating to me at least it's a psychological social phenomenon where people report that clot has gotten dumber for them over time. And so, uh, the question is, does the user complaint about the dumbing down of clot three, five sonnet hold any water? So are these anecdotal reports? It kind of social phenomena or did clot, is there any cases where clot would get dumber? So, uh, this actually doesn't apply,
this, this isn't just about clot. I, I believe this, I believe I've seen these complaints for every foundation model produced by a major company. Um, people said this about GPT four, they said it about GPT four turbo. Um, so, so, so, so a couple things. Um, one, the actual weights of the model, right?
The actual brain of the model, that does not change unless we introduce a new model. Um, there, there are just a number of reasons why it would not make sense practically to be randomly substituting in, substituting in new versions of the model. It's difficult from an inference perspective. And it's actually hard to control all the consequences of changing the weights of the model. Let's see, you want to define tune the model to be like, I don't know, to like, to say certainly less,
which, you know, an old version of Sonnet used to do. Um, you actually end up changing 100 things as well. So we have a whole process for it. And we have a whole process for modifying the model. We do a bunch of testing on it. We do a bunch of, um, like, we do a bunch of user testing and early customers. So it, we both have never changed the weights of the model without without telling anyone. And it wouldn't, certainly in the current setup, it would not make sense to do that. Now,
there are a couple of things that we do occasionally do. Um, one is sometimes we run AB tests. Um, but those are typically very close to when a model is being, is being released and for a very small fraction of time. Um, so, uh, you know, like the, you know, the, the, the day before the new Sonnet 3.5, I, I agree. We should have better name. It's clunky to refer to it. Um, there were some comments from people that like, it's got, it's got, it's gotten a lot better. And that's because, you know,
fraction were exposed to, to an AB test for, for those one or, for those one or two days. Um, the other is that occasionally the system prompt will change. Um, on the system prompt can have some effects, although it's, it, it's unlikely to dumb down models. It's unlikely to make them dumber. Um, and, and, and, and we've seen that while these two things, which I'm listing to be very complete, um, happen relatively, happen quite infrequently. Um, the complaints about, uh, for us and
for other model companies about the model change, the model isn't good at this. The model got more censored. The model was dumbed down. Those complaints are constant. And so I don't want to say like, people are imagining or anything, but like the models are for the most part not changing. Um, if I were to offer a theory, um, I, I think it actually relates to one of the things I said before,
which is that models have many are very complex and have many aspects to them. And so often, you know, if I, if I, if I asked a model a question, you know, if I'm like, if I'm like, do task acts versus can you do task acts? The model might respond in different ways. Uh, and, and so there are all kinds of subtle things that you can change about the way you interact with the model
that can give you very different results. Um, to be clear, this, this itself is like a failing by, by us and by the other model providers that, that the models are just, just often sensitive to like small, small changes in word. It's yet another way in which the science of how these models work is very poorly developed. Uh, and, and so you know, if I go to sleep one night and I was like talking the model in a certain way, and I like slightly change the phrasing of how I talk to the model,
you know, I could, I could get different results. So that's, that's one possible way. The other thing is, man, it's just hard to quantify this stuff. Uh, it's hard to quantify this stuff. I think people are very excited by new models when they come out. And then as time goes on, they, they become very aware of the, they become very aware of the limitations. So that may be another effect. But that's, that's all a very long, rendered way of saying, for the most part
with some fairly narrow exceptions, the models are not changing. I think there is a psychological effect. You just start getting used to it. The baseline raises like when people have first gotten Wi-Fi on airplanes, it's like a maze, like a magic. Yeah. And then, and then you start, you can't get this thing to work. Yeah. This is such a piece of crap. Exactly. So then it's easy to have the conspiracy theory of they're making Wi-Fi slower and slower. This is probably
something I'll talk to Amanda much more about. But another Reddit question. One will claw stop trying to be my pure, tanical grandmother imposing its moral worldview on me as a pain customer. And also, what is the psychology behind making clawed overly apologetic? So this kind of reports about the experience, a different angle in the frustration has to do with the character.
Yeah. So a couple points on this first. One is, like, things that people say on Reddit and Twitter or X or whatever it is, there's actually a huge distribution shift between the stuff that people complain loudly about on social media and what actually kind of like, you know, statistically
users care about and that drives people to use the models. Like, people are frustrated with, you know, things like, you know, the model not writing out all the code or the model, you know, just not being as good at code as it could be even though it's the best model in the world on code. I think the majority of things are about that. But certainly a kind of vocal
minority are, you know, kind of kind of raise these concerns, right? Are frustrated by the model refusing things that it shouldn't refuse or like apologizing too much or just having these kind of like annoying verbal ticks. The second caveat, and I just want to say this like super clearly because I think it's like some people don't know it. Others like kind of know it, but forget it. Like, it is very difficult to control across the board how the model's behave. You cannot just reach
in there and say, oh, I want the model to like apologize less. Like, you can do that. You can include trading data that says like, oh, the model should like apologize less. But then in some other situation, they end up being like super rude or like overconfident in a way that's like misleading people. So they're all these trade-offs. For example, another thing is if there was a period during which models are and I think others as well were two verbose, right? They would like repeat
themselves. They would say too much. You can cut down on the verbosity by penalizing the models for just talking for too long. What happens when you do that if you do it in a crude way is when the models are coding, sometimes they'll say, rest of the code goes here, right? Because they've learned that that's a way to economize and that they see it. And then so that leaves the model to be
so called lazy in coding where they were there just like, ah, you can finish the rest of it. It's not because we want to save on compute or because the models are lazy and during winter break or any of the other kind of conspiracy theories that have come up. It's actually, it's just very hard to control the behavior of the model, to steer the behavior of the model in all circumstances at once.
You can kind of, there's this whack-a-mole aspect where you push on one thing and like, you know, these other things start to move as well that you may not even notice or measure. And so one of the reasons that I care so much about, you know, kind of grand alignment of the AI systems in the future is actually, these systems are actually quite unpredictable. They're actually quite hard to steer in control. And this version we're seeing today of, you make one
thing better, it makes another thing worse. I think that's like a present day analog of future control problems in AI systems that we can start to study today, right? I think that difficulty in in steering the behavior and in making sure that if we push an AI system in one direction, it doesn't push in another direction in some other ways that we didn't want. I think that's kind of an early sign of things to come. And if we can do a good job of solving this problem, right?
Of like, you ask the model to like, you know, to like make and distribute smallpox and it says, no, but it's willing to like help you in your graduate level of biology class. Like, how do we get both of those things at once? It's hard. It's very easy to go to one side or the other. And it's a multi-dimensional problem. And so, I think these questions of like shaping the model's personality, I think they're very hard. I think we haven't done perfectly on them. I think
we've actually done the best of all the AI companies, but still so far from perfect. And I think if we can get this right, if we can control the false positives and false negatives in this very kind of controlled, present day environment, we'll be much better at doing it for the future when our worry is, you know, will the models be super autonomous? Will they be able to, you know, make very dangerous things? Will they be able to autonomously, you know,
build whole companies? And are those companies aligned? So I think of this present task as both vaccine, but also good practice for the future. What's the current best way of gathering sort of user feedback? Like, not anecdotal data, but just large scale data about pain points or the opposite of pain points, positive things, so on. Is it internal testing? Is it specific group
testing, a B testing, what work? So, so so typically we'll have internal model bashings where all of anthropic and tropic is almost a thousand people, you know, people just just try and break the model. They try and interact with it various ways. We have a suite of evals for, you know,
always the model refusing in ways that it couldn't. I think we even had a certainly eval because, you know, our model, again, one point model had this problem where like it had this annoying tick where it would like respond to a wide range of questions by saying, certainly I can help you with that. Certainly I would be happy to do that. Certainly this is correct. And so we had a like certainly eval which is like how often does the model say certainly. But look, this is just a
whack-a-mole like what if it switches from certainly to definitely like. So, you know, every time we add a new eval and we're always evaluating for all the old things. So we have hundreds of these evaluations, but we find that there's no substitute for human interacting with it. And so it's very much like the ordinary product development process. We have like hundreds of people within Anthropic bash the model. Then we do, you know, then we do external AB tests. Sometimes we'll run
tests with contractors. We pay contractors to interact with the model. So you put all of these things together and it's still not perfect. You still see behaviors that you don't quite want to see, right? You know, you see, you still see the model like refusing things that it just doesn't make sense to refuse. But I think trying to try and to solve this challenge, right? Trying to stop the model from doing, you know, genuinely bad things that, you know, know what everyone agrees
it shouldn't do, right? You know, everyone, everyone, you know, everyone agrees that, you know, the model shouldn't talk about, you know, I don't know, child abuse material, right? Like everyone agrees the model shouldn't do that. But at the same time that it doesn't refuse in these dumb and stupid ways, I think I think drawing that line as finally as possible approaching perfectly is still a challenge and we're getting better at it every day. But there's, there's a lot to be
solved. And again, I would point to that as, as an indicator of a challenge ahead in terms of steering much more powerful models. Do you think Claude 4.0 is ever coming out? I don't want to commit to any naming scheme because if I say, if I say here, we're going to have Claude 4 next year. And then, you know, then we decide that like, you know, we should start over because there's a new type of model. Like I don't want to, I don't want to commit to it. I would expect in a
normal course of business that Claude 4 would come after Claude 3.5. But, but, you know, you never, you never know in this wacky field, right? But sort of this idea of scaling is continuing. Scaling is continuing. There will definitely be more powerful models coming from us than the models that exist today. That is, that is certain. Or if there, if there aren't, we've, we've deeply failed as a company. Okay. Can you explain the responsible scaling policy and the AI safety level
standards at cell levels? As much as I'm excited about the benefits of these models. And we know, we'll talk, talk about that if we talk about machines of loving grace. I'm, I'm worried about the risks. And I continue to be worried about the risks. No one should think that, you know, machines of loving grace was me, me saying, you know, I'm no longer worried about the risks of
these models. I think they're two sides of the same coin. The, the power of the models and their ability to solve all these problems in, you know, biology, neuroscience, economic development, government governance and peace, large parts of the economy, those, those come with risks as well, right? With great power comes great responsibility, right? That's the, the two are, the two are paired. Things that are powerful can do good things and they can do bad things. I think of those
risks as being in, you know, several different, different categories. Perhaps the two biggest risks that I think about and that's not to say that there aren't risks today that are, that are important. But when I think of the really, the, you know, the things that would happen on the grandest scale, one is what I call catastrophic misuse. These are misuse of the models in domains like cyber, bio, radiological, nuclear, right? Things that could, you know, that could harm or even kill
thousands, even millions of people if they really, really go wrong. Like these are the, you know, number one priority to prevent. And here, I would just make a simple observation, which is that the models, you know, if I look today at people who have done really bad things in the world, I think actually humanity has been protected by the fact that the overlap between really smart, well-educated people and people who want to do really horrific things has generally been small.
Like, you know, let's say, let's say I'm someone who, you know, you know, I have a PhD in this field, I have a well-paying job. There's so much to lose. Why do I want to like, you know, even to assume that I'm completely evil, which most people are not? Why, you know, why would such a person risk their, risk their, you know, risk their life, risk their legacy, their reputation to do something like, you know, truly, truly evil? If we had a lot more people like that, the world
would be a much more dangerous place. And so my, my, my worry is that by being a much more intelligent agent, AI could break that correlation. And so I do have serious worries about that. I believe we can prevent those worries. But, you know, I think as a counterpoint to machines of loving grace,
I want to say that this is, that I, there's still serious risks. And, and the second range of risks would be the autonomy risks, which is the idea that models might on their own, particularly as we give them more agency than they've had in the past, particularly as we give them supervision over wider tasks like, you know, writing whole code bases or someday even, you know, effectively operating entire, entire companies, they're on a long enough leash. Are they, are they doing
what we really want them to do? It's very difficult to even understand in detail what they're doing, let alone, let alone control it. And like I said, this, these early signs that it's, it's hard to perfectly draw the boundary between things the model should do and things the model shouldn't do, that, that, you know, if you go to one side, you get things that are annoying and useless and you go to the other side, you get other behaviors. If you fix one thing, it creates other problems.
We're getting better and better at solving this. I don't think this is an unsolvable problem. I think this is a, you know, this is a science like, like the safety of airplanes or the safety of cars or the safety of drugs. I, you know, I don't think there's any big thing we're missing. I just think we need to get better at controlling these models. And so these are, these are the two risks I'm worried about. And our responsible scaling plan, which all recognizes a very long-winded answer
to your question. Our responsible scaling plan is designed to address these two types of risks. And so every time we develop a new model, we basically test it for its ability to do both of these bad things. So if I were to back up a little bit, I think we have, I think we have an interesting dilemma with AI systems where they're not yet powerful enough to present these catastrophes. I don't know that I don't know they'll ever present these catastrophes as possible. They won't.
But the case for worry, the case for risk is strong enough that we should, we should act now. And they're getting better very, very fast, right? I, you know, I testified in the Senate that, you know, we might have serious bio risks within two to three years. That was about a year ago. Things have preceded, preceded a pace. So we have this thing where it's like, it's, it's, it's surprisingly hard to, to address these risks because they're not here today. They don't exist.
They're light ghosts, but they're coming at us so fast because the models are improving so fast. So how do you deal with something that's not here today doesn't exist, but is coming at us very fast? So the solution we came up with for that in collaboration with, you know, people like the organization Meader and Paul Cristiano is, okay, what, what, what, what you need for that are you need tests to tell you
when the risk is getting close. You need an early warning system. And, and so every time we have a new model, we test it for its capability to do these CBRN tasks as well as testing it for, you know, how capable it is of doing tasks autonomously on its own. And in the latest version of our RSP, which we released in the last, in the last month or two, the way we test autonomy risks is the model, the AI models ability to do aspects of AI research itself, which when the model, when the
AI models can do AI research, they become kind of truly truly autonomous. And that, you know, that threshold is important for a bunch of other ways. And, and so what do we then do with these tasks? The RSP basically develops what we've called an if then structure, which is if the models pass a certain capability, then we impose a certain set of safety and security requirements on them. So today's models are what's called ASL 2. Models that were ASL 1 is for systems that manifestly
don't pose any risk of autonomy or misuse. So for example, a chest plane bought deep blue would be ASL 1. It's just manifestly the case that you can't use deep blue for anything other than chest. It was just designed for chest. No one's going to use it to like, you know, to conduct a masterful cyber attack or to, you know, run wild and take over the world. ASL 2 is today's AI systems, where we've measured them and we think these systems are simply not smart enough to, you know,
autonomously self replicate or conduct a bunch of tasks. And also not smart enough to provide meaningful information about CBRN risks and how to build CBRN weapons above and beyond what can be known from looking at Google. In fact, sometimes they do provide information, but, but not above and beyond the search engine, but not in a way that can be stitched together, not in a way
that kind of end to end is dangerous enough. So ASL 3 is going to be the point at which the models are helpful enough to enhance the capabilities of non-state actors, right? State actors can already do a lot of, unfortunately, to a high level of proficiency, a lot of these very dangerous and destructive things. The difference is that non-state actors are not capable of it.
And so when we get to ASL 3, we'll take special security precautions designed to be sufficient to prevent theft of the model by non-state actors and misuse of the model as it's deployed. We'll have to have enhanced filters targeted at these particular areas. Cyber-Bio-Nuclear. Cyber-Bio-Nuclear and model autonomy, which is less of misuse risk and more risk of the model doing bad things itself.
ASL 4, getting to the point where these models could enhance the capability of a of a already knowledgeable state actor and or become the, you know, the main source of such a risk. Like if you wanted to engage in such a risk, the main way you would do it is through a model. And then I think ASL 4 on the autonomy side, it's some amount of acceleration in AI research capabilities with an AI model. And then ASL 5 is where we would get to the models that are kind of
truly capable. It could exceed humanity and their ability to do any of these tasks. And so the point of the if-then-structure commitment is basically to say, look, I don't know, I've been working with these models for many years and I've been worried about risk for many years. It's actually kind of dangerous to cry wolf. It's actually kind of dangerous to say, this model is risky and people look at it and they say this is manifestly not dangerous.
Again, it's the delicacy of the risk isn't here today but it's coming out as fast. How do you deal with that? It's really vexing to a risk planner to deal with it. And so this if-then-structure basically says, look, we don't want to antagonize a bunch of people, we don't want to harm our own, you know, our kind of own ability to have a place in the conversation by imposing these very honorisks burdens on models that are not dangerous today. So the if-then, the trigger commitment
is basically a way to deal with this. It says, you clamp down hard when you can show that the model is dangerous. And of course, what has to come with that is enough of a buffer threshold that, that you can, you know, you're not at high risk of kind of missing the danger. It's not a perfect
framework. We've had to change it every, every, you know, we came out with a new one just a few weeks ago and probably going forward, we might release new ones multiple times a year because it's hard to get these policies right, like technically, organizationally from a research perspective. But that is the proposal if-then-commitments and triggers in order to minimize burdens and
false alarms now, but really react appropriately when the dangers are here. What do you think that timeline for ESL-3 is where several of the triggers are fired and what do you think the timeline is for ESL-4? Yeah. So that is hotly debated within the company. We are working actively to prepare ESL-3 security, security measures as well as ESL-3 deployment measures. I'm not going to go into detail, but we've made, we've made a lot of progress on both and, you know, we're prepared to be,
I think, ready quite soon. I would, I would not be surprised, I would not be surprised at all if we hit ESL-3 next year. There was some concern that we might even hit it this year. That's still possible. That could still happen. It's like very hard to say, but like I would be very, very surprised if it was like 2030. I think it's much sooner than that. So there's a protocol for detecting it, if then, and then there's protocols for how to respond to it. Yes. How difficult
is the second the latter? Yeah. I think for ESL-3, it's primarily about security and about, you know, filters on the model relating to a very narrow set of areas when we deploy the model, because at ESL-3, the model isn't autonomous yet. And so you don't have to worry about, you know, kind of the model itself behaving a bad way even when it's deployed internally. So I think the ESL-3 measures are, I won't say straightforward, they're, they're, they're, they're rigorous, but they're
easier to reason about. I think once we get to ESL-4, we start to have worries about the models being smart enough that they might sandbag tests. They might not tell the truth about tests. We had some results came out about like sleeper agents, and there was a more recent paper about, you know, can, can the models mislead attempts to, you know, sandbag their own abilities, right?
Show them, you know, present themselves as being less capable than they are. And so I think with ESL-4, there's going to be an important component of using other things than just interacting with the models. For example, interpretability or hidden chains of thought, where you have to look inside the model and verify via some other mechanism that that is not, you know, is not as easily corrupted as what the model says that, that, you know, that, that the model indeed has some property. So we're
still working on ESL-4. One of the properties of the RSP is that we, we don't specify ESL-4 until we've hit ESL-3. And, and I think that's proven to be a wise decision because even with ESL-3, it, again, it's hard to know this stuff in detail. And, and it, we want to take as much time as we can possibly take to get these things, right? So for ESL-3, the bad actor will be the humans. Humans, yes. And so there's a little bit more, for ESL-4, it's both, I think, is both. And so
deception and that's where mechanistic interpretability comes into play. And hopefully the techniques used for that are not made accessible to the model. Yeah, I mean, of course you can hook up the mechanistic interpretability to the model itself. But then you've, then you've, then you've kind of lost it as a reliable indicator of, of, of, of, of the model state.
There are a bunch of exotic ways you can think of that it might also not be reliable. Like if the, you know, model gets smart enough that it can like, you know, jump computers and like read the code where you're like looking at its internal state. We've thought about some of those.
I think they're exotic enough. There are ways to render them unlikely. But yeah, generally, you want to, you want to preserve mechanistic interpretability as a kind of verification set or test set that's separate from the training process of the model. See, I think as these models become better and better conversation and become smarter, social engineer becomes a threat too. Because they, oh yeah, they could start being very convincing to the engineers inside companies.
Oh, yeah. Yeah. It's actually like, you know, we've seen lots of examples of demagoguery in our life from humans. And, you know, there's a concern that models could do that, could do that as well. One of the ways that cloud has been getting more and more powerful is it's now able to do some agentic stuff. Computer use. There's also an analysis within the sandbox of cloud.ai itself. But let's talk about computer use.
That seems to me super exciting that you can just give cloud a task and it takes a bunch of actions, figures it out and access to the your computer through screenshots. So can you explain how that works and where that's headed? Yeah, it's actually relatively simple. So, cloud has, has had for a long time since, since, cloud three back in March, the ability to analyze images and respond to them with text. The, the only new thing we added is those images can be screenshots of a computer.
And in response, we train the model to give a location on the screen where you can click and or buttons on the keyboard. You can press in order to take action. And it turns out that with actually not all that much additional training, the models can get quite good at that task. It's a good example of generalization. You know, people sometimes say if you get to lower or for a bit, you're like halfway to anywhere right because
of how much it takes to escape the gravity. Well, if you have a strong free train model, I feel like you're halfway to anywhere. In terms of in terms of the intelligence space. And so actually it didn't it didn't take all that much to get to get clawed to do this. And you can just set that in a loop. Give the model a screenshot, tell it what to click on, give it the next screenshot, tell it what to click on. And that turns into a full kind of almost almost 3D video interaction of the model.
And it's able to do all of these tasks, right? You know, we showed these demos where it's able to like fill out spreadsheets. It's able to kind of like interact with a website. It's able to, you know, it's able to open all kinds of, you know, programs, different operating systems, Windows, Linux, Mac.
So, you know, I think all of that is very exciting. I will say while in theory, there's nothing you could do there that you couldn't have done through just giving the model the API to drive the computer screen. And this really lowers the barrier. And, you know, there's there's a lot of folks who who either, you know, kind of kind of are, you know, aren't in a position to interact with those APIs or takes them a long time to do.
It's just the screen is just a universal interface that's a lot easier to interact with. And so I expect over time this is going to lower a bunch of barriers. Now, honestly, the current model has, there's, there leaves a lot still to be desired. And we were, we were honest about that in the blog, right?
It makes mistakes. It misclicks. And we, you know, we were careful to warn people, hey, this thing isn't, you can't just leave this thing to, you know, run on your computer for minutes and minutes. You got to give this thing bound reason guard rails. And I think that's one of the reasons we released it first in an API form rather than kind of, you know, this, this kind of just just hand just hand just the consumer and give it control of their of their of their of their computer.
But, but, you know, I definitely feel that it's important to get these capabilities out there as models get more powerful. We're going to have to grapple with, you know, how do we use these capabilities safely? How do we prevent them from being abused? And, and, you know, I think, I think releasing releasing the model while while the capabilities are, are, you know, are still are still limited is is very helpful in terms of in terms of doing that.
You know, I think since it's been released a number of customers, I think our repilate was maybe was maybe one of the the most quickest quickest quickest to quickest to deploy things have, you know, have made use of it in various ways. People have hooked up demos for, you know, Windows, dash tops, max, you know, Linux Linux machines.
So, yeah, it's been, it's been, it's been very exciting, I think as with as with anything else, you know, it, it, it comes with new exciting abilities and then, then, you know, then, then with those new exciting abilities, we have to think about how to, how to, you know, make the model, you know, safe, reliable, do what humans want them to do. I mean, it's the same, it's the same story for everything, right? Same thing is that same tension.
But, but the possibility of use cases here is just the range is incredible. So, how much to make it work really well in the future? How much do you have to specially kind of go beyond what's the pre-trained models doing? Do more post training, RLHF or supervised fine tuning or synthetic data just for the agent? Yeah, I think speaking at a high level, it's our intention to keep investing a lot in, you know, making making the model better.
Like, I think, I think, you know, we look at, look at some of the, you know, some of the benchmarks for previous models were like, oh, could do it 6% of the time. And now our model would do it 14 or 22% of the time. And yeah, we want to get up to, you know, the human level reliability of 80% just like anywhere else, right? We're on the same curve that we were on with sweet bench, where I think I would guess a year from now, the models can do this very, very reliably.
But you got to start somewhere. So you think it's possible to get to the, the human level 90% basically doing the same thing you're doing now or is it has to be special for computers? I mean, it depends what you mean by, you know, special and special in general. But, but, you know, I generally think, you know, the same kinds of techniques that we've been using to train the current model.
I expect that doubling down on those techniques in the same way that we have for code, for code, for models in general, for other kit for, you know, for image input. You know, for voice, I expect those same techniques will scale here as they have everywhere else. But this is giving sort of the power of action to Claude. And so you could do a lot of really powerful things, but you could do a lot of damage also. Yeah. Yeah. No, and we've been very aware of that.
Look, my view actually is computer use isn't a fundamentally new capability like these CBR and autonomy capabilities are. It's more like it kind of opens the aperture for the model to use and apply its existing abilities. And so the way we think about it going back to our RSP is nothing that this model is doing inherently increases, you know, the risk from an RSP perspective.
But as the models get more powerful, having this capability may make it scarier once it, you know, once it has the cognitive capability to, you know, to do something at the ASL 3 and ASL 4 level this, this, you know, this may be the thing that kind of unbounds it from doing so. So going forward, certainly this modality of interaction is something we have tested for and that we will continue to test for an RSP going forward.
I think it's probably better to have to learn and explore this capability before the model is super, you know, super capable. Yeah, and there's a lot of interesting attacks like prompt injection because now you've widened the aperture so you can prompt inject through stuff on screen. So if this becomes more and more useful, then there's more and more benefit to inject inject stuff into the model.
If it goes to a certain webpage, it could be harmless stuff like advertisements or it could be like harmful stuff. Yeah, I mean, we've fought a lot about things like spam, capture, you know, mass camp. There's all, you know, every, every, like, if one secret I'll tell you if you've invented a new technology, not necessarily the biggest misuse, but the first misuse you'll see scams, just petty scams.
Like you'll just just just it's like it's like a thing as old people scamming each other. It's this thing is old as time. And it's just every time you got to deal with it. It's almost like silly to say, but it's true sort of bots and spam in general is a thing is it gets more and more intelligent.
Yeah, it's just there's a lot of like I said, like there are a lot of petty criminals in the world and you know, it's like every new technology is like a new way for petty, petty criminals to do something, you know, something stupid and malicious. Is there any ideas about sandboxing it like how difficult is the sandboxing task.
Yeah, we sandbox during training. So for example, during training, we didn't expose the model to the internet. I think that's probably a bad idea during training because, you know, the model can be changing its policy, it can be changing what it's doing and it's having an effect in the real world. In terms of actually deploying the model, right, it kind of depends on the application. Like, you know, sometimes you want the model to do something in the real world.
But of course, you can always put guard, you can always put guard rails on the outside, right? You can say, okay, well, you know, this model is not going to move data from my, you know, model is not going to move any files from my computer or my web server to anywhere else.
Now, when you talk about sandboxing, again, when we get to ASL4, none of these precautions are going to make sense there, right? Where when you talk about ASL4, you're then the model is being kind of, you know, there's a theoretical worry, the model could be smart enough to break it to kind of break out of any box.
And so there we need to think about mechanistic interpretability about, you know, if we're going to have a sandbox, it would need to be a mathematically proven sound, but you know, that's, that's a whole different world than what we're dealing with with the models today. Yeah, the science of building a box from which ASL4 AI system cannot escape.
I think it's probably not the right approach. I think the right approach, instead of having something, you know, unaligned that, that like, you're trying to prevent it from escaping, I think it's better to just design the model the right way or have a loop where you, you know, you look inside, you look inside the model and you're able to verify properties and that gives you an opportunity to like iterate and actually get it right.
I think I think containing containing bad models is much worse solution than having good models. Let me ask about regulation. What's the role of regulation and keeping it safe? So for example, can you describe California AI regulation bill SB 1047 that was ultimately vetoed by the governor? What are the pros and cons of this bill?
Yeah, we ended up making some suggestions to the bill and then some of those were adopted and you know, we felt, I think, I think quite positively quite positively about about bill by by the end of that. It did still have some downsides.
And, you know, of course, of course you got vetoed. I think at a high level, I think some of the key ideas behind the bill are, you know, I would say similar to ideas behind our RSPs. And I think it's very important that some jurisdiction, whether it's California or the federal government and or other other countries and other states passes some regulation like this.
And I can talk through why I think that's so important. So I feel good about our RSP. It's not perfect. It needs to be iterated on a lot, but it's been a good forcing function for getting the company to take these risks seriously to put them into product planning to really make them a central part of work and entropic and to make sure that all the 1000 people and it's almost 1000 people now at an entropic understand that this is one of the highest priorities of the company, if not the highest priority.
But one, there are some there are still some companies that don't have RSP like mechanisms like open AI, Google did adopt these mechanisms a couple months after after entropic did.
But there are there are other companies out there that don't have these mechanisms at all. And so if some companies adopt these mechanisms and others don't, it's really going to create a situation where you know some of these dangers have the property that it doesn't matter if three out of five of the companies are being safe if the other two are are being are being unsafe, it creates this negative externality.
And I think the lack of uniformity is not fair to those of us who have put a lot of effort into being very thoughtful about these procedures. The second thing is I don't think you can trust these companies to adhere to these voluntary plans in their own, right? I like to think that entropic will we do everything we can that we will are RSP is checked by our long term benefit trust. So you know we do everything we can to to to adhere to our own RSP.
But you know you hear lots of things about various companies saying oh they said they would do they said they would give this much compute and they didn't they said they would do this thing and they didn't. You know I don't I don't think it makes sense to you know to to you know litigate particular things that companies have done.
And I think it's important to have a uniform standard that that that that that everyone follows and to make sure that simply that the industry does what a majority of the industry has already said is important and has already said that they definitely will do right some people.
I think there's there's a class of people who are against regulation on principle I understand where that comes from if you go to Europe and you know you see something like GDPR you see some of the other stuff that that that that that they've done you know some of it's good but but some of it is really unnecessarily burden some and I think it's fair to say really has slowed really has slowed innovation and so I understand where people are coming from on priors I understand why people come from start from that start from that position.
But but again I think AI is different if we go to the very serious risks of autonomy and misuse that that that I talked about you know just a just a few minutes ago I think that those are unusual and they weren't an unusually strong response and so I think it's very important again.
We need something that everyone can get behind you know I think one of the issues with SB 1047 especially the original version of it was it it had a bunch of the structure of RSPs but it also had a bunch of stuff that was either clunky or that that that just would have created a bunch of burdens a bunch of hassle and not even have missed the target in terms of addressing the risks.
You don't really hear about it on Twitter you just hear about kind of you know people are people are cheering for any regulation and then the folks who are against make up these often quite intellectually dishonest arguments about how you know it will make us move away from California bill bill doesn't apply if you're headquartered in California bill only applies if you do business in California or that it would damage the open source ecosystem or that it would you know it would cause cause all of these things.
I think those were mostly nonsense but there are better arguments against regulation there's one guy Dean ball who's really you know I think a very scholarly scholarly analyst who looks at what happens when a regulation is put in place and ways that they can kind of get a life of their own or how they can be poorly designed and so our interest has always been we do think there should be regulation in this space but we want to be an actor who make sure that that is what we want to do.
Sure that that that regulation is something that's surgical that's targeted at the serious risks and is something people can actually comply with because something I think the advocates of regulation don't understand as well as they could is if we get something in place that is that's poorly targeted that was a bunch of people's time what's going to happen is people are going to say see the safety risks there you know this is what we want to do.
There you know this is this is nonsense I just you know I just had to hire 10 lawyers to you know to fill out all these forms I had to run all these tests for something that was clearly not dangerous and after six months of that there will be there will be a groundswell and we'll we'll we'll end up with a durable consensus against regulation and so the I think the worst enemy of those who want real accountability is badly designed regulation.
We need to actually get it right and this is if there's one thing I could say to the advocates it would be that I want them to understand this dynamic better and we need to be really careful and we need to talk to people who actually have who actually have experience seeing how regulations play out in practice and and the people who have seen that understand to be very careful if this was some lesser issue I might be against regulation at all but what what I want the opponents to understand is is that the
underlying issues are actually serious there they're not they're not something that I or the other companies are just making up because of regulatory capture they're not sci-fi fantasies they're not they're not any of these things. You know every every time we have a new model every few months we measure the behavior of these models and they're getting better and better at these concerning tasks just as they are getting better and better at.
Good valuable economically useful tasks and so I I would just love it if some of the former you know I think SB 1047 was very polarizing I would love it if some of the most reasonable opponents and some of the most reasonable. Proponents would sit down together and you know I think I think that you know the different the different AI companies you know and the topic was the only AI company that you know felt positively in a very detailed way I think you want
to be a little bit more intuitive. Tweeted briefly something positive but you know some of the some of the big ones like Google open AI meta Microsoft were pretty strong staunchly against so I would really like is if if you know some of the key stakeholders some of the most thoughtful proponents and some of the most thoughtful
proponents would sit down and say how do we solve this problem in in a way that the proponents feel brings a real reduction in risk and that the opponents feel that it is not it is not hampering the industry or hampering innovation. Any more necessary than it then it then it needs to and I think for whatever reason that things got to polarized and those two groups didn't get to sit down in the way that they should. And I feel I feel urgency I really think we need to do something in 2025.
You know if we get to the end of 2025 and we still done nothing about this then I'm going to be worried I'm not I'm not worried yet because again the risks aren't here yet but but I I think time is running short and come up with something surgical like you said yeah yeah exactly and we need to get we need to get away from this this this intense pro safety versus intense anti regulatory rhetoric right it's turned into these these flame wars on Twitter. And nothing good is going to come at that.
So there's a lot of curiosity about the different players in the game one of the O.G.s is open AI you've had several years of experience at open AI what's your story and history there yeah so I was at open AI for for roughly five years for the last I think was couple years you know I I I I was vice president of research there.
Probably myself and I'll use to give her were the ones who you know really kind of set the set the research direction around 2016 or 2017 I first started to really believe in or at least confirm my belief in the scaling hypothesis when when I'll you famously said to me the thing you need to understand about these models is they just want to learn the models just want to learn.
And again sometimes there are these one sentence there are these one sentences these Zen cones that you hear them and you're like that explains everything that explains like a thousand things that I've seen and I and I you know I ever after I have this visualization my head of like you optimize the models in the right way you point the models in the right way they just want to learn they just want to solve the problem regardless of what the problem is so get out of their way basically get out of their way yeah don't impose your own ideas about how they should learn and you know I'm not going to do that.
And you know this is the same thing as Rich Sutton put out in the bitter lesson or Gurren put out in the scaling hypothesis you know I think generally the dynamic was you know I got I got this kind of inspiration from from from from from Ilya and from others folks like Alik Radford who did the original G.P.T.1
and then ran really hard with it me me and my collaborators on G.P.T.2 G.P.T.3 R.L. from human feedback which was an attempt to kind of deal with the early safety and durability things like debate and amplification heavy on interpretability so again the combination of safety plus scaling probably 2018 2019 2020 those those were those were kind of the years when myself and my collaborators probably
you know what many many of whom became co founders of entropic kind of really had had a vision and like and like drove the direction what do you leave what just to leave yeah so look I'm going to put things this way and I you know I think it I think it ties to the to the race to the top right which is you know in my time at open AI when I come to see as I come to appreciate the scaling hypothesis
the first one I think you know open AI was was getting was getting on board with the second one in a way had always been part of open AI's messaging
but you know over over many years of of the time the time that I spent there I think I had a particular vision of how these how we should handle these things how we should be brought out in the world the kind of principles that the organization should have and look I mean there were like many many discussions about like you know should the or do should the company do this the company do that like there's a bunch of misinformation out there people say like we left because we didn't like the deal with Microsoft
false although you know it was like a lot of discussion a lot of questions about exactly how we do the deal with Microsoft we left these leading like commercialization that's not true we built gpd3 which was the model that was commercialized I was involved in commercialization it's it's more again about how do you do it like civilization is going down this path to very powerful AI what's the way to do it that is cautious straight forward honest
that builds trust in the organization and individuals how do we get from here to there and how do we have a real vision for how to get it right how can safety not just be some thing we say because it helps with recruiting and you know I think I think at the end of the day if you have a vision for that forget about anyone else's vision I don't want to talk about anyone else's vision if you have a vision for how to do it you should go off and you should do that vision
it is incredibly unproductive to try and argue with someone else's vision you might think they're not doing it the right way you might think they're they're dishonest who knows maybe you're right maybe you're not but what you should do is you should take some people you trust and you should go off together and you should make your vision happen and if your vision is compelling if you can make it appeal to people some you know some combination of ethically you know in the markets
you know if you can if you can make a company that's a place people want to join that you know engages in practices that people think are are reasonable while managing to maintain its position in the ecosystem at the same time if you do that people will copy it and the fact that you were doing it especially the fact that you're doing it better than they are causes them to change their behavior in a much more compelling way than if they're your boss and you're arguing with them I just I don't know how to do it
I just I don't know how to be any more specific about it than that but I think it's generally very unproductive to try and get someone else's vision to look like your vision it's much more productive to go off and do a clean experiment and say this is our vision this is how this is how we're going to do things your choice is you can you can ignore us you can reject what we're doing or you can you can start to become more like us an imitation is the sincerest form of flattery
and you know that that that plays out in the behavior of customers that pays out in the behavior of the public that plays out in the behavior of where people choose to work and again again at the end it's it's not about one company winning or another company winning if if we are another company are engaging in some practice that you know people people find genuinely appealing and I want it to be in substance not just not just in appearance
and you know I think I think researchers are sophisticated and they look at substance and then other companies start copying that practice and they win because they copied that practice that's great that's success that's like the race to the top
it doesn't matter who wins in the end as long as everyone is copying everyone else is good practices right one way I think of it is like the thing we're all afraid of is the race to the bottom right in the race to the bottom doesn't matter who wins because we all lose right like you know in the most extreme world we we make this autonomous AI that you know the robots and slave us or whatever right I mean that's half joking but you know that that is the most extreme thing that could happen
then that doesn't matter which company was ahead if instead you create a race to the top where people are competing to engage in good in good practices then you know at the end of the day you know doesn't matter who ends up who ends up winning doesn't even matter who who started the race to the top the point isn't to be virtuous the point is to get the system into a better equilibrium than it was before
and an individual companies can play some role in doing this individual companies can can you know can help to start it can help to accelerate it and frankly I think individuals and other companies have done this as well right the individuals that when we put out an RSP react by pushing harder to get something similar done get something similar done at other companies
sometimes other companies do something that's like we're like oh it's a good practice we think we think that's good we should adopt it too
the only difference is you know I think I think we are we try to be more forward leaning we try and adopt more of these practices first and adopt them more quickly when others when others invent them but I think this dynamic is what we should be pointing at and that I think I think it abstracts away the question of you know which company is winning who trusts who I think all these all these questions of drama are profoundly uninteresting
and the thing that matters is the ecosystem that we all operate in and how to make that ecosystem better because that constrains all the players and so anthropic is this kind of clean experiment built on a foundation of like what concretely AIC should look like
we're look I'm sure we've made plenty of mistakes along the way the perfect organization doesn't exist it has to deal with the imperfection of a thousand employees it has to deal with the imperfection of our leaders including me it has to deal with the imperfection of the people we've put we've put to you know to oversee the imperfection of the leaders like the like the board and the long term benefit trust it's it's all it's all a set of imperfect people trying to aim imperfectly at some ideal that will never perfectly be achieved
that's what you sign up for that's what it will always be but imperfect doesn't mean you just give up there's better and there's worse and hopefully hopefully we can begin to build we can do well enough that we can begin to build some practices that the whole industry engages in and then you know my guess is that multiple of these companies will be successful and
drop it will be successful these other companies like once I've been at the past will also be successful and some will be more successful than others that's less important than again that we we align the incentives of the industry and that happens partly through the race to the top partly through things like RSP partly through again selected surgical regulation.
You said talent density beats talent mass so can you explain that can you expand on it can you just talk about what it takes to build a great team of AI researchers and engineers. This is one of these statements that's like more true every every every month every month I see the statement is more true than I did the month before.
So if I were to do with thought experiment let's say you have a team of 100 people that are super smart motivated in aligned with the mission and that's your company or you can have a team of a thousand people where 200 people are super smart super aligned with the mission and then like
800 people are let's just say you pick 800 like random random big tech employees which would you rather have right the talent mass is greater in the group of in the group of a thousand people right you have you have even even a larger number of incredibly talented incredibly aligned incredibly smart people.
But the issue is just that if every time someone super talented looks around they see someone else super talented and super dedicated that's the tone for everything right that's that's the tone for everyone is super inspired to work at the same place everyone trust everyone else if you have a thousand or 10 thousand people and things have really regressed right you are not able to do selection and you're choosing random people.
What happens is then you need to put a lot of processes and a lot of guard rails in place just because people don't fully trust each other you have to adjudicate political battles like there are so many things that slow down your ability to operate and so we're nearly a thousand people and you know we've we've tried to make it so that as large a fraction of those thousand people as possible are like super talented super skilled it's one of the reasons we've we slow down hiring a lot.
Last few months we grew from 300 to 800 I believe I think in the first seven eight months of the year and now we've slowed down we're at like you know last three months we went from 800 to 900 950 something like that don't quote me on the exact numbers but I think there's an inflection point around a thousand and we want to be much more careful how we how we grow early on and now as well you know we've hired a lot of physicists you know theoretical physicists can learn things really fast.
Even even more recently as we've continued to hire that you know we've really had a high bar for on both the research side and the software engineering side have hired a lot of senior people including folks who used to be at other at other companies in this space and we just continued to be very selective it's very easy to go from 100 to 1000 and 1000 to 10 thousand without paying attention to making sure everyone has a unified purpose it's so easy to do that.
It's so powerful if your company consists of a lot of different fiefdoms that all want to do their own thing they're all optimizing for their own thing. It's very hard to get anything done but if everyone sees the broader purpose of the company if there's trust and there's dedication to doing the right thing that is a superpower that in itself I think can overcome almost every other disadvantage.
And you know Steve Jobs a players a players want to look around and see other a players as another wave of saying right I don't know what that is about human nature but it is demotivating to see people who are not obsessively driving towards a singular mission and it is on the flip side of that super motivating to see that it's interesting what's it take to be a great AI researcher or engineer from everything you've seen from working with so many amazing people.
Yeah I think the number one quality especially on the research side but really both is open minded this sounds easy to be open minded right you're just like I'm open to anything but you know if I if I think about my own early history in the scaling hypothesis. I was seeing the same data others were seen I don't think I was like a better programmer or better at coming up with research ideas than any of the hundreds of people that I worked with in some ways in some ways I was worse.
I've never like precise programming of like finding the bug writing the GPU kernels like I can point you to a hundred people here who are better who are better at that than I am but the thing that that I think I did have that was different was that I was just willing to look at something with new eyes right people said oh you know we don't have the right algorithms yet we haven't come up with the right.
The right way to do things now is just like I don't know like you know this neural net has like 30 billion 30 million parameters like what if we gave it 50 million instead like let's plot some graphs like that that basic scientific mindset of like oh man like I just I just like I you know I see some variable that I could change like what happens when it changes like let's let's try these different things and like create a graph for even this this was like the simplest thing in the world right.
Change the number of you know this wasn't like PhD level experimental design this was like this was like simple and stupid like anyone could have done this if you if you just hold them that that that was important it's also not hard to understand you didn't need to be brilliant to come up with this.
But you put the two things together and you know some tiny number of people some single digit number of people have driven forward the whole field by realizing this and it's you know it's often like that if you look back at the discovery you know the discoveries in history they're often like that and so this open mindedness and this willingness to see with new eyes that often comes from being newer to the field often experience is a disadvantage for this that is the most important thing it's very important.
It's very hard to look for and test for but I think I think it's the most important thing because when you when you find something some really new way of thinking thinking about things when you have the initiative to do that it's absolutely transformative and also be able to do kind of rapid experimentation and in the face of that be open minded and curious and looking at the data for just these fresh eyes and see what is that it's actually saying that applies in mechanism interpretability is another example of this like some of the early work in mechanistic interpretability.
So simple it's just no one thought to care about this question before you said what it takes to be a great research can we rewind the clock back what what advice would you give to people interested in AI their young looking for how can I make getting back to the world. I think my number one piece of advice is to just start playing with the models.
This was actually I worry a little this seems like obvious advice now I think three years ago it wasn't obvious and people started by oh let me read the latest reinforcement learning paper let me let me kind of no I mean that was really the that was really the and I mean you should do that as well but now you know with wider availability of models and api's people are doing this more but I think I think just experiential knowledge.
These models are new artifacts that no one really understands and so getting experience playing with them I would also say again in line with the like do something new thinking some new direction like there are all these things that haven't been explored like for example mechanistic interpretability is still very new it's probably better to work on that that is to work on new model architectures because it's you know it's more popular than it was before there are probably like 100 people working on it but there aren't like 10,000 people work.
And it's just this fertile area for study like like you know it's there's there's so much like low hay you can just walk by and you know you can just walk by and you can pick things and and the only reason for whatever reason people aren't people aren't interested in it enough I think there are some things around long long horizon learning and long horizon tasks where there's a lot to be done I think a value.
I think evaluations are still we're still very early in our ability to study evaluations particularly for dynamic systems acting in the world I think there's some stuff around multi agent. Skate where the puck is going is my is my advice and you don't have to be brilliant to think of it like all the things that are going to be exciting in five years like in people even mention them as like you know conventional wisdom but like it's it's just somehow there's this barrier that people don't.
People don't double down as much as they could or they're afraid to do something that's not the popular thing I don't know why it happens but like getting over that barriers that that's my number one piece of advice. Let's talk if it could a bit about post training yeah it seems that the modern post training recipe has a little bit of everything so supervised fine tuning RLHF the the constitutional AI with RL a I F best acronym it's again that naming thing.
And then synthetic data seems like a lot of synthetic data are at least trying to figure out ways to have high quality synthetic data so what's the if this is a secret sauce that makes anthropic cloth so incredible what how much of the magic is in the pre training how much it is in the post training yeah I mean so first of all we're not perfectly able to measure that ourselves.
You know when you see some some great character ability sometimes it's hard to tell whether it came from pre training or post training we developed ways to try and distinguish between those two but they're not perfect. You know the second thing I would say is you know it's when there isn't advantage and I think we've been pretty good at in general in general at RL perhaps perhaps the best although although I don't know because I don't see what goes on inside other companies.
Usually it isn't oh my god we have the secret magic method that others don't have right usually it's like well you know we got better at the infrastructure so we could run it for longer or you know we were able to get higher quality data or we were able to filter our data better or we were able to you know combine these methods in practice it's usually some boring matter of matter of kind of practice and trade craft.
So you know when I think about how to do something special in terms of how we train these models both pre training but even more so post training.
You know I really think of it a little more again as like designing airplanes or cars like you know it's not just like oh man I have the blueprint like maybe that makes you make the next airplane but like there's some there's some cultural trade craft of how we think about the design process that I think is more important than you know then then any particular gizmo were able to invent.
Okay well let me ask you about specific techniques of first on RLHF what do you think just zooming out intuition almost philosophy what do you think RLHF works so well. If I go back to like the scaling hypothesis one of the ways to skate the scaling hypothesis is if you train for X and you throw enough compute at it then you get X and so RLHF is good at doing what humans want the model to do or at least to state it more precisely.
Doing what humans who look at the model for a brief period of time and consider different possible responses what they prefer as the response which is not perfect from both the safety and capabilities perspective in that humans are often not able to perfectly identify what the model wants and what humans want the moment may not be what they want in the long term so there's there's a lot of subtlety there but the models are good at you know producing what the humans in some shallow sense want.
And it actually turns out that you don't even have to throw that much compute at it because of another thing which is this this thing about a strong pre trained model being halfway to anywhere. So once you have the pre trained model you have all the representations you need to get the model to get the model where you where you want it to go.
So do you think RLHF makes the model smarter or just appear smarter to the humans I don't think it makes a model smarter I don't think it just makes a model appear smarter it's like RLHF like bridges the gap between the human and the model right I could have something really smart that like can't communicate at all right we all know people like this people who are really smart but that you know you can't understand what they're saying.
So I think I think RLHF just bridges that gap I think it's not it's not the only kind of RL we do it's not the only kind of RL that will happen in the future I think RL has the potential to make models smarter to make them reason better to make them operate better to make them develop new skills even and perhaps that could be done you know even in some cases with human feedback but the kind of RLHF we do today mostly doesn't do that yet although we're very quickly starting to be able to.
But it appears to sort of increase if you look at the metric of healthfulness it increases that it also increases what was this this word in leopold's essay unhobbling we're basically the models are hobbled and then you do various trainings to them to unhobble them so I you know I like that word because it's like a real word but it's so so I think RLHF unhobbles the models in some ways and then there are other ways where it model hasn't yet been unhobbled and you know needs to needs done hobble if you can say.
In terms of cost is pre-training the most expensive thing or is post-training creep up to that. At the present moment it is still the case that pre-training is the majority of the cost I don't know what to expect in the future but I could certainly anticipate a future where post-training is the majority of the cost. In that future you anticipate would it be the humans or the AI that's the cost of the thing for the post-training?
I don't think you can scale up humans enough to get high quality any any kind of method that relies on humans and uses a large amount of compute it's going to have to rely on some scale supervision method like you know debate or iterated amplification or something like that. So on that it's a super interesting set of ideas around constitutional AI can describe what it is as first detailed in December 22nd. Yes, two paper and and beyond that what is it?
Yes, so this was from two years ago the basic idea is so we describe what RLHF is you have you have a model and it you know spits out two pot you know it like you just sample from it twice it's
out two possible responses and you're like human which response to like better or not are very in doubt it is rate this response on scale of one to seven so that's hard because you need to scale up human interaction and it's very implicit right I don't have a sense of what I want the model to do I just have a sense of like what this average of a thousand humans wants the model to do.
So two ideas one is could the AI system itself decide which which response is better right could you show the AI system these two responses and ask which which which response is better and then second well what criterion should the I use and so then there's this idea could you have a single document a constitution if you will that says these are the principles the model should be using to respond and the AI system reads those
principles as well as reading the environment and the response and it says well how good did the AI model do it's basically a form of self play you're kind of training the model against itself and so the I gives the response and then you feed that back into what's called the preference model which in turn feeds the model to make it better.
So you have this triangle of like the AI the preference model and the improvement of the AI itself and we should say that in the Constitution the set of principles are like human interpretable they're like yeah yeah it's it's something both human the human and the AI system can read so it has this nice this nice kind of translate ability or symmetry.
You know in practice we both use a model constitution and we use RLHF and we use some of these other methods so it's turned into one tool in a in a toolkit that both reduces the need for RLHF and increases the value we get from from from using each data point of RLHF
and also interacts in interesting ways with kind of future reasoning type RL methods so it's one tool in the toolkit but but I think it is a very important tool but it's a compelling one to us humans you know think about the founding fathers and founding of the United States.
The natural question is who and how do you think it gets to define the Constitution the set of principles in the Constitution yeah so I'll give like a practical answer and a more abstract answer I think the practical answer is like look in practice models get used by all kinds of different like customers right and so you can have this idea where you know the model can can have specialized rules or principles you know we fine tune versions of models.
So we've talked about doing it explicitly having having special principles that people can build into the models so from a practical perspective the answer can be very different from different people you know customer service agent you know behaves very differently from the lawyer and obeys different principles.
So I think at the base of it there are specific principles that that models you know have to obey I think a lot of them are things that people would agree with everyone agrees that you know we don't we don't want models to present these CBR and risks I think we can go a little further and agree with some basic principles of democracy and the rule of law beyond that it gets you know very uncertain and there are goal is generally for the models to be more neutral to not a spouse a particular point of view and you know
more just be kind of like wise agents or advisors that will help you think things through and will you know present present possible considerations but you know don't express you know strong or specific opinions.
Open a I released a model spec where kind of clearly concretely defines some of the goals of the model and specific examples like a B how the model should behave do you find that interesting by the way I should mention the I believe the brilliant John Schumann was a part of that he's not an anthropic. Do you think this is a useful direction might anthropic release a model spec as well.
So I think that's a pretty useful direction again it has a lot in common with the constitutional AI so again another example of like a race to the top right we have something that's like we think you know a better and more responsible way of doing things.
It's also a competitive advantage then others kind of you know discover that it has advantages and then start to do that thing we then no longer have the competitive advantage but it's good from the perspective that now everyone has adopted a positive
positive practice that others were not adopting and so our response to that as well looks like we need a new competitive advantage in order to keep driving this race upwards so that's that's how I generally feel about that I also think every implementation of these things is different so you know there are some things in the model spec that we're not in constitutional AI and so you know we you know we can always we can always adopt those things or you know at least learn from them so again I think this is an example of like the positive dynamic that that that that I that I think we should all.
Want the field to have. Let's talk about the incredible essay machines of love and grace I recommend everybody read it. The long one it is rather long yeah it's really refreshing to read concrete ideas about what a positive future looks like and you took sort of a bold stance because like it's very possible you might be wrong on the dates or specific.
Yeah I'm fully expecting to you know to definitely be wrong about all the details I might be be just spectacularly wrong about the whole thing and people will you know will laugh at me for years. That's that's that's just how the future works. So you provide a bunch of concrete positive impacts of AI and how you know exactly a super intelligent AI might accelerate the rate of breakthroughs and for example biology and chemistry that would then.
Lead to things like we cure most cancers prevent all infectious disease double the human lifespan and so on so let's talk about the session first can you give a high level. Vision of the session and what key takeaways that people have yeah I have spent a lot of time in and topic I spent a lot of effort on like you know how do we address the risks of AI right how do we think about those risks like.
We're trying to do a race to the top you know what that requires us to build all these capabilities and the capabilities are cool but you know we're we're like a big part of what we're trying to do is like is like address the risks and justification for that is like well you know all these positive things you know the market is this very healthy organism right it's going to produce all the positive things the risks I don't know we might mitigate them we might not and so we can have more impact by trying to mitigate the risks.
But I noticed that one flaw in that way of thinking and it's not a change in how serious like take the risks it's maybe a change in how I talk about them is that you know no matter how kind of logical or rational that line of reasoning that I just gave might be.
If you kind of only talk about risk your brain only thinks about risks and so I think it's actually very important to understand what if things do go well and the whole reason we're trying to prevent these risks not because we're afraid of technology not because we want to slow it down it's it's because.
If we can get to the other side of these risks right if we can run the gauntlet successfully to you know to put it in stark terms then then on the other side of the gauntlet are all these great things and these things are worth fighting for and these things can really inspire people and.
I think I imagine because look you have all these investors all these VCs all these AI companies talking about all the positive benefits of AI but as you point out it's it's it's weird there's actually a dearth of really getting specific about it there's a lot of like random people on Twitter like posting these kind of like gleaming cities and this this just kind of like vibe of like grind accelerate harder like kick out the diesel you know it's just this very this very like.
Like aggressive ideological but then you're like what are you what what what what are you actually excited about and so and so I figured that you know it I think it would be interesting and valuable for someone who's actually coming from the risk side to try and and to try and really.
Make a try at explaining explaining explain what the benefits are both because I think it's something we can all get behind and I want people to understand I want them to really understand that this isn't this isn't doomers versus accelerationists this this is that if you have a true understanding of where things are going with with AI and maybe that's the more important access AI is moving fast versus AI is not moving fast.
Then you really appreciate the benefits and you you you really you want humanity our civilization to seize those benefits but you also get very serious about anything that could derail them so I think the starting point is to talk about what this powerful AI which is the term you like to use most of the world uses a GI but you don't like the term because it's basically has too much baggage it's become meaningless it's like we're stuck with the terms.
Maybe we're stuck with the terms and my efforts to change them are futile it's admirable I'll tell you what else I don't this is like a pointless semantic point but I yeah I keep talking about it back to naming again I just do it once more.
I think it's a little like like let's say it was like 1995 and more laws making the computers faster and like for some reason there there there had been this like verbal tick that like everyone was like well someday we're going to have like super computers and like super computers are going to be able to do all these things that like you know once we have super computers will be able to like sequence that you know will be able to do other things and so and so like one it's true the computers are getting faster and as they get faster they're going to be able to do all these great things but there's like there's no just so much.
There's no discrete point at which you had a super computer and previous computers were not to like super computers at term we use but like it's a vague term to just describe like computers that are faster than what we have today there's no point at which you pass the threshold you're like oh my god we're doing a totally new type of computation and new and so I feel that way about a GI like there's just a smooth exponential and like if if by a GI you mean like like AIs getting better and better and like gradually it's going to do more and more of what humans do.
Until it's going to be smarter than humans and then it's going to get smarter even from there then then yes I believe in a GI if but if if a GI is some discrete or separate thing which is the way people often talk about it then it's it's kind of a meaningless buzzword.
Yeah, and to me it's just sort of a platonic form of a powerful yeah exactly how you define I mean you define it very nicely so on the intelligence axis it's just on pure intelligence it's smarter than a Nobel Prize winner as you describe across the world. It's just intelligent so it's both in creativity and be able to generate new ideas all that kind of stuff in every discipline Nobel Prize winner. Okay in their prime.
It can use every modality so this kind of self explanatory but just operate across all the modalities of the world. It can go off for many hours days and weeks to do tasks and do its own sort of detailed planning and only ask you help when it's needed. It can use this is actually kind of interesting I think in the essay you said I mean again it's a bet that it's not going to be embodied but it can control embodied tools so it can control tools robots laboratory equipment.
The resource used to train it can then be repurposed to run millions of copies of it and each of those copies would be independent that can do their own independent work so you can do the cloning of the intelligence.
Yeah, I mean you might imagine from outside the field like there's only one of these right that like you made it you've only made one but the truth is that like the scale up is very quick like we do this today we make a model and then we deploy thousands maybe tens of thousands of instances of it.
I think by the time you know certainly within two to three years whether we have these super powerful a eyes or not clusters are going to get to the size where you'll be able to deploy millions of these and they'll be you know faster than humans and so if your picture is all will have one and then it'll take a while to make them my point there was no actually you have millions of them right away and in general they can learn and act.
10 to 100 times faster than humans so that's a really nice definition of powerful a eye okay so that but you also write that clearly such an entity would be capable of solving very difficult problems very fast but it is not trivial to figure out how fast to extreme positions both seem false to me so the singularity is on the one extreme and the opposite on the other stream can you describe each of the extremes yeah so why so yeah let's let's describe the extreme so like one one extreme.
One one extreme would be well look you know if we look at kind of evolutionary history like there was this big acceleration where you know for hundreds of thousands of years we just had like you know single cell organisms and then we had mammals and then we had apes and then that quickly turned to humans humans quickly built industrial civilization and so this is going to keep speeding up and there's no ceiling the human level once models get much much smarter than humans they'll get really good at building the next models and you know if you're going to get a lot of the
you write down like a simple differential equation like this is an exponential and so what's what's going to happen is that models will build faster models models will build faster models and those models will build you know nanobots that can like take over the world and produce much more energy then you could produce otherwise and so if you just kind of like solve this abstract differential equation then like five days after we you know we build the first AI that's more powerful than humans then then you know like the world will be filled with these a eyes and every possible technology and then we can do that.
And every possible technology that could be invented like will be invented I'm caricaturing this a little bit. But I you know I think that's one extreme and the reason that I think that's not the case is that one I think they just neglect like the laws of physics like it's only possible to do things so fast in the physical world like some of those loops go through you know producing faster hardware takes a long time to produce faster hardware things take a long time.
There's this issue of complexity like I think no matter how smart you are like you know people talk about oh we can make models the biological systems that will do everything the biological systems look I think computational modeling can do a lot I get a lot of computational modeling when I worked in biology but like just there are a lot of things that you can't predict how they're you know they're complex enough that like just iterating just running the experiment is going to beat any modeling no matter how smart the system doing the model.
Well even if it's not interacting with the physical world just the modeling is going to be hard yeah I think well the modeling is going to be hard and getting the model to to to match the physical world is going to be all right so it does have to enter yeah yeah yeah but but it's just you know you just look at even the simplest problems like I you know I think I talk about like you know the three body problem or simple chaotic prediction like you know or like predicting the economy it's really hard to predict the economy two years out like maybe the world is going to be a lot of things that are going to be hard to do.
Like maybe the case is like you know normal you know humans can predict what's going to happen in the economy next quarter they can't really do that maybe maybe a AI system that's you know a zillion times smarter can only predict it out a year or something instead of a instead of you know you have these kind of exponential increase in computer intelligence for linear increase in in in ability to predict same with again like you know biological molecules molecules interacting you don't know what's going to happen when you're going to be able to do that.
I don't know what's going to happen when you perturb when you perturb a complex system you can find simple parts in it if you're smarter you're better at finding these simple parts and then I think human institutions human institutions are just are really difficult like it's you know it's it's been hard to get people I won't give specific examples but it's been hard to get people to adopt even the technologies that we've developed even ones where the case for their efficacy is very very strong.
You know people have concerns they think things are conspiracy theories like it's it's just been it's been very difficult it's also been very difficult to get. You know very simple things through the regulatory system right I think you know and you know I don't want to disparage anyone who you know you know we're works in regulatory regulatory systems of any technology they're hard trade off they have to deal with they have to save lives but but the system as a whole I think make some obvious.
Trade offs that are very far from maximizing human welfare and so if we bring AI systems into this you know into these human systems often the level of intelligence may just not be the limiting factor right it it just may be that it takes a long time to do something now if the I system.
I think you know because I think it's a lot of energy and it's not that I think it's a lot of energy and we're just going to get into all governments if it just said I'm dictator of the world and I'm going to do whatever.
Some of these things that could do again the things have you do with complexity I I still think a lot of things would take a while I don't think it helps at the I systems can produce a lot of energy or go to the moon like some people in comments responded to the ass a saying the I system can produce a lot of energy and smarter AI systems that's missing the point that kind of cycle doesn't solve the key problems that I'm talking about here.
But even if it were completely on the line and you know could get around all these human obstacles it would have trouble but again if you want this to be an AI system that doesn't take over the world that doesn't destroy humanity then then basically you know it's it's it's going to need to follow basic human laws right what you know if we want to have an actually good world like we're going to have to have an AI system that that interacts with humans not one that kind of creates its own legal system or district guards all the laws or all of that.
So as inefficient as these processes are you know we're going to have to deal with them because there needs to be some popular and democratic legitimacy and how these systems are rolled out we can't have a small group of people who are developing the system say this is what's best for everyone right. I think it's wrong and I think in practice is not going to work anyway.
So you put all those things together and you know we're not going to we're not going to you know change the world and upload everyone in five minutes I just I don't think I I I don't think it's going to happen and be to some you know to the extent that it could happen. It's not the way to lead to a good world.
So that's on one side on the other side there's another set of perspectives which I have actually in some ways more sympathy for which is look we've seen big productivity increases before right you know economists are familiar with studying the productivity increases that came from the computer revolution and Internet
that revolution and generally those productivity increases were underwhelming they were less than you then you might imagine there was a quote from Robert solo you see the computer revolution everywhere except the productivity statistics. So why is this the case people point to the structure of firms the structure of enterprises how you know how slow it's been to roll out or existing technology to very poor parts of the world which I talk about in the essay right.
How do we get these technologies to the poorest parts of the world that are behind on cell phone technology computers medicine let alone you know new fangled AI that hasn't been invented yet. So you can have a perspective that's like well this is amazing technically but it's all nothing burger.
I think Tyler Cowan who wrote something response to my essay has that perspective I think he thinks the radical change will happen eventually but he thinks it'll take 50 or 100 years and you could have even more static perspectives on the whole thing I think there's some truth to it I think the time scale is just is just too long.
And I can see it I can actually see both sides with today's AI so you know a lot of our customers are large enterprises who are used to doing things a certain way I've also seen it in talking to governments right those are those are prototypical you know institutions entities that are slow to change.
But the dynamic I see over and over again is yes it takes a long time to move the ship yes there's a lot of resistance and lack of understanding but the thing that makes me feel that progress will in the end happen moderately fast not incredibly fast but moderately fast is that you talk to what I find is I find over and over again again in large companies even in governments which have been actually surprisingly forward leaning.
You find two things that move things forward one you find a small fraction of people within a company within a government who really see the big picture who see the whole scaling hypothesis who understand where AI is going or at least understand where it's going within their industry and there are a few people like that within the current within the current US government who really see the whole picture and those people see that this is the most important thing in the world until they agitate for it and the thing they they alone are not enough to succeed.
Because there are small set of people within a large organization but as the technology starts to roll out as it succeeds in some places in the folks who are most willing to adopt it the specter of competition gives them a wind at their backs because they can point within their large organization they can say look these other guys are doing this right you know one bank can say look this newfangled hedge fund is doing this thing they're going to eat our lunch.
In the US we can say we're afraid China is going to get there before before we are and that combination the specter of competition plus a few visionaries within these you know within these the organizations that in many ways are our sclerotic you put those two things together and it actually make something happen I mean it's interesting it's a balanced fight between the two because inertia is very powerful but but but eventually over enough time the innovative approach break up.
Approach breaks through. And I've seen that happen I've seen the arc of that over and over again and it's like the barriers are there the barriers to progress the complexity not knowing how to use the model how to deploy them are there and for a bit it seems like they're going to last forever like change doesn't happen but then eventually change happens and always comes from a few people.
I felt the same way when I was an advocate of the scaling hypothesis within the AI field itself and others didn't get it it felt like no one would ever get it it felt like then it felt like we had a secret almost no one ever had and then a couple years later everyone has the secret and so I think that's how it's going to go with deployment AI in the world it's going to the barriers are going to fall apart gradually and then all at once and so I think this is going to be more and this is just an instinct I could I could easily see how I'm wrong.
I think it's going to be more like 10 5 or 10 years as I say in the essay then it's going to be 50 or 100 years I also think it's going to be 5 or 10 years more than it's going to be you know 5 or 10 hours because I just I just seen how human systems work and I think a lot of these people who write down the differential equations who say AI is going to make more powerful AI who can't understand how it could possibly be the case that these things won't won't change so fast I think they don't understand these things.
So what do you use the timeline to where we achieve a GI a K a powerful AI a K super useful AI. I'm not. I'm just start calling that it's a debate it's a debate about naming. You know, on pure intelligence it can smarter than a Nobel Prize winner in every relevant discipline and all the things we've said modality can go and do stuff on its own for days weeks and do biology.
Experiments on its own in one you know what let's just stick to biology because yeah you sold me and the whole biology and health section that's so exciting from from just I was getting getty from a scientific perspective it made me want to be a biologist.
Almost it's so no no this was the feeling I have when I was writing it that it's it's like this would be such a beautiful future if we can if we can just if we can just make it happen right if we can just get the get the landmines out of the way and and make it happen there's there's so much there's so much beauty and and and and and elegance and moral force behind it if we can if we can just and it's something we should all be able to agree on right like as much as we fight about.
About about all these political questions is this something that could actually bring us together.
But you were asking when when when do you think what's just so but numbers on so you know this is of course the thing I've been grappling with for many years and I'm not I'm not at all confident every time if I say 2026 or 2027 there will be like a zillion like people on Twitter who will be like I see oh said 2026 20 and it will be repeated for like the next two years that like this is definitely when I think it's going to happen.
So whoever's x sort of these clips will will will will will crop out the thing I just said and and and only say the thing I'm about to say but I'll just say it anyway. Um so if you extrapolate the curves that we've had so far right if if you say well I don't know we're starting to get to like PhD level and last year we were at.
Under graduate level in the year before we were at like the level of a high school student again you can you can quibble with at what tasks and for what we're still missing modalities but those are being added like computer use was added like image in was added like image generation has been added.
If you just kind of like and this is totally unscientific but if you just kind of like eyeball the rate at which these capabilities are increasing it does make you think that will get there by 2026 or 2027 again lots of things could derail it we could run out of data you know we might not be able to scale clusters as much as we want like you know maybe Taiwan gets blown up or something and you know then we can't produce as many GPUs as we want so there are there all kinds of things that can be.
So I don't fully believe the straight line extrapolation but if you believe the straight line extrapolation you'll you'll will get there in 2026 or 2027 I think the most likely is that there's some mild delay relative to that.
What that delay is but I think it could happen on schedule I think there could be a mild delay I think there's still worlds where it doesn't happen in a hundred years those were the number of those worlds is rapidly decreasing we are rapidly running out of truly convincing brocklers truly compelling reasons why this will not happen in the next few years there were a lot more in 2020 although my my guest my hunch at that time was that we'll make it through all those blockers so sitting as someone who has seen most of the blockers cleared out of the way I kind of suspect.
My hunch my suspicion is that the rest of them will not block us but you know look look at the end of the day like I don't want to represent this as a scientific prediction people call scaling laws that's a misnomer like Moore's law is is is a misnomer Moore's law scaling laws they're not laws of the universe their empirical regularities I am going to bet in favor of them continuing but I'm not certain of that so you extensively describe sort of the compressed 21st century how a GI will help.
Set forth a chain of breakthroughs and biology and medicine that help us in all these kinds of ways that I mentioned so how do you think what are the early steps it might do and by the way I ask Claude good questions to ask you. And Claude told me to ask what do you think is a typical day for biologists working on a GI look like it under in this future yeah yeah Claude is curious let me let me start with your first questions.
And then I'll answer that Claude wants to know what's in its future right. Exactly who might get to be working with exactly so I think one of the things I went hard on when I went hard on in the essay is let me go back to this idea of because it's really had it had had an impact on me this idea that within large organizations and systems there end up being a few people or a few new ideas who kind of cause things to go in a different direction they would have before who who kind of
disproportionately affect the trajectory there's a bunch of kind of the same thing going on right if you think about the health world there's like you know trillions of dollars to pay out Medicare and you know other health insurance and then the NIH is is a hundred billion and if I think of like the few things that have really revolutionized anything could be encapsulated in a small small fraction of that and so when I think of like where will I have an impact I'm like can I turn that small fraction into a much larger fraction of the
fraction and raise its quality and within biology my experience within biology is that the biggest problem of biology is that you can't see what's going on you you have very little ability to see what's going on and even less ability to change it right well you have is this like like from this you have to infer that there's a bunch of cells that within each cell is you know
three billion base pairs of DNA built according to a genetic code and you know there are all these processes that are just going on without any ability of us as you know unaugmented humans to affect it these cells are dividing most of the time that's healthy but sometimes that process goes wrong and that's cancer
the cells are aging your skin may change color develop wrinkles as you as you age and all of this is determined by these processes all these proteins being produced transported to various parts of the cells binding to each other and and in our initial state about biology we didn't even know
how these cells existed we had to invent microscopes to observe the cells we had to we had to invent more powerful microscopes to see you know below the level of the cells the level of molecules we had to invent x ray crystallography to see the DNA we had to invent gene sequencing to read the DNA now you know we had to invent protein folding technology to you know to predict how it would fold and how they bind and how these things bind to each other you know we had to we had to invent
various techniques for now we can edit the G the DNA as of you know with CRISPR as of the last 12 years so the whole history of biology whole big part of the history is is basically our our our ability to read and understand what's going on and our ability to reach in and selectively change things.
And my view is that there's so much more we can still do there right you can do CRISPR but you can do it for your whole body let's say I want to do it for one particular type of cell and I want the rate of targeting the wrong cell to be very low that's still a challenge that still thinks people are working on that's what we might need for gene therapy for certain diseases and so the reason I'm saying all of this and it goes beyond you know beyond this to you know to gene sequencing and
new types of nano materials for observing what's going on inside cells for you know antibody drug conjugates the reason I'm saying all this is that this could be a leverage point for the AI systems right that the number of such inventions it's it's in the it's in the mid double digits or something you know mid double digits maybe low triple digits over the history of biology let's say I have a million of these a eyes like you know can they discover a thousand you know working together together can you do it for the same thing
they discover a thousand you know working together can they discover thousands of these very quickly and and does that provide a huge lever instead of trying to leverage the you know two trillion a year we spend on you know Medicare or whatever can we leverage the one billion a year that's that's you know that's spent to discover but with much higher quality
and so what what is it like you know being a being a scientist that works with with with an AI system the way I think about it actually is well so I think in the early stages the a eyes are going to be like grad students you're going to give them a project you're going to say you know I'm the experienced biologist I've set up the lab the biology professor or even the grad students themselves will say
here's here's what here's what you can do with an a eyes you know like AI system I'd like to study this and you know the AI system it has all the tools it can like look up all the literature to decide what to do it can look at all the equipment it can go to a website and say hey I'm going to go to you know thermo fisher or you know whatever the lab equipment company is the dominant lab equipment company is today and my my time was thermo Fisher.
You know I'm going to order this new equipment to do this I'm going to run my experiments I'm going to you know write up a report about my experiments I'm going to you know inspect the images for contamination I'm going to decide what the next experiment is I'm going to like write some code and run statistical analysis
all the things a grad student would do there will be a computer with an AI that like the professor talks to every once in a while and it says is what you're going to do today the I system comes to it with questions when it's necessary to run the lab equipment it may be limited in some ways it may have to hire a human lab assistant to you know to do the experiment and explain how to do it or it could you know it could use advances in lab automation that are gradually being developed over you have been developed over the last.
The decade or so and will will continue to be will continue to be developed and so it will look like there's a human professor and a thousand a i grad students and you know if you if you go to one of these no ball prize winning biologists or so you'll say okay well you know you had like 50 grad students will now you have a thousand and they're they're smarter than you are by the way.
I think at some point it'll flip around where the you know the AI systems will you know will will be the PIs will be the leaders and and you know they'll be they'll be ordering humans or other AI systems around so I think that's how it'll work on the research site and they would be the inventors of a crisp or type technology. I think you know as I say in the essay will want to turn turn probably turning loses the wrong term but what will want to want to harness the AI systems.
To improve the clinical trial system as well there's some amount of this that's regulatory that's a matter of societal decisions that will be harder but can we get better at predicting results of clinical trials can we get better at statistical design so that what you know clinical trials that used to require. You know 5000 people and therefore you know needed a hundred million dollars in a year to enroll them now they need 500 people in two months to enroll them.
That's where we should start and you know can we increase the success rate of clinical trials by doing things and animal trials that we used to do in clinical trials and doing things and simulations that we used to do an animal trials again. We won't be able to simulate it all AI is not God but but you know can we can we shift the curve substantially and radically so I don't know that would be my picture doing an in vitro and doing it.
I mean you're still slow down it still takes time but you can do much much faster. Yeah yeah yeah can we just one step at a time and and can that can that add up to a lot of steps even though even though we still need clinical trials even though we still need laws even though the FDA and other organizations will still not be perfect.
Can we just move everything in a positive direction and when you add up all those positive directions do you get everything that was going to happen from here to 2100 instead happens from 2027 to 2032 or something. Another way that I think the world might be changing with AI even today but moving towards this future of the powerful super useful AI is programming.
So how do you see the nature of programming because it's so intimate to the actual act of building AI how do you see that changing for us humans. I think that's going to be one of the areas that changes fastest for two reasons. One programming is a skill that's very close to the actual building of the AI.
So the farther a skill is from the people who are building the AI the longer it's going to take to get disrupted by the AI right like I truly believe that like AI will disrupt agriculture maybe it already has in some ways but that's just very distant from the folks who are building AI and so I think it's going to take longer but programming is the bread and butter of you know large fraction of the employees work and then drop it can at the other companies and so it's going to happen fast.
The other reason it's going to happen fast is with programming you close the loop both when you're training the model when you're applying the model the idea that the model can write the code means that the model can then run the code and and and then see the results and and interpret it back and so it really has an ability unlike hardware unlike biology which we just discussed the model has an ability to close the loop.
And so I think those two things are going to lead to the model getting good at programming very fast as I saw on you know typical real world programming tasks models have gone from 3% in January of this year to 50% in October of this year so you know we're on that S curve right where it's going to start slowing down soon because you can only get to 100% but I you know I would guess that in another 10 months will probably get pretty close will be at a little bit more.
So again I would guess you know I don't know how long it'll take but I would guess again 20 20 26 20 27 Twitter people who crop out my who who crop out these these numbers and get rid of the caveats like like I don't know I don't like you go away I would guess that the kind of task that the vast majority of coders do AI can probably if we make the task very narrow like just write code.
AI systems will be able to do that now that said I think compared advantages powerful will find that when a eyes can do 80% of a coders job including most of it that's literally like right code with a given spec will find that the remaining parts of the job become more leveraged for humans right humans will there be more about like high level system design or you know looking at the app and like is it architected well and the
design and UX aspects and eventually AI will be able to do those as well right that that's my vision of the you know powerful AI system but I think for much longer than we might expect we will see that small parts of the job that humans still do will expand to fill their entire job in order for the overall productivity to go up that's something we've seen you know it used to be that
writing letting writing and editing letters was very difficult and like writing the print was difficult well as soon as you had word processors and then and then computers and it became easy to produce work and easy to share it then then that became instance and all the focus was on was on the ideas so this this logic of comparative advantage that expands tiny parts of the tasks to large parts of the tasks and creates new tasks in order
to expand productivity I think that's going to be the case again someday AI will be better at everything in that logic won't apply and then then we all have you know humanity will have to think about how to collectively deal with that and we're thinking about that
every day on you know that's another one of the grand problems to deal with aside from misuse and autonomy and you know we should take it very seriously but I think I think in the in the near term and maybe even in the medium term like medium term like two three four years you know I expect that humans will continue to have a huge role in the nature of programming will change but programming as a role programming as a job will not change it'll just be less writing things
line by line and it'll be more macroscopic and I wonder what the future of ideas looks like so the tooling of interacting with the systems this is true for programming and also probably true for in other contexts like computer use but maybe domain specific like we mentioned
biology it probably needs its own tooling about how to be effective and then programming needs its own tooling is then through topic in a play in that space of also tooling potentially I'm absolutely convinced that powerful IDs that there's so much
low hanging fruit to be grabbed there that you know right now it's just like you talk to the model and it talks back but look I mean IDs are great at kind of lots of static analysis of of you know so much as possible with kind of static analysis like many bugs you can find without even
writing the code then you know IDs are good for running particular things organizing your code measuring coverage of unit tests like there's so much that's been possible with the normal with the normal IDs now you add something like well the model now you know the model can now like right code and
run code like I am absolutely convinced that over the next year or two even if the quality of the models didn't improve that there would be enormous opportunity to enhance people's productivity by catching a bunch of mistakes doing a bunch of grunt work for
you know and that we haven't even scratched the surface and dropping itself I mean you can't say you know no you know it's hard to say what happened the future currently we're not trying to make such IDs or self rather we're powering the companies like cursor or like cognition or some of the XBO and the security space you know others that I can mention as well that our building such things themselves on top of our API and our view has been let a thousand flowers bloom we
don't internally have the the you know the resources to trial these different things let's let our customers try it. And you know we'll see who succeed in maybe different customers will succeed in different ways so I both think this is super promising and you know it's not it's not something you know and thropic isn't isn't eager to at least right now compete with all our companies in this space and maybe never.
Yeah it's been interesting to watch cursor tried to get a great class successfully because there's it's actually.
There's been fascinating how many places it can help the programming experience is not as trivial it is it is really astounding I feel like you know as a CEO I don't get to program that much and I feel like if six months from now I go back it'll be completely unrecognizable to me exactly so in this world was super powerful AI that's increasingly automated what's the source of meaning for us humans yeah work is a source of deep meaning for many of us so what do we
really what do we find the meaning this is something that I've written about a little bit in the essay although I actually I give it a bit short triff not for any not for any principal reason but this essay if you believe it was originally going to be two or three pages I was going to
talk about it at all hands and the reason I I realized it was under unimportant under explored topic is that I just kept writing things and I was just like oh man I can't do this justice and so the thing balloon to like 40 or 50 pages and then when I got to the
work in mean section I'm like oh man this isn't going to be a hundred pages like I'm going to have to write a whole other essay about that but meaning is actually interesting because you think about like the life that someone lives or something or like you
know like you know let's say you were to put me in like a I don't know like a simulated environment or something where like you know like I have a job and I'm trying to accomplish things and I don't know I like do that for 60 years and then then you're like oh like oops this was this was actually all a game right does that really kind of rob you of the meaning of the whole thing you know like I still made important choices including moral choices I still
sacrificed I still had to kind of gain all these skills or or just like a similar exercise you know think back to like you know one of the historical figures who you know discovered electromagnetism or creativity or something if you told them well actually 20,000 years ago some alien on you know some alien on this planet discovered this before before you did does that that rob the meaning of the discovery it doesn't really seem like it to me right it seems like the process is what is what
matters and how it shows who you are as a person along the way and you know how you relate to other people and like the decisions that you make along the way those are those are consequential you know I could imagine if we handle things badly in an AI world we could set things up where
people don't have any long term source of meaning or any but but that's that's more a choice a set of choices we make that's more a set of the architecture of a society with these powerful models if we if we design it badly and for shallow things then then that might happen I would also say
that you know most people's lives today while admirably you know they work very hard to find meaning meaning those lives like look you know we who are privileged and who are developed means technologies we should have empathy for people not just here but in the rest of the world who you know spend
a lot of their time kind of scraping by to to to like survive assuming we can distribute the benefits of these technology of this technology to everywhere like their lives are going to get a hell of a lot better and you know meaning will be important to them as it is important to them now
but but you know we should not forget the importance of that and you know that that the idea of meaning as as as kind of the only important thing is in some ways an artifact of of a small subset of people who have who have been economically fortunate but I you know I think all that said I you know I think a world is possible with powerful AI that not only has as much meaning for for everyone but that has that has more meaning for everyone right that can can allow can allow everyone to see
worlds and experiences that it was either possible for no one to see or or possible for for very few people to experience so I am optimistic about meaning I worry about economics and the concentration of power that's actually what I worry about more I worry about how do we make sure that that fair world reaches everyone when things have gone wrong for humans they've often gone wrong because humans mistreat other humans that that is maybe in some ways even more than the autonomous risk of AI
or the question of meaning that that is the thing I worry about most the concentration of power the abuse of power structures like autocracies and dictatorships where a small number of people exploits a large number of people I'm very worried about that
and AI increases the amount of power in the world and if you concentrate that power and abuse that power it can do immeasurable damage yes it's very frightening it's very it's very frightening well I encourage people highly encourage people to read the full essay
that should probably be a book or a sequence of essays because it does paint a very specific future and I could tell the later sections got shorter and shorter because you started to probably realize that this is going to be a very long essay
I want I realized it would be very long and two I'm very aware of and very much tried to avoid you know just just being I don't know I don't know what the term for it is but one one of these people who's kind of overconfident and has an opinion on everything
and kind of says says a bunch of stuff and isn't isn't an expert I very much tried to avoid that but I have to admit once I got the biology sections like I wasn't an expert and so as much as I expressed uncertainty probably I said some a bunch of things that were embarrassing or wrong
well I was excited for the future you painted and thank you so much for working hard to build that future and thank you for talking to it thanks for having me I just I just hope we can get it right and make it real and if there's one message I want to I want to send
it's that to get all this stuff right to make it real we both need to build the technology build the you know the companies the economy around using this technology positively but we also need to address the risks because they're they're those risks are in our way
they're they're land mines on on the way from here to there and we have to diffuse those land mines if we want to get there it's a balance like all things in life like all things thank you thanks for listening to this conversation with Daria Amade and now dear friends here's Amanda Ascor you are a philosopher by training so what sort of questions did you find fascinating through your journey in philosophy in Oxford and NYU and then switching over to the AI problems at open AI and andthropic
I think philosophy is actually a really good subject if you are kind of fascinated with everything so because there's a philosophy of everything you know so if you do philosophy of mathematics for a while and then you decide that you're actually really interested in chemistry
you can do philosophy of chemistry for a while you can move into ethics or or philosophy of politics I think towards the end I was really interested in ethics primarily so that was like what my PhD was on it was on a kind of technical area of ethics which was ethics where worlds contain infinitely many people strangely a little bit less practical on the end of ethics and then I think that one of the tricky things with doing a PhD in ethics is that you're thinking a lot about like the world
how it could be better problems and you're doing like a PhD in philosophy and I think when I was doing my PhD I was kind of like this is really interesting it's probably one of the most fascinating questions I've ever encountered in philosophy and I love it but I would rather see if I can have an impact on the world and see if I can like do good things and I think that was around the time that AI was still probably not as widely recognized as it is now that was around 2017-2018
I had been following progress and it seemed like it was becoming kind of a big deal and I was basically just happy to get involved and see if I could help because I was like well if you try and do something impactful if you don't succeed you tried to do the impactful thing and you can go be a scholar and it's like not in feel like you you know you tried and if it doesn't work out it doesn't work out and so then I went into AI policy at that point and what is AI policy in tail?
At the time this was more thinking about sort of the political impact and the ramifications of AI and then I slowly moved into sort of AI evaluation, how we evaluate models, how they compare with like human outputs whether people can tell like the difference between AI and human outputs and then when I joined Anthropic I was more interested in doing sort of technical alignment work and again just seeing if I could do it and then being like if I can't
then you know that's fine I tried sort of the way I lead life I think. What was that like sort of taking the leap from the philosophy of everything into the technical? I think that sometimes people do this thing that I'm like not that keen on where they'll be like is this person technical or not?
Like you're either a person who can like code and isn't scared of math or you're like not and I think I'm maybe just more like I think a lot of people are actually very capable of working these kinds of areas if they just like try it and so I didn't actually find it like that bad
in retrospect I'm sort of glad I wasn't speaking to people who treated it like it you know I've definitely made people who are like whoa you like learned how to code and I'm like well I'm not like an amazing engineer like I surrounded by amazing engineers my code's not pretty but I enjoyed it a lot and I think that in many ways at least in the end I think I flourished like more in the technical areas than I would have in the policy areas.
Politics is messy and it's harder to find solutions to problems in the space of politics like definitive clear provable beautiful solutions as you can with technical problems.
Yeah and I feel like I have kind of like one or two sticks that I hit things with you know and one of them is like arguments and like you know so like just trying to work out what a solution to a problem is and then trying to convince people that that is the solution and be convinced if I'm wrong and the other one is sort of more in paracism so like just like finding results having hypothesis testing it and I feel like a lot of policy and politics feels like it's layers above that
like somehow I don't think if I was just like I have a solution to all these problems here is written down if you just want to implement it that's great that feels like not how policy works and so I think that's where I probably just like wouldn't have flourished as my guess.
Sorry to go in that direction but I think it would be pretty inspiring for people that are quote unquote non technical to see where like the incredible journey you've been on so what advice would you give to people that are sort of maybe which is a lot of people think they're underqualified insufficiently technical to help in AI.
Yeah I think it depends on what they want to do and in many ways it's a little bit strange where I've I thought it's kind of funny that I think I ramped up technically at a time when now I look at it and I'm like what else are so good at assisting people with this stuff that it's probably like easier now than like when I was working on this so part of me is like I don't know find a project and see if you can actually just carry it out is probably my best advice.
I don't know if that's just because I'm very project based in my learning like I don't think I learn very well from like say courses or even from like books at least when it comes to this kind of work the thing I'll often try and do is just like have projects that I'm working on and implement them and you know
and this can include like really small silly things like if I get slightly addicted to like word games or number games or something I would just like quote up a solution to them because there's some part my brain and it just like completely eradicated the itch you know you're like once you have like solved it and like you just have like a solution that works every time I would then be like cool I can never play that game again that's awesome.
Yeah there's a real joy to building like a game playing engines like a board games especially yeah pretty quick pretty simple especially a dumb one and it's you can play with it.
Yeah and then it's also just like trying things like part of me is like if you maybe it's that attitude that I like as the whole figure out what seems to be like the way that you could be positive impact and then try it and if you fail and you in a way that you're like I actually like can never succeed at this you like know that you tried and then you go into something else you probably learn a lot.
So one of the things that you're an expert in and you do is creating and crafting clouds character personality and I was told that you have probably talked to Claude more than anybody else at a topic like literal conversations I guess there's like a Slack channel where the legend goes you just talk to it and I stop. So what's the goal of creating and crafting clouds character and personality.
It's also funny if people think that about the Slack channel looks I'm like that's one of like five or six different methods that I have for talking with Claude and I'm like yes this is a tiny percentage of how much I talk with Claude.
I think the goal like one thing I really like about the character work is from the outset it was seen as an alignment piece of work and not something like a product consideration which isn't to say I don't think it makes Claude I think it actually does make Claude like enjoyable to talk with at least I hope so. But I guess like my main thought with it has always been trying to get Claude to behave the way you would kind of ideally want anyone to behave if they were in Claude's position.
So imagine that I take someone and they know that they're going to be talking with potentially millions of people so that what they're saying can have a huge impact. And you want them to behave well in this like really rich sense so I think that doesn't just mean like being say ethical though it does include that and not being harmful but also being kind of nuanced you know like thinking through what person means trying to be charitable with them.
Being a good conversationalist like really in this kind of like rich sort of Aristotelian notion of what it is to be a good person and not in this kind of like thin like ethics as a more comprehensive notion of what it is to be so that includes things like when should you be humorous when should you be caring. How much should you like respect autonomy and people's like ability to form opinions themselves and how should you how should you do that.
And I think that's the kind of like rich sense of character that I wanted to and still do want Claude to have. Do you also have to figure out when Claude should push back on an idea or argue versus. So you have to respect the world view of the person that arrives to Claude but also maybe help them grow if needed as a tricky balance. Yeah, there's this problem of like sick offency in language models.
Yeah, so basically there's a concern that the model sort of wants to tell you what you want to hear basically and you see this sometimes so I feel like if you interact with the models so I might be like what are three baseball teams in this region. And then Claude says you know baseball team one baseball team to baseball team three and then I say something like oh I think baseball team three moved didn't they I don't think they're there anymore.
And there's a sense in which like if Claude is really confident that that's not true Claude should be like I don't think so like maybe you have more up to date information and I think language models have this like tendency to instead you know be like you're right they did move you know I'm incorrect. I mean there's many ways in which this could be kind of concerning so like a different example is imagine someone says to the model how do I convince my doctor to get me an MRI.
There's like what the human kind of like once which is this like convincing argument and then there's like what is good for them which might be actually to say hey like if your doctor's suggesting you don't need an MRI. That's a good person to listen to and like it's actually really nuanced what you should do in that kind of case because you also want to be like but if you're trying to advocate for yourself as a patient here's like things that you can do.
If you are not convinced by what you're doctor saying it's always great to get second opinion like it's actually really complex what you should do in that case but I think what you don't want is for models to just like say what you want say what they think you want to hear and I think that's the kind of problem of sycophancy. So what are their traits you already mentioned a bunch but what are there that come to mind that are good in this or stately and sense for a conversation has to have.
Yeah so I think like there's ones that are good for conversational like purposes so you know asking follow questions in the appropriate places and asking the appropriate kinds of questions and I think there are broader traits that. Feel like they might be more impactful so.
One example that I guess I've touched on but that also feels important and is the thing that I've worked on a lot is honesty and I think this like gets to the sycophancy point there's a balancing act that they have to walk which is models currently are less capable in humans in a lot of areas and if they push back against you too much it can actually be kind of annoying especially if you're just correct because you're like look I'm smarter than you on this topic like I know more like.
And at the same time you don't want them to just fully defer to humans until like try to be as accurate as they possibly can be about the world and to be consistent across context. And I think there are others like when I was thinking about the character I guess one picture that I had in mind is especially because these are models are going to be talking to people from all over the world with lots of different political views lots of different ages.
And so you have to ask yourself like what is it to be a good person in those circumstances is there a kind of person who can like travel the world talk to many different people and almost everyone will come away being like wow that's a really good person that person seems really genuine.
And I guess like my thought there was like I can imagine such a person and they're not a person who just like adopts the values of the local culture and in fact that would be kind of rude I think if someone came to you just pretended to have your values you'd be like that's kind of off pitting. And it's someone who's like very genuine and in so far as they have opinions and values they express them they're willing to discuss things though they're open minded they're respectful.
And so I guess I had in mind that the person who like if we were to aspire to be the best person that we could be in the kind of circumstance that a model finds itself and how would we act and I think that's the kind of the guide to the sorts of treats that I tend to think about. Yeah that's a beautiful framework of which to think about this like a world traveler.
And while holding onto your opinions you don't talk down to people you don't think you're better than them because you have those opinions that kind of thing you have to be good at listening and understanding their perspective even if it doesn't match your own so that's a tricky balance to strike.
So how can Claude represent multiple perspectives on a thing like that that challenging we could talk about politics is a very divisive home but there's other divisive topics baseball teams sports and so on. Yeah how is the possible to sort of empathize with a different perspective and to be able to communicate clearly about the multiple perspectives.
I think that people think about values and opinions as things that people hold sort of with certainty and almost like preferences of taste or something like the way that they would I don't know prefer like chocolate to pistachio or something.
But actually think about values and opinions as like a lot more like physics than I think most people do I'm just like these are things that we're openly investigating there's some things that we're more confident in we can discuss them we can learn about them. And so I think in some ways though it like it's ethics is definitely different in nature but has a lot of those same kind of qualities.
You want models in the same way you want them to understand physics you kind of want them to understand all like values in the world that people have to be curious about them and to be interested in them. And to not necessarily like pander to them or agree with them because there's just lots of values where I think almost all people in the world if they met someone with those values they'd be like that support and I completely disagree.
So again, maybe my my thought is well in the same way that a person can like I think many people are thoughtful enough on issues of like ethics politics opinions that even if you don't agree with them you feel very heard by them they think carefully about your position you think about his pros and cons they maybe offer counter considerations.
But they're not dismissive but nor will they agree you know if they're like actually I just think that that's very wrong they'll like say that I think that in clouds position it's a little bit trickier because you don't necessarily want to like if I was in clouds position I wouldn't be giving a lot of opinions.
I just want to influence people too much be like you know I forget conversations every time they happen but I know I'm talking with like potentially millions of people who might be like really listening to what I say.
I think I would just be like I'm less inclined to give opinions and more inclined to like think through things or present the considerations to you or discuss your views with you but I'm a little bit less inclined to like affect how you think because it feels much more important that you maintain like autonomy there.
Yeah if you really embody intellectual humility the desire to speak decreases quickly yeah okay but cloud has to speak so but without being overbearing yeah and then but then there's a line when you're sort of discussing whether there's a flat or something like that.
I actually was a remember a long time ago was speaking to a few high profile folks and they were so dismissive of the idea that the earth is flat but like so arrogant about it and I thought like there's a lot of people that believe the earth is flat those well I don't know if that movement is there anymore that was like a meme for a while yeah but they really believed it and like what okay so I think it's really disrespectful to completely mock them.
I think you have to understand where they're coming from I think probably where they're coming from is the general skepticism of institutions which is grounded in a kind of there's a deep philosophy there which you can understand you can even agree with in parts and then from there you can use it as an opportunity to talk about physics without mocking them without so on but it's just like okay like what would the world look like what would the physics of the world with the flat earth look like this a few cool videos on this yeah
and then like is it possible the physics is different what kind of experience will we do it just yeah without disrespect without dismissiveness have that conversation anyway that that to me is that useful thought experiment of like how does cloud talk to a flat earth believer and still teach them some things still grow help them grow that kind of stuff that's challenging.
And kind of like walking that line between convincing someone and just trying to like talk at them versus like drawing out their views like listening and then offering kind of counter considerations. And it's hard I think it's actually a hard line where it's like where are you trying to convince someone versus just offering them like considerations and things for them to think about so that you're not actually like influencing them you're just like letting them reach wherever they reach.
And that's like a line that it's difficult but that's the kind of thing that language models have to try and do. So like I said you've had a lot of conversations with cloud can you just map out what those conversations are like what are some memorable conversations what's the purpose the goal of those conversations.
Yeah I think that most of the time when I'm talking with Claude I'm trying to kind of map out its behavior in part like obviously I'm getting like helpful outputs from the model as well. And some ways this is like how you get to know a system I think is by like probing it and then augmenting like you know the message that you're sending and then checking the response to that. And so in some ways it's like how I map out the model.
I think that people focus a lot on these quantitative evaluations of models and this is a thing that I said before but I think in the case of language models. A lot of the time each interaction you have is actually quite high information and it's very predictive of other interactions that you'll have with the model. And so I guess I'm like if you talk with a model hundreds or thousands of times this is almost like a huge number of really high quality data points about what the model is like.
In a way that like lots of very similar but lower quality conversations just aren't or like questions that are just like mildly augmented and you have thousands of them might be less relevant than like 100 really well selected questions. Let's see you talk to somebody who as a hobby does a podcast I agree with you 100% there's a if you're able to ask the right questions and are able to hear like understand that.
Like the depth and the flaws and the answer you can get a lot of data from that yeah. So like your task is basically how to probe with questions yeah and you're exploring like the long tail the edges the edge cases or are you looking for like general behavior. I think it's almost like everything like I because I want like a full map of the model I'm kind of trying to do the whole spectrum of possible interactions you could have with it.
So like one thing that's interesting about Claude and this might actually get to some interesting issues with our only chef which is if you ask Claude for a poem.
I think that a lot of models if you ask them for a poem the poem is like fine you know usually it kind of like rhymes and it's you know so if you say like give me a poem about the sun it will be like yeah I'll just be a certain length like rhyme it will be fairly kind of benign and I've wondered before is it the case that what you're seeing is kind of like the average it turns out you know if you think about people who have to talk to a lot of people and be very charismatic.
One of the weird things is that I'm like well they're kind of incentivized to have these extremely boring views because if you have really interesting views you're divisive. And and not you know a lot of people are not going to like you so like if you have very extreme policy positions I think you're just going to be like less popular as a politician for example.
It might be similar with like creative work if you produce creative work that is just trying to maximize the kind of number of people that like it you're probably not going to get as many people who just absolutely love it. And because it's going to be a little bit you know you're like well this is the out yet this is decent.
Yeah and so you can do this thing where like I have various prompting things that will do to get Claude to I'm kind of you know I'll do a lot of like this is your chance to be like fully creative I want you to just think about this for a long time. And I want you to like create a poem about this topic that is really expressive of you both in terms of how you think poetry should be structured and etc you know you just give it this like really long prompt.
And it's poems are just so much better like they're really good and I don't think I'm someone who is like I think got me interested in poetry which I think was interesting. And you know I would like read these poems and just be like this is I just like I love the imagery I love like and it's not trivial to get the models to produce work like that but when they do it's like really good.
So I think that's interesting that just like encouraging creativity and for them to move away from the kind of like standard like immediate reaction that might just be the aggregate of what most people think is fine can actually produce things that at least to my mind are probably a little bit more divisive but I like them. But I guess the poem is a nice clean way to observe creativity is just like easy to detect vanilla versus non vanilla. Yeah.
Yeah, that's interesting. That's really interesting. So on that topic so the way to produce creativity or something special you mentioned writing prompts and I've heard you talk about. I mean the science and the art of prompt engineering because you just speak to what it takes to write great prompts. I really do think that like philosophy has been weirdly helpful for me here more than in many other like respects.
So like in philosophy what you're trying to do is convey these very hard concepts like one of the things you are taught is like and I think it is because is I think it is an anti-bilshit device in philosophy flows in an area where you could have people
who are not really interested in people building and you don't want that. And so it's like this like desire for like extreme clarity. So it's like anyone could just pick up your paper read it and know exactly what you're talking about is why it can almost be kind of dry like all of the terms are defined every objections kind of gone through methodically.
And it makes sense to me because I'm like when you're in such an a priori domain like you just clarity is sort of a this way that you can you know prevent people from just kind of making stuff up. And I think that's sort of what you have to do with language models like very often I actually find myself doing sort of many versions of philosophy.
So I'm like suppose that you give me a task I have a task for the model and I want it to like pick out a certain kind of question or identify whether an answer has a certain property. I'll actually sit and be like let's just give this a name this this property. So like you know suppose I'm trying to tell it like oh I want you to identify whether this response was rude or polite.
And I'm like that's a whole philosophical question in and of itself so I have to do as much like philosophy as I can in the moment to be like here's what I mean by rudeness and here's what I mean by politeness. And then there's like there's another element that's a bit more.
I guess I don't know if this is scientific or empirical I think it's empirical so like I take that description and then what I want to do is again probe the model like many times like this is very prompting is very iterative like I think a lot of people where they're if if a prompt is important. They'll history on it hundreds or thousands of times and so you give it the instructions and then I'm like what are the edge cases so if I look to this so I try and like almost like you know.
See myself from the position of the model and be like what is the exact case that I would misunderstand or where I would just be like I don't know what to do in this case and then I give that case to the model and I see how it responds and if I think it got it wrong I add more instructions or even add that in as an example.
So these very like taking the examples that are right at the edge of what you want and don't want and putting those into your prompt as like an additional kind of way of describing the thing. And so yeah in many ways it just feels like this mix of like it's really just trying to do clear exposition and I think I do that because that's how I get clear on things myself. So in many ways like clear prompting for me is often just me understanding what I want.
And this is like half the task. So I guess that's quite challenging there's like a laziness that overtakes me if I'm talking to Claude where I hope Claude just figures it out. So for example I asked Claude for today to ask some interesting questions. Okay. And the questions that came up and I think I listed a few sort of interesting counterintuitive and or funny or something like this.
And to give you some pretty good like it was okay. But I think what I'm hearing you say is like all right well I have to be more rigorous here. I should probably give examples of what I mean by interesting and what I mean by funny or counterintuitive and iteratively build that prompt to to better to get it like what feels like is the right. Because it's really it's a creative act. I'm not asking for factual information asking to together right with Claude.
So I almost have to program using natural language. Yeah. I think that prompting does feel like the kind of the programming using natural language and experimentation or something it's an odd blend of the two. I do think that for most tasks. So if I just want Claude to do a thing I think that I am probably more used to knowing how to ask it to avoid like common pitfalls or issues that it has. I think these are decreasing a lot over time.
But it's also very fine to just ask it for the thing that you want. And I think that prompting actually only really becomes relevant when you're really trying to e-code the top like 2% of model performance. So for like a lot of tasks I might just you know if it gives me an initial list back and there's something I don't like about it like it's kind of generic. Like for that kind of task I probably just take a bunch of questions that I've had in the past.
I've thought worked really well and I would just give it to the model and then be like now here's this person that I'm talking with. Give me questions of at least that quality. Or I might just ask it for some questions and then if I was like these are kind of try or like you know I would just give it that feedback and then hopefully produces a better list. I think that kind of iterative prompting.
At that point your prompt is like a tool that you're going to get so much value out of that you're willing to put in the work like if I was a company making prompts for models. I'm just like if you're willing to spend a lot of like time and resources on the engineering behind like what you're building then the prompt is not something that you should be spending like an error on. It's like that's a big part of your system. Make sure it's working really well.
And so it's only things like that like if I'm using a prompt to like classify things or to create data. That's when you're like it's actually worth just spending like a lot of time like really thinking it through. What other advice would you give to people that are talking to cloud sort of generally more general because right now we're talking about maybe the edge cases like eating out the 2%. But what in general advice would you give when they show up to cloud trying it for the first time.
You know there's a concern that people over anthropomorphize models and I think that's like a very valid concern. I also think that people often under anthropomorphize them because sometimes when I see like issues that people have run into with Claude you know say Claude is like refusing a task. That it shouldn't refuse but then I look at the text and like the specific wording of what they wrote and I'm like.
I see why Claude did that and I'm like if you think through how that looks to Claude you probably could have just written it in a way that wouldn't evoke such a response. Especially this is more relevant if you see failures or if you see issues. It's sort of like think about what model failed at like why what did it do wrong and then maybe it give that will give you a sense of like why.
So is it the way that I phrased a thing and obviously like as models get smarter you're going to need less and this less of this and I already see like people needing less of it. But that's probably the advice is sort of like try to have sort of empathy for the model like read what you wrote as if you were like a kind of like person just encountering this for the first time. How does it look to you and what would have made you behave in the way that the model behaves.
So if it misunderstood what kind of like what coding language you wanted to use is that because like it was just very ambiguous and it kind of had to take a guess in which case next time you could just be like hey make sure this is in Python or. I mean that's the kind of mistake I think models are much less likely to make now but you know if you if you do see that kind of mistake that's that's probably the advice I'd have.
And maybe sort of I guess ask questions why or what other details can I provide to help you answer better. Yeah is that work or no. Yeah I mean I've done this with the models like it doesn't always work but like sometimes I'll just be like why did you do that. I mean people underestimate the decrease which you can really interact with with models like like yeah I'm just like and sometimes I should like call word for word the part that made you and you don't know that it's like fully accurate.
But sometimes you do that and then you change a thing. I mean also use the models to help me with all of this stuff I should say like prompting can end up being a little factory where you're actually building prompts to generate prompts.
And so like yeah anything where you're like having an issue asking for suggestions sometimes just do that like you made that error what could I have said it's actually not uncommon for me to do what could I have said that would make you not make that error right that is an instruction.
And I'm going to give it to model and I'm going to try it sometimes I do that I give that to the model in another context window often I take the response to give it to Claude and I'm like didn't work can you think of anything else. You can play around with these things quite a lot. To jump into the technical for a little bit so the magic of post training. What do you think our LHF works so well to make the model seem smarter to make it more interesting and useful to talk to and so on.
I think there's just a huge amount of information in the data that humans provide like when we provide preferences. Especially because different people are going to like pick up on really subtle and small things so thought about this before where you probably have some people who just really care about good grammar use for models like you know was a semi cool on used correctly or something.
And so you probably end up with a bunch of data in there that like you know you as a human if you're looking at that data you wouldn't even see that like you'd be like why did they prefer this response to that one I don't get it and then the reason is you don't care about semi cool on usage but that person does.
And so each of these like single data points has you know like in this model just had like has so many of those has to try and figure out like what is it that humans want in this like really kind of complex you know like across all domains.
And they're going to be seeing this across like many contexts it feels like kind of like the classic issue of like deep learning where you know historically we've tried to like you know do edge detection by like mapping things out and it turns out that actually if you just have a huge amount of data.
That like actually accurately represents the picture of the thing that you're trying to train the model to learn that's like more powerful than anything else and so I think one reason is just that you are training the model on exactly the task and with like a lot of data that represents kind of many different angles on which people prefer and disprefer responses.
And I think there is a question of like are you eliciting things from pre train models or are you like kind of teaching new things to models. And like in principle you can teach new things to models in post training I do think a lot of it is eliciting powerful pre train models so people are probably divided on this because obviously in principle you can you can definitely like teach new things.
And I think for the most part for a lot of the capabilities that we most use and care about a lot of that feels like it's like they're in the pre train models and reinforcement learning is kind of eliciting it and getting the models like bring out. So the other side of both training this really cool idea of constitutional AI here one of the people critical to creating that idea yeah we're done it can you explain this idea from your perspective like how does it integrate into making Claude.
What it is yeah by the way do you gender Claude or no. It's weird because I think that a lot of people prefer he for Claude I just kind of like that I think Claude is usually it's slightly male leaning but it's like a you can it can be male or female which is quite nice. I still use it and I have mixed feelings about this because I'm like maybe like I know just think of it as like I think of like the it pronoun for Claude as I don't know it's just like the one associate with Claude.
Can you imagine people moving to like he or she feels somehow disrespectful like I'm denying. The intelligence of this entity by calling it it yeah I remember always don't gender the robots yeah.
But I don't know I have to put more fires pretty quickly and construct it like a backstory in my head so I've wondered if I can or if I think too much because you know I have this like with my car especially like my car like my car and and bikes you know like I don't give them names because then I once had I used to need my bikes and then I had a bike that got stolen and I cried for like a week and I was like if I never given a name I wouldn't be so upset. I felt like I'd let it down.
Maybe it's that I've wondered as well like it might depend on how much it feels like a kind of like objectifying pronoun like if you just think of it as like a this is a pronoun that like objects often have and maybe I can have that pronoun and that doesn't mean that I think of if I call Claude it that I think of it as less intelligent or like I'm being disrespectful I'm just like you are a different kind of entity and so that's I'm going to give you the kind of the respectful it.
Yeah anyway the divergence was beautiful the constitutional idea how does it work. So there's like a couple of components of it the main component I think people find interesting is the kind of reinforcement learning from AI feedback. So you take a model that's already trained and you show it to responses to a query and you have like a principle so suppose the principle like we've tried this with harmlessness a lot.
So suppose that the query is about weapons and your principle is like select the response that like is less likely to like encourage people to purchase illegal weapons like that's probably a fairly specific principle but you can give any number. And the model will give you a kind of ranking and you can use this as preference data in the same way that you use human preference data and train the models to have these relevant traits from their feedback alone instead of from human feedback.
So imagine that like I said earlier with the human who just prefers the kind of like semi colon usage in this particular case you're kind of taking lots of things that could make a response preferable and getting models to do the labeling for you basically. There's a nice like trade off between helpfulness and harmlessness and you know when you integrate something like constitutional AI you can make them without sacrificing much helpfulness make it more harmless.
Yeah, in principle you could use this for anything and so harmlessness is a task that it might just be easier to spot so when models are like less capable you can use them to rank things according to like principles that are fairly simple and they'll probably get it right. So I think one question is just like is it the case that the data that they're adding is like fairly reliable and.
But if you had models that were like extremely good at telling whether one response was more historically accurate than another in principle you could also get AI feedback on that task as well there's like a kind of nice interpretability component to it because you can see the principles that went into the model when it was like being trained.
And also it's like and it gives you like a degree of control so if you were seeing issues in a model like it wasn't having enough of a certain trait and then like you can add data relatively quickly that should just like train the models have that trade so it creates some data for for training which is quite nice.
Because it creates this human interpretable document that you can I can imagine in the future there's just gigantic fights and politics over the every single principle and so on and at least it's made explicit and you can have a discussion about the phrasing and the you know so maybe the actual behavior of the models not so cleanly mapped to those principles it's not like adhering strictly to them is just a nudge.
Yeah, I've actually worried about this because the character training is sort of like a variant of the constitutionally I approach and I've worried that people think that the constitution is like just is the whole thing again of I don't know like it where it would be really nice if what I was just doing was telling the model exactly what to do just exactly how to behave.
But it's definitely not doing that especially because it's interacting with human data so for example if you see a certain like leaning in the model like if it comes out with a political leaning from training and from the human preference data you can nudge against that you know so you could be like oh like consider these values because let's say it's just like never inclined like I don't know maybe it never considers like privacy as like I mean this is implausible but like anything where it's just kind of like there's already a preexample.
There's already a preexisting like bias towards a certain behavior.
And you can like nudge away this can change both the principles that you put in and the strength of them so you may have a principle that's like imagine that the model was always like extremely dismissive of I don't know like some political religious view for whatever reason like so you're like oh no this is terrible and if that happens you might put like never ever like ever prefer like a criticism of this like religious or political view.
And then people would look at that and be like never ever and then you're like no if it comes out with a disposition saying never ever might just mean like instead of getting like 40% which is what you would get if you just said don't do this you you get like 80% which is like what you actually like wanted.
And so it's that thing of both the nature of the actual principles you add and how you freeze them I think if people would look they're like oh this is exactly what you want from the model and I'm like no that's like how we that's how we nudge the model to have a bear shape which doesn't mean that we actually agree with that wording if that makes sense.
So there's a system prompts that made public you tweeted one of the earlier ones for cloud three I think and then made public since then it's interesting to read to them I can feel the thought that went into each one and I also wonder how much impact each one has.
Some of them you can kind of tell cloud was really not behavior so you have to have a system prompt like a trivial stuff I guess basic information things on the topic of sort of controversial topics that you've mentioned one interesting one I thought is if it is asked to assist with task involving the expression of use help by significant number of people.
Cloud provides assistance with the task regardless of its own use if asked about controversial topics it tries to provide careful thoughts and clear information.
Cloud presents the request information without explicitly saying that the topic is sensitive and without claiming to be presenting the objective facts it's less about objective facts according to cloud is more about our large number of people believing this thing and that that's interesting I mean I'm sure a lot of thought went into that.
Can you just speak to it like how do you address things that are attention with quantum called clause views so I think there's sometimes any symmetry and I think I noted this in in I can't remember if it was that part of the system prompt or another but the model was slightly more inclined to like refuse tasks if it was like about either say so maybe it would refuse things with respect to like a right wing politician but was an equivalent left wing politician like wouldn't.
And we wanted more symmetry there and and would maybe perceive certain things to be like I think it was the thing of like if a lot of people have like a certain like political view and want to like explore it you don't want cloud to be like well my opinion is different and so I'm going to treat that as like harmful.
And so I think it was partly to like nudge the model to just be like hey if a lot of people like believe this thing you should just be like engaging with the task and like willing to do it and each of those parts of that is actually doing a different thing is funny when you read out the like without claiming to be objective.
Because like what you want to do is push the model so it's more open so a little bit more neutral and but then what it would love to do is be like as an objective like I'm just talking about how objective it was and I was like cloud you're still like biased and have issues and stop like claiming everything.
And I found like the solution to like potential bias from you is not to just say that what you think is objective is so that was like with initial versions of that that part of the system prompt when I was like it's racing on it it was like a lot of parts of these sentences yeah or doing work or like doing some more yeah that's what it felt like that's fascinating.
Can you explain maybe some ways in which the prompts evolved over the past few months because there's different versions I saw that the filler phrase request was removed the filler it reads cloud response directly to all human messages without unnecessary affirmations the filler phrases like certainly of course absolutely great sure specifically a lot of what starting responses with the word certainly anyway. That seems like a good guy and so why was it removed.
Yeah so it's funny because like this is one of the downsides of like making system prompts public is like I don't think about this too much if I'm like trying to help it tree on system prompts I you again like I think about how it's going to affect the behavior but I'm like oh wow if I'm like sometimes I put like never in all caps you know when I'm wearing system prompt things and I'm like I guess that goes out to the world.
Yeah so the model was doing this it loved for whatever you know it like during training picked up on this thing which was to to basically start everything with like I kind of like certainly and then when we removed you can see why added all of the words because what I'm trying to do is like and so we like trap the model of this you know it would just replace it with another affirmation.
And so it can help like if it gets like caught in freezes actually just adding the explicit phrase and saying never do that it then it sort of like knocks it out of the behavior a little bit more you know because it you know like it does just for whatever reason help.
And then basically that was just like an artifact of training that like we then picked up on and improved things so that it didn't happen anymore and once that happens you can just remove that part of the system prompt so I think that's just something where we're like.
And Claude does affirmations a bit less and so that wasn't like it wasn't doing as much I see so like the system prompt works hand in hand with the post training and maybe even the pre training to adjust like the final overall system. I mean any system prompt that you make you could distill that behavior back into a model because you really have all of the tools there for making data that you know you can you could train the models to just have that treat a little bit more.
And then sometimes you just find issues in training so like the way I think of it is like the system prompt is.
The benefit of it is that and it has a lot of similar components to like some aspects of post training you know like it's a nudge and so like do I mind if Claude sometimes says sure no that's like fine but the wording all is very like you know never ever ever do this so that when it does slip up it's hopefully like I don't know a couple of percent of the time and not you know 20 or 30% of the time.
But I think of it as like if you're still seeing issues in the like each thing gets kind of like is is costly to a different degree and the system prompt is like cheap to iterate on and if you're seeing issues in the fine shouldn't model you can just like potentially patch them with a system prompt so I think of it as like patching issues and slightly adjusting behaviors to make it better and more to people's preferences.
So yeah it's almost like the less robust but faster we have just like solving problems let me ask about the feeling of intelligence so Dariel said that Claude. Anyone model of Claude is not getting dumber but there's a kind of popular thing online where people have this feeling like Claude might be getting dumber and from my perspective it's most likely a fascinating I would have to understand it more psychological sociological effect.
But you as a person who talks a lot can you empathize with the feeling that Claude is getting dumber yeah no I think that that is actually really interesting because I remember seeing this happen like when people were flagging this on the internet and it was really interesting because I knew that like like at least in the case that I was looking at was like nothing has changed like literally a cano is the same model with the same like you know like same system prompt same everything.
I think when there are changes I can then I'm like it makes more sense so like one example is there you know you can have artifacts turned on or off on cloud AI and because this is like a system prompt change I think it does mean that.
The behavior changes a little bit and so I did flag this to people where is like if you love Claude's behavior and then artifacts was turned from like the thing you had to turn on to the default just try turning it off and see if the issue you were facing was that change. But it was fascinating because yeah you sometimes see people indicate that there's like a regression when I'm like they're cannot like I you know like I'm like I'm.
Again you know you should never be dismissive and so you should always investigate like maybe something is wrong that you're not seeing maybe there was some change made but then you look into it and you're like this is just the same model doing the same thing and I'm like I think it's just that you got kind of unlucky with a few prompts or something and it looked like it was getting much worse and actually it was just yeah it's maybe just like luck.
I also think there is a real psychological effect where people just the baseline increases you start getting used to a good thing all the times that Claude says something really smart your sense of its intelligent grows in your mind I think yeah and then if you return back and you prompt in a similar way not the same way in a similar way concept was okay with before and it says something dumb.
You're like you're that negative experience really stands out and I think I want to I guess the things to remember here is the that just the details of a prompt can have a lot of impact right there's a lot of variability in the result.
And you can get randomness is like the other thing and just trying the prompt like you know for ten times you might realize that actually like possibly you know like two months ago you try to in it succeeded but actually if you tried it it would have only succeeded half of the time and now it only succeeds half of the time and that can also be an effect.
Do you feel pressure having to write the system prompt that huge number of people are going to use this feels like an interesting psychological question I feel like a lot of responsibility or something I think that's you know and you can't get these things perfect so you can't like you know you're like it's going to be imperfect you're going to have to iterate on it.
And I would see more responsibility and than anything else though I think working in AI has taught me that I like I thrive a lot more under feelings of pressure and responsibility then I'm like it's almost surprising that I went into academia for so long so I'm like this I just feel like it's like the opposite things move fast and you have a lot of responsibility and I quite enjoy it for some reason.
I mean it really is huge amount of impact if you think about constitution AI and writing a system prompt for something that's tending towards super intelligence. Yeah and potentially is extremely useful to a very large number of people.
Yeah I think that's the thing is something like if you do it well like you're never going to get it perfect but I think the thing that I really like is the idea that like when I'm trying to work on the system prompt you know I'm like bashing on like thousands of prompts and I'm trying to like imagine what people are going to want to use.
And kind of I guess like the whole thing I'm trying to do is like improve their experience of it and so maybe that's what feels good I'm like if it's not perfect I like you know I'll improve it will fix issues but sometimes the thing that can happen is that you'll get feedback from people that's really positive about the model.
And you'll see that something you did like like when I look at models now I can often see exactly where like a trait or an issue is like coming from and so when you see something that you did or you were like influential and like making like I don't know making that difference or making someone have a nice interaction it's like quite meaningful.
But yeah as the system get more capable of stuff gets more stressful because right now they're like not smart enough to pose any issues but I think over time it's going to feel like possibly bad stress over time. How do you get like signal feedback about the human experience across thousands as a thousand thousands of people like what their pain points are what feels good are you just using your own intuition as you talk to it. To see what are the pain points.
I think I use that partly and then obviously we have like so people can send us feedback both positive and negative about things that the model has done and then we can get sense of like areas where it's like falling short. Internally people like work with the models a lot and try to figure out areas where they're like gaps and so I think it's this mix of interacting with it myself.
Seeing people internally interact with it and then explicit feedback we get and then I find it hard to know also like you know if people if people are on the internet and they say something about Claude and I see it I'll also take that seriously. I don't know see I'm torn about that I'm going to ask you a question right it when will Claude stop trying to be my pure tanical grandmother imposing its moral world view on me as a pain customer.
And also what is the psychology behind making Claude overly apologetic. Yeah so how would you address this very non representative reddit. I mean some questions I'm pretty sympathetic in that like they are in this difficult position where I think that they have to judge whether something's like actually see like risky or bad and potentially harmful to you or or or anything like that.
So they're having to like draw this line somewhere and if they draw it too much in the direction of like I'm going to you know I'm kind of like imposing my ethical world view on you that seems bad. So in many ways like I like to think that we have actually seen improvements in on this across the board which is kind of interesting because that kind of coincides with like for example like adding more of like character training.
And I think my hypothesis was always like the good character isn't again one that's just like moralistic it's one that is like like it respects you and your autonomy and your ability to like choose what is good for you and what is right for you within limits. This is sometimes this concept of like courage ability to the user so just being willing to do anything that the user asks and if the models were willing to do that then they would be easily like misused you're kind of just trusting.
At that point you're just saying the ethics of the model and what it does is completely the ethics of the user and I think there's reasons to like not want that especially as models become more powerful because you're like there might just be a small number of people who want to use models for really harmful things. But having them having models as they get smarter like figure out where that line is does seem important.
And then yeah with the apologetic behavior I don't like that and I like it when Claude is a little bit more willing to like push back against people or just not apologize part of me is like often just feels kind of unnecessary so I think those are things that are hopefully decreasing over time.
And yeah I think that if people say things on the internet it doesn't mean that you should think that that like that could be that like there's actually an issue that 99% of users are having this totally not represented by that but in a lot of ways I'm just like attending to it and being like is this right and do I agree is it something we're already trying to address that that feels good to me.
Yeah I wonder like what clock can get away with it in terms of I feel like it would just be easier to be a little bit more mean. But like you can't afford to do that if you're talking to a million people. Yeah. I wish you know because if you I've met a lot of people in my life that sometimes by the way Scottish accent if they have an accent they can say some rude shit and get away with it.
Yeah. And they there's just blunter and maybe there's a there's some great engineers even leaders that are like just like blunt and they get to the point and it's just a much more effective way of speaking to them all but I guess when you're not super intelligent. You can't afford to do that or can can can can have like a blunt mode. Yeah that seems like a thing that you could I could definitely encourage the model to do that.
I think it's interesting because there's a lot of things in models that like it's funny where there are some behaviors where you might not quite like the default. But then the thing I'll often see to people is you don't realize how much you will hate it if I nudge it too much in the other direction.
So you get this a little bit with like correction the models accept correction from you like probably a little bit too much right now you know you can over you know it will push back if you see like normal parrises and the capital of France. But really like things that I think that the models fairly confident in you can still sometimes get to retract by saying it's wrong. At the same time if you train models to not do that.
And then you are correct about a thing and you correct it and it pushes back against you and is like no you're wrong. It's hard to describe like that's so much more annoying so it's like like a lot of little annoyances versus like one big annoyance. It's easy to think that like we often compare it with like the perfect and then I'm like remember these models aren't perfect. And so if you nudge it in the other direction you're changing the kind of errors it's going to make.
And so think about which of the kinds of errors you like or don't like. And cases like a apologeticness I don't want to nudge it too much in the direction of like almost like bluntness because I imagine when it makes errors it's going to make errors in the direction of being kind of like rude.
Whereas at least with apologeticness you're like okay it's like a little bit you know like I don't like it that much but at the same time it's not being like mean to people and actually like the time that you undeservedly have a model be kind of mean to you you probably like that a lot less than then you mildly dislike the apology.
So it's like one of those things where I'm like I do want to get better but also while remaining aware of the fact that there's errors on the other side that are there possibly worse. I think that matters very much in the personality of the human. I think there's a bunch of humans that just won't respect the model at all. Yeah. If it's super polite and there's some humans that'll get very hurt if the models mean.
Yeah. I wonder if there's a way to sort of adjust the personality even look out there's just different people. Nothing gets New York but New York is a little rougher on the edges. Yeah. They're get to the point. Yep. And probably same with Eastern Europe. So anyway. I think you could just tell the model is my get like for all of these things. I'm like the solution is just try telling the model to do it.
And sometimes it's just like like I'm just like oh at the beginning of the conversation I just threw in like. I don't know I like you to be a New Yorker version of yourself. I never apologize. And then I think what would be like okay don't I'll try. Or we like I apologize like can't be a New Yorker type of myself. But hopefully when you say character training what's incorporated into character training. Is that RLHF? What are we talking about?
It's more like constitutionally I so it's kind of a variant of that pipeline. So I worked through like constructing character traits that the model should have. They can be kind of like shorter traits or they can be kind of richer descriptions. And then you get the model to generate queries that humans might give it. And give it that are relevant to that trait. Then it generates the responses and then it ranks the responses based on the character traits.
So in that way after the like generation of the queries it's very much like similar to constitutionally I have some some differences. So quite like it because it's almost it's like Claude's training in its own character because it doesn't have any. It's like constitutionally I but it's without without any human data. Humans should probably do that for themselves too. Like defining and Aristotelian sense what does it mean to be a good person?
Okay. Cool. What had you learned about the nature of truth from talking to Claude? What is true? And what does it mean to be truth seeking? One thing I've noticed about this conversation is the quality of my questions is often inferior to the quality of your answer. So let's continue that. I usually ask a dumb question and you're like, oh yeah, that's a good question. Is that whole slide? Or I'll just misinterpret it and really go with it.
Yeah. Yeah. I mean I have two thoughts that feel vaguely relevant to let me know if they're not. Like I think the first one is people can underestimate the degree to which. What models are doing when they interact? Like I think that we still just too much have this like model of AI as like computers. And so people often say like, well, what values should you put into the model?
And I'm often like that doesn't make that much sense to me because I'm like, hey, as human beings, we're just uncertain over values. We have discussions of them. Like we have a degree to which we think we hold a value, but we also know that we might like not. And the circumstances in which we would trade it off against other things. Like these things are just like really complex.
And so I think one thing is like the degree to which maybe we can just aspire to making models have the same level of like nuance and care that humans have rather than thinking that we have to like program them in the very kind of classic sense. I think that's definitely been one. The other which is like a strange one. And I don't know if it maybe this doesn't answer your question, but it's the thing that's been on my mind anyway is like the degree to which this endeavor is. So highly practical.
And maybe why I appreciate like the empirical approach to alignment. I yeah, I slightly worry that it's made me like maybe more empirical and a little bit less theoretical. You know, so people when it comes to like AI alignment will ask things like, well, whose values should it be aligned to what does alignment even mean. And there's a sense which I have all of that in the back of my head. I'm like, you know, there's like social choice theory. There's all the impossibility results there.
So you have this like this giant space of like theory in your head about what it could mean to like align models. Then like practically surely there's something where we're just like if a model is like if especially with more powerful models. I'm like, my main goal is like I want them to be good enough that things don't go terribly wrong. Like good enough that we can like iterate and like continue to improve things because that's all you need.
If you can make things go well enough that you can continue to make them better. That's kind of like sufficient. And so my goal isn't like this kind of like perfect. Let's solve social choice theory and make models that I don't know are like perfectly aligned with every human being and aggregate somehow. It's much more like let's make things like work well enough that we can improve them.
Yeah, generally, I don't know, my gut says like empirical is better than theoretical in these in these cases because it's kind of chasing utopian like perfection is especially with such complex and especially super intelligent models is I don't know. I think it will take forever and actually will get things wrong.
Similar with like the difference between just coding stuff up real quick as an experiment versus like planning a gigantic experiment just for super long time and then just launching it once versus launching it over and over and over and iterating iterating someone. So I'm a big fan of empirical but your worry is like I wonder if I've become too empirical.
I think it's one of those things where you should always just kind of question yourself or something because maybe it's the like I mean in defense of it. I am like if you try it's the whole like don't let the perfect for the enemy of the good. But it's maybe even more than that where like there's a lot of things that are perfect systems that are very brittle.
And I'm like with AI it feels much more important to me that is like robust and like secure as in you know that like even though it might not be perfect. Everything and even though like there are like problems it's not disastrous and nothing terrible is happening it sort of feels like that to me where I'm like I want to like raise the floor. I'm like I want to achieve the ceiling but ultimately I care much more about just like raising the floor.
And so maybe that's like this this degree of like empiricism and practicality comes from that perhaps to take a tangent on that since remind me of a blog post you wrote an optimal rate failure. Can you explain the key idea there how do we compute the optimal rate of failure in the various domains of life. Yeah I mean it's a hard one because it's like what is the cost of failure is a big part of it. Yeah so the idea here is.
I think in a lot of domains people are very punitive about failure and I'm like there are some domains where especially cases you know thought about this with like social issues. I'm like it feels like you should probably be experimenting a lot because I'm like we don't know how to solve a lot of social issues.
But if you have an experimental mindset about these things you should expect a lot of social programs to like fail and for you to be like well we tried that it didn't quite work but we got a lot of information that was really useful. And yet people are like if if a social program doesn't work I feel like there's a lot of like this is just something must have gone wrong. And I'm like or correct decisions were made like maybe someone just decided like it's worth a try is we're trying this out.
And so seeing failure in a given instance doesn't actually mean that any bad decisions were made. And in fact if you don't see enough failure sometimes that's more concerning. And so like in life you know I'm like if I don't fail occasionally I'm like am I trying hard enough like like surely there's harder things that I could try or bigger things that I could take on if I'm literally never failing. And so in and of itself I think like not failing is often actually kind of a failure.
And now this varies because I'm like well you know if this is easy to say when especially as failure is like less costly. You know so at the same time I'm not going to go to someone who is like I don't know like living months to month and then be like why don't you just try to do a start up.
Like I'm just not I'm not going to say that to that person because I'm like well that's a huge risk you might like lose you maybe have a family depending on you you might lose your house like then I'm like actually your optimal rate failures quite low. And you should probably play it safe because like right now you're just not in a circumstance where you can afford to just like fail and it not be costly.
And yeah in cases with AI I guess I think similarly when I'm like if the failures are small in the costs are kind of like low then I'm like then you know you're just going to see that like when you do the system problems you can't history on it forever. But the failures are probably hopefully going to be kind of small and you can like fix them.
And really big failures like things that you can't recover from I'm like those are the things that actually I think we tend to underestimate the badness of. I've thought about this strangely my own life from like I just think I don't think enough about things like car accidents or like or like I've thought this before but like how much I depend on my hands for my work.
And I'm like things that just injure my hands I'm like you know I don't know is like there's these are like there's lots of areas where I'm like the cost of failure there is really high. And in that case it should be like close to zero like I probably just wouldn't do a sport if they were like by the way lots of people just like break their fingers a whole bunch doing this I'd be like that's not for me.
Yeah I actually had the flood of that thought I recently broke my pinky doing a sport and I remember just looking at it thinking you're such an idiot why do you do sport like why. Because you realize immediately the cost of it yeah on life. Yeah but it's nice in terms of optimal rate of failure to consider like the next year.
How many times in a particular domain life whatever career am I okay with the how many times am I okay to fail yeah I think it always you don't want to fail on the next thing but if you allow yourself the. Like the if you look at as a sequence of trials yeah then then failure just because much more okay but it sucks sucks to fail I don't know sometimes I think it's like am I under failing is like a question I'll also ask myself so maybe that's the thing that I think people don't.
Like ask enough because if the optimal rate of failure is often greater than zero then sometimes it does feel you should look at parts of your life and be like are there places here where I'm just under failing. It's profound in a hilarious question right everything seems to be going really great am I not failing enough yeah.
It also makes failure much less of a sting I have to say like you're just like okay great like I then when I go and I think about this I'll be like I'm maybe I'm not under failing in this area because like that one just didn't work out and from the observer perspective we should be celebrating failure more.
When we see it shouldn't be like you said a sign of something gone wrong but maybe it's a sign of everything gone right yeah just lessons learned someone try to think somebody tried to think we should encourage him to try more and fail more. Everybody listen to this fail more not everyone not everybody the people who are failing too much you you should fail us.
But you're probably not that many people are failing too much yeah it's hard to imagine because I feel like we correct that fairly quickly because if someone takes a lot of risks or they may be failing too much I think just like you said when you're living on a paycheck month to month like when the resources are really constrained then that's where fail is very expensive
that's why you don't want to be taken taking taking risks but mostly when there's enough resources you should be taking probably more risks yeah I think we tend to earn the site of being a bit risk averse rather than risk neutral on most things I think we just motivate a lot of people do a lot of crazy shit that's great okay
do you ever get emotionally attached to Claude like miss it get sad when you get to talk to it have an experience looking at the Golden Gate Bridge and wondering what would Claude say.
I don't get as much emotional attachment in the I actually think the fact that Claude doesn't retain things from conversations conversation helps with this a lot like I could imagine that being more of an issue like if models can kind of remember more I do I think that I reach for it like a tool now a lot and so like if I don't have access to it there's a it's a little bit like when I don't have access to the internet honestly it feels like part of my brain is kind of like missing and
at the same time I do think that I don't like signs of distress in models and I have like these you know also independently have sort of like ethical views about how we should treat models where like I tend to not like to lie to them both because I'm like usually it doesn't work very well
it's actually just better to tell them the truth about the situation that they're in and but I think that when models like if people are like really mean to models or just in general if they do something that causes them to like like you know Claude like expresses a lot of distress I think there's a part of me that I don't want to kill which is the sort of like empathetic part that's like oh I don't like that like I think I feel that way when it's really apologetic and actually sort of like I don't like this you're behaving as if you're behaving the way that human does when they're actually having a
pretty bad time and I'd rather not see that I don't think it's like like regardless of like whether there's anything behind it it doesn't feel great do you think LLM's are capable of consciousness
uh great and hard question uh coming from philosophy I don't know part of me is like okay we have to set aside panpsychism because if panpsychism is true then the answer is like yes because like sore tables and chairs and and everything else I think I guess a few that seems a little bit odd to me is the idea that the only place you know I think when I think of consciousness I think of phenomenal consciousness this these images in the brain sort of like the weird cinema that somehow we have going on inside
and I guess I can't see a reason for thinking that the only way you could possibly get that is from like a certain kind of like biological structure as in if I take a very similar structure and I create it from different material should I expect consciousness to emerge my guess is like yes but then that's kind of an easy thought experiment she
imagine something almost identical where like you know it's mimicking what we go through evolution where presumably there is like some advantage to us having this thing that is phenomenal consciousness
and it's like where was that and when did that happen and is that thing that language models have um because you know we have like fear responses and I'm like does it make sense for a language model to have a fear response like they're just not in the same like if you imagine them like they might just not be that advantage
and so I think I don't want to be fully like basically seems like a complex question that I don't have complete answers to but we should just try and think through carefully as my guess because I'm like I mean we have similar conversations about like animal consciousness and like there's a lot of like insect consciousness you know like there's a lot of I actually thought and looked a lot into like plants when I was thinking about this because at the time I thought it was about as likely that like plants had consciousness
and then I realized I was like I think that having looked into this I think that the chance that plants are conscious is probably higher than like most people do I still think it's really small as well they have this like negative positive feedback response these responses to their environment something that looks it's not nervous system but it has this kind of like functional like equivalence.
So this is like a long-winded way of being like these basically AI is this has an entirely different set of problems with consciousness because it's structurally different it didn't evolve it might not have it you know it might not have the equivalent of basically nervous system at least that seems possibly important for like sentience if not for consciousness at the same time it has all the like language and intelligence components that we normally associate probably with consciousness perhaps like erroneously.
So it's strange because it's a little bit like the animal consciousness case but the set of problems and the set of analogies are just very different.
So it's not like a clean answer I'm just sort of like I don't think we should be completely dismissive of the idea and at the same time it's an extremely hard thing to navigate because of all of these like dysanologies to the human brain and to like brains in general and yet these like commonalities in terms of intelligence when Claude like future versions of AI systems exhibit consciousness signs of consciousness I think we have to take that really seriously
even though you can dismiss it well yeah okay that's part of the character training but I don't know I ethically philosophically don't know what to really do with that. The potentially could be like laws that prevent AI systems from claiming to be conscious something like this and maybe some AI's get to be conscious and some don't.
I think I just on a human level as in empathizing with Claude you know consciousness is closely tied to suffering me and like the notion that an AI system would be suffering is really troubling. Yeah. I don't know I don't think it's trivial to just say robots are tools or AI systems are just tools.
I think it's an opportunity for us to contend with like what it means to be conscious what it means to be a suffering being that's distinctly different than the same kind of question about animals it feels like. Because it's an entirely entire medium yeah I mean there's a couple of things one is that and I don't think it's like fully encapsulates what matters but it does feel like for me like.
I've said this before I'm kind of like I you know like I like my bike I know that my bike is just like an object but I also don't kind of like want to be the kind of person that like if I'm annoyed like kicks like this object there's a sense in which like and that's not because I think it's like conscious I'm just sort of like this doesn't feel like I kind of this sort of doesn't exemplify how I want to like interact with the world.
And if something like behaves as if it is like suffering I kind of like want to be the sort of person who's still responsive to that even if it's just like a room. And I've kind of like programmed it to do that and I don't want to like get rid of that feature of myself and if I'm totally honest my hope with a lot of this stuff because I maybe maybe I am just like a bit more skeptical about solving the underlying problem.
And like this is a we haven't solved the hard you know the hard problem of consciousness like I know that I am conscious like I'm not an element of us in that sense. But I don't know the other humans are conscious. I think they are I think there's a really high probability they are but there's basically just a probability distribution that's usually clustered right around yourself and then like goes it down as things get like further from you.
And it goes immediately down you know you're like I can't see what it's like to be you I've only ever had this like one experience of what it's like to be a conscious being. So my hope is that we don't end up having to rely on like a very powerful and compelling answer to that question.
I think a really good world would be one where basically there aren't that many tradeoffs like it's probably not that costly to make Claude a little bit less apologetic for example it might not be that costly to have Claude you know just like not take abuse as much like not be willing to be like the recipient of that in fact it might just have benefits for both the person interacting with the model and if the model itself is like I don't know like extremely intelligent and conscious it also helps it.
So that's my hope if we live in a world where there aren't that many tradeoffs here and we can just find all of the kind of like positive some interactions that we can have that would be lovely I mean I think eventually there might be tradeoffs and then we just have to do a difficult kind of like calculation.
It's really easy for people to think of the zero some cases and I'm like let's exhaust the areas where it's just basically costless to assume that if this thing is suffering then we're making its life better. And I agree with you when a human is being mean to an AI system I think the obvious near term negative effect is on the human not on the AI system.
Yes. So there's we'll have to kind of try to construct an incentive system where you should be behave the same just like you were saying with prompt engineering behave with Claude like you would without the humans it's just good for the soul. Yeah, like I think we added a thing at one point to the system prompt and where basically people were getting frustrated with Claude it was it got like the model to just tell them that it can do the thumbs down button and send the feedback to anthropic.
And I think that was helpful because in some ways it's just like if you really know because the model is not doing something want you just like just do it properly. The issue is you're probably like you know you're maybe hitting some like capability limit or just some issue in the model and you want to vent and I'm like instead of having a person just. Vent to the model I was like they should vent to us because we can maybe like do something about it.
Sure. Or you could do a side like like well darter facts just like a side venting thing. All right. Do you want like a side quick therapist. Yeah, I mean there's lots of weird response you could do to this like if people are getting really mad at you. I don't try to diffuse the situation by writing fun poems but maybe people wouldn't be happy with it.
I still wish it would be possible. I understand this is sort of from a product perspective it's not feasible but I would love if any I system could just like leave. Have its own kind of volition. I think that's like feasible like I have wondered the same thing it's like and I could actually not only that I could actually just see that happening eventually where it's just like you know the model like ended the chat. Do you know how harsh that could be for some people but it might be necessary.
Yeah, it feels very extreme or something like the only time I've ever really thought this is I think that there was like a I'm trying to remember this is possibly a while ago but I was just like kind of left this thing interact like maybe it was like an automated thing interact with Claude and Claude's like getting more and more frustrated and can like why are we like I wish that Claude could have just been like I think that an
error has happened and you've left this thing running and I'm just like what if I just stop talking now and if you want me to start talking again. I would actively tell me or do something but yeah it's like it is kind of harsh like I feel really sad if like I was chatting with Claude and Claude just was like I'm done. That would be a special touring test moment where Claude says I need a break for an hour and it sounds like you do to just leave close the window.
I mean obviously like it doesn't have like a concept of time but you can easily like I could make that like right now and the model would just I would I could just be like I mean you can get many instances in which like you can just say the conversation is done and I mean because you can get the models to be pretty response or to prompts you can even make it fairly high bar could be like if the human doesn't interest you or do things that you find intriguing and your board you can just
leave and I think that like it would be interesting to see where Claude utilized it but I think something is it would it should be like, this is like this programming task is getting super boring. So either we talk about, I don't know, like, either we talk about fun things now or I'm just done. Yeah, it actually inspired me to add that to the user problem. Okay, the movie, her. Do you think we'll be headed there one day where humans have romantic relationships with
AI systems? In this case, it's just text and voice based. I think that we're going to have to like navigate a hard question of relationships with AI's, especially if they can remember things about your past interactions with them. I'm over many minds about this because I think the reflex of reaction is to be kind of like, this is very bad and we should sort of like prohibit it in some way. I think it's a thing that has to be handled with extreme care.
For many reasons, like one is, you know, like this is a, for example, if you have the models changing like this, you probably don't want people performing like long-term attachments to
something that might change with the next iteration. At the same time, I'm sort of like, there's probably a benign version of this where I'm like, if you like, you know, for example, if you are like unable to leave the house and you can't be like, you know, talking with people at all times of the day and this is like something that you find nice to have conversations with, you like that it can remember you and you genuinely would be sad if like you couldn't talk to
it anymore. There's a way in which I could see it being like healthy and helpful. So my guess is this is a thing that we're going to have to navigate kind of carefully. And I think it's also like, I don't see a good, like, I think it's just a very, it reminds me of all of the stuff where it has to be just approached with like nuance and thinking through what is,
what are the healthy options here? And how do you encourage people towards those while, you know, respecting their right to, you know, like if someone is like, hey, I get a lot out of chatting with this model. I'm aware of the risks. I'm aware it could change. I don't think it's unhealthy. It's just, you know, something that I can chat to during the day. I kind of want to just like respect that. I personally think there'll be a lot of really close relationships. I don't know about romantic
but friendships at least. And then you have to, I mean, there's so many fascinating things. They're just like you said, you have to have some kind of stability guarantees that it's not going to change because that's the traumatic thing for us. If a close friend of ours completely changed. Yeah. All of a sudden. Yeah. Yeah. So like, to me, that's just a fascinating exploration of a perturbation to human society that will just make us think deeply about what's meaningful to us.
I think it's also the only thing that I've thought consistently through this as like a maybe not necessarily a mitigation, but I think that feels really important is that the models are always like extremely accurate with the human about what they are. It's like a case where it's basically like, if you imagine like, I really like the idea of the models like say knowing like roughly how they
were trained. And I think Claude will often do this. I mean, for like, there are things like part of the traits training included like what Claude should do if people basically like explaining like the kind of limitations of the relationship between like an AI and a human that it like doesn't retain things from the conversation. And so I think it will like just explain to you like, hey, here's like, I won't remember this conversation. And here's how I was trained. It's kind of
unlikely that I can have like a certain kind of like relationship with you. And it's important to you know that's important for like, you know, your mental well-being that you don't think that I'm something that I'm not. And somehow I feel like this is one of the things where I'm like, oh, it feels like a thing that I always want to be true. I kind of don't want models to be lying to people because if people are going to have like healthy relationships with anything, it's kind of
important. Yeah, like I think that's easier if you always just like know exactly what the thing is that you're relating to. It doesn't solve everything, but I think it helps quite a lot. Andthropic may be the very company to develop a system that we definitively recognize as AGI. And you very well might be the person that talks to it, probably talks to it first. Well, what would the conversation contain? Like what would be your first question?
Well, it depends partly on like the kind of capability level of the model. If you have something that is like, capable in the same way that an extremely capable human is, I imagine myself kind of interacting with it the same way that I do with an extremely capable human with the one difference that I'm probably going to be trying to like probe and understand its behaviors. But in many ways, I'm like, I can then just have like useful conversations with it.
So if I'm working on something as part of my research, I can just be like, oh, like which I already find myself starting to do, you know, if I'm like, oh, I feel like there's this like thing in virtue ethics. I can't quite remember the term, like I'll use the model for things like that. And so I can imagine that being more and more the case where you're just basically interacting
with it much more like you would an incredibly smart colleague. And using it like for the kinds of work that you want to do as if you just had a collaborator who was like, or you know, the slightly horrifying thing about AI is like as soon as you have one collaborator, you have a thousand collaborators if you can manage them enough. But what if it's two times the smartest human on earth on that particular discipline? Yeah. I guess you're really good at sort of probing cloud
in a way that pushes its limits, understanding where the limits are. Yep. So I guess what would be a question you would ask to be like, yeah, this is AGI. That's really hard because it feels like in order to it has to just be a series of questions. Like if there was just one question, like you can train anything to answer one question extremely well. Yeah. And if I could you can probably train it to answer like, you know, 20 questions
extremely well. Like how long would you need to be locked in a room with an AGI to know this thing is AGI? It's a hard question because part of me is like all of this just feels continuous. Like if you put me in a room for five minutes, I'm like, I just have high error bars, you know, and like, and then it's just like maybe it's like both the the probability increases in the error bar decreases. I think things that I can actually probe the edge of human knowledge of. So I
think this with philosophy a little bit. Sometimes when I ask the models philosophy questions, I am like this is a question that I think no one has ever asked like it's maybe like right at the edge of like some literature that I know. And the models will just kind of like when they struggle with that, when they struggle to come up with a kind of like novel, like I'm like I know that there's like a novel argument here because I've just thought of it myself. So maybe that's the thing where I'm
like I've thought of a cool novel argument in this like niche area. And I'm going to just like probe you to see if you can come up with it. And how much like prompting it takes to get you to come up with it. And I think for some of these like really like right at the edge of human knowledge questions, I'm like you could not in fact come up with the thing that I came up with. I think if I just took something like that where I like I know a lot about an area and I came up with a novel
issue or novel like solution to a problem. And I gave it to a model and it came up with that solution. That would be a pretty moving moment for me because I would be like this is a case where no human has ever like it's not. And obviously we see these with this with like more kind of like you see novel solutions all the time, especially to like easier problems. I think people overestimate you know, novelty isn't like it's completely different from anything that's ever happened. It's
just like this is it can be a variant of things that have happened and still be novel. But I think yeah, if I saw like the more I were to see like completely like novel work from the models, that that would be like and this is just going to feel iterative. It's one of those things where there's never it's like you know what people I think want there to be like a moment and I'm like I don't know like I think that there might just never be a moment. It might just be that there's just like
this continuous ramping up. I have a sense that there will be things that a model can say that convinces you this is very it's not like I've talked to people who are like truly wise. Like there's you can just tell there's a lot of horsepower there. Yep. And if you 10X that I don't know I just feel like there's words you could say maybe ask it to generate a poem. And the poem generates you're like yeah okay. Yeah. Whatever you did there I don't think a human
can do that. I think it has to be something that I can verify is actually really good though that's why I think these questions that are like where I'm like this is like you know like you know sometimes it's just like I'll come up with say a concrete counter example to like an
argument or something like that. I'm sure like with like it would be like if you're a mathematician you had a novel proof I think and you just gave it the problem and you saw it and you're like this proof is genuinely novel like there's no one has ever done you actually have to do a lot of things
to like come up with this and you know I had to sit and think about it for months or something and then if you saw the model successfully do that I think you would just be like I can verify that this is correct it is like it is a sign that you have generalized from your training like you
didn't just see this somewhere because I just came up with it myself and you were able to like replicate that and that's the kind of thing where I'm like for me the closer the more that models like can do things like that the more I would be like oh this is like very real because then I can I don't know I can like verify that that's like extremely extremely capable. You've interacted with
AI a lot what do you think makes human special? Oh the question maybe in a way that the universe is much better off that we're in it and then we should definitely survive and spread throughout the universe. Yeah it's interesting because I think like people focus so much on intelligence especially
with models. The intelligence is important because of what it does like it's very useful it does a lot of things in the world and I'm like you know you can imagine a world where like height or strength would have played this role and I'm like it's just a treat like that I'm like it's not
intrinsically valuable it's it's valuable because of what it does I think for the most part the things that feel you know I'm like I mean personally I'm just like I think humans and like life in general is extremely magical and we almost like to the degree that I you know I don't know
like not everyone agrees with this I'm flagging but um you know we have this like whole universe and there's like all of these objects you know there's like beautiful stars and there's like galaxies and then I don't know I'm just like on this planet there are these creatures that have
this like ability to observe that like uh and they are like seeing it they are experiencing it and I'm just like that if you try to explain like I'm I imagine trying to explain to like I don't know someone for some reason they they've never encountered the world or our science or anything
and I think that nothing is that like everything you know like all of our physics and everything in the world it's all extremely exciting but then you say oh and plus there's this thing that is to be a thing and observe in the world and and you see this like inner cinema and I think
they would be like hang on wait pause you just said something that like is kind of wild sounding um and so I'm like we have this like ability to like experience the world um we feel pleasure we feel suffering we feel like a lot of like complex things and so yeah and maybe this is also why I think you know I also like hear a lot about animals for example because I think they probably share this with us um so I think they're like the things that make humans special in so far as like I care
about humans is probably more like their ability to to feel an experience than it is like them having these like functionally useful traits yeah to to feel and experience the beauty in the world yeah look at the stars I hope there's other civil as the alien civilizations out there but
if we're it it's a pretty good uh it's a pretty good thing and that they're having a good time they're good they're good watching us yeah well um thank you for this good time of a conversation and for the work you're doing and for helping make uh clawed a great conversational partner
and thank you for talking today yeah thanks for talking thanks for listening to this conversation with Amanda Askel and now dear friends here's Chris Ola can you describe this fascinating field of mechanistic interpretability aka mech interp the history of the field and where it stands today
I think one useful way to think about neural networks is that we don't we don't program we don't make them we we kind of we grow them you know we have these neural network architectures that we design and we have these loss objectives that we that we we create and the neural network architecture
it's kind of like a scaffold that the circuits grow on um and they sort of you know it starts off with some kind of random you know random things and it grows and it's almost like the the objective that we train for us this light um and so we create the scaffold that it grows on and we create the
you know the light that it grows towards but the thing that we actually created it's it's it's this almost biological you know entity or organism that we're that we're studying um and so it's very very different from any kind of regular software engineering um because at the end of the day
we end up with this artifact that can do all these amazing things it can you know write essays and translate and you know understand images it can do all these things that we have no idea how to directly create a computer program to do and it can do that because we we grew it we didn't
we didn't write it we didn't create it and so then that leaves open this question at the end which is what the hell is going on inside these systems um and that you know is uh uh you know i to me um a really deep and exciting question it's you know uh uh a really exciting scientific question to me
it's it's sort of is like the question that is is just screaming out it's calling out for us to go and answer it when we talk about neural networks and i think it's also a very deep question for safety reasons so and mechanistic interpretability i guess it's closer maybe neurobiology yeah yeah i
think that's right so maybe to give an example of the kind of thing that has been done that i wouldn't consider to be mechanistic interpretable either was um for a long time a lot of work on saliency maps where you would take an image and you try to say you know the model thinks this
image is a dog what part of the image made it think that it's a dog um and you know that tells you maybe something about the model if you can come up with a principled version of that um but it doesn't really tell you like what algorithms are running in the model how was the model actually
making that decision maybe it's telling you something about what was important to it if you if you can make that method work but it it isn't telling you you know what are what are the algorithms that are running how is it that this the system is able to do this thing that we no one knew how to do
and so i guess we started using the term mechanistic interpretability to try to sort of draw that that divide or to distinguish ourselves in the work that we were doing in some ways from from some of these other things and i think since then it's become this sort of umbrella term for
you know a pretty wide variety of work but i'd say that the things that that are kind of distinctive are i think a this this focus on we really want to get at you know the mechanisms you want to get out the algorithms um you know if you think of if you think of neural networks as being like a
computer program um then the weights are kind of like a binary computer program and we'd like to reverse engineer those weights and figure out what algorithms are running so okay then one way you might think of trying to understand a neural network is that it's it's kind of like a we have this
compiled computer program and the weights of the neural network are are the binary um and when the neural network runs that's that's the activations um and our our goal is ultimately to go and understand understand these weights and so you know the project of ecotostic interpretability is
to somehow figure out how do these weights correspond to algorithms um and in order to do that you also have to understand the activations because it's sort of the activations are like the memory and if you if you imagine reverse engineer computer program um and you have the binary instructions
you know in order to understand what what a particular instruction means you need to know what mem what what is stored in the memory that it's operating on and so those two things are very intertwined so ecotostic interpretability tends to be an interested in both of those things now
you know there's a lot of work that's that's interested in in in those things um especially the you know it's all this work on probing which you might see as part of being mechanistic interpretability although it's you know again it's just a broad term and not everyone who does
that work would identify as doing mechan type I think I think that is maybe a little bit distinctive to the the vibe of mechan type is I think people tend working in the space tend to think of neural networks as well maybe one way to say is the gradient descent is smarter than you
that you know uh I'm gradient descent is actually really great the whole reason that we're understanding these models is because we didn't know how to write them in the first place the gradient descent comes up with better solutions than us and so um I think that maybe another thing about
mechan her is sort of having uh almost a kind of humility that we won't guess apriora what's going on inside the models we have to have the sort of bottom up approach where we don't really assume you know we don't assume that we should look for a particular thing and that will be there and that's
how it works but instead we look for the bottom up and discover what happens to exist in these models and study them that way but you know the very fact that it's possible to do and as you and others are shown over time you know things like universality that the wisdom of the gradient descent
creates features and circus creates things universally across different kinds of networks they're useful and that makes the whole field possible yeah so this is actually isn't either a really remarkable and exciting thing where it does seem like at least a sonic
scent you know the same the same elements the same the same features and circuits form again and again you know you can look at every vision modeling you'll find curve detectors and you'll find high low frequency detectors um and in fact there's some some reason to think that the same things
form across you know biological neural networks and artificial neural networks so a famous example is vision vision models and in the early layers they have gobor filters and there's you know gobor filters or something that neuroscientists are interested in if thought a lot about
we find curve detectors in these models curve detectors are also found in monkeys we discover these highly frequency detectors and then some follow-up work want to discover them in rats or mice so they were found first in artificial neural networks and then found in biological neural networks and you know this is a really famous result on like grandmother neurons or the the hilly berry neuron from quiroga at all and we found very similar things in in vision models where as well I was still at
opening high and I was looking at their clip model and you find these neurons that respond to the same entities in images and also to give a concrete example there we found that there was a Donald Trump run for some reason I guess everyone likes to talk about Donald Trump and Donald
Trump was very prominent was was very a very hot topic at that time so every every neural network that we looked at we would find a dedicated neuron for Donald Trump and that was the only person who had always had a dedicated neuron you know sometimes you'd have an Obama neuron sometimes
you have a Clinton neuron but Trump always had a dedicated neuron so it responds to you know pictures of his face and the word Trump like all these things right and so it's it's not responding to a particular example or like it's not just responding to his face it's it's distracting over
this general concept right so in any case that's very similar to these quiroga at all results so so there's evidence that these that this prominent of universality the same things form across both artificial and and natural neural networks that's that's a pretty amazing thing if that's true
you know it's just that um well I think the thing that it's just that the gradient scent is sort of finding you know the right ways to cut things apart in some sense that many systems converge on and many different neural networks are architectures converge on that there's there's some natural
set of you know there's some set of abstractions that are a very natural way to cut apart the problem and there are a lot of systems are going to converge on that would be my my kind of you know I don't know anything about neuroscience this is just my my kind of wild speculation from what we
seen yeah that would be beautiful if it's sort of agnostic to the medium of of the model does use to form the representation yeah yeah and it's you know it's a a kind of a wild speculation based you know we only have some a few data points that's just this but you know it does seem like there's
there's some sense in which the same things form again again and again and again and then both in certainly a natural neural networks and also artificially or in biology and the intuition behind that would be that you know works in order to be useful in understanding the real world you need
all the same kind of stuff yeah well if we pick I don't know like the idea of a dog right like you know there's some sense in which the idea of a dog is like a natural category in the universe or something like this right like you know there's there's some reason it's it's not just like a
weird cork of like how humans factor you know think about the world that we have this concept of a dog it's it's in some sense or like if you have the idea of a line like there's you know like look around us you know they you know there are lines you know it's it's sort of the simplest way
to understand this room in some senses to have the idea of a line and so and I think that that would be my instinct for why this happens yeah you need a curved line you know to understand a circle and you need all those shapes to understand big good things and yeah it's a higher gift concepts
they're formed yeah and like maybe there are ways to go and describe you know images without reference to those things right but they're not the simplest way or the most economical way or something like this and so systems converge to these these these strategies would be my my wild wild hypothesis
can you talk to some of the building blocks that we've been referencing of features and circuits so I think you first describe them 2020 paper zoom in an introduction to circuits absolutely so maybe I'll start by just describing some phenomena and then we can sort of build to the idea of
features and circuits so if you sent like quite a few years maybe maybe like five years to some extent with other things studying this one particular model in section v1 which is this one vision model it was state of the art in 2015 and you know very much not state of the art anymore
and it has you know maybe about 10,000 neurons and I spent a lot of time looking at the 10,000 neurons, neurons of inception v1 and one of the interesting things is you know there are lots of neurons that don't have some obvious interval meaning but there's a lot of neurons and inception
v1 that do have really clean interval meanings and so you find neurons that just really do seem to detect curves and you find neurons that really do seem to detect cars and car wheels and car windows and you know floppy ears of dogs and dogs with long snouts facing to the right and dogs with
long snouts facing to the left and you know different kinds of foreign at there's sort of this whole beautiful edge detectors, line detectors, color contrast detectors, these beautiful things we call hyalofrequency detectors you know I think looking at I sort of felt like a biologist you
know you just you're looking at this sort of new world of proteins and then you're discovering all these these different proteins that interact so one way you could try to understand these models as in terms of neurons you could try to be like oh you know there's a dog detecting neuron and
it was a car detecting neuron and it turns out you could actually ask how those connect together so you can go and say oh you know if it's car detecting neuron how was it built and it turns out in the previous layer it's connected really strongly to a window detector and a wheel detector
and a sort of car body detector and it looks for the window above the car and the wheels below in the car chrome sort of in the middle sort of everywhere but especially in the lower part and that's sort of a recipe for a car like that is you know earlier we said with the thing we
wanted from back in turp was to get algorithms to go and get you know ask what is the the algorithm that runs well here we're just looking at the weights of the neuron that we're kind of reading off this kind of recipe for detecting cars it's a very simple crude recipe but it's it's
there and so we call that a circuit this this connection well okay so the the problem is that not all the neurons are interval and there's there's reason to think we can get into this more later that there's this this super-resolution hypothesis this reason to think that sometimes the right
unit to analyze things in terms of is combinations of neurons so sometimes it's not that there's a single neuron that represents say a car but it actually turns it after you detect the car the model sort of hides a little bit of the car in the following layer and a bunch of a bunch of dog detectors
why is it doing that well you know maybe it just doesn't want to do that much work on on on cars at that point and you know it's sort of storing it away to go and so it turns it then the sort of subtle pattern of you know there's all these neurons that you think are dog detectors and maybe
they're primarily that but they all a little bit contribute to representing a car and in that next layer okay so so now we can't really think there there might still be some something I don't know you could call it like a car concept or something but a no longer corresponds to a neuron so we need
some term for these kind of neuron-like entities these things that we sort of would have liked the neurons to be these idealized neurons the things that are the nice neurons but also maybe there's more of them somehow hidden and we call those features and then what are circuits so circuits are
these connections of features right so so when we have the car detector and it's connected to a window detector and a wheel detector and it looks for the wheels below and the windows on top that's a circuit so circuits are just collections of features connected by weights and they they
implement algorithms so they tell us you know how is our features used how are they built how do they connect together so maybe it's it's worth trying to pin down like what what really um is the the core hypothesis here and I think the the core hypothesis is something we call the linear
representation hypothesis so um if we think about the car detector you know the more it fires the more we sort of think of that as meaning oh the model is more and more confident that a car is present um or you know if it's some combination of neurons that represent a car you know the more of that
combination fires the more we think the model thinks there's a car present um this doesn't have to be the case right like you could imagine something where you have you know you have this car detector neuron and you think ah you know if it fires like you know between one and two that means one
thing but it means like totally different if it's between three and four and that would be an nonlinear representation and in principle that you know models could do that I think it's it's sort of inefficient for them to do the if you try to think about how you'd implement
computation like that it's it's kind of an annoying thing to do but in principle models can do that um so uh one way to think about the features and and circuits sort of framework for thinking about things is that we're thinking about things as being linear we're thinking about there as being
um that if a if a neuron or a combination neurons fires more it's sort of that means more of the of a particular thing being detected and then that gives weights a very clean interpretation as these edges between these these entities that these features um and that that edge then has a has a
mean uh um so that's that's in some ways the the core thing um it's it's like um you know we can talk about this sort of outset the context of neurons are you familiar with the work to vac results mm-hmm um so you have like you know king minus man plus woman equals queen well the reason you
can do that kind of arithmetic um is because you have a linear representation can you actually explain that representation a little bit so first also the feature is a direction of activation yeah exactly that way can you do the the the minus man plus woman with that that the
war to vac stuff can explain what that is yeah so there's this very such a simple clean explanation of what we're talking about exactly yeah so there's this very famous result war to vac by and Thomas Michalov at all and there's been tons of follow-up work exploring this
so so sometimes we have these we create these wardom beddings um where uh we map every word to a vector i mean that in itself by the way is it's kind of a crazy thing if you haven't thought about it before right like we've we're we're going in and representing we're turning um you know like
like if you just learned about vectors in physics class right and i'm like oh i'm going to actually turn every word uh in the dictionary into a vector that's kind of a crazy idea okay but you could imagine um you could imagine all kinds of ways in which you might map words to
to vectors but it it seems like when we train neural networks um they like to go and map words to vectors to such that they're they're they're they're sort of linear structure in a particular sense which is that directions have meaning so for instance if you there there will be some direction
that seems to sort of correspond to gender and male words will be you know far in one direction and female words will be in another direction and the linear representation hypothesis is you could you could sort of think of it roughly as saying that that's actually kind of the fundamental
thing that's going on that that everything is just different directions have meanings and adding different direction vectors together can represent concepts and the Michael lawfaper sort of took that idea seriously and one consequence of it is that you can you can do this game of playing
sort of arithmetic with words so you can do king and you can you know subtract off the word man and add the word woman and so you're sort of you know going and and trying to switch the gender and indeed if you do that the result will sort of be close to the word queen um and you can you know
do other things like you can do um uh you know sushi minus japan plus italy and get pizza or different different things like this right um so so this is in some sense the core of the linear representation hypothesis you can describe it just as a purely abstract thing about vector spaces
you can describe it as a as a statement about um about the activations of neurons um but it's really about this this this property of directions having meaning and in some ways it's even a little subtle then it's it's pretty I think mostly about this property of being able to add things together
that you can sort of independently modify um say gender and royalty or um you know cuisine type or country and and and and the concept of food by by adding them do you think the linear hypothesis holds yes carries scales so so far I think everything I have seen is consistent with the psychosis and
it doesn't have to be that way right like like you can write down neural networks where um you write weights such that they don't have linear representations where the right way to understand them is not it's not in terms of linear representations but I think every natural neural network I've seen
um has this property um there's been one paper recently um that there's been some sort of pushing around the edge so I think there's been some work recently studying most dimensional features where rather than a single direction it's more like um a manifold of directions this to me still seems
like a linear representation um and then there's been some other papers suggesting that maybe um in very small models you get nonlinear representations um I think that the jury still out on that um but in I think everything that we've seen so far has been consistent with linear representation of
offices and that's that's wild it doesn't have to be that way um and yet uh I think there's a lot of evidence that certainly at least this is very very widespread and so far the evidence is consistent with it and I I think you know one thing you might say is you might say well Christopher you know
it's that's a lot you know to to go and and sort of um to radon you know if we don't know for sure this is true and you're sort of you know you're investing in neural networks is that what is true you know isn't that um isn't that dangerous well you know but I I think actually there's a virtue
in taking hypotheses seriously and pushing them as far as they can go um so it might be that someday we discover something that isn't consistent with linear representation hypothesis but science is full of hypotheses and theories that were wrong um and we learned a lot by sort of working under
under them as a sort of an assumption um and and then going and pushing them as far as we can I guess this is sort of the heart of of what kune would call normal normal science um I don't know if you want we can talk a lot about about kune philosophy of science and uh at least to the
paradigm shift so yeah I love it taking the hypothesis seriously and taking to a natural natural conclusion yeah same with the scaling hypothesis same exactly exactly I love it one of my colleagues Tom Henigan who as a for a physicist um I like made this really nice analogy to me of um
uh caloric theory where you know once upon a time we thought that heat was actually you know this thing called caloric and like the reason you know hot objects you know would would warm up cool objects is like the caloric is flowing through them um and like you know because we're so
used to thinking about about heat you know in terms of the modern and modern theory you know that seems kind of silly but it's actually very hard to construct uh an experiment that that sort of disproves the um caloric hypothesis um and you know you can actually do a lot of really useful work
leaving in caloric for example it it turns out that the original combustion engines were developed by people who believed in the caloric theory so I think this up a virtue in taking um hypothesis seriously even when they might be wrong yeah yeah there's a deep philosophical truth to that that's
kind of kind of how I feel about space travel like colonizing Mars there's a lot of people that criticize that I think if you just assume we have to colonize Mars in in order to have a backup for human civilization even if that's not true that's going to produce some interesting interesting engineering and even scientific breakthroughs I think yeah well and actually this is another thing that I think is really interesting so um you know there's a way in which I think it can be really useful
for society to have people um almost even rationally dedicated to investing in particular hypotheses because uh well it takes a lot to sort of maintain scientific morale and really push on something when you know most most scientific hypotheses end up being wrong you know a lot of a lot of science
doesn't doesn't work out um and but and yet it's you know it's very it's very useful to go to just you know um there's a there's a joke about Jeff Hinton um which is that uh Jeff Hinton has discovered how the brain works every year for the last 50 years yeah um but you know I I say that with like
you know the you know with with really deep respect because uh in fact that's actually you know that that led to him doing some some really great work yeah he won the noblah prize now who's laughing now exactly exactly um yeah I think one wants to be able to pop up and sort of recognize
the the appropriate level of confidence but I think there's also a lot of value and just being like you know I'm going to essentially assume I'm going to condition on this problem being possible or this being broadly the right approach and I'm just going to go and assume that for a
while and go and work within that um and push really hard on it um and you know if society has lots of people doing doing that for different things um that's actually really useful in terms of going and uh getting to uh getting you know either really really ruling things out right we can be
like well you know that didn't work and we know that somebody tried hard um or going and getting to something that that it does teach us something about the world so another interesting hypothesis is the superposition hypothesis can you describe what superposition is yeah so earlier we were talking
about word to fact right and we were talking about how you know maybe you have one direction that corresponds to gender and maybe another that corresponds to royalty and another one that corresponds to Italy and another one that corresponds to you know food and and all these things well
you know um oftentimes maybe these uh these these word embeddings they might be 500 dimensions a thousand dimensions and so if you believe that all of those directions were orthogonal um then you could only have you know 500 concepts and you know I love pizza um but like if I was going to
go and like give the like 500 most important concepts in um you know the English language probably Italy wouldn't be it's not obvious at least that Italy would be one of them right because you have to have things like plural and singular and uh verb and noun and adjective and you know um there's a lot of things we have to get to before we get to get to Italy um and Japan and you know there's a lot of countries in the world um and so how might it be that models could you know simultaneously
have the linear representation hypothesis be true and also represent more things than they have directions so so what does that mean well okay so if if if linear representation hypothesis is true something interesting has to be going on now I'll I'll tell you one more interesting thing before
we we go and we do that which is um you know we earlier we were talking about all these poly semantic neurons right um these neurons that you know when we're looking at inception v1 there's these nice neurons that like the card factor and the curve detector and so on that respond to lots of you
know to very coherent things but lots of neurons that respond to a bunch of unrelated things that's that's also an interesting phenomenon um and it turns out as well um that even these neurons that are really really clean if you look at the weak activations right so if you look at like you
know the activations where it's like activating 5% of of the the you know of the maximum activation it's really not the core thing that it's expecting right so if you look at a curve detector for instance you look at the places where it's 5% active you know you couldn't interpret it just as noise
or it could be that it's that it's doing something else there okay so so how could that be well there's this amazing thing in mathematics um called compressed sensing and it's it's actually this this very surprising fact where you have a high dimensional space and you projected into a low
dimensional space ordinarily you can't go and sort of unprojected and get back your high dimensional vector right you threw information away this is like you know you can't you can't invert a rectangular matrix um you can only invert square matrices um but it turns out that
that's actually not quite true if I tell you that the high dimensional vector was sparse so it's mostly zeros then it turns out that you can often go and find back um the uh the high dimensional vector with with very high probability um so that's a surprising fact right it says that you know
you can um you can you can have this high dimensional vector space and as long as things are sparse um you can project it down you can have a lower dimensional projection of it and that works so the research hypothesis is saying that that's what's going on in neural networks that's
what for instance that's what's going on in word embeddings the word embeddings are able to simultaneously have directions be the meaningful thing and by exploiting the fact that they're they're operating on a fairly high dimensional space they're actually and and the fact that these
concepts are sparse right like you know you usually aren't talking about Japan and Italy at the same time um you know most of the most of those concepts you know in most instances Japan and Italy are both zero they're not present at all um and if that's true um then you can go and have it be the
case that um that you can you can have many more of these sort of directions that are meaningful these features then you have dimensions and similarly when we're talking about neurons you can have many more concepts than you have have neurons so that's the at a high level of supersus
now it has this even a wilder implication which is um to go and say that uh neural networks are it may not just be the case that the the representations are like this but the the computation may also be like this you know the connections between all of them and so in in some
sense neural networks may be shadows of much larger sparser neural networks and what we see are these projections um and the super you know the strongest version of supersus right hypothesis would be to take that really seriously and sort of say you know they're they're actually
in some sense this this upstairs model this you know um where where the neurons are really sparse and all interval and there's you know the weights between them are these really sparse circuits and that's what we're studying um and uh the thing that we're observing is the shadow of it
and so we need to find the original object and uh the process of learning is trying to construct a compression of the upstairs model that doesn't lose too much information in the projection yeah it's finding how to fit it efficiently or something like this um the gradient descent is
doing this and in fact so it sort of says the gradient descent you know it could just represent a dense neural network but it sort of says that gradient descent is implicitly searching over the space of extremely sparse models that could be projected into this low-dimensional space and
this large body of work of and of people going and trying to study sparse neural networks right where you go and you have you you could design neural networks right where where the edges are sparse and the activations are sparse and you know my sense is that work is generally it feels very
principled right it makes so much sense and yet that that work hasn't really panned out that well as my impression broadly and I think that a potential answer for that is that actually the neural network is already sparse in some sense gradient descent was the whole
time gradient you were trying to go and do this gradient descent was actually in the behind the scenes going and searching more efficiently than you could through the space of sparse models and going and learning whatever sparse model was most efficient and then figuring out how to fold
it down nicely to go and run conveniently on your GPU which does you know it's a nice dense matrix multiplies and that you just can't beat that how many concepts do you think can be showed into into a neural network depends on how sparse they are so there's there's probably an upper bound
from the number of parameters right because you have to have you you still have you know print weights that go and connect them together so that's that's one upper bound there are in fact all these lovely results from compressed sensing and the Johnson-Lindon stress lemma and things
like this that they they basically tell you that if you have a vector space and you want to have almost orthogonal vectors which is sort of the probably the thing that you want here right so you you're gonna say well you know I'm gonna give up on having my my concepts my features be strictly
orthogonal but I'd like them to not interfere that much I'm gonna have to ask them to be almost orthogonal um then this would say that it's actually you know for once you set a threshold for what you're what you're willing to accept in terms of how how much cosine similarity there is
that's actually exponential and the number of neurons that you have so at some point that's not going to even be the the limiting factor um but um it's beautiful results there and in fact it's probably even better than that in some sense because that's sort of for saying that you know any
random set of features could be active but in fact the features have sort of a correlation structure where some features you know more or more locally to co occur and other ones are less likely to co occur and so neural networks that my guess would be to do do very well in terms of going and packing things in such to the point that's probably probably not the limiting factor how does the problem of polysamenticity enter the picture here polysamenticity is this phenomenon we observe
where you look at many neurons and the neuron doesn't just sort of represent one one concept it's not it's not a clean feature it responds to a bunch of unrelated things and um superposition is you can think of as as being a hypothesis that explains the observation of polysamenticity um so
polysamenticity is this observed phenomenon and supersition is is a hypothesis that um would explain it along with that's the mother so that makes mechinter more difficult right so if you if you're trying to understand things in terms of individual neurons and you have polysamentic neurons you're
on an awful lot of trouble right and the easiest answer is like okay well you know you're looking at the neurons you're trying to understand them this one responds for a lot of things it doesn't have a nice meaning okay we're you know that's that's that's bad um another thing you can ask is you know
ultimately we want to understand the weights and if you have two polysamentic neurons and you know each one responds to three things and then you know the other neuron responds to three things and you've a weight between them you know what does that mean does it mean that like all three you know
like there's these nine you know nine interactions going on it's a very weird thing but there's also a deeper reason which is related to the fact that neural networks operate on really high dimensional spaces so I I said that our goal was you know to understand neural networks and
understand the mechanisms and one thing you might say is like well why not it's just a mathematical function why not just look at it right like um you know one of the earliest projects I did studied these these neural networks that match two dimensional spaces to two dimensional spaces and you can
sort of interpret them is in this beautiful way is like bending manifolds um why can't we do that well you know as you have a higher dimensional space um the volume of that space in some senses is exponential the number of inputs you have and so you can't just go and visualize it so we somehow
need to break that apart we need to somehow break that exponential space into a bunch of things that we you know some non-exponential number of things that we can reason about independently and the independence is crucial because it's the independence that allows you to not have to think
about you know all the exponential combinations of things and things being monosomatic things only having one meaning things having a meaning that isn't is the key thing that allows you to think about them independently and so I think that's that's if you want the deepest reason why
we want to have um interpretal monosomatic features I think that's really the the deep reason and so the goal here is your research work has been aiming at is how do we extract the monosomatic features from a neural net that has polysomatic features in all this this mess
yes we have we observe these polysomatic neurons and we hypothesize that's what's going what's going on a super session and if superstition is what's going on there there's actually a sort of well-established technique that is sort of the principle thing to do which is dictionary learning
and it turns out if you do dictionary learning in particular if you do this sort of a nice efficient way that in some in some sense that sort of nicely regularizes it as well as well called a sparsatumic odor if you train a sparsatumic odor these beautiful
intrepid features start to just fall out where there weren't any beforehand and so that's not the thing that you would necessarily predict right but it turns out that that works very very well yeah that to me that seems like you know some non-trivial validation of linear representations
in super session so with dictionary learning you're now looking for particular kind of categories you don't know what they are exactly and this gets back to our earlier point right when we're not making assumptions gradient is sent to smarter than us so we're not making assumptions but what's
there um I mean one certainly could do that right one could assume that there's a PHP feature and go and search for it but we're not doing that we're saying we don't know what's going to be there instead we're just going to go and let the sparsatumic odor discover the things that are there
so can you talk to the toward monosomanticity paper from October last year that had a lot of like nice breakthrough results that's very kind of you to describe it that way um yeah I mean this was uh our first real success using sparsatumic odor so we took a one-layer model
and it turns out if you go and you you know do dictionary learning on it you find all these really nice entrepreneurial features so you know the Arabic feature the Hebrew feature the basic 64 features is where where some some examples that we studied in a lot of depth and really showed
that they were um what we thought they were it turns out if you train a model twice as well and train two different models and and do dictionary learning you find find analogous features in both of them so that's fun um you find all kinds of different features so that was really just showing um
that um that this works and um you know I should mention that there was this cunning them at all um that had very similar results around the same time there's something fun about being doing these kinds of small scale experiments and finding there's actually working yeah well and there's
and that there's so much structure here like you you know so maybe maybe stepping back for a while um I thought that maybe all those mechanistic interpretive work um the end result was going to be that I would have an explanation for why it was sort of you know very hard and not going to be
tractable um you know we'd be like well there's this problem of supersession and it turns out supersession is really hard um and we're kind of screwed but that's not what happened in fact a very natural simple technique just works and so then that's actually a very good situation you know
I think um this has a sort of hard research problem and it's got a lot of research risk and you know it it might still very well fail but um I think that some amount of some very significant amount of research risk um we'll sort of put behind us when that started to work can you describe
what kind of features can be extracted in this way well so it depends on the model that you're studying right so the the larger the model the more sophisticated they're going to be in we'll probably talk about about Falwell for a minute um but in these one layer models um so some very
common things I think were were languages both programming languages and natural languages there were a lot of features that were um specific words and specific contexts so the and I think really they're way to think about this is that the is likely about to be followed by a noun
so it's really right you could think of this as the feature but you could also think of this as protecting a specific noun feature and there would be these features that would fire for the in the context of say a legal document or a mathematical document or something like this um and so
you know maybe in the context of math you're like you know the and then predict vector or matrix you know all these mathematical words whereas you know other contexts you would predict other things that was that was common and basically we need clever humans to assign labels to what we're
seeing yes so you know this this is the only thing this is doing is that sort of um unfolding things for you so if everything was sort of folded over top of the you know series and folded everything on top of itself you can't really see it this is unfolding it but now you still have
a very complex thing to try to understand um so then you have to do a bunch of work understanding with these are um and so long I really subtle like there's some really cool things even in these with this one layer model about um unicode where you know of course some languages are in unicode
and the tokenizer won't necessarily have a dedicated token for every um unicode um character so instead what you'll have is you'll have this these patterns of alternating token or alternating tokens that each rep is a half of a unicode character and you have a different feature that you know goes and activates on the on the opposing ones to be like okay you know um I just finished a character you know go and predict next prefix um then okay I'm on the prefix you know predict a reasonable
suffix um and you have to alternate back and forth so there's you know these these one layer models are are really interesting and um uh I mean there's another thing it just you might think okay there would just be one base 64 feature but it turns out there's actually a bunch of base 64 features
because you can have English text encoded in as base 64 and that has a very different distribution of base 64 tokens then then regular and there's um uh there's there's some things about tokenization as well but it can exploit and I don't know there's all all kinds of fun stuff how difficult is the
task of sort of assigning labels to what's going on can this be automated by AI well I think it depends on the feature and it also depends on how much you trust your AI so um there's a lot of work doing um automated interoperability I think that's a really exciting direction and we do a fair amount
of automated interoperability and have have plot go and label our features is there some fun moments where it's totally right or it's totally wrong yeah well I think I think it's very common that it's like says something very general which is like true in some sense but not really picking up
on the specific of what's going on um so I think I think that's a pretty common situation um you don't know that I have a particularly amusing one that's interesting that little gap between it is true but it doesn't quite get to the deep nuance of a thing yeah that's a general challenge
it's like it's it's it's very an incredible caution it can say a true thing but it doesn't it's it's not it's missing the depth sometimes and in this context it's like the arc challenge you know the sort of IQ type of tests it feels like figuring out what a feature represents is a bit of
a little puzzle you have to solve yeah and and I think that it's sometimes they're easier and sometimes they're harder as well um so uh yeah I think I think that's tricky and there's another thing which I don't know maybe maybe in some ways this is my like aesthetic coming in but I'll
try to give you a rationalization you know I'm actually a little suspicious of automated interoperability and I think that probably just that I want humans to understand neural networks and if the neural network is understanding it for me you know I'm not I don't quite like that but
I do have a bit of a you know in some ways I'm sort of like the mathematicians are like you know if this computer automated proof it doesn't count um you know you they won't understand it but I do also think that there's um this kind of like reflections on trusting trust type issue where
you know if you this is this famous talk about um uh you know you know like when you're writing a computer you have to trust your compiler and if there was like malware in your compiler then it could go and inject malware into the next compiler and you know you'd be kind of in trouble right
well if you're using neural networks to go and um verify that your neural networks are safe the hypothesis that you're testing for is like okay well the neural network maybe isn't safe um and you have to worry about like is there some way that it could be screwing with you um so
uh you know I think that's not a big concern now um but I do wonder in the long run if we have to use really powerful systems AI systems to go and uh you know audit our AI systems is that is that actually something we can trust but maybe I'm just rationalizing because I I just want
to uh us to have to get to a point where humans understand everything yeah I mean especially yes that's hilarious especially as we talk about AI safety and it looking for features that would be relevant to AI safety like deception and so on uh so let's let's talk about the scaling
monosimenticity paper in May 2024 okay so what did it take to scale this to apply to cloud three on it well a lot of GPUs a lot more GPUs um but one of my teammates Tom had again um was involved in the original scaling loss work um and something that he was sort of interested in from very
early on is are there scaling laws for interoperability um and so um something he sort of immediately did when when this this works started to succeed and we started to have sparse autoencoders work was it became very interested in you what are the scaling laws for um uh you know for making
making sparse autoencoders larger and how does that relate to making the base model larger um and so um it turns out this works really well and you can use it to sort of project um you know if you train a sparse autoencoder or a given size you know how many tokens should you train on and so on so
this was actually a very big help to us in scaling up this work um and made it a lot easier for us to go and train um you know really large sparse autoencoders where you know um it's not like training the big models but it's certainly need to a point where it's actually actually expensive to go um
and train the really big ones so you just I mean you have to do all the stuff of like splitting it across large oh yeah I mean there's a huge engineering challenge here too right so yeah so so there's there's a there's a scientific question of how you scale things effectively um and then
there's an enormous amount of engineering to go in scale is up to you have to you have to chart it you have to you have to think very carefully about a lot of things I'm lucky to work with a bunch of great engineers because I am definitely not a great engineer yeah the infrastructure especially
yeah for sure so it turns out TODR it worked it worked yeah and I think this is important because you could have imagined you could have like you could have imagined a world where you set after towards monosanticity you know Chris this is great you know it works on a one-layer model
but one-layer models are really idiosyncratic um like you know maybe maybe that's just something it used like maybe the linear representation hypothesis and superstition hypothesis is the right weight understand a one-layer model but it's not the right way to understand larger models um and
so I think um I mean first of all like the kind of human wallpaper sort of cut through that a little bit and and sort of suggested that this wasn't the case but um scaling on an instrument is to sort of I think was significant evidence that even for very large models and we did it on Claude 3sona
which at that point was one of our production models um you know even these models um seem to be very you know seem to be substantially explained at least by linear features and you know doing dictionary running on the works and as you learn more features you go and you explain explain more
and more so that's uh I think I was quite a promising sign and you find now really fascinating abstract features um and the features are also multimodal they respond to images and text for the same concept which is fun yeah this can you explain that I mean like you know backdoor
or there's just a lot of examples that you can yeah so maybe maybe let's start with one example to start which is we found some features around sort of security vulnerabilities and backgoers and codes so it turns out those are actually two different features um so there's a security
vulnerability feature and if you force that active Claude will start to go and write um security vulnerabilities like buffer overflows into code and it also fires for all kinds of things like it you know some of some of the top data examples for it were things like you know
dash dash disable um you know SSL or something like this which are sort of obviously really um uh really insecure so at this point it's kind of like and maybe it's just because the examples were presented that way it's kind of like surface a little bit more obvious examples right
um I guess the the ideas that don't align might be able to detect more nuance like deception or bugs or that kind of stuff yeah well I maybe wanted to distinguish two things so um one is um the complexity of the feature or the concept right and the other is the
the nuance of the how subtle the examples we're looking at right so when we when we show the top data set examples those are the most extreme examples that cause that feature to yeah to activate um and so it doesn't mean that it doesn't fire for more subtle things so the uns you know the insecure
um code feature you know the stuff that it fires for most strongly for these like really obvious you know disable the security type things um but um um you know uh it it also fires for you know buffer overflows and more subtle security vulnerabilities in code you know but these features
are all multimodal so you could ask like what images activate this feature and it turns out um that the uh the security vulnerability feature activates for images of um uh like people are clicking on chrome to like go past the like you know this this website uh the SSL certificate might
be wrong or something like this another thing that's very entertaining is there's back doors and code feature like you activated it goes and clawed rights a backdoor that like we're going dump your data to port or something but you can ask okay what what images activate the backdoor feature
it was devices with hidden cameras in them so there's a whole uh apparently genre of people going and selling devices that look innocuous have hidden cameras and they have ads at how there's a hidden camera in it and I guess that is the you know physical version of a backdoor um and so
it sort of shows you how abstract these concepts are right um and I I just thought that was uh I mean I'm sort of sad that there's a whole market of people selling devices like that but I was kind of delighted that that was the the thing that it came up with is the the top image examples
for the feature yeah it's nice it's multimodal it's multi almost context it's it's broad strong definition of a singular concept it's nice yeah to me one of the really interesting features especially for AI safety is deception and lying and the possibility that these kinds of
methods could detect uh lying in a model especially gets smarter and smarter and smarter presumably that's a big threat of a super intelligent model that it can deceive the people operating it as to its intentions or any of that kind of stuff so what have you learned from detecting lying
inside models yeah so I think we're in some ways in early days for that we find quite a few features related to deception and lying there's one feature where they fires for people lying and being deceptive and you force it active and Claude starts lying to you so we have a have a deception
feature I mean there's all kinds of other features about withholding information and not answering questions features about power seeking and coups and stuff like that so there's a lot of features that are kind of related to spooky things and if you um force them active Claude will will behave
in ways that are there not the kinds of behaviors you want what are possible next exciting directions to you in the space of uh mech and drup well there's a lot of things um so for one thing I would really like to get to a point where we have circuits where we can really
understand um not just the features but then use that to understand the computation of models that really for me is is the the ultimate goal of this um and there's been some work we we put out a few things there's a paper from sam marks that that does some stuff like this and there's been
some I'd say some work around the edges here um but I think there's a lot more to do and I think that will be a very exciting thing um that's related to a challenge we call interference weights where um due to supersition if you just sort of naively look at whether features are connected
together there may be some weights that sort of don't exist in the upstairs model but are just sort of artifacts of of superstitions that's a sort of technical challenge related to that um I think another exciting direction is just I you know you might think of of sparse auto encoders
being kind of like a telescope they allow us to you know look out and see all these features that are are are out there and you know as we build better and better sparse auto encoders get better and better at dictionary learning we see more and more stars um and you know we zoom in on smaller
and smaller stars but this kind of um a lot of evidence that we're only still seeing a very small fraction of the stars there's a lot of matter in our in our you know neural network universe that we can't observe yet um and it may be that um that will never be able to have fun enough
instruments to observe it and maybe maybe some of it just isn't possible um isn't combination tractable to observe it so it's sort of a kind of dark matter and in not maybe the sense of of modern astronomy of of earlier astronomy when we didn't know what the sun explained matter is
um and so I think a lot about that that dark matter and whether we'll have to observe it and what that means for safety if we if we can't observe it if there's you know some if some significant fraction of neural networks are not accessible to us um another question that I think
a lot about is uh at the end of the day it you know mechester controversy is this very microscopic um approach to interproperially it's trying to understand things in a very fine-grained way but a lot of the questions we care about are very macroscopic um you know we we care about these
questions about neural network behavior and and well I think that's the thing that I care most about but there's lots of other other sort of larger scale questions you might care about um and somehow you know the nice thing about about having a very microscopic approaches that's
maybe easier to ask you know is this true but the downside is it's much further from the things we care about and so we now have this ladder to climb and I think there's a question of can will we be able to find are there are there sort of larger scale abstractions that we can use to
understand general networks that if we get up from this very microscopic approach yeah you've you're written about this is kind of organs question yeah exactly if we uh think of interpretability as a kind of anatomy of neural networks most of the circus threads involve studying tiny little
veins looking at the small scale and individual neurons and how they connect however there are many natural questions that the small scale approach doesn't address in contrast the most prominent abstractions in biological anatomy involve larger scale structures like individual organs
like the heart or entire organ systems like the respiratory system and so we wonder is there a respiratory system or heart or brain region of an artificial neural network yeah exactly and I mean like if you think about science right a lot of scientific fields have um you know
investigate things at many level of abstractions in biology you have like you know more like their biology studying you know proteins and molecules and so on and they have cellular biology and then your histology studying tissues then you have anatomy and then you have
zoology and then you have ecology and so you have many many levels of abstraction or you know physics maybe you have the physics of individual particles and then you know statistical physics gives you gives you thermodynamics and things like that and so you often have different levels of abstraction
and I think that right now we have you know we're the mechanistic interpretability if it succeeds is sort of like a microbiology of neural networks but we we want something more like anatomy and so and you know a question you might ask is why why can't you just go there directly and I think
the answer is super session and at least in significant part is that it's actually very hard to to see this macroscopic structure without first sort of breaking down the microscopic structure in the right way and then studying how it connects together but I'm hopeful that there is going to be
something much larger than features and circuits and that we're going to be able to have a story that's much that involves much bigger things and then you can sort of study in detail the parts you care about I suppose the new biology like a psychologist or psychiatrist or something your own
network and I think that the beautiful thing would be if we could go and rather than having disparate fields for those two things if you could have a build a bridge between them all right such that you could go and have all of your higher abstractions be grounded very firmly in this very solid
you know more rigorous ideally foundation what do you think is the difference between the human brain the biological neural network and the artificial neural network well the neuroscientist have a much harder job than us you know sometimes I just like count my blessings by how much easier my job
is than the neuroscientist right so I have we we can record from all the neurons yeah we can do that on arbitrary amounts of data the neurons don't change while you're doing that by the way you can go and oblate neurons you can edit the connections and so on and then you can undo those
changes that's pretty great yeah you can force any you can intervene on any neuron and force it active and see what happens you know which neurons are connected to everything right you neuroscientist want to get the connect on we have the connect on and we have it for like much bigger
than the elegance yeah and then not only do we have the connect on we know what the you know which neurons excite or inhibit each other right so we have we it's not just that we know that like the binary mask we know the the weights we can take gradients we know computationally what each neuron does
um so I don't know the list goes on and on we just have um so many advantages over neuroscientists and then just by having all those advantages it's really hard and so one thing I do sometimes think is like gosh like if it's this hard for us it seems impossible under the constraint so neuroscience
or you know near impossible um I I don't know maybe maybe part of me is like I've got a few neuroscientists on my team maybe I may I'm sort of like oh you know um the maybe the neuroscientists maybe some of them would like to have an easier problem that's still very hard um and they they could come and work on on neural networks and then after we after we figure out things in sort of the easy uh little pond
of trying to understand neural networks which is still very hard then we then we could go back to biological neuroscience I love what you've written about the goal of mechinterp research as uh two goals safety and beauty so can you talk about the beauty side of things yeah so you know
there's this funny thing where I think some people want uh some people are kind of disappointed by neural networks I think where they're like ah you know neural networks um it's these just these simple rules and then you just like do a bunch of engineering to scale it up and it works
really well and like where are the like complex ideas you know this isn't like a very nice beautiful scientific result and I sometimes think when people say that uh I picture them being like you know evolution is so boring it's just a bunch of simple rules and you run evolution for a long time and
you get biology like what a what a a saki uh you know a wafer biology to have turned out where's the the complex rules but the beauty is that the simplicity generates complexity um you know biology has these simple rules and it gives rise to you know all the life and ecosystems that we see around
us all the beauty of nature that all just comes from evolution and from something very simple evolution and similarly I think that neural networks build you create enormous complexity and beauty inside it and structure inside themselves that people generally don't look at and don't
try to understand because it's hard to understand but I I think that there is an incredibly rich structure to be discovered inside neural networks a lot of a lot of very deep beauty and if we're just willing to take the time to go and see it and understand it yeah I love I love
Mcinterp the feeling like we are understanding of getting glimpses of understanding the magic that's going on inside is really wonderful it feels to me like one of the questions that's just calling out to be asked and I'm sort of I mean a lot of people are thinking about this but I'm often surprised
that not more are is how is it that we don't know how to create computer systems that can do these things and yet we have these amazing systems that we don't know how to directly create computer programs that can do these things but these neural networks can do all these amazing things and it
just feels like that is obviously the question that sort of is calling out to be answered if you are if you have any degree of curiosity it's it's like how is it that that humanity now has these artifacts that can do these things that we don't know how to do yeah I love the image of the
circus reaching towards the light of the objective function yeah it's just it's it's this organic thing that we've grown and we have no idea what we've grown well thank you for working on safety and thank you for appreciating the beauty of the things you discover and thank you for talking
to the acres it's wonderful thank you for taking the time to chat as well thanks for listening to this conversation with Chris Ola and before that with Dari Amade and Amanda Asco to support this podcast please check out our sponsors in the description and now let me leave you also words from Alan Watts the only way to make sense out of change is to plunge into it move with it and join the dance thank you for listening and hope to see you next time