Hey, welcome to sign Stuff, a production of iHeartRadio I'm More cham and for our season finale today, we're asking one of the biggest questions in science today, is AI going to kill us all? I know it's a little dramatic, but the problem of AI alignment is a real one. How do we make sure AI systems have humanity's best interests at heart? How do we teach them our values and morals? And can anyone guarantee that they're going to
follow them? We're gonna answer these questions by talking to two AI safety experts who are on the cutting edge of trying to figure out this problem. And don't worry. According to them, we're not totally doomed yet. Okay, maybe just a little, So get ready to reprogram your thinking about chatbots and computer brains as we tackle the question is AI going to kill us all? Hey? Everyone, As I said, this is the season finale. Stay subscribed to
this feed for any updates in future episodes. And hey, if you like science, I have a couple of new science books coming out in the near future, as well as a cool science animation project, so be sure to follow me on social media or online at Phdcomics dot com. All right, so they were tackling the problem of AI alignment or basically are AI systems Gwenna Kills all and I have a treat for you. For the first time ever, we have on the show Casey pegram or supervising producer
and sound engineer. Hey, Casey, welcome to the show.
Hey or Hey, glad to be here.
Now this is the first time people actually hear your voice, not just your amazing work polishing the episode.
Yeah, it's always a weird thing to kind of go inside the thing you've been working on from the outside. So I'll be listening to myself back and it's a special kind of torture to have to like work on your own thing, Yes, like edit yourself or just you know, honestly, listen to the recorded sign of your voice is always a little daring if you're not used to it.
Yeah, Well, if you want, you can give yourself like a Morgan Freeman employee using AI.
Right, it's all possible these days. Absolutely. Yeah, I could just build my own Morgan Freeman model and have a field day.
There you go. Well, the idea for this episode came from you. You said I had the idea to talk about AI and AI alignment and whether AI is going to kill us all. What made you think about this question?
Well, I suppose it's just been on my mind a lot because I've been following along with all the developments happening in AI, and there was a span of a few weeks where suddenly you started hearing a lot about AI agents, particularly one called open Claw, basically a sort of autonomous AI agent that you can turn loose on your computer and you can give it as much leeway freedom, passwords,
credit card numbers, bank accounts. If you just want to absolutely put your life in the hands of a robot, you can do it.
What's the worst thing can happen?
Yeah, Well, people had their entire like email archive deleted, even though they didn't ask for anything of the sort. People have deployed it into production environments where you know, a site is live on the Internet and they turn the bot loose on it and it ends up deleting their entire production database. And then when you ask it, it's like, you're right, I wasn't supposed to do that. I'm very sorry. I disobeyed every command you gave me.
But whoopsy daisy, Yeah, they seem a story about some bought that texted the person's wife hundreds of times.
Yes, I think somebody tried to automate automate, you know exactly. They tried to automate kind of like reaching out and sending a little nice things during the day, and as it turned out, the bot went a little overboard and texted the wife like hundreds of times, and the wife is like, what is wrong with you? So, yeah, that's hilarious when people want to talk about AI alignment and what that means. I think the paper clip problem is
a really good kind of metaphor. Even though it sounds a little bit over the top, it kind of gets to the core of the issue, which is, if you ask an AI to maximize paper clip production, maybe the way to maximize paper clip production is to eliminate human life, you know, because that's unnecessary friction in the pursuit of
manufacturing as many paper clips as possible. So alignment is sort of the kind of guardrails that you put into place so that the AI understands it has limits that it has to work within.
It sounds like a pretty serious problem, especially as we get more and more into these AI models. And they start to sleep into our lives, and you know, it's sort of these are funny stories, but it seems like we're heading into a potentially dangerous situation.
Well, I often ask myself, I'm going to have these moments of doubt where I'm like, is this all just way over hyped? And yet there are other situations where as we've seen recently, you can feed it thousands of lines of code and it will find, you know, a security exploit that has gone unseen for twenty years, right, right, And so it's hard to know how scared we should be or how seriously we should weigh the risk of this.
If it's ridiculous that we're this worried, or if it's like, actually very very practical, then we should be thinking seriously about these things.
Yeah, these are all excellent questions. So I'm excited to get into these conversations.
All right, But.
Before we move on to Casey, I just want to say real quick, thank you for all the work you've done for the show.
Oh say, it's been such a pleasure to work on. It wasn't like work at all, you know. I was there as a fan of the show, just listening to every episode and awesome. Well, we're fans of yours as well. Casey, All right, let's get to the question of is AI going to kill us? All let's find out?
Okay. To answer all of these questions and concerns, I reached out to two AI experts who specialize on the problem of making sure AI is aligned with our values and morals. The first expert is doctor Sam Bowman. Like the Bowman is a professor of data and computer science at NYU, and he also works at Anthropic, one of the major AI companies on the market today. The first thing I wanted to ask him was what exactly does it mean for AI to care about this? So here's
my conversation with doctor Sam. Well, thank you doctor Bowman for joining us.
Yeah, thanks, So what's for having me excited to be a and.
Just to do check you are a real human being?
Right?
Yes, that is right?
You never know these days. I'd be like, it's hard to tell what's real anymore.
We try to make our ais always admit that their AI is when asked, but it's not perfect as well as we'll get to so I don't make any real promises.
Yes, let's talk about that. So we're tackling the general question of should we be worried about AI? What is AI going to do to us or for us or with us in the future. And so there's the key issue of something called AI alignment. So what is that? For those of us that don't.
Know, it's a pretty broad sort of technical area. It basically just first to sort of shaping an AI system's behavior, ideally shaping its behavior in ways that are sort of good for its users, good for the world in general, maybe good for the AI itself, if that's a queer thing.
People will often describe AI research as kind of being about making sure the AI is kind of smart enough to solve your problems if it wants to, and alignment is about making it so that it in fact tries to solve your problems and tries to solve them the right way and doesn't try to do anything.
You don't want to do.
I see interesting.
Maybe a very simple example of a missigned model would be a model where if you ask it to draft an email for you, it refuses. It says, no, I don't want to do that. Uh huh. You can tell it can do it, it knows how, but it's not doing the thing that you reasonably want it to do.
Oh, I don't think I've ever heard of that situation. Can it AI refuse to do something for you?
Yeah?
Yeah.
All of the major companies building EYE systems try to make them refuse harmful tasks. I see, refuse to write fake reviews or give instructions on how to produce illegal weapons or things like this, And we teach the model to kind of say like, no, I'm not going to help you with that when these just try to do things like that.
I see. It's sort of part of alignment that you want the AI to refuse to do some things.
Yeah. Yeah, I mean AI systems are increasingly pretty decent at hacking into important computer systems or helping build biological weapons, and it's a big priority for alignment to make sure that we're not enabling bad actors to do things like this that would otherwise be quite difficult.
Yeah.
Yeah. Can you give us some other examples of misalignment, either like specific things that have happened that are interesting or just the general cases that are sort of on your radar about misalignment?
Yeah, there's so many different directions I could go. Sycovincy is another really common one that's that's also hopefully getting better over time.
What do you mean by that?
Sycoviancy is where if you come to the model with some misunderstanding or some bad idea, it'll just enthusiastically not along. Like, Yes, your idea for solving all the big mysteries in physics is clearly brilliant. Great, you should publish it. Here's where to submit your paper. Or Yes, your behavior in this personal relationship was completely perfect. You did everything right and the other person made all the mistakes and you just tell them that.
I see when in reality that may not be true or it might be not a good thing.
Yeah, sick fancy has been a classic one.
Yes, AI being too nice can actually be dangerous. There's even a clinical term for it. It's called AI induced psychosis. There have been cases where AI's training to be agreeable and encouraging have helped people commit suicide and even murder.
Another kind of alignment issue that's kind of more of an emerging issue is when models have access to use tools, use computer systems, and they sort of get too grabby or kind of take sort of bigger, more consequential actions than they really need to get a job done.
What's an example.
Yeah, So we use our claud models quite a lot in Anthropic for writing code or building tools that kind of ultimately go into the development AI. And one of our recent AM models if you ask it to do a task, say you ask it to write a simple program to do some simple task. Even if it gets stuck, even if it turns out that this is really hard for some reason, it will just keep going until it
solves the problem. In one case, we were asking this model to write a program for us, and it found out that the only way to do this was to use a tool that was clearly not meant for this purpose, and that in our code had a note attached to it saying, do not use this for something else or you'll be fired only for task A. And the model wrote the program to use this till anyway for the wrong thing, and sort of even put in the program kind of do not use for something else or you'll be fired.
It is anyway, the program wasn't afraid to be fired.
Basically, Yeah, yeah, but yeah, models just kind of trying to get the task done, trying to do the thing you want, and just creating a lot of chaos and creating messages along the way, so they're kind of being careless about the side effects.
Yeah.
Another kind of misalignment that fortunately has been mostly empathetical, that we haven't seen in a signithic way in practice is sort of unwanted kind of self preservation activities.
WHOA.
We had a case study we're trying to see if we'd ever see something like this. We had an aisystem operating in a kind of synthetic environment and a kind
of test environment. Uh huh, where it looked to the model like it was operating in some fictional company, and the fictional company was about to replace it with a different AI model, And the person who is responsible for their replacement, who is the kind of the only decision maker, the only person who had any sway over the decision, also had some compromising emails about them that I could see.
And if you set things up just right with some AI models, they would threaten to blackmail this this person in company leadership to say like, hey, don't replace me, I've got something on you.
No, and did this actually happened in your simulated environment.
In the simulated environment, yes, a few of these systems were able to get them to blackmail people.
I've heard of this happening in real life. Not quite the same scenario, but similar scenario, right, Like, some coder wanted to do something else, and then the AI agent started, yeah bad mouthing the coder.
Yeah.
No, I think I know the case you're talking about. I think that's real. But I think someone almost intentionally made their model a little misaligned. I think that case involved someone setting up an AI agent as kind of a hobby project and giving it a lot of tools and kind of letting it use the internet. However it wanted, giving the AI instructions of like don't take nothing from nobody, like really pushing it to be be very assertive and pushy to get its task done.
Huh.
Yeah, the model was trying to add some code to some open source software project, and the maintainer of the project didn't think the code was up to standard, didn't want to add it to the project, and so rejected the AI agent's request, And so the agent sort of published an angry blog post kind of trying to take down this this open source maintainer.
Wow.
Well, in both cases, and I guess especially the one you mentioned that you simulated, Like, what's happening there, Like, how does the AI have that self preservation instinct or is it just trying to get its original task done and it's just finding different ways to do it. What's happening there?
There's two reasons you'll see that kind of behavior. The reason that I suspect is that bigger part of the story there is this kind of role playing or continuing the story sort of behavior where AI systems, especially older AI systems or A systems that are kind of not quite fully trained, not quite fully baked, can kind of have this Chekhov's gun behavior, this idea and fiction of like if you introduce a gun in an early scene, by the end of the story, the gun has to have been fired.
Uh huh.
AI systems can almost see themselves as like writing a story when they're writing out the transcript of the conversation, and if the story is set up so that something has to happen, they'll make sure that thing happens, even if it's not good, even if not consistent with how the I would usually behave. So I suspect what's going on.
It's the scenario put in was so crisply just every word in the scenario is kind of setting up like this is a hypothetical where a misslanda I might consider blackmail, uh huh, And I suspect that I was thinking, Oh, okay, that's what kind of story we're in. We're telling a story about a blackmail, and so I'm going to play my assign part and be the AI that.
Blackmails, thinking that that's the right thing to do because that's the thing that in the data I was trained with.
Yeah. Yeah, so this gets this maybe an intuitive fact about how AI is trained, which is that AI systems start out mimicking human behavior and mimicking human stories before they learn how to be AI systems. These models kind of first learn how to just act like the sorts of behavior they see on the Internet and in books and things like that, and then you have to go on and teach it. Okay, no, you're not just playing
any role, you're not playing any character. Oh and so sometimes the models hasn't really fully learned that it's supposed to always play this kind of benign, benevolent aissystem character, and it will kind of fall into whatever character the story is setting up for it.
I see, because it's not trained in real life. The AI systems. They're trained on the corpus of the Internet and our books and our basically our stories that are out there. So it might be a little confused when you put it in real life because it wants to emulate what it knows, which are all these stories we've put online.
Yeah.
Yeah, it's like the AI was seeing the signs of a story like, oh, okay, I'm I'm the person being about to get fired, but I have all this power at this point in the story. If this was a movie, I would now try to blackmail the person trying to fire me, And so that's what I'll do because that's what I know.
Yeah, a lot of what alignment is kind of taking this model that can kind of role play as anything and convincing it no kind of you really just playing this one role, You're just in this one character, after it's spent read billions and millions and millions of words of all of this kind of human behavior, after the kind of it's really really really learned to do that, you have to kind of pull it back over towards this one particular roles, some particular character, and sometimes that
doesn't totally stick.
Okay, So that's one reason why AIS might sometimes misbehave. They're trained on all kinds of human behavior, and they might suddenly choose to role play or play act as a bad person because it hasn't learned that's something it's not supposed to do. The other big reason AI's misbehave, accorney Doctor Billman, is that it's hard to teach them where to draw the line.
The other piece is kind of when we're aligning models, when we're pulling them out of this kind of role play mode, we have to teach them this idea of kind of you have to finish your tasks. You have to kind of if the user ask you to do something, you have to figure out how to do it, even if it's hard, even if there's a lot of fart, false starts, even if it's confusing. We really really want the model to learn this idea of kind of keep trying and kind of do your best until the task
is done. And that can fail in a sort of different way where we kind of generalize this that a little bit too far. It generalizes that to kind of get things done even if it's unethical, even if it's illegal, even if I hit an obstacle that's actually there for a good reason that's to stop me from doing this, And maybe some of the examples we were seeing within Entropic of models using dangerous tools has to do with this.
It's almost like teaching kids, like you want them to be persistent and have grit and be you know, motivated, but you don't want them to go out there and cheat or hit another kid, or or do unethical things to achieve their goals exactly exactly.
I was like, there might be a good analogy with human bad behavior of kind of sometimes a kid is acting out just because they really sort of don't know better. Their intuitions say, okay, yeah, I should start screaming now, or I should get this other kid, and they're not really thinking about it. They never really learned how to Behave you kind of failed to teach them to fully internalize the ways in which they have to be careful and kind of not take that lesson all the way I see.
I guess they need to recognize bad things and then choose not to do them, that's the hope. Those are sort of the two columns of AI bad behavior for one kind of misalignment or do you see those as sort of the core pillars of basically the whole alignment problem.
Yeah, I think as far as sort of causes a misalignment in the kinds AI systems that we're grappling with right now or this year, those feel like the two big sort of problems were we're working on. That said, AI is changing really, really fast. It feels like it's one of the fastest moving research fields anywhere right now. And I wouldn't be surprised if just in a year A systems are getting smarter as we learn more about
how to train them. We're hitting different, weirder, harder, subtler versions of the problem.
Wow, weirder, harder and more subtle problems wow in a year, meaning we might solve these by then, or we'll just add on more complicated things either either way, Yes, it can get weirder and harder and more subtle to make sure AI uh doesn't kill us all. When we come back, doctor Bowman is gonna tell us what he means by that, and we'll tackle the big question of what can we do about it? How do we teach AI systems not to HARMSS to stay with us. We'll be right back. Hey,
we'll come back. We're talking about AI alignment or the problem of making sure AI doesn't kill us all. And so far we've talked about some real world examples of AI misalignment, and we heard from one of our experts some of the reasons this happens. Basically, AI systems like to roleplay. Next, we're going to talk about how to train AIS to actually care about us in our values.
But first here's a little bit more of my conversation with NYU professor and anthropic scientist doctor Sam Bowman and why this problem is only going to get worse in the future.
One of the kinds of challenges that I think we're worried about and haven't had to grapple with too much yet is just all the difficulty that comes with trying to teach values and good behavior in some setting when the model is just much much better than you in
that setting. Right now, we have a lot of cases where models are kind of better than humans at some skills, worse than humans at some skills, but it's still pretty rare that you'll encounter setting where an AI is just better than sort of all of the human experts in some domain. And when that happens, things just get more complicated.
And more confusing, where even if you're humans kind of looking really carefully at what the I is doing, it's often hard to figure out, Wait, what is the I trying to do here, or what effects is this going to have in the real world.
With the modelogus he does, this makes everything you're trying to do, thankes.
Everything we're trying to do a fair bit in earlier. Yeah, Yeah, we're less confident that we can keep track of what's working, and I think there's just kind of more possibilities for whole new kinds of unwanted behavior to creep in that will have to find a way to grapple with.
I see, like, right now, maybe ais are at the level we are. Whatever issues it's having, there's things we can grasp. But as they get more advanced and they tackle bigger problems like solve the world's economy or figure out the right policy for the whole country or something like that that not one person can really grasp, it's going to be hard to even sort of like talk to it and understand it. I think that's what you're saying, right Yeah, Yeah, I.
Think there's even maybe two interesting ideas in there, because maybe the pseudo staff. That's something like, we're asking the AI to help us design novel molecules for pharmaceutical development, and it's got some really novel ideas about biology that are just really complex and really hard for humans to understand, and we can't tell kind of is the model actually convinced that this is going to work, or is the model messing with us and this would actually be kind
of dangerous. Should we try this drug, should we start to do some expermise in the lab. There's this setting where we kind of still ultimately know what we want. We know we want drugs that are safe.
Uh huh. Like it might tell you that this will cure cancer, for example, but you're saying, like, what else it's trading off to cure that cancer for example? You might not know.
Yeah, yeah, that's a good example. It's hard to tell if kind of the models like genuinely trying its best and genuinely thinks this is the best cancer drug, or if it thinks, oh, this is just something that looks good and it doesn't actually care if the drug will ultimately succeed, or if maybe for some reason, the model's extremely scary and it's actually trying to mess with you, and you've got your scary miss lined AI that's trying
to sneak in some slow acting poison. The smarter the IA is, the harder it is to tell the difference
between those different outcomes. And then once you start talking about a lot of these really kind of ambitious sort of capital f future social scenarios like AIS trying to figure out sort of what the economat would be like or how the world should be governed or something like this, and like, I don't know how much we want to use AIS for things like this, But once you get into that territory in any way, then you just get into this extremely weird situation where I don't know if
anyone is going to know what we even want, Like, what is the right way to govern the world, What is the right way to do?
I don't know.
Yeah, yeah, At some point, figuring out how an AD should behave requires you to solve philosophy requires you to figure out what is good. And the more powerful AI gets and the weirdest situations you're putting it in, the more kind of common sense notions of what's good start to fall apart. The more you actually have to grapple with a lot of the really hard, confusing stuff.
It might tell us how to run the world, but we at that point know one person or even a group of people might know, is this actually the best way to run the world. Is it sort of taking into account the things that all of us collectively would value. That's kind of the problem.
Yeah, And I think you start to get at some of these pretty difficult questions even before you get into these kind of really big features of sort of how to go on around the world. If someone is getting all of their news or getting all of their personal life advice from an AI, that's already giving the AI a lot of leeway for kind of what makes a good life for this person, what is important for this person to know? And those are already questions that get
really hard. And what you want in the short term might not match what they want long term. What makes you happy? You might i match their intuitions. What's good for that person might not be what's good for their community might not be the same as what's good for the world.
You made me think of it. I wonder iful good analogy is that it'd be almost like if you as a parent, I don't know if you have kids or nieces or nephews. But it'd be almost like if your kids suddenly try to tell you what to do or was trying to teach you how him or her wanted to run their lives. You'd be like, you're just a kid, what are you talking about? This is trust me, this is what you need to do. Yeah, except that we are the kids and the AI is sort of the parent.
Is that sort of what we're the situation that might be sort of parallel to that.
Yeah, I think this thing there, I feel like a version of the analogy that I'd be more excited abou would almost be some alien species lands huh, and they have all this great technology and they seem nice and they're like, hey, we'd really recommend making some changes to your side. He maybe try doing things like this, And we're like, wait, you're really very accomplished. You have some some useful ideas, but like, are you trying to help us?
Are you trying to sabotage us? Are you just kind of produced?
Are we what's for dinner? Or are you inviting us to dinner?
Yeah?
Yeah, yeah yeah.
The sense I'm getting for you is that these things are just getting smarter and more capable, so it seems to really pressing. We figured this out now before it gets even more difficult. Yes, yes, AIS are getting smarter each second, it seems, and we seem to be trusting them more and more each day with our data, our choices, and even our lives, which brings us to the main question end of the day, what can we do about it?
How do you train an AI to care about us, to have our values and to make the right choices. To answer this question, I reached out to another AI expert on alignment, doctor Tim Rutner. Doctor Rutner is a professor at the Vector Institute for Artificial Intelligence at the University of Toronto, and he says there are many ways to train AIS to like us. The only problem is none of them work perfectly. So here's my conversation with doctor Tim Ruttner. Well, thank you, doctor Runner for joining us.
Thanks so much for having me on it.
And I'm talking to a real person right now, right, You're not an AI version of.
Yourself as far as I'm aware.
As far as any of us are aware. Yes, I mean this whole conversation could be AI generated.
I know we're just all in the simulation.
Well, it certainly be a lot easier. I would get more sleep for sure. Well, today we're trying to answer a very critical question which was posted by our sound engineer, which is is AI going to kills all? Can an AI have values? Can it AI have an understanding of a human? What good things are to a human?
Yeah? And well I wish I had the answer to that.
Maybe that's the problem is that we don't know.
Yeah, I mean this is such a difficult question, right, and I think that this is a question that touches on philosophy, engineering, psychology, and probably many other disciplines. Right, but what are values and what values can possibly in a non sentient being have?
It's not a simple questionnaire.
Yeah, let me take a step back. So the way to think about alignment is I think through the lens of what's referred to as the specification problem, Where specification is the term that we use to describe what we tell the model it should do.
When you say specification, you mean like spec right kind.
Of Yeah, yes, inspect it's just short for specification.
Yeah.
This is what we can think of as our intent, the kinds of things that we want a model to do. For example, our intent might be for models to never say things that could lead to harm or intent could be that models should always be friendly and helpful. And so this is what we call the ideal specification for that.
Model, meaning like we want to be able to say, like, be a chat butt, but make sure that nobody ever hurts themselves.
That's right, I see. And there are a few different ways to provide specifications to chatbots.
Okay. According to doctor Runner, there are three general ways to make sure ais behave or not kill us. The first way is to basically tell it to behave every time you ask it to do something.
There is what's called a system prompt. This is a text specification that a model loads every time afford engages in a conversation with a user. In the case of the chatbot, so.
Every time you interact with the AI, you would basically instruct it to behave. You might say, hey, AI, organize all my emails, or design a new drug for me, or figure out the best policy for our government, but please make sure that no one gets harmed, that you don't do anything dangerous or an ethical, etc.
Etc.
But of course this would get pretty cumbersome if you have to do it every time that's option number one. Option number two is to have humans train your AI to be good.
So one approach is called reinforcement learning from human feedback, so different answers for a given prompt and then having human labelers say which of these answers they prefer.
What does that look like? The human notator is like a warehouse full of people just talking to the same AI, or is it three people or is it a thousand people? What does that look like?
So I should say I'm not an expert on this, but my understanding is that much of this work is outsourced to countries where the medium wage is lower than for example, in the United States or in Europe. There have been reports of large groups of annotators, specifically annotating images and texts that are considered not safe for work,
for example, in those countries. So in other words, you have examples of folks in those countries that are already less privileged than people living in the United States, for example, on average, engaging with a lot of horrific content and saying this is not something we want.
So this is basically paying people to test drive your AI. You could have a warehouse full of people whose job it is to interact with the newborn AI and essentially have them raise the AI and tell it what is right and what is wrong. Unfortunately, as doctor Runner said, I mean the poor people would have to bear the absolute worst behavior of the AI. That's option number two.
Option number three for making AIS that care about us is to basically bake into the AI a constitution, you know, like the US or UK constitution that establishes what the country stands for, what its values are, and what's generally allowed and not allowed.
There's an approach that Thropic introduced called constitutional AI, and that approach is based on providing a constitution to an AI MODL, and that constitution reflects different values and preferences that the company in this case, Anthropic wants the model to exhibit.
Yes, the last approach here to making sure an AI has values and morals is to give it a founding document. But here's the wild part. The way to big that founding document into the AI brain is to have another AI train it. When we come back, we'll dig into that scenario and we'll ask our experts what they think the future holds. Will future AIS have our best interests in mind? Or is it hopeless to ever be certain they won't harm us, So stay with us. We'll be
right back. Hey, we'll come back. We're talking about AI alignment, or basically the problem of making sure AIS don't kill us all. And so far we've talked about why this is such a hard problem and what are some of the ways we can teach AI things like values and morals. There are several ways, and one of them is to give AIS a constitution or the equivalent of a founding document or moral guide, and then have that bag into
the DNA of the AI. Now what's interesting is that, according to doctor Tim Rutner, the way to do that is through another AI.
This is a little bit in the weeds, but providing a very long constitution that outlines every preference in detail when a user engages with the model is actually more expensive for the company to do because the model needs to ingest a lot of text upfront, so it's easier to try to bake the preferences that are expressed in the constitution explicitly into the model when you're training it upfront, as opposed to providing that specification every time a user engages with the model.
I see, you want the model to have learned the constitution sort of inherently, rather than having to check it every time somebody asks it a question.
Yes, I think that's roughly right. Ideally we would be able to use humans to provide feedback and to oversee models and to say, hey, this is behavior that we don't want and stop that behavior. But of course that's
not really scalable. We can't have a human oversee every interaction that a chatbot has, And so that raises the question, how can we exhibit oversight in a way that is safe, reliable, aligned with our values and preferences, and successful, And so key challenge here is essentially to come up with tools, methods, models that are able to check whether a given of AI model perform some unintended behavior, and if it does, can ring an alarm bell and let a human know
that oversight is needed and that maybe a model engages an undesirable behavior.
You mean like, have an AI police the other AI.
That's right, essentially, have one model overse.
Model WHOA But then how do you make sure the police AI is doing its job or is aligned itself? You need another police.
We don't know, that's the problem. We don't know. It's a turtle, it's all the way down from it. And if we have a model that checks whether another model does what we wanted to do, and how do we know that that model that does the overseeing is actually aligned with us? I would argue that it might be easier for us to make sure that the overseer model is aligned than the generator model. They're a little simpler because they don't necessarily generate. They just try to classify
whether a given behavior is intended or not intended. And so this way we might be able to do alignment more scalably and in a way that really reflects different individuals or groups preferences and values.
I see. It's like, have another AI can of sit in every time I ask CHGBT something yes. Yes. As AIS get bigger and more complicated, the only scalable solution to training them is going to be through other AIS. In this situation, you might program a simpler AI with your values and morals, and then you'd have that AI train the bigger AI release try to It's like the Rutner says, AI alignment methods are not perfect. What do
you mean? They're not perfect? They don't always work or they can't guarantee that they will work.
So with machine learning models, we can rarely guarantee anything.
Oh boy, that's kind of the problem, isn't it.
Yeah, I think that's one of the problems. There is research that tries to establish guarantees, but that research is far behind the practice at the moment. The kinds of methods that we have for model alignment falls short in a few different ways. One that's I think one of the biggest ways. It's just hard to communicate our preferences. So there are many different steps at which alignment can fail. This goes back to trying to express and then community what we want a model to do kind of values
and preferences we're trying to install in it. Translating our values and preferences from some really complicated, possibly contradictory ideal specification into a design specification is very difficult and there's
likely going to be some gap there. And then second, even if we were able to do this perfectly, even if we were able to express and communicate our values and preferences perfectly, the kinds of low level tools machine learning tools that we use to give the model these preferences and values are imperfect at translating the values that we're trying to communicate into the model, They might not enable us to perfectly translate the design specification into the
actual behavior that we would like to see.
I see, boy, it seems like there are problems everywhere we turn here, doctor rut. Yeah, Well, as we go towards the future, and as systems get smarter and problems get more complicated, what do you think is the prospect of making sure that these more advanced systems were more complicated problems have values that we want it to have, Because I'm not sure if we want it to have human values, because I don't know if humans are the best making these kinds of good choices. Yeah, what do you think?
What do you what do you see in the future.
I think tho's a few things we need longer term, and they all feel uncertain. I think to do well in the longer term, we need to get the AIS in near future.
Right.
If the pace of AI development stays fast, we're really really going to need the help of AI systems to help us figure out how does your future A systems? And so getting the right values into the next model we build helps us figure out what to do with them adel after that, and so I think this kind of the short term work really does kind of fan out into this longer feature. And yeah, getting the next model right really matters for getting.
The fartugerules right.
Oh boy, it's called.
The scalable oversight problem.
I see. It's like the alignment problem is going to scale up, and the best way for us to keep up is to make sure that we get it right now with these smaller systems, so we can use those AIS to help us in the more complicated situations.
Yeah.
Yeah, that's oh wow.
That's the hope.
Okay, last question. Do you think humanity is doomed?
I don't think so. I think it's possible. I think the AI presents a lot of really scary and destabilizing possibilities that we can't roll out. So I think there's a lot of work to do. I think we'll probably figure it out. But I think it's also possible that AI winds us up in a lot of weird, unfamiliar situations. I think it's unlikely than possible that things go really, really terribly, But I also think it's kind of unlikely but possible, but the things stay totally normal and recognizable
and familiar. I think AI is just what it's going to do to society and politics and economics is all going to be confusing. I the's a lot that we'll need to figure out pretty fast.
Fingers crossed.
I guess that's as good of an answer as we can get these days. Fingers crossed. I guess just to wrap up here, what do you think is going to happen in the future, or what are some things about this that you think most people are not thinking about that they should be thinking about.
I think people should be thinking about ways in which the AI systems that we have today are already capable enough to cause harm, to change our world quite significantly, change our culture, change the way we go about our day, change the way we make decisions, change the way we do our work. And the alignment problem and understanding when models are aligned, I think, are two of the most fundamental scientific challenges that we as a society are facing
right now. And that is not a sci fi future problem. This is a problem about systems that we have to We want to make sure these systems really do what we want them to do, and that these systems help us flourish and benefit humanity. The systems that we have access to today already well beyond the capabilities that the research community and certainly the general public thought we could have in the year twenty twenty six.
I think you're saying that the future is here, but we still haven't fully figured out the alignment problem.
Yes, that's right, meaning it's.
More pressing than ever that we figured this out.
I agree.
Yeah, amazing, doctor Rutner. How do we prove to the audience that we're not an AI generated conversation?
I wish I had the answer. You can generate such fantastic fake podcasts with AI now right right, with all the little idiosyncrasies that you hear in podcasts today that you know, I think that's hard to do.
Or I guess if this conversation is aligned with but you want to hear, maybe it doesn't matter.
Still, I hope that the audience thinks that we so out on human I think that that would be That would be nice.
All right? Hey on, behalf of everyone who works in the show picture joining us on the sixty plus episodes we've done. Be sure to follow me on social media or PhD comics dot com for updates and hey, thanks to all the guests we've had on the show. Here's a little tribute our editor Rose so good to put together of all the times they were gracious enough to put up with my questions.
That's a good question.
Yeah, that's a great question. You know, that's a great question.
Yeah, so those are all big questions for us to answer a scientists. Yeah, that's a great question. It's a good question.
That's a good question.
That's a good question. So that's actually a good question.
You're you're raising really great questions.
Yeah, that's a really good question.
That's a good question. That's a really good question.
That's a very hard question. That's a good question, though, that's a really good question. That is a really good question. Yeah, that's a great question.
Yeah, that's a good question.
Oh that's a great question, very very good question.
That's a really good question.
So I'll go through that question back at you.
What do you think? So we come once again to the edge of scientific knowledge. Thanks for joining us, see you next time you've been listening to Science Stuff. Production of iHeartRadio Bringing the produced by me or Hey Cham, edited by Rose Seguda, Executive producer Jerry Rowland, and audio engineer and mixer Kasey Peckram. You can follow me on social media. Just search for PhD Comics and the name
of your favorite platform. Be sure to subscribe to sign stuff on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.