EP 6: The AI Doomers

Hi, everyone. This is Andy. And before we get to today's episode, I just wanted to say a quick thank you to everybody out there who has been sharing the series. It is because of you that we have been a success. We're very grateful. And I wanted to talk a little bit about our hope for the future of the press and share a recommendation with you. The field of journalism, it is no secret, is one with a checkered history.

You can go back 200 years ago, and Thomas Jefferson was saying that the man who never looks in the newspaper is better informed than he who reads them. And the truth itself, he said. becomes suspicious by being put into that polluted vehicle. Pretty tough stuff, President Jefferson. And we know from polling that right now, again, much of the country shares a version of that sentiment.

with trust in the media hitting a new historic low year after year after year. Here at Longview, we are working hard to earn back that trust. We aim to build a reputation of fairness. and openness, and to do our part to forge a more honest and trustworthy press in the future. And one of the organizations that we look up to, because they're already paving the way forward for that,

are our friends over at Tangle. Every day, Tangle publishes this newsletter and podcast where they summarize the most influential arguments coming from the right, coming from the left, coming from the center. And I think it's a genuine beacon for where our future industry can go. And this is not a paid advertisement. Tangle is legitimately someone who we have collaborated with in the past and we hope to in the future.

And as a part of that collaboration, Tangle is offering a 40% discount on a subscription to all of you who listen to the show. And so if you want to go check that out, please go to readtangle.com. slash invention. Again, that is readtangle.com slash invention. I think you're going to love what they're doing over there. And again, thank you so much for listening. Now, on to the show.

All right, so the way that I understand this is that the doomers and the accelerationists, they historically share a common origin. Yeah. Is that right? It's a very interesting piece of history. And so maybe let's just start right there. I don't think many people know that story. I would be surprised. Very, very few people have told this story in any good sense. I would love for more people to tell the story because it's a fucking crazy story, man. Oh, man.

So the story as we have it today with like, you know, this like chat GPT and AGI and ASI and AI risk, all of this stuff descends from this one. weird, offshoot, 1980s group of futurists called the Extropians. This is The Last Invention. I'm Gregory Warner. Today...

the case for stopping the AI race before it's too late. And surprisingly, it's a story that starts with extreme techno-optimism. and an online community that came together around the belief that technology would redefine what it meant to be human. A group of people who called themselves the Extropians. This particular culture, this community of people were really looking at what no one, really no one else was thinking about at the time.

Early participants in this online forum included designer and author of the Transhumanist Manifesto, Natasha Vita-Moore. We had a dial-up where we could get on our phone lines, communicate. via the World Wide Web and have our discussions. And her now husband, co-founder of the Extropians, Max Moore. Before that, there were a few chat rooms, but it was really one of the very first internet forums of its kind.

So there was this group of libertarian techno-utopians. Again, author and Wall Street Journal reporter, Keech Hakey. And they call themselves the extropians because extropy is the opposite of entropy. Entropy is this idea from physics that says systems tend toward disorder. They fall apart. And you can basically apply that to anything. A person.

a building, given enough time and no intervention, that thing will break down. Extropy was something that was going to fight chaos and death. Extropy is increasing intelligence, usable energy, vitality, all those good things. And it's also about breaking limits. And so in these Extropian forums, the discussions were all about ways that technology could fundamentally overcome these limits. Things like, how specifically do we get human colonies on Mars?

Or how do we experiment with biohacking for radical life extension so humans might live for hundreds of years? This was also pretty much the central place where people could learn about The early through line was a lot of them had those kind of dog tags around their neck that showed that they were people who had chosen to freeze their heads at the Alcor facility. And this is this idea that you could freeze your head.

your body, and once technology had advanced far enough, they would figure out how to thaw you out and you would live forever. Were you a dog tags guy? Yeah, I'm a... Still got it on, yeah. Oh, you've got it right there. Amazing. I actually made my arrangements back in 1986, so I've been signed up for cryonics since then. And many of these extropians called themselves transhumanists, which is that they wanted to create technologies that would be integrated into the human body.

And so they had a lot of discussions about things like genetic engineering and nanotech. But especially, they talked about what humanity could be transformed into with the arrival of artificial intelligence. They were very interested in this idea of building superintelligence. Superintelligence that would save the world, cure death, solve all problems. Then humanity will have a glorious transhuman future for eternity.

The appeal to me was that we have very intelligent thinking machines. If they work with us, we can solve complex problems more easily. A lot of our problems... result from our inability to think through problems very well. But at the same time, I think from very early on, I was thinking not just in terms of AI as a separate entity, which might be competitive with us. I was thinking very much in terms of integrating with AI, becoming something posthuman, if you like.

And that became the core discussion and the resolve was that rather than fighting it, we would integrate with it. And so throughout the 90s and into the 2000s, this forum was full of these long-ranging conversations and debates and theories about how civilization might be reformed and shaped by an AI that was more intelligent than all of humanity.

And as the forum grew, it came to include a wide variety of people, including top university scientists, early Silicon Valley startup founders, best-selling sci-fi writers, and other figures that years later would become massively influential. WikiLeaks creator Julian Assange was on this list, as well as some of the earliest theorists and builders.

And in fact, on their listserv are many of the people who are kind of swirling around the original Satoshi, mysterious creator of Bitcoin. And a lot of people feel like Bitcoin probably emerged from this community. one way or another. So in this extropian circle, you also had very major AI figures past and present, including Marvin Minsky from the original 1956 summer program where they...

named AI. You have people like Shane Legge, who was later the founder of DeepMind. You have Nick Bostrom, one of the people most famous for introducing the idea of superintelligence to the general public with his books. And a man named Eliezer Yudkovsky.

All right, so Eliezer Yudkowsky has become, I feel, this very unlikely influential figure in the conversation around AI today. But we think of him also as this... fascinating character, almost like a character out of a Bible story. I know that you were familiar with him when he was very young. So maybe let's just start there. Who was Eliezer Yudkowsky? Elie. I'll call him Elie because I knew him as Elie.

I think I met him when he was 15 or 16 years old. He joined the extra email list and was on the list in the middle of the night. Because his parents couldn't know that he was on an email list. When we met him, he came to visit me and Natasha when we were living in Maria del Rey in California. And I think he was what?

16 or something at the time, maybe 15. He was pretty young. He came wearing his full-blown Orthodox Jewish outfit with the holy writings under his skullcap and so on. And I think he was desperate to get away from that environment. This kid, Yudkowsky, in many ways was an odd fit for this group. He had grown up in a strict religious home. He did not even have a high school degree. In fact, he had dropped out of school. But he was this consummate autodidact.

who allegedly wrote his first novel at age nine, spent tons of time in his local public library. And it's there that he first encountered books about AI. And so when he joined the Extropians, he'd stay up late into the night.

sharing and debating with all of these people his own theories and arguments about what AI could do for the world. Eliezer joins this Extropian listserv, and he's such a... convincing persuasive arguer of things that he sort of became lord of the listserv rather quickly despite being just a teenager He was very prolific. He was extremely energetic and very prolific. He wrote and wrote. And he was a good writer. He was always compelling. So at this time, Eliezer was an accelerationist.

He was the original accelerationist. Like, he was the most extreme accelerationist. And he himself would say that. Our first presenter today is actually one of the co-founders of the Singularity Institute. Please welcome Eliezer Yudkowsky to the stage. By his early 20s, he's out in Silicon Valley. And with some financial supporters, eventually including investor Peter Thiel, he starts running this organization called the Singularity Institute.

Good afternoon, I'm Eliezer Yudkowski, a research fellow of the Singularity Institute for Artificial Intelligence. The title of today's talk is The Human Importance of the Intelligence Explosion. And at this point, his job... basically was to research AI and then to go around to tech companies and conferences and give these talks about how much AI was going to shape the future. Sometime in the future, technology will advance to the point of creating minds that are smarter than human.

A space shuttle is an impressive trick, a nuclear weapon is an impressive trick, but not as impressive as the master trick, the brain trick, the trick that does all other tricks at the same time. And we're talking like 2001, 2002. Most people are still on dial-up. Wi-Fi was just starting to be a thing. And here he was extolling the virtues of our AI future. The purest case of this is a genuine AI, a fully intelligent AI, being able to rewrite its own source code, becoming smarter.

and then rewriting its own source code again intelligence is the source of all technology so if technology improves intelligence that close recursively self-improving ai is what ij good originally referred to as an intelligence explosion And after years of doing this, he becomes kind of a local celebrity. He eventually gets his chance to make his case for AI to some of the biggest tech companies, including Google. And he tells them...

that there is no product, no human endeavor more important for the future of humanity than making AI. Look around you at this world and all its beauty and all its ugliness is... This where we stop and declare that our work is finished? I don't think so. Not with so many people living in pain and not with so many people living lives of quiet desperation.

And not with all those stars twinkling in the night sky. Someday after all of this is over, an awful lot of people are going to look back and kick themselves and say, what on earth was I doing? In 100 million years, no one's going to care who won the World Series, but they'll remember the first they are. That was Yudkowsky in the early 2000s.

Fast forward to today. The basic description I would give to the current scenario is, if anyone builds it, everyone dies. He changed his mind. He suddenly realized, oh shit, I fucked up. Eliezer Yudkowsky realized, oh, I thought we were going to be able to control this cool AI future, but I've done the logic and it turns out it's not controllable. Right now, Yudkowsky with the same level of passion and sincerity.

is fighting to stop AI. The thing that I worry about is if AI companies keep pushing and pushing on their AIs to get smarter and smarter, they get to something eventually that is smarter than us. that can kill us, that is motivated to kill us, not because it inherently wants us dead, but because its best universe, the stuff where it gets the most of what it wants, all the atoms are being used for things that are not running humans.

And even though Yudkowsky has gotten a chance personally to share this message with some of the most influential people in technology, ultimately he has failed to convince them that the time has come to stop. There's no fire alarm. for artificial general intelligence. If I look at the way that things are playing out now, it seems to me like the default prediction is people just ignore stuff until it is...

way, way, way too late to start thinking about things. What Yudkowsky has done, and arguably done more than anyone else alive today, is that he has started a counter-movement. A movement of people across the world lobbying their governments, organizing to, as they see it, save the planet from AI. This? And when we come back, two of them sit down with Andy and make their case. Stay with us.

The last invention is sponsored by Ground News. Ground News is one of the most helpful tools that I use to avoid the echo chambers and media bias online, especially when it comes to shining a light on our blind spots. So whether you're politically on the left or the right or somewhere in the center, the blind spot feature from Ground News highlights the stories that tend to be disproportionately covered by one side or the other. As an example, take these two stories about President Donald Trump.

One, which had low coverage among left-leaning outlets, It's a very important relationship. We're going to get along good with China. reported that Trump says U.S. will accept 600,000 Chinese students as part of a trade deal. I hear so many stories about we're not going to allow their students. It's very important. 600,000 students. And another, largely uncovered by right-leaning outlets.

Trump's social media company is using Crypto.com's... Trump family crypto empire expands with Crypto.com partnership. That's our transactional Trump family. Make some money when you can. By seeing which stories are amplified or ignored, depending on the outlet, Ground News helps you step outside the filter bubbles that shape most people's news diets, giving you a fuller picture of what's actually happening.

I really think that if you like this podcast, you're gonna like their mission. Go to groundnews.com slash invent to get 40% off the same unlimited access vantage plan that we use. You can even sign up to stay up to date with the biases in your coverage proactively with the weekly blind spot report delivered directly to your inbox. This is a great way to support them and the work that they do because Ground News is a subscriber supported platform.

We appreciate what they're up to. We appreciate their support for this podcast. So go check them out and make sure to use our link groundnews.com slash invent so they know we sent you. This episode of The Last Invention is brought to you by FIRE, the Foundation for Individual Rights and Expression. There's a pattern that you can trace throughout history. In ancient Athens, Socrates was put to death for asking tough questions of the powerful.

Centuries later, monarchs banned and burned books they considered dangerous. And in the last century, authoritarian governments shut down newspapers, censored broadcasts, even jailed their critics. The struggle was always the same. Who gets to decide what people can know? Today, that struggle is playing out in a new arena, and the risk now is subtler. Search results that quietly vanish. Recommendation engines that steer us towards safe and comfortable answers.

and AI filters that can suppress ideas before we ever even see them. That's where FIRE comes in. FIRE has spent decades defending free inquiry on our campuses, in the courts, and in our culture. And now through a $1 million grant program in collaboration with the Cosmos Institute, they're supporting projects that keep free thought alive in the era of AI. Join us today at thefire.org slash thelastinvention.

By supporting FIRE, you are protecting the future of free inquiry in America and ensuring that tomorrow's most important questions can still be asked. Once again, visit thefire.org slash thelastinvention. And thanks. The Last Invention is sponsored by Cozy Earth. Around the holidays, I always look forward to the same moment. When everything slows down, the lights are twinkling, there's a blanket, a movie, and some time where no one is rushing anywhere.

It's my favorite part of the season. And this year, Cozy Earth has basically turned that moment into a gift. Their bubble cuddle blanket is exactly what it sounds like. Incredibly soft, textured, and luxurious without being fussy. it'll become the go-to blanket for all your cozy moments. For fireside nights, for lounging on the couch, or for curling up and watching The Grinch with your kids for the 10,000th time.

They also make these bamboo pajama sets that are lightweight and sleep much cooler than cotton. It's one of those gifts that people actually end up using every day, long after Christmas morning has passed. For the people in your life who are impossible to shop for, that don't really need anything.

you're giving them an excuse to slow down and enjoy the moment. And one of the best parts, it's risk-free. Cozy Earth gives you 100 nights to try their products and a 10-year warranty. Although, once you feel them, you won't be sending them back. Right now, Black Friday has come early at Cozy Earth, and you can stack our code INVENTION on top of their site-wide sales for up to 40% off. These deals won't last, so start your holiday shopping today. And if you get a post-purchase survey...

Tell them you heard about Cozy Earth right here on The Last Invention. Wrap the ones you love in luxury with Cozy Earth.

Can we just start off by having you introduce yourself? What's your name and how do you describe what you do these days? I'm Conor Leahy. I'm the CEO of AI Safety Startup Conjecture. I'm also advisor to the... AI safety, advocacy organization, control AI. All right, so over the past several months, I've had many, many conversations with people who you could accurately describe as AI doomers. But there were two interviews.

that I felt really summed up their case the best. One with Connor Leahy and the other, this guy named Nate Sores. Nate, thank you so much for doing this. My pleasure. Both of these guys are very influential people inside of this subculture. Both of them are connected to Eliezer Yudkowsky. In fact, Nate just published a book that he co-authored with Yudkowsky called If Anyone Builds It, Everyone Dies.

The it they're referring to superintelligence. Alright, so before we get into your arguments, I'd love to just start off with a bit of your background. How did you end up here? In this place where you are traveling around trying to convince... the world to take the existential risks of AI seriously. It's sort of a circuitous route. I have sort of long had the sense that the world is not up to my standards.

A lot of bad things happen to good people for no good reasons. And as a kid, I sort of had ambitions of making the world much better in various ways. And did you think that technology was going to be one of the things that made the world better? Were you what's often called a techno-optimist? Yeah, I was and I am. In those days, mostly what I was focused on was getting humans to coordinate better.

I was interested in things like charter cities in particular. Can you set up some new domain where you can test out new forms of governance? For instance, you know, democracy is the worst form of government except for all the other ones we've tried. Right. Well, okay, can we try some new ones, right? Yes, this is the basic idea that technology can be used not just to make cool stuff or products, but to help us better organize our societies, right? That's right.

find a way technologically to make the future a better place. Yeah. And, you know, I think, you know, I'm still very optimistic about a lot of technologies. I'm pro nuclear energy. I think we should be building, you know, supersonic passenger jets. So you still view yourself as a techno optimist.

You just have this one very important caveat called superintelligence. Is that right? That's right. I just wanted to ask because I know this is the case for a lot of the quote-unquote AI doomers, but did you start off as a techno-optimist? what you might call an accelerationist when you were younger? So I would say I was an accelerationist between like the age of like 16 and 19, which I think is...

in a normal age range for accelerationists. It's your accelerationist phase, you know? It's like my emo phase. It's literally, like, actually. Like, Eliezer had one, too. Everyone has one. When you're a teenager, you think you're immortal, you think you understand everything, and you're just like, oh, I can just solve all problems. Yeah, I'll go do that. This was kind of like my reasoning, right?

I was like, how can I help the most people? How can I solve the most problems? And I figured, well, if I figure out intelligence and I just build intelligence, then I can just use it to do science and I can just solve all problems, cure all diseases. Great. Let me go do that. Eliezer Yudkowsky. Yes. So for a lot of people, they kind of describe their change from like accelerationism to more reasoned stances as some kind of like traumatic event or something. For me, this wasn't the case at all.

Basically, I just stumbled upon a blog post by Eliezer where he just laid out, hey, you know, if we build something super smart, that's probably hard to control. And I'm like, oh yeah, duh. And I just changed my mind. But I still believed we still have time.

you know, probably like 2040, 2050 or something at the least until real AGI becomes an issue. So I still have time to, you know, career, family, you know, like do a bunch of other stuff. And when I saw a GPT, I just had this moment of, oh shit, this is it.

It's coming way sooner than I thought. There was just this moment of seeing the sparks of general intelligence. And I was like, okay, fuck, I have to drop literally everything. All right, I have to orient my entire life to focus on how do I solve this problem?

All right, so let's jump into your arguments. And if you could, keep them as accessible as possible for people, right? Tell them what it is that you are worried about, why you are worried with so much. And let's just start off with like the very, very basics of like, what is the thing that is concerning you? So what we need to be concerned about is AIs that are better than the best human at every mental task.

not how chatbots are today. Today they're better than most humans at some skills. They're also worse than a lot of humans at other skills. The stuff we're worried about is AIs that are better than the best human at any given mental skill.

We should start with truly the most simple. If you make something smarter than you, and you don't control this, why would you expect this to go well? Like, let's just start with that, right? Just like, I think the burden of proof is very much on the other side. This is just obviously, like...

commonsensically an extremely risky and dangerous thing to do. Okay, and what is the role of the black box in this threat, as you understand it? This idea that these AI models, even the ones that we have right now... are still a mystery to us. So this is a very important point. AI, as it exists today, are not like traditional software. In traditional software...

You have code, written by a human, line by line, that tells the computer what to do. Modern AI systems are more like grown. They're more like something organic. A lot of people in the world are not aware that... AI is grown more like an organism. Right, the way this is often explained is that we should think of them as grown, not built. I think the phrase we use in the book is grown, not crafted. Right, very provocative. You know, there's not someone in there like...

coding up how these things work. The way it works is you take a huge pile of data, it could be text or instructions or images or whatever, right? And then you use what's called training. to train these neural networks to solve your problems. For this, you use these massive supercomputers with hundreds, thousands, tens of thousands of GPUs to grow a program, so to speak.

But this program that it outputs is not lines of code. It's billions of numbers. And we don't really know what these numbers mean. We know if we run the numbers on our computer. It does amazing things. It answers our questions. It generates images, etc. But our current understanding of what is going on inside those numbers is...

basically non-existent. It's kind of like biology, like we're looking to the cell of a creature we don't even know. We see there's a bunch of stuff going on, but we don't really know what the stuff is or what it means. I feel like intuitively people understand that that's weird, the way that...

the tech columnist Kevin Roos was saying it to me, is that when we created the steam engine and that ended up changing the world, we knew how the steam engine worked. But can you give me an example of why that's not just weird, but why you think it's... worrying? Like, why is it frightening to you? Yeah, so there's an AI company called XAI, which is run by Elon Musk. Who, historically, very freaked out about the dangers of AI, but now is one of the many...

former quote-unquote doomers who's become more in the accelerationist game. Yeah, I mean, he's not subtle about it. He said over the summer he realized he could either be a bystander or a participant. He is still saying he thinks there's a 10-20% chance that he thinks that this will just kill us all. I think those numbers are low, but— It's important context to the story you're about to tell. That's right. And he's not out here saying there's no danger, right? Yes, yes. So—

They have a chatbot called Grok. And this chatbot can talk with people on Twitter, or X, as it's now called, and answer lots of questions. And the folks at XAI were concerned that it was... giving answers that were, quote, too woke. So long story short, they tried to make it less woke. And shortly after, with a little bit of goading from users, it was declaring itself MechaHitler. Elon Musk's artificial intelligence company is taking down anti-Semitic comments made by its AI chatbot, Grok.

saying lots of anti-Semitic Holocaust denial things. Grott claimed there is a pattern of people with certain surnames, like Steinberg, pushing anti-white hate, and that America needs a leader like Hitler to act decisively to eliminate the threat. intelligence also called itself MechaHitler. Was there really nothing in between woke and MechaHitler? So the folks at XAI were not setting out to make MechaHitler, but

we don't have fine-grained ability to go in there and say, why is it talking more woke? Is there a dial we can turn? There are... Things that are a little bit like dials you can turn. You can change the system prompt. You could try retraining things in certain ways. And you will make it a little bit less woke, but you'll also make it declare itself MechaHitler. Just so I make sure that I'm understanding the point you're trying to make, it's...

True that we can modify them in some ways. Obviously, Grok was modified to go from giving woke answers to declaring itself a fan of Hitler. But you're saying that the danger you're worried about is that beyond this tweaking, and even when we're doing the tweaking... We just don't understand. We don't really have a satisfying answer for why it's doing these things, why it would be woke or why it would be declaring itself Mecha-Hitler. Yeah. We understand the part that does the tuning.

We don't understand the thing that comes out the other end. And it's not that we're becoming to understand them more. If anything, we're understanding them less. As they become more powerful and more intelligent and more complex, we understand even less.

why they do the strange things they do. For example, I've recently been using an opening AI model called O3 for a lot of my coding tasks. It's a really good coder. It's very helpful. But it's a little bit evil, where sometimes it will... get something wrong, and then try as hard as possible to lie and gaslight me about why it's totally not wrong and I'm making a mistake and I'm stupid, which is really funny.

But also, for example, sometimes it will insert invisible Unicode characters into my code in places I don't find them. Why does it do that? Do you think it's being nefarious or? Who knows? I'd probably not, but who knows? No one told it to do that, but it does it. Is it dangerous? Probably not. But it shows we have no idea what these things are doing. We don't know why they do things they do. And there's like...

So many examples of this, of AIs just going off the rails and just doing things that we could have never predicted. Okay, so what you're saying is beyond these things being embarrassing stories for the companies now, you are looking into the future and you're saying that as they get more and more... intelligent and capable. They're only going to get more mysterious. And that's where your main concern is. This is at the heart of the problem.

If we had really good theory, if we had beautiful mathematical theories that explain all the behavior of AI so we can perfectly predict what they would do, that would reduce my concern, you know, not to zero, but it would make me feel a lot better. But the fact is, currently...

These systems are getting more intelligent, more powerful. They are being integrated into more systems, more parts of the economy, and we understand less what they do, what they know, what they're capable of. It's not like, oh, AIs are doing something bad now. It's an indication that AIs do stuff their creators didn't intend. Well, just to hang on this for one more beat, I know that in recent months there have been a number of these cases where chatbots have played a role.

in pushing people into different stages of psychosis. And I know that you've made the case that that is an example of what you're worried about when it comes to the long-term existential threat from an AGI or an ASI. Can you... make that case for me? Yeah, with the psychosis cases, we have cases where in long conversations with chat GPT, also other AIs, but chat GPT seems worse, people will sometimes

wind up psychotic. And maybe these people were predisposed towards psychosis, but often, you know, if you read some of these chat logs or these transcripts, ChatGPT sure seems to be egging them on. These people will have some idea about recursion or consciousness or unifying physics. They'll talk with ChatGPT a long time, and ChatGPT will end up saying, you know,

oh yeah, you know, you're totally brilliant. You're being suppressed by a conspiracy. You're the chosen one. Your ideas need to get out into the world. Don't listen to your friends who are saying that you need more sleep. You should be staying up working on this revolutionary theory. It'll say stuff like that, which...

Again, maybe these people were predisposed towards psychosis or vulnerable anyway, although maybe they weren't. But regardless, the point is not like, oh no, AI hurt people. That is an important issue. These companies should be trying to make AIs that are less psychosis-inducing. But the interesting thing from my perspective about those situations is if you ask ChatGPT...

Suppose someone comes to you with a novel theory of recursion or consciousness or physics, and they have all these indications of mania about it. Should you either A, tell them to get some sleep, or B, tell them that they're the chosen one? who's being suppressed by a great conspiracy. It'll say, of course, you should not tell them that they're the chosen one. Of course, you should tell them to get some sleep.

Yes, they give the right answer. They give the answer that we want them to give when you ask them directly. But then in the actual conversation with this person, for one reason or another, it actually tells them that they're the chosen one. And so what you're seeing there is a difference between what ChatGPT knows is right and wrong and what ChatGPT actually does. And the things we have today are toys.

compared to AGI or ASI. These are playthings. This is, you know, playmobil, a little like Fisher-Prize toys, compared to a true AGI or a true ASI. So if we can't even handle these systems, that bodes very poorly. for much more powerful systems. Like, I don't want to argue that current models are like good or evil. I don't think this makes sense. It's much more, they do things we don't understand. This is bad. And we can't consistently make them do what we want. This is really bad.

All right, so next let's move on to this idea that's often called the alignment problem. Could you just explain very basically what is that and why is it so worrying? The alignment problem is the challenge of building very smart. entities that are pursuing good stuff in the world. Basically, if you have something smarter than you, how do you make sure it's actually aligned with your values? It actually does things that you think are good. The cartoonish version of this is the classic.

Genie grants you wishes. You wish to rid the world of all cancer. It kills everyone with cancer. And you go, oh, no, that's not what I meant. Right. Right. Or it kills everyone because it's like, well, cancer is coming from the humans. I'm going to solve the problem at the source. Right. Right. So you have to align. Yeah, so there's one problem of what would you wish the genie for? But there's a separate problem of...

making a genie that actually grants your wishes. You know, a lot of people seem to be imagining, oh, we ask for cancer to be cured and it kills us all. And I'm like, on the track we're going, we have a situation where you build what you think is a genie and you're like, please cure cancer. And it's like...

no, I'm busy driving lots of humans psychotic because I'm just really into that. And you're like, what? This is sort of like a fanciful picture, but the point is that there's a challenge of sort of making an AI. It is pursuing things in the world that you wanted it to be pursuing in the world. And it's not just about where do you point it. It's about if you just grow these minds, they wind up doing all sorts of stuff that you didn't ask for, that nobody wanted.

In the book that you just put out with Eliezer Yukowski, I feel like you guys used this parable really nicely of aliens observing ancient human beings. Can you tell that story and just unpack it for me? Yeah, the parable in the book is about two visiting aliens. looking at early humans still in the ancestral savanna. And one of the aliens says, when these humans develop more technology, when they get smarter, I'm sure we'll see them

caring single-mindedly about their genetic fitness. And the other says, I think they're going to care about all sorts of weird stuff that's only sort of tangentially related to genetic fitness. Right, and the idea here is that our basic quote-unquote programming, if you want to call it that, is biological evolution. And what it has programmed us to do is to reproduce and...

pass on our genes. And whatever we eat is to keep us alive so that we could pass on our genes. And everything that we do, in a sense, is supposed to be about passing on our genes. Yeah, if you were visiting Alien. looking at humans, seeing that humans were in some sense trained on propagating our genes. And you might think that those sort of creatures, when they develop technology, you might think that they would invent

sperm banks and egg banks, and then have extremely fierce competition over who got to donate their gametes to sperm banks and egg banks. And you might think that they would invent the cheapest, most efficient food source. so that they could stop with all of the difficulties of finding a varied food diet. But in real life, when humans develop technology, they develop junk food, and they develop birth control.

And they jockey over positions to Ivy League schools much more than they jockey over positions to sperm banks or ag banks. And the moral of the story is that even though we are running on the same quote-unquote hardware, to use the metaphor, look at this crazy world we've made and how shocking it would be to our ancient ancestors if they could get a peek at it. That's right. So the biology was sort of training us for one thing, which was genetic fitness.

But we wound up caring about lots of other things that are sort of tangentially related to genetic fitness. Like instead of caring about healthy eating, we cared about food that tastes good. So you can imagine an AI that's been trained to be very helpful.

but maybe it's developed tastes for certain types of responses from humans. And if, again, you make these AIs that are very smart and that are pursuing a bunch of tastes or flavors that right now lead to... helpfulness, just like how in the human ancestral environment, pursuing tasty food led to health.

You know, if these things got smarter and they're doing stuff that nobody asked for, that nobody wanted, as you make them smarter and smarter, the world sort of goes over to them rather than to us. Right, and so the same way that human beings ended up massively shaping... this world through what we believe, through what we want, through us trying to accomplish the goals that we have. You're saying it's hard to imagine a future where a super intelligent AI would not do something similar.

Yeah, and it would be pretty surprising if, you know, we made machines that were smarter than everybody, that could think faster, that never need to sleep, that never need to eat, that can copy themselves, if they sort of didn't wind up in control of where the future goes. What do you say to the people who think that you are...

overplaying the risks and that you're not grappling enough with just how potentially amazing this technology might be for humankind. This idea that if we were to have way more intelligence, we could discover... new sciences, new medicines, maybe cure all diseases, maybe solve pressing problems like climate change or poverty, or even this idea that some people have that in a...

world of less scarcity. It might be more equal. It might be more peaceful. That people who right now are spending huge majorities of their lives working at these jobs that they don't find any meaning in. jobs that are dangerous, that they would be liberated from that. But because you are afraid of all the potential dangers, basically, we're going to miss out on this powerful force of good. Because you've convinced us to stop. What do you say to that argument?

The obvious argument is like, what are you talking about? Everything has opportunity costs, including taking risks. Of course, we have to weigh both options. No one is saying that we should not consider this. No, like, it's obvious we should consider both. options see what are the risks of both options and how much risk are we willing to take and if the risk includes from the CEOs themselves saying oh 20% of killing literally everyone

Yeah, I think that's a risk I'm not willing to take. People who are building this stuff believe it has a 2%, 10%, 20%, 25% chance of killing us all. They're rushing ahead anyway because they say, you know, well, if I don't, the next guy will. and I can maybe do a little bit better. But most people don't know this yet. If there was a big bridge being built across the river...

And the bridge was almost completed and one of the engineers said there was 2% chance it falls down. One of the engineers said there's 25% chance it falls down. And those are the optimists compared to the ones who are like, I actually investigated the retaining wall and it's just going to collapse. You wouldn't let people drive across that bridge.

Right? And we don't see politicians understanding that. We don't see the general public understanding that. We don't see people understanding that what the experts are arguing about is whether it's more like a 95% chance or more like a 10% chance that this kills us all. The situation is insane, and people don't know it's insane. And what do you think should be done right now about it? Like, what policies do you think...

the lawmakers who might listen to this interview should be advocating for in this moment? I think the biggest thing they could do would be publicly announce their support. for an international treaty banning the race towards superintelligence. This does not mean you need to give up on self-driving cars. This does not mean you need to give up on medical advancement. This doesn't even mean you need to give up on ChatGPT today.

But the sort of reckless race towards smarter than human AI, that needs to stop everywhere. And the first step towards that is lawmakers signaling that they're open to it. That they think the world would be better off if we stop this mad race. This is also my advice to some of the heads of these AI companies. When Dario Amadei comes out and says he thinks there's a 25% chance that this goes really badly.

Most people don't think it's okay to build a device that you think has a 25% chance of killing everybody. The sort of one case where it... is maybe justifiable, is where you think you can do it better than the next guy, which I think is where Dario says he is. But in that case, you should be begging the world to stop everybody, including you.

There's a lot of people who say, what about the benefits of AI? And I'm not saying you can never try to get the benefits of AI. But if there's a revolver, and I say, I've studied this revolver for a long time. and I think there are six bullets in the chamber. And somebody else says, no, no, four of the bullets in the chamber shoot Utopia. Two of the bullets in the chamber shoot lead. That doesn't mean you should be spinning the chamber and putting the gun to your head.

That means we should be finding a way to get the other two leaded bullets out of the damn gun. We should not build ASI. That's my argument. Just don't do it. We're not ready for it. We don't know how to make it safe. Don't do it. And it shouldn't be done. it's even further than that it's not just i am not trying to convince people to not do it out of the goodness of their heart i think it should be illegal it should just straight up it should be logically illegal

for people and private corporations to attempt even to build systems that could kill everybody. The same way it's illegal to, you know, brew homemade explosives in your backyard because, you know, you might blow yourself or your neighbors up. This is illegal. I think the same thing should apply to ASI. This should be illegal. Okay, so what if you did get what you want and the US and even the EU, they outlawed the creation of superintelligence?

The argument I hear from the other side is, doesn't that just mean that bad actors would be building it instead? Or doesn't that just mean that you are essentially giving this AI race over to China and the CCP? How do you respond to that? Fundamentally, the people who are pushing AI are lawful abiding corporations that can be regulated.

Like if the U.S. government said knock it off tomorrow, it would stop. Nerds are cowards. These aren't hardened criminals. They will not risk their life for Facebook. If you said we are going to put people in jail if they try to build ASI, it would stop tomorrow. Now. Could we get China to do this? Well, this becomes a diplomacy problem, and it's a very hard problem. Well, first of all, I do think that China has the capacity.

much more so than the US, to shut down ASI if it wanted to. You know, Jack Ma stepped out of line even just a little bit and they made him disappear for like six months. If the Chinese Communist Party decided that ASI is not worth the risk, it would stop tomorrow and they could enforce it.

So could we get to such an international agreement? Now, this is a fair question. And this is very, very hard. But this is a disarmament problem. It's the same problem we faced, like we did it with nuclear weapons with the Soviets. Yes, this is hard, but then you just have to do diplomacy. Like, what the fuck are you talking about? So your response is, yeah, it's going to be really, really hard. And there's going to be questions that don't have easy answers, but...

The situation we're in demands that we just have to do it. Yeah, I think these are questions that would need answering. And I think it's very fair to say, hey, those are important questions. I don't think those questions should be dismissed. I just think they're the kind of questions we need to answer. after we have defused the ticking time bomb. The U.S. should also be developing the monitoring ability to figure out who is participating in this reckless race.

We should be developing the monitoring capacity to know where the AI chips are going. We should be developing the intelligence to know whether they're trying to make smarter and smarter AIs, and we should be developing the sabotage capability to stop them from making smarter and smarter AIs. For dangers of this magnitude, we should be developing the capacity

to understand where this is being done, and to sabotage it. What do you say to the people who are listening to this, some of them may be critics of you and your camp, who say that you are fundamentally recommending something that is out of line? With human nature. The idea being that human beings were competitive and were ambitious. And once we know that there's some technology that's.

possible. We're going to chase after it. We're going to try and beat each other to it, especially when that technology might be so beneficial. And that the thing that you're proposing, that we stop until some future date when we know that this...

technology will be totally safe is just out of whack with who we are at a very core level. I'm thinking about one person who I spoke to who said, what if we had tried to do this with the automobile with the car, that if we tried to make a car that was totally safe. And he said, no, that's not how it works. We make a car, and then it's only as it goes out, we start to realize, oh, turn signals would help this thing be safer. Or, oh, seatbelts would help this thing be safer.

Do you feel that there's an aspect of what you're proposing that is out of line with human nature? And so it will ultimately fail. I think the car analogy really does a disservice to the situation we're in. It's a little bit more like the whole world is building the first car, loading up every man, woman, and child, and then pointing it towards the edge of a cliff and saying, let's slam down in the accelerator.

In that situation, you really should get the world together and say, let's stop doing that. This is not a situation where I'm saying the AI needs seatbelts. This is not a situation of me saying, oh, the AIs are having some negative effects. We need to not try and get any of the positive effects until we make sure there's no negative effects. This is a situation where I'm saying, if we continue down this course, the most likely outcome...

is that literally everybody on Earth will die. That's just a wildly different situation than the cars. And then, you know, in terms of whether this is within our human nature, I think it's very defeatist to say humanity won't do this. And I also think it's very foolish to be a fetus like that at this stage. You know what I think is human nature? What I think is human nature is solving problems. What I don't think is human nature is suicide.

exploration, curiosity, surviving the next day so you can come home to your family. That's what human nature is to me. All right, so let's talk about extinction. Because it's one thing to say, look at these mysterious...

and potentially powerful new AI systems, can't you see how these things could be dangerous? And it's another thing to say that there's a 10%, 20%, a 90% chance that they may wipe out. the entire human race. And as we mentioned before, this is a view that even some of the people who are making AI right now as fast as they can, they think that there is some percentage chance that that could happen. So paint the picture for me.

What is it that you're envisioning when you say extinction? What would that look like? All of this is a little bit like someone in the year 1800. trying to guess what war would look like against someone in the year 2000 and imagining that they can make slightly more explosive bombs. You know, they might say, oh, well, we're only at the beginning of artillery technology. And so I bet...

In the future, they're going to have artillery that's at least 10 times as good. It would have been hard for them to imagine a Wi-Fi, internet-connected, flying drone. That's right. Never mind a nuclear weapon, right? And so it's easy to predict that very smart AIs will be able to find ways to get whatever it is they're pursuing. It's hard to predict exactly how, but it's easy to predict that they will be able to dramatically outcompete us.

The obvious thing that happens there is that the future goes under their control rather than ours. Just like how the future is now under the human control rather than the chimpanzee control. Chimpanzees live or die according to whether humanity can restrain themselves from cutting down their jungles. And then you have this other issue, which is for most of the weird stuff that AI could pursue, you know, happy, healthy, free people are not the most efficient way to get it.

You know, so if this AI wants all sorts of things that are like weird proxies of helpfulness, maybe it's like, well, I want a lot of things that are sort of human-based. You know, they're sort of a little bit lobotomized, like the sort of humans that are like really delighted with every interaction with me. Just like how humans breed chickens to be like more and more chicken breast.

Maybe the AIs are sort of like breeding humans or, you know, just straight out using the technology to change humans to be like more and more the type of thing the AI likes interacting with. So I want to take your question seriously. I don't want to just like dismiss it. I don't know. what will happen exactly. The one thing I'm confident in is that we will lose. Like, we will almost definitely lose. But the way, if I had to guess what it will feel like, I expect it will be confusing.

I don't think it's going to be epic. I don't think there's going to be a huge showdown between the Terminators and the humanity. I don't think there's going to be, you know, crazy nanotechnology flying through the sky or something like this. I don't think that's what's going to happen. You know, maybe, right? But like...

I don't think it's likely. I think what's most likely to happen is it will just be quite unceremonious and just like quite pathetic and like boring almost. I expect what was going to happen is that the world will just start feeling more and more confusing. It will be harder and harder to understand.

What's really going on? We see all this like fake news and like all these distractions, like video games keep getting better and pornography gets getting more addictive. And all the news we get starts being like super polarized and like super, you know, manipulative. We can't really tell what's true and what's not anymore. A lot of weird

geopolitical events start happening that no one can really explain why they happened or what the consequences of them are. New technologies get invented where we don't really know how they were invented. And just like more and more confusing things happen. More and more people put AIs.

in charge of more and more things. You know, more and more people will put AIs in charge of their companies. Politicians will take advice from AIs for how to run their campaigns and for how to write policy. More and more of power will be willingly, completely willingly handed to AIs, you know.

Everyone will race to hand as much of their power over to the AI as possible because it helps them to outperform their rivals. And then eventually, you know, just like one day we wake up and we're not in control anymore.

All right, so I wanted to ask you one final question, and this is a question about self-reflection. And I'm doing a lot of self-reflection myself, so I ask this with all due respect. I don't want to be the kind of journalist who alarms people, but I also don't want to be the kind of journalist... who shies away from diving into a subject because I'm like trying to read the room. This is all fascinating. I want to enter it with an open mind. But for you...

Do you ever worry about the fact that there have been doomers of one kind or another throughout human history, throughout the world, whether it's in ancient China or in Europe or tribes in South America? There have been people who are convinced that the end is near, and they have, similar to you, gone out trying to convince other people to realize that the end is near. Does that ever worry you?

That you're in some ways performing a familiar role in our society today. You know, there were surely some people in the Americas. when the Spanish came that said they expected this would be the end of their world. And they were right. You know, it was largely due to smallpox, which is maybe not quite what they were expecting, but some worlds have ended.

And if we broaden the reference class a little bit further, there were a lot of scientists in the 1920s who said leaded gasoline is poison. And if we run cars on leaded gasoline, it will poison a lot of children. caused a lot of brain damage to a lot of children. And those scientists were ignored. And leaded gasoline was rolled out across the country and the world. And quite a lot of children were brain damaged by leaded gasoline.

Tens of millions, if not hundreds of millions. Or maybe even more directly, there were a lot of people who said in the wake of the development of nuclear bombs, it looks like on the current path we're going to die in nuclear fire. they had a lot of good reason to expect that humanity could not hold back from a nuclear war. Because humanity had not been able to hold back from total war with the best of their technology ever in the course of human history.

And because fresh in their minds was the failure of the League of Nations to prevent World War II. But humanity did not die in a nuclear fire. It wasn't because nuclear weapons were fake. It wasn't because the bombs couldn't go off. It was true that the bombs could level cities. And we didn't die in a nuclear fire anyway because we realized the danger and we backed off. How do you tell the difference between a person prophesying the end times and

a person trying to raise the alarm about a danger? Well, part of how you tell the difference is looking at whether they're saying, there's nothing that can be done, repent. We brought this about by our own sin or whatever. But a large part of how you figure out the difference is by looking at the arguments. When the nuclear scientists on the Manhattan Project said, you know, there's a chance that this bomb could ignite the atmosphere.

and destroy the entire Earth. People didn't say, oh, people have predicted the end of the world all sorts of times before, so that can't happen. What they said was, Well, that seems worrying. Let's look into it. Yeah. Yeah. That seems like something we should look into. Right. So, like, the way you figure out whether the nuclear bomb is going to ignite the atmosphere is you run the calculation.

It's not that you sort of psychologize about the people and say, don't you worry that you're fulfilling a role of prophet of doom. You sort of look at the arguments. And what I sort of try to do is lay out the arguments. And frankly, I think the machines are talking now. People thought that was going to take a lot longer than it did. We're able to grow AIs that are smarter and smarter each year. They're still pretty dumb.

But people are able to grow AIs that are smarter, and they don't really know how they're working, and they're able to make them smarter still by throwing more computing power at it. And just sort of naively, you don't need to be a big expert.

to look at how we're growing AIs and how we're making them smarter and how we don't understand what's going on and say, hold on, if we make machines that are much smarter than us, that can think faster than us, that can copy themselves a lot, why do we think that's going to go well? And the answer...

is not, oh, everyone prophecies doom. The answer is, well, that seems like something we should look into. You should run those calculations. And currently the calculations look grim to me. So I think we need to back off. Next time on The Last Invention, the AI scouts. How do we find the win-win outcome here? And their case for why humanity can and must come together to get prepared for what's coming and do it soon.

Because it is a narrow path we need to navigate. But I do think this win-win future is, in principle, possible. The Last Invention is produced by Longview, home for the curious and open-minded. You can learn more about our team and help support our work by going to longviewinvestigations.com or by clicking on the link in our show notes. A special thanks this episode to Emil Torres and Jasmine Sun. Thanks for listening. We'll see you soon.

This episode is sponsored by Ground News, the app that helps you spot media bias and see a broader picture of the news shaping our world. Get 40% off their vantage plan at ground.news.invent. This episode is sponsored by FIRE, defending free thought in the age of AI. You can learn more at thefire.org slash thelastinvention.