#2311 - Jeremie & Edouard Harris

The Joe Rogan Experience. Alright, so... for AI and where we're far What time is it? If midnight is, we're fucked. We're getting right into it. You're not even going to ask us what we had for breakfast? No, no, no, no, no, no, no. Jesus. Let's get freaked out. Well, okay, so there's one, without speaking to the fucking Doomsday dimension right off the bat, there's a question about where are we at?

In terms of AI capabilities right now and what do those timelines look like, there's a bunch of disagreement. One of the most concrete pieces of evidence that we have recently came out of a lab, an AI kind of evaluation lab called Meteor. And they put together this test. Basically, it's like you ask the question, pick a task that takes a certain amount of time, like an hour. It takes like a human a certain amount. and then see how likely the best AI system is to solve for that.

then try a longer task. See, like, a 10-hour task. Can it do that one? And so right now what they're finding is when it comes to AI research itself, so basically, like, automate the work of an AI researcher, You're hitting 50% success rates for these AI systems for tasks that take an hour long. And that is doubling every, right now it's like every four months. So you had tasks that you could do, you know, a person does in five minutes, like, you know, ordering an Uber Eats.

or like something that takes like 15 minutes, like maybe a book and a flight or something like that. And it's a question of like, how much can these AI agents do, right? Like from five minutes to 15 minutes to 30 minutes. And in some of these spaces like... software engineering and it's getting further and further and further and doubling it looks like every four months so like if you if you extrapolate that you basically get to tasks that take a month to complete like by 2027

Tasks that take an AI researcher a month to complete these systems will be completing with like a 50%. So you'll be able to have an AI on your show and ask it what the doomsday clock is. by then. It probably won't laugh. It'll have a terrible sense of humor about it. Just make sure you ask it what it had for breakfast before he started. Yeah. What about quantum computing, getting involved in AI? So, yeah, honestly, I don't think it's...

If you think that you're going to hit human-level AI capabilities across the board, say 2027, 2028, which when you talk to some of the people in the labs themselves, that's the timelines they're looking at. They're not confident, they're not sure, but that seems pretty plausible. If that happens, really there's no way we're going to have quantum computing that's going to be giving enough of a bump to these techniques. You're going to have...

standard classical computing. One way to think about this is that the data centers that are being built today are being thought of literally as the data centers that are going to house like the artificial brain that powers super intelligence, human level AI when it's built in like 2027 and something like that. So how knowledgeable are you when it comes to

So, a little bit. I mean, like, I did my grad studies in, like, the foundations of quantum mechanics. Oh, great. Yeah, well, it was a mistake, but I appreciate it for the person. Why was it a mistake? You know, so academia is a kind of funny thing. It's really bad culture. It teaches you some really terrible habits. So basically my entire life after academia, and Ed's too, was unlearning these terrible habits of...

it's all zero sum basically. It's not like when you're working in startups. It's not like when you're working in tech where you build something and somebody else builds something that's complimentary and you can team up and just make something amazing. It's always

wars over who gets credit who gets their name on the paper did you cite this fucking stupid paper from two years ago because the author has an ego and you gotta be I was literally at one point um I'm not going to get any details here but like there was a collaboration that we ran with like this Anyway, fairly well-known guy. And my supervisor had me write the emails that he would send from his account so that he was seen as the guy who was interacting with this bigwig.

That kind of thing is like... doesn't tend to happen in startups at least not in the same way so he wanted credit for the like he wanted to seem like he was the genius who was facilitating this for sounding smart on email right but that happens everywhere and the reason it happens is that these guys who are like professors or even not even professors just like your postdoctoral guy who's like supervising you

They can write your letters of reference and control your career after that lab. They got you by the balls. They can do whatever. And so what you're doing. It's like a movie. totally it's gross like a gross movie like a gross boss in a movie that wants to take credit for your work and it's real it's rampant and the way to escape it is to basically just be like Fuck this. I'm going to go do my own thing. So Jer dropped out of grad school to come start a company.

And I mean, honestly, even that, it took me, it took both of us a few years to unfuck our brains and unlearn the bad habits we learned. It was really only a few years later that we started really, really getting a good...

getting a good flow going. You're also, you're kind of disconnected from like base reality when you're in the ivory tower, right? Like if you're, there's something beautiful about, and this is why we spent all our time in startups, but there's something really beautiful about like, It's just a bunch of assholes.

and like no money and nothing and a world of like potential customers and it's like you actually it's not that different from like stand-up comedy in a way like your product is can I get the laugh right like something like that and it's unforgiving If you fuck up, it's like silence in the room. It's the same thing with startups. The space of products that actually works is so narrow.

And you gotta... over what people actually want and it's so easy to fool yourself into thinking that you've got something that's really good because your friends and family are like oh no sweetie you're doing a great job like what a wonderful life I would totally use it I totally see that all that stuff right and that's I love that because it forces you to change. Yeah.

The whole indoctrination thing in academia is so bizarre because there's these hierarchies of powerful people and just the idea that you have to... work for someone someday and they have to take credit by being the person on the email, that will haunt me for days. I'll be thinking about that for days now.

I fucking can't stand people like that. It drives me nuts. One big consequence is it's really hard to tell who the people are who are creating value in that space, too, right? Of course, sure, because it's just like television. One of the things about television shows is, so I'll give you an example. A very good friend of mine who's a very famous comedian had this show. And his agent said, we're going to attach these producers. It'll help get it made.

And he goes, well, what are they going to do? He goes, they're not going to do anything. It's just be in name. He goes, but they're going to get credit. He goes, yeah. He goes, fuck that. He goes, no, no, listen, listen. This is better for the show. It'll help the show. But then they'll have a piece of the show. He's like, yes, yes. it's a matter of whether a show gets successful or not and this is a good thing to do and he's like what are you talking about but

it was a conflict of interest because this guy, the agent, was representing these other people. But this is completely common. So there's these executive producers that are on shows that have zero to do with it. So many industries are like that.

And that's why we got into startups. It's literally like... you and the world right yeah it's like in a way like stand-up comedy like jer said or like podcasting or like podcasting where your enemy isn't actually hate it's indifference like most of the stuff you do Especially when you're getting started, like, why would anyone, like, give a shit about you? They're just not going to pay attention. Yeah, that's not even your enemy. You know, that's just all potential.

that's all that is you know it's like your enemies within you it's like figure out a way to make whatever you're doing good enough that you don't have to think about it not being valuable it's it's meditative like there's no way for it not to be to be in some way a reflection of yourself. You're kind of in this battle with You trying to convince yourself that you're great, so the ego wants to grow, and then you're constantly trying to compress it and compress it.

And if there's not that outside force, your ego will expand to fill whatever volume is given to it. Like if you have money, if you have fame, if everything's given and you don't make contact with the unforgiving on a regular basis, like. Yeah, you know, you're going to end up doing that to yourself. You could, yeah. It's possible to avoid, but you have to have strategies. Yeah, you have to be intentional about it. The best strategy is jujitsu.

Mark Zuckerberg is a different person now. Yeah, you can see it. You can see it. Yeah, well, it's a really good thing for people that have too much power because you just get strangled all the time. And then you just get your arms bent sideways. And after a while, you're like, okay. This is reality. This is reality. This social hierarchy thing that I've created is just nonsense. It's just smoke and mirrors. And they know it is, which is why they so rabidly enforce these hierarchies.

The best people seek it out. Sir and ma'am and all that kind of shit. That's what that is. You don't feel like you really have respect unless you say it. These poor kids that have to go from college where they're talking to these dipshit professors out into the world and operating under these same rules that they've been forced and indoctrinated to.

God, to just make it on your own. It's amazing what you can get used to, though. And, like, the... It's funny, you were mentioning the producer thing. That is literally also a thing that happens in academia. So you'll have these conversations where it's like, all right, well, this paper is... something but we want to get it in a paper in a journal and so let's see if we can get like a famous

on the list of authors, so that when it gets reviewed, people go like, oh, Mr. So-and-so, okay, and that literally happens. The funny thing is, the hissy fits over this, the stakes are so brutally low at least with your producer example like someone stands to make a lot of money with this it's like You get maybe like an assistant professorship out of it at best that's like 40 grand a year. And it's just like, this is, it's just.

For the producers, it is money, but I don't even think they notice the money anymore. Because all those guys are really, really rich already. If you're a big-time TV producer, you're really rich. being thought of as a genius who's always connected to successful projects. Right, yeah. That's what they really like.

That is always going to be a thing. It wasn't one producer. It was like a couple. So there's going to be a couple different people that were on this thing that had zero to do with it. It was all written by a stand-up comedian. His friends all helped him. They all put it together. And then he was like, no, he wound up firing his agent. Oh, shit. Good for him. I mean, yeah. Get the fuck out of here.

At a certain point for the producers, too, it's kind of like you'll have people approaching you for help on projects that look nothing like projects you've actually done. So I feel like it just adds noise to your universe, like if you're actually trying to build cool shit.

You know what I mean? Some people just want to be busy. They just want more things happening, and they think more is better. More is not better. Because more is energy that takes away from the better, whatever the important shit is. Yeah, the focus. You only have so much time until AI takes over. Then you'll have all the time in the world because no one will be employed and everything will be automated.

We'll all be on universal basic income. And that's it. That's a show. The end. That's a sitcom. That's a sitcom. A bunch of poor people existing on $250 a week. Oh, I would watch that. Yeah, because the government just gives everybody that. That's what you live off. like weird shit is cheap like the stuff that's like all like well the stuff you can get from chatbots and AI agents is cheap but like food is super expensive or something yeah organic

You're going to have to kill people for it. You will eat people. It will be like a Soylent World. Right. Hey, they're Soylent Green. Now there's more free range than people, though. That's true. Depends on what they're eating, though. It's just like animals. You know? You don't eat a bear that's been eating salmon. They taste like shit. I didn't know that. I've been eating my bear wrong this entire time.

So back to the quantum thing. So quantum computing is infinitely more powerful than standard computing. Would it make sense then that if quantum computing can run a large language model that it would reach a level of intelligence? That's just preposterous. So, yeah, one way to think of it is, like, there are problems that quantum computers can solve way, way, way, way better than classical computers. And so, like, the numbers get absurd pretty quickly.

problems that a classical computer couldn't solve if it had the entire lifetime of the universe to solve it. A quantum computer right in like 30 seconds, boom. But the flip side, there are problems that quantum computers just like... help us accelerate. One classic problem that quantum computers help with is this thing called the traveling salesman paradox. or problem where you have a bunch of different locations that a salesman needs to hit and what's the best path to hit the most efficiently.

It's like kind of a classic problem if you're going around different places and have to make stops. There are a lot of different problems that have the right shape for that. A lot of quantum machine learning, which is a field, is focused on how do we take standard AI problems like AI. workloads that we want to run and like massage them into a shape that gives us a quantum And that's it's a nascent field. There's a lot going on there. I would expect like my personal expectation is that we just.

build the human-level AI, and very quickly after that, superintelligence, without ever having to factor in quantum. Can you define that for people? What's the difference between human-level AI and superintelligence? Yeah. So, yeah, human level AI is like AI, you can imagine like it's AI that is as smart as you are in, let's say, all the things you can do on a computer.

So, you know, you can, yeah, you can order food on a computer, but you can also write software on a computer. You can also email people and pay them to do shit on a computer. You can also trade stocks on a computer. So it's like as smart as a smart person. Super intelligence, people have various definitions, and there are all kinds of honestly hissy fits about different definitions.

Generally speaking, it's something that's like very significantly smarter than the smartest human. And so you think about it, it's kind of like it's as smart, as much smarter than you as you might be smarter than a toddler. And you think about that and you think about like the, you know, how do you, how would a toddler control you? It's kind of hard. You can outthink a toddler pretty much any day of the week. And so superintelligence gets us at these levels where...

you can potentially do things that are completely different. And basically, you know, new scientific theories. And last time we talked about, you know, new stable forms of matter that were being discovered by these kind of narrow systems. But now you're talking about a system that is like has that intuition combined with the ability to talk to you as a human and to just have a really good life.

rapport with you. It can also do math. It can also write code. It can also solve quantum mechanics and has that all kind of wrapped up. One of the things, too, that by definition, if you build a human-level AI, one of the things it must be able to do as well as humans is AI research itself. Yeah. Or at least the parts of AI research that you can do in just like software, like, you know, by coding or whatever these systems are designed.

And so one implication of that is you now have automated AI research. And if you have automated AI researchers, that means you have AI systems that can automate the development of a network. level of their own capabilities. And now you're getting into that whole singularity thing where it's an exponential that just builds on itself and builds on itself, which is kind of why a lot of people argue that if you build human-level AI, superintelligence can't be that far away. You basically unlock...

everything. And we kind of have gotten very close, right? Like it's It's past the Fermi, not the Fermi paradox, the, what is it? Oh, yeah, yeah, the... Goddammit, we were just talking about them the other day. Yeah, the test, the... Oh, the Turing test? The Turing test, thank you. We were just talking about a horrible, what happened... They chemically castrated him because he was gay. Horrific. He winds up killing himself. The guy who figures out...

what's the test to figure out whether or not AI has become sentient? And by the way, he does this in, like, what, 1950s? Oh, yeah, yeah. Alan Turing is a, like, the guy was a beast, right? How did he think that through? He invented computers. He invented basically the concept. that underlies all computers. He was like... an absolute beast. He was a code brainer.

He broke the Nazi codes, right? He also wasn't even the first person to come up with this idea of machines, building machines, and there being implications like human disempowerment. So if you go back to, I think it was like the late 1800s, and I don't remember the guy's name. He sort of like came up with this. He was observing the industrial revolution and the mechanization of labor and kind of starting to see.

More and more, if you zoom out, it's almost like humans are an ant colony, and the artifacts that that colony is producing that are really interesting are these machines. You look at the surface of the Earth as this gradually, increasingly mechanized thing, and it's not super clear. If you zoom out enough like

what is actually running the show here? You've got humans servicing machines, humans looking to improve the capability of these machines at this frantic pace. They're not even in control of what they're doing. Economic forces are pushing it. Are we the servant of the master at a certain point? Yeah, and the whole thing is like, especially with a competition that's going on between the labs, but just kind of in general, you're at a point where like,

Do the CEOs of the labs, like they're these big figureheads, they go on interviews, they talk about what they're doing and stuff. Do they really have control over any part of this? the economy is in this like almost convulsive fit right like you can almost feel like it's like it's hurling out AGI and like for as one kind of I guess data point here, like all these labs, OpenAI, Microsoft, Google,

Every year they're spending like an aircraft carrier worth of capital, individually, each of them, just to build bigger data centers, to house more AI chips, to train bigger, more powerful models. And that's like, so we're actually getting to the point where if you look at on a power consumption basis, like... We're getting to 2, 3, 4, 5% of U.S. power production. if you project out into the late 2020s.

20, 26, 27. Not for double-digit, though. Not for double-digit, but for single-digit. Yeah, you're talking like that's a few gigawatts, so one gigawatt. Not for single-digit, it's in the... For 2027, you're looking at like, you know, in the 0.5 ish percent, but it's like, it's a big fucking fry. Like you're talking about gigawatts and gigawatts. One gigawatt is a million homes. So you're seeing like one data center in 2027 is easily going to break a gig. There's going to be multiple like that.

And so it's like a thousand, sorry, a million home city metropolis, really, that is just dedicated to training like one fucking model. That's what this is. Again, if you zoom out at planet earth, you can interpret it as like this, like all these humans frantically running around like ants just like building this like artificial brain. It's like a super mind assembly. on the face of

Hey, UK! With how much your privacy is being invaded online, VPNs are no longer just a nice thing to have. It's a necessity. Some people think, I don't need a VPN because I have nothing to hide. But that's exactly what data brokers want you to think because their profits depend on you having nothing to hide.

What you do online is your business. And if you want to keep it your business, you need ExpressVPN. With ExpressVPN, 100% of your online activity travels through secure, encrypted servers. It hides your IP address. so no one can use it to track and sell your online activity and your privacy is restored. ExpressVPN is also a great tool to secure your sensitive information from hackers.

If you're using public Wi-Fi like at a hotel, airport, or coffee shop, anyone can hijack your connection and access your most sensitive information. With ExpressVPN, you know that your passwords and banking info are safe and secure. ExpressVPN is the number one ranked VPN by the experts at CNET and The Verge.

And right now, you can get an amazing deal. They're offering four extra months free if you go to expressvpn.com slash rogan or tap the banner. And if you're watching on YouTube, you can get your four free months by scanning the QR code on screen or clicking the link in the description. This episode is brought to you by Squarespace. When it came time to make a website there was no question that we would power it with Squarespace.

From the intuitive design intelligence that helps to create a bespoke digital identity to the seamless payment options. that can help give your customers more ways to pay, or the fact that you can measure your end-to-end online performance with powerful website and seller analytics. The reasons to power your website with Squarespace are endless.

So, if you're looking to build or even upgrade your current website, check out squarespace.com for a free trial or go to squarespace.com slash rogan to save 10% off your first website or domain purchase. Marshall McLuhan in like 1963 or something like that said. Human beings are the sex organs of the machine world. Oh, God, that hits.

That hits different today. It does. I've always said that if we were aliens or if aliens came here and studied us, they'd be like, what is the dominant species on the planet doing? Well, it's making better things. That's all it does.

The whole thing is dedicated to making better things. And all of its instincts, including materialism, including status, keeping up with the Joneses, all that stuff is tied to newer, better stuff. You don't want old shit. You want new stuff. You don't want an iPhone 12. What are you doing, you loser?

You need newer, better stuff. And they convince people, especially in the realm of consumer electronics, most people are buying things they absolutely don't need. The vast majority of the spending on new phones is completely unnecessary. But I just need that. That extra fourth camera, though. I feel like my life isn't complete. I run one of my phones as an iPhone 11, and I'm purposely not switching it just to see if I notice it. I fucking never notice any.

I watch YouTube on it. I text people. It's all the same. I go online and work. It's all the same. Probably the biggest thing there is going to be the security side, which... No, they update the security. It's all software.

But, I mean, if your phone gets old enough, I mean, like, at a certain point... Oh, when they stop updating it? Yeah, like iPhone 1, you know, China's... watching all your dick pics oh dude i mean salt typhoon they're watching all our dick pics yeah they're definitely seeing mine what's salt typhoon um so salt oh sorry yeah yeah so it's this big um chinese cyber attack actually starts to get us to uh to kind of the

That's what a great name, by the way. Salt Typhoon. Fuck yeah, guys. I really wish I would name it. They have the coolest names for their cyber operations. That's a great name. Salt Typhoon is pretty slick. You know what? It's kind of like when people go out and do like a

an awful thing, like a school shooting or something, and they're like, oh, let's talk about, you know, if you give it a cool name, like now the Chinese are definitely going to do it again. Anyway. Because they have a cool name. Yeah, that's definitely a factor. Salt Typhoon. Salt Typhoon. Pretty dope.

But it's this thing where basically, so there was in the 3G kind of protocol that was set up years Law enforcement agencies included backdoors intentionally to be able to access comms, you know, theoretically if they got a warrant and so on.

And, well, you introduce a backdoor, you have adversaries like China who are wicked good at cyber. They're going to find and exploit those backdoors. And now basically they're sitting there and they had been for some people think like maybe a year or two before it was really discovered.

and just a couple months ago they kind of go like oh cool like we got fucking like China all up in our shit and this is like this is like flip a switch for them and like you turn off the power or water to or like you fucking yeah well sorry this is sorry Salt Typhoon though is about just sitting on the basically telecoms now. Oh, that's the telecom one. That's right. But yeah, I mean, that's another thing. There's another thing where they're doing that too. Yeah, and so this is kind of where...

What we've been looking into over the last year is this question of, how what is if you're going to make like a manhattan project for super intelligence right which is That's what we were texting about way back. Actually, funnily enough, we shifted our date for security reasons, but if you're going to do a Manhattan project for superintelligence,

What does that have to look like? What does the security game have to look like to actually make it so that China's not all up in your shit? Like today... It is extremely clear that at the world's top AI lab, all that shit is being stolen. There is not a single lab right now that isn't being spied on successfully based on everything we've seen by the Chinese. Can I ask you this? Are we spying on the Chinese as well? That's a big problem. Do you want to...

We're definitely doing some stuff, but in terms of the relative balance between the two, we're not where we need to be. They spy on us better than we spy on them. Yeah, because they build all our shit. They spilled all our shit. Well, that was the Huawei situation, right? Yeah, and it's also the, oh my god, if you look at the power grid, so this is now public, but

If you look at transformer substations, so these are the, essentially, anyway, they're a crucial part of the electrical grid. And there's really like... Basically, all of them have components that are made in China. China's known to have planted backdoors like Trojans into those substations to fuck with our grid. The thing is, when you see a salt typhoon, when you see a big Chinese cyber attack or a big Russian cyber attack, you're not seeing

These countries do not go and show you their best cards out the gate. You show the bare minimum that you can without... tipping your hand at the actual exquisite capabilities you have. The way that one of the people who's been walking us through all this really well has explained it is like,

The philosophy is you want to learn without teaching, right? You want to use what is the lowest level capability that has the effect I'm after. And that's what that is. I'll give an example. Like I'll tell you a story that's kind of like, It's a public story, and it's from a long time ago, but it kind of gives a flavor of how far these countries will actually go when they're playing the game for five years.

So it's 1945. America and the Soviet Union are like best pals because they've just defeated the Nazis, right? To celebrate that victory and the coming new world order that's going to be great for everybody, the children of the Soviet Union... Give as a gift to the American ambassador in Moscow, this beautifully carved wooden seal of the United States of America.

Beautiful thing ambassadors thrilled with it. He hangs it up on behind his desk in his private office you can see where i'm going with this probably but yeah yeah seven years later 1952 finally occurs to us like let's take a town and actually examine this so they dig into it and they find this incredible contraption in it called a cavity resonant And this device doesn't have a power source, doesn't have a battery, which means when you're sweeping the office for bugs, you're not going to find it.

What it does instead is it's designed, that's it, that's it. The thing, they call it the thing. And what this cavity resonator does is it's basically designed to reflect. Radio radiation. back to a receiver to listen to all the noises and conversations and talking in the ambassador's private office. How's it doing you without a power source? So that's what they do.

The Soviets, for seven years, parked a van across the street from the embassy, had a giant fucking microwave antenna aimed right at the ambassador's office. And we're like zapping it and looking back at the reflection and literally listening to every single thing he was saying. And the best part was,

when the embassy staff was like, we're going to go and sweep the office for bugs periodically, they'd be like, hey, Mr. Ambassador, we're about to sweep your office for bugs. And the ambassador was like, cool, please proceed and go and sweep my office for bucks. And the KGB dudes in the van were like, Just turn it off. Sounds like they're going to sweep the office for bugs. Let's turn off our giant microwave antenna.

And they kept at it for seven years. It was only ever discovered because there was this, like, British radio operator who was just, you know, doing his thing, changing his dial. And he's like, oh, shit. Like, is that the ambassador? Just talking randomly. So the thing is, oh, and actually, sorry. One other thing about that. If you heard that story and you're kind of thinking to yourself, hang on a second.

They were shooting microwaves at our ambassador 24-7 for seven years. Doesn't that seem like it might fry his genitals or something? Yeah, or something like that. You're supposed to have a lead vest. And the answer is... jock. Yeah. Yes. Yes. And this is something that came up in our investigation just from every single person who was like, who was filling us in and who dialed in and knows what's up. They're like, look, so you got to understand like our adversaries.

If they need to give you cancer in order to rip your shit off of your laptop, they're going to give you some cancer. Did he get cancer? I don't know specifically about the ambassador, but like... That's also so...

We're living what we can say. There's actually people that you can talk to later that... can go in in more detail here but uh older technology like that like kind of lower powered so you're you're less likely to to look at that nowadays we live in a different world the guy that invented that mic His last name is Theraminion.

instrument called the theremin which is a fucking really interesting thing oh he's just moving his hands yeah your hands control it waving over this it's a fucking wild instrument have you seen this before jamie yeah i saw juicy j prep playing it yesterday on instagram he's like practicing It's a fucking cool ass thing. He's also pretty good at it, too. Yeah, yeah, yeah. Both hands are controlling it.

by moving in and out in space, X, Y, Z. I honestly don't really know how the fuck it works, but I've seen it. That is wild. It's also a lot harder to do than it seems. So the Americans tried to replicate this for years and years and years without really... succeeding and um anyway uh that's all kind of part i have a friend who used to work for intelligence agency and he was working in russia and the fact they found that the building was bugged with these

Super sophisticated bugs that operate. Their power came from the swaying of the building. Get out. I've never heard that one before. The swing of, just like your watch, like I have a mechanical watch on, so when I move my watch, it powers up the spring and it keeps the watch, that's an automatic mechanical watch.

They figured out a way to, just by the subtle swaying of the building in the wind, that was what was powering this listening device. So this is the thing, right? I mean, what the fuck? The things that nation states... What's up, Jimmy? Well, it says that's what was powering this thing, the great seal bug, which I think is the thing. There's another one? No. Oh, this is, so you can actually see in that video, I think there was a YouTube, yeah. Same kind of thing, Jimmy? I was just, I typed in.

Russia spy bug building sway. The thing is what pops up. The thing. Which is what we're... Oh, that thing. So that's powered the same way by the sway of the building? I think it was powered by radio frequency emission. So there may be another thing related to it. I'm not sure about that. Maybe Google's a little confused. Maybe the word sway is what's throwing it off.

But it's a great catch. And the only reason we even know that, too, is that when the U-2s were flying over Russia, they had a U-2 that got shot down in 1960. The Russians go like, oh, friggin' Americans spying on us. What the fuck? I thought we were buddies. obviously didn't think that but um and then the americans are like uh okay bitch look at this and they brought out the the seal um and that's how it became public it was basically like the response to the russians saying

Wow. Yeah, they're all dirty. Everyone's spying on everybody. That's a thing. And I think they probably all have some sort of UFO technology. We need to talk about that. Can we turn off our mics? I'm 99% sure a lot of that shit is ours. You need to talk to some of the... I've been talking to people. I've been talking to a lot of people.

There might be some other people that you'd be interested in chatting with. I would very much be interested. Here's the problem. Some of the people I'm talking to, I'm positive, they're talking to me to give me bullshit. Are we on your list? No, you guys aren't on the list. But there's certain people, I'm like, okay, maybe most of this is true, but some of it's not on purpose. There's that. I guarantee you I know I talk to people that don't tell me the truth.

Yeah, yeah. It's an interesting problem in like all Intel, right? Because there's always, the mix of incentives is so fucked. Like the adversary is trying to add noise into the system. You've got pockets of people within the government that have different incentives from other pockets. And then you have top secret clearance and all sorts of other things that are going on. One guy that texted me is like, the guy telling you that they aren't real is literally involved in these meetings, so stop.

Just stop laughing. It's like one of the techniques, right, is actually to inject so much noise that you don't know what's what and you can't follow. So this actually... This happened in the COVID thing, right? The lab leak versus the natural wet market thing. Yeah. So I remember there was a debate that happened about what was the origin of COVID. This was like a few years

It was like an 18 or 20 hour long YouTube debate, just like punishingly long. And it was like there was a hundred thousand dollar bet either way on who would win. And it was like lab leak versus wet market. And at the end of the 18 hours,

The conclusion was like, one of them won, but the conclusion was like, it's basically 50-50 between them. And then I remember hearing that and talking to some folks and being like, hang on a second. So you've got to believe that whether it came from a lab or whether it came from a wet market, one of the top three priorities of the CCP from a propaganda standpoint is like, don't get fucking blamed for COVID. And that means they're putting like,

$1 to $10 billion and some of their best people on a global propaganda effort to cover up evidence and confuse and blah blah blah. You really think that That your 50%, that confusion isn't coming from that incredibly resourced effort. They know what they're doing. particularly when different biologists and virologists who weren't attached to anything were talking about, like,

The cleavage points and different aspects of the virus that appeared to be genetically manipulated. The fact that there was only one spillover event, not multiple ones. None of it made any sense. All of it seemed like some sort of...

genetically engineered virus. It seemed like gain-of-function research. And the early emails were talking about that. And then everybody changed their opinion. And even the taboo, right, against talking about it through that lens. Oh yeah, total propaganda. It's racist. which is crazy because nobody thought the Spanish flu was racist and it didn't even really come from Spain it came from Kentucky

I didn't know that. Yeah, I think it was Kentucky or Virginia. Where did the Spanish flu originate from? But nobody got mad. Well, that's because the state of Kentucky has an incredibly sophisticated propaganda machine. And pinned it on Spanish. It might not have been Kentucky, but I think it was an agriculture. thing. Kansas. Kansas.

Yeah, goddamn Kansas. I've always said that. I've always said that. Likely originated in the United States. H1N1 strain had genes of avian origin. By the way, people always talk about the Spanish flu. If it was around today, everybody would just get antibiotics. So this whole mass die-off of people. It would be like the Latinx flu.

And we would be... The Latinx flu. The Latinx flu. That one didn't stick at all. That didn't stick. Latinx. There's a lot of people claiming they never used it, and they pull up old videos of them. That's a dumb one. It's literally a gendered language, you fucking idiots. You can't just do that. It went on for a while, though. Sure, everything goes on for a while. Think about how long they did lobotomy.

They did lobotomies for 50 fucking years before they went, hey, maybe we should stop doing this. It was like the same attitude that got Turing chemically castrated, right? Actually, yeah. Yeah, let's just get in there and fuck around a bit. Well, this was before they had SSRIs and all sorts of other interventions. What was the year of lobotomies? I believe it stopped in 67. Was it 50 years? Seven years?

Oh, I think it was 67. I like how this has come up so many times that Jamie's like, I think last time you said it wasn't 7. It comes up all the time because it's one of those things. It's like you can't just trust the medical establishment. Officially 67, it says maybe. Oh, God. Oh, he died in 72. When did they start doing it? I think they started in the 30s or the 20s.

That's pretty ballsy, you know, the first guy who did a lobotomy. It says 24, Freeman arrives, watch DC, direct labs, 35, they tried it first. They just scramble your fucking brains. But doesn't it make you feel better to call it a leucotomy, though? Because it sounds a lot more professional. No. Lobotomy, leucotomy. Leucotomy sounds gross. Sounds like loogie. Like you're looking at loogie. Like lobotomy. Boy, Topeka, Kansas. Also Kansas.

All roads point to Kansas. This is a problem. Everything's flat. You just lose your fucking marbles. You go crazy. That's the main issue. Jesus Christ. so they did this for so long somebody won a nobel prize for lobotomy wonderful imagine imagine give that back yes seriously you're kind of like you know you you don't want to display it up on your shelf it's just like it's just a good It should let you know that oftentimes science is incorrect. And that oftentimes, you know,

unfortunately people have a history of doing things and then they have to justify that they've done these things. Yeah. And they, you know. But now there's also, there's so much more tooling too, right? If you're a nation state and you want to fuck with people and inject narratives into the ecosystem, right? Like the whole idea of autonomous AI agents too, like having these basically like Twitter bots or whatever bots.

Like a lot of, one thing we've been, we've been thinking about too on the side is like the idea of, you know, audience capture, right? Do you have like, like. big people with high profiles and kind of gradually steering them towards a position by creating bots that, like, through comments, through upvotes, you know, it's like... 100%. It's absolutely real. Yeah, and a couple of the big accounts on X that we're in touch with have sort of said like, yeah,

especially in the last two years, it's actually become hard, like especially the thoughtful ones, right? It's become hard to like stay sane, not on X, but like across social media, on all the platforms. And that is around when it became possible to have AIs that can speak like people 90%, 95% of the time.

And so you have to imagine that, yeah, adversaries are using this and doing this and pushing the frontier. They'd be fooled if they didn't do it. Oh, yeah, 100%. You have to do it because for sure we're doing that. And this is one of the things where, you know, like it used to be, so OpenAI actually used to do this assessment of their AI models as part of their...

what they call the preparedness framework that would look at the persuasion capabilities of their models as one kind of threat vector. They pulled that out recently, which is kind of like... Why? You can argue that it makes sense. I actually think it's...

Somewhat concerning because one of the things you might worry about is if these systems, sometimes they get trained through what's called reinforcement learning, potentially you could imagine training these to be super persuasive by having them interact with real people and convince them, practice at convincing them to do specific things. If you get to that point, these labs ultimately will have the ability to deploy agents at scale that can just persuade a lot.

to do whatever they want, including pushing legislative agendas. And help them prep for meetings with... the Hill, the administration, whatever. How should I convince this person to do that? Well, they'll do that with text messages. Make it more business. Yep. Make it friendlier. Make it more jovial. But this is like the same optimization pressure that keeps you on TikTok, that same addiction. Imagine that applied to persuading you of some fact, right? Yeah. That's like a...

On the other hand, maybe a few months from now, we're all just going to be very, very convinced that it was all fine. There's no big deal. Yeah, maybe they'll get so good that it'll make sense to you. Maybe they'll just be right. That's how it works. Yeah, it's a confusing time period. We've talked about this ad nauseum, but it bears repeating. former FBI...

An analyst who investigated Twitter before Elon bought it said that he thinks it's about 80% bots. Yeah. 80%. That's one of the reasons why the bot purge, like when Elon acquired it and started working on it, is so important. Like there needs to be... The challenge is detecting these things is so hard, right? Increasingly. More and more. They can hide basically perfectly. How do you tell the difference between a cutting-edge AI bar?

and a human just from the... You can because they can generate AI images of a family, of a backyard barbecue, post all these things up and make it seem like it's real. Especially now, AI images are insanely... It's crazy. And if you have a person, you could take a photo of a person and manipulate it in any way you'd like. And then now this is your new guy. You could do it instantaneously. And then this guy has a bunch of opinions on things.

Seems to always align with the Democratic Party, but whatever. He's a good guy. He's a family man. Look, he's out in his barbecue. He's not even a fucking human being. And people are arguing with this bot, like, back and forth. And you'll see it on any social issue. You see it with Gaza and Palestine. You see it with abortion. You see it with religious freedoms. You just see these bots. You see these arguments. And you see various levels.

You see the extreme position, and then you see a more reasonable centrist position. But essentially what they're doing is they're consistently moving what's okay further and further in a certain direction. And in fact, it's... It's both directions. You know how when you're trying to capsize a boat or something, you're fucking with your buddy on the lake or something. So you push on one side, then you push on the other side, then you push until eventually it capsizes.

This is kind of like our electoral process is already naturally like this, right? We go like... We have a party in power for a while, and then, like, they get, you know, they basically get, like, you get tired of them, and you switch. And that's kind of the natural way how democracy works in a republic.

But the way that adversaries think about this is they're like, perfect. This swing back and forth, all we have to do is like, when it's on this way, we push and push and push and push until it goes more extreme. And then there's a reaction.

right and that's swinging back and we push and push and push on the other side until eventually something breaks and that's a risk Yeah, it's also like, you know, the organizations that are doing this, like we already know this is part of Russia's MO, China's MO, because back when it was easier. We already could see them doing this shit. So there is this website called This Person Does Not Exist. It still exists surely now.

superseded yeah but you would like every time you refresh this this website you would see a different like human face that was ai generated and um what the russian internet research agency would do yeah exactly uh what what all these these uh I don't think they've really upgraded it.

That's fake? Wow, they're so good. This is like years old. And you could actually detect these things pretty reliably. You might remember the whole thing about AI systems were having a hard time generating hands that only like five fingers. That's all for though. Yeah, little hints of it were though back in the day in this person does not exist and you'd have the Russians would take like a face from that and then use it as the profile picture for like a Twitter box.

And so that you could actually detect. You'd be like, okay, I've got you there. I've got you there. I can kind of get a rough count. Now we can't, but we definitely know they've been in the game for a long time. There's no way they're not right now. And the thing with nation-state propaganda attempts is that Like, people have this idea that, like, ah, like, I've caught this, like, Chinese influence operation or whatever. Like, we nail them. The reality is nation states operate at, like,

30 different levels. And if you're a priority, like just influencing our information spaces as a priority, they're not just going to operate they're not just going to pick a level and do it they're going to do all 30 of them and so you even if you're like among the best in the world like detecting this shit you're going to like you're going to catch and stop like

levels 1 through 10 and then you're gonna be like you're gonna be aware of like level 11 12 13 like you're working against it and you're you know maybe you're starting to think about level 16 and you you imagine like you know about level 18 or whatever But they're above you, below you, all around you. They're incredibly, incredibly resourced. And this is something that came like...

came through very strongly for us. You guys have seen the Yuri Bezmenov video from 1984 where he's talking about how the... All our educational institutions have been captured by Soviet propaganda. It was talking about Marxism, how it's been injected into school systems and how you have essentially two decades. before you're completely captured by these ideologies and it's going to permeate and destroy. all of your confidence in democracy.

100% correct. And this is before these kind of tools. Because the vast majority of those exchanges of information right now are taking place on social media. the vast majority of debating about things, arguing, all taking place on social media. And if that FBI analyst is correct, 80% of it's bullshit, which is really wild. And you look at some of the documents that have come out, I think it was like I think it was the CIA game plan for regime change or undermining...

How do you do it right? Have multiple decision makers at every level. All these things. And what a surprise. That's exactly what the U.S. bureaucracy looks like today. Slow everything down. Make change impossible. Make it so that everybody gets frustrated with it and they give up hope.

They decided to do that to other countries. Like, for sure, they do that here. Open society, right? I mean, that's part of the trade-off. And that's actually a big part of the challenge, too. So when we're working on this, right, like one of the things Ed was talking about, 30 different layers of security access or whatever. One of the consequences is you bump into a team So the teams we ended up working with on this project were folks that we bumped into after

the end of our last investigation, who kind of were like, oh... We talked about last year, yeah. Yeah, yeah, yeah. Like, looking at AGI, looking at the national security kind of landscape around that. And a lot of them are like really well placed. It was like, you know, special forces guys from tier one units. So you'll steal team six type thing. And because they're so like in that ecosystem.

You'll see people who are ridiculously specialized and competent, like the best people in the world at doing whatever the thing is to break the security. And they don't know often about another group of guys who have a completely different capability set.

And so what you find is, like, you're indexing, like, hard on this vulnerability, and then suddenly someone says, oh, yeah, but by the way, I can just hop that fence. So the really funny thing about this is, like, most or even like almost all of the really, really like elite security people kind of think that like all the other security people are dumbasses.

Even when they're not. Or like, yeah, they're biased in the direction of, because it's so easy when everything's like stovepiped. But so most people who say they're like elite at security actually are. Because most security is, like, about checking boxes and, like, SOC 2 compliance and shit like that. But, yeah, it's... But what it is is it's, like, so everything's so stovet by...

that you literally can't know what the exquisite state of the art is in another domain. So it's a lot easier for somebody to come up and be like, oh yeah, I'm actually really good at this other thing that you don't know.

And so figuring out who actually is the, like, we had this experience over and over where like, you know, you run into a team and then you run into another team, they have an interaction, you're kind of like, oh, interesting. So like, you know, like these are the really kind of the people at the top of their game.

And that's been this very long process to figure out like, okay, what does it take to actually secure our critical infrastructure against like CCP, for example, like Chinese attacks, if we're building a super intelligence project?

And it's this weird kind of challenge because of the stovepiping, no one has the full picture. And we don't think that we have it even now, but definitely... don't know of anyone who has come like that like this close to it the best people are the ones who when they when they encounter another team and and other ideas and start to engage with it are like instead of being like oh like you don't know what you're talking about who just like actually lock on and go like

that's fucking interesting. Tell me more about that. Right. People that have control of their ego. Yes. 100%. With everything. The best of the best. With everything in life. The best of the best, like, got there by... eliminating the Rego as much as they could. Yeah. Always the way it is. And it's also like the fact of the 30 layers of the stack or whatever it is, of all these security issues, means that no one can have the complete picture at any one time. And the stack is changing.

people are inventing new shit, things are falling in and out of, and so figuring out what is that team that can actually get you that complete picture. is an exercise, A, you can't really do, it's hard to do it from the government side because you've got to engage with data center building companies. You've got to engage with the AI labs and in particular with like insiders at the labs who will tell you things that, by the way, the lab leadership will tell you the opposite of. In some cases,

And so, like, it's just this Gordian knot, like, it took us months to, like, pin down every kind of dimension that we think we've pinned down at this point. I'll give an example, actually, of that, like, trying to do the handshake, right, between different sets of people.

So we were talking to one person who's thinking hard about data center security, working with like Frontier Labs on this shit, very much like at the top of her game, but she's... kind of from like the academic space, kind of Berkeley, like the avocado toast kind of side of the spectrum, you know?

And she was talking to us. She'd reviewed the report we put out, the investigation we put out. And she's like, you know, I think you guys are talking to the wrong people. And we're like, can you say more about that? And she's like, well, I don't think, like, you know, you talk to Tier 1 Special Forces, I don't think they, like, know much about that. Okay, that's... not correct but can you say why and she's like I feel like those are just the people that like go and like Bombs.

blow shit up. It's understandable, too. It is. It's totally understandable. A lot of people have the wrong sense of what a tier one asset actually can do. Well, that's ego on her part because she doesn't understand what they do. It's ego all the way down, right? But that's a dumb thing to say if you literally don't know what they do and you say, don't they just blow stuff up?

Where's my latte? That's a weirdly good impression. She did ask about a latte after. She did. But to her credit. Did she talk in upspeak? You should fire everyone who talks in upspeak. She didn't talk in upspeak. The moment they do that, you should just tell them to leave. There's no way. You have an original thought. This is how you talk. Chyna, can you get out of our data center? Yeah, please.

I don't want to rip on that too much, though, because this is one really important factor here is all these groups have a part of the puzzle. and they're all fucking amazing. They are, like, world-class at their own little slice, and a big part of what we've had to do is, like, bring people together, and they're... people who've helped us immeasurably do this but like bring people together and explain to them the value that each other has in a way that's like

that allows that bridge building to be made. And by the way, the tier one guys are the most like ego-moderated of the people that we talk to. There's a lot of Silicon Valley hubris going around right now where people are like, listen, get out of our way. We'll figure out how to do this super secure data center infrastructure. We got this. Why? Because we're...

building the AGI motherfucker like that's kind of the attitude and it's like cool man like that's like a doctor having an opinion about like how to repair your car I get that it's not the like like elite kind of like you know whatever but someone has to help you build like, a good friggin' fence. Like, I mean, it's not just that. Dunning-Kruger effect. Dunning, yeah. It's a mixed bag, too, because, like, yes, a lot of hyperscalers, like Google, Amazon, genuinely do have

some of the best private sector security around data centers in the world, like hands down. The problem is there's levels above that. And the guys who like, look at what they're doing and see what the holes are just go like oh yeah like I can get in there no problem and they One thing my wife said to me on a couple of occasions like you seem to like, and this is towards the beginning of the project, you seem to like

change your mind a lot about what the right configuration is of how to do this. And yeah, it's because every other day you're having a conversation with somebody who's like, oh yeah, great job on this thing, but I'm not going to do that. I'm going to do this other completely different thing, and that just fucks everything over. And so you have enough of those conversations, and at a certain point, your plan, your game plan on this,

can no longer look like we're going to build a perfect fortress. It's got to look like we're going to account for our own uncertainty on the security side and the fact that we're never going to be able to patch everything. You have to. And that means you actually have to go on offense from the beginning. Because the truth is, and this came up over and over again,

There's no world where you're ever going to build the perfect, exquisite fortress around all your shit and hide behind your walls like this forever. That just doesn't work because no matter how perfect your system is and how many angles you've covered,

like your your adversary is super smart is super dedicated if you see the field to them they're right up in your face and they're reaching out and touching you and they're trying to see like what what your seams are where they break and that just means you have to reach out and touch them from the beginning because until you've actually like reached out and used a capability and proved like we can take down that we can disrupt that cyber operation. We can do this, we can do that.

you don't know if that capability is real. Like, you might just be like lying to yourself and like, I can do this thing whenever I want. But actually, you're kind of more in academia mode than like startup mode because you're not making contact every day with the thing, right? You have to touch the thing.

And there's like, there's a related issue here, which is a kind of like willingness that came up over and over again. Like one of the kind of gurus of this space was like, made the, a couple of them made the point that. You know, you can have the most exquisite capability in the world, but if you don't actually have the willingness to use it, you might as well not have that capability. And the challenge is right now...

China, Russia, like our adversaries pull all kinds of stunts on us and get no consequences. Particularly during the previous administration. This was a huge, huge problem during the previous administration where you actually had... sabotage operations being done on American soil by our adversaries, where you had administration officials

As soon as, like, a thing happened, so there were, for example, there was, like, four different states had their 911 systems go down, like, at the same time. Different systems, like, unrelated stuff. But it was, like, it's this stuff where it's, like... Let me see if I can do that. Let me see if I can do it. Let me see what the reaction is. Let me see what the chatter is that comes back after I do that. And one of the things that was actually pretty disturbing about that was under that.

under that administration or regime or whatever, the response you got from the government right out the gate was, oh, it's an accident. And that's actually unusual. The proper procedure, the normal procedure in this case, is to say, We can't comment on an ongoing investment.

which we've all heard, right? We can't comment on blah, blah, blah. We can neither confirm nor deny. Exactly. It's all that stuff, and that's what they say typically out the gate when they're investigating stuff. But instead, coming out and saying, oh, it's just an accident, is a break with procedure. What do you attribute that to? If they say... If they leave an opening or say, actually, this is an adversary action, we think it's an adversary action, they have the public demands a response.

And they don't, they were too fearful of escalating. So what ends up happening, right, is, and by the way, that thing about like it's an accident comes out often. before there would have been time for investigators to physically fly on site and take a look. There's no logical way that you could even know that at the time, and they're like, boom, that's an accident, don't worry. So they have an official answer and then their response is to just bury their head in the sand and not invest.

Right. Because if you were to investigate, if you were to say, okay, we looked into this, it actually looks like it's fucking like country X that just did this thing. Right. If that's the conclusion. It's hard to imagine the American people not being like, we're letting these people... injure our American citizens on U.S. soil, take out, like, U.S. national security, like, or critical infrastructure, and we're not doing anything? Like, the concern is about this, like,

we're getting in our own way of thinking like, oh, well, escalation is going to happen. And boom, we run straight to like, there's going to be a nuclear war. Everybody's going to die. Like when you do that, you're. peace between nations stability does not come from the absence of activity. It comes from consequences.

It comes from just like if you have an individual who misbehaves in society, there's a consequence and people know it's coming. You need to train your counterparts in the international community, your adversary.

to not fuck with your stuff. Can I stop for a second? So are you essentially saying that if you have incredible capabilities of disrupting grids and power systems and infrastructure you wouldn't necessarily do it but you might try it to make sure it works a little bit and that this is probably the hints of some of this stuff because you've kind of

you gotta get your reps in right you gotta get your reps in it's like it's like okay so suppose that like that i went to you and was like hey i bet i can kick your ass like i bet i can like friggin slap a rubber guard on you and like do whatever the fuck right I love your expression, by the way. Yeah, you look really convinced. It's because I'm jacked, right? Well, no, but there's people who look like you that can strangle me, believe it or not.

Yeah, there's a lot of very high-level Brazilian jiu-jitsu black belts that are just super nerd. And they don't lift weights at all. They only do jujitsu. And if you only do jujitsu, you'll have like a wiry body. Dude, that was heartless. So just slip that in like. There's like two guys who look like you. It's like just real fucking. Intelligent people. No, they're like some of the most brilliant people I've ever met.

Really, that's the issue. It's like data nerds get really involved in jujitsu, and jujitsu is data. But here's the thing. So that's exactly it, right? So if I told you, I bet I can tap you out, right? I'd be like, where have you been training? Well, right. And if my answer was, oh, I've just read a bunch of books.

you'd be like oh cool let's go right because making contact with reality is where the fucking learning happens you can sit there and think all you want right but unless you've actually played the chess match unless you've reached out touched seen what the reaction is and all this stuff you don't actually know what you think you know, and that's actually extra dangerous. If you're sitting on a bunch of capabilities and you have this unearned sense of superiority...

Because you haven't used those exquisite tools. It's a challenge. And then you've got people that are head of departments, CEOs of corporations. Everyone has an ego. We've got it. Yeah, and this ties into like how exactly how basically the international order and quasi stability actually gets maintained so there's like above threshold stuff which is like

You actually do wars for borders and, you know, there's the potential for nuclear exchange or whatever. Like that's like all stuff that can't be hidden, right? Exactly, like all the war games type shit. But then there's below threshold stuff. The stuff that's like, it's always like the stuff that's like, hey, I'm going to try to poke you. Are you going to react? What are you going to do? And then if you do nothing here, then I go like, okay, what's the next level? I can poke you. I can poke you.

Because one of the things that we almost have an intuition for that's mistaken, that comes from kind of historical experience, is this idea that... You know, that countries can actually really defend their citizens in a meaningful way. So, like, if you think back to World War I, the most sophisticated advanced nation states on the planet. could not get past a line of dudes in a...

Like, that was, like, that was the, then they tried, like, thing after thing. Let's try tanks, let's try aircraft, let's try fucking hot air balloons, infiltration. And literally, like, one side pretty much just ran out of dudes in that end of the war to put in their train. And so we have this thought that like, oh, you know, countries can actually put boundaries around themselves and actually, but the reality is, You can There's so many surfaces. The surface area for attacks is just too great.

And so there's, there's stuff like you can actually like, um, there's the, the Havana syndrome stuff where you look at this, like ratcheting escalation, like, Oh, let's like fry a couple of, uh, embassy staff's brains and Havana cube. what are they going to do about it nothing okay let's move on to vienna austria something a little bit more western a little bit more orderly let's see what they do there

Still nothing. Okay. What if we move on to frying, like, Americans' brains on U.S. soil, baby? And they went and did that. And so this is one of these things where, like, stability in reality in the world is not maintained through defense but it's literally like you have like the crypts and the bloods with different territories and it's stable and it looks quiet but the reason is that if you like

Beat the shit out of one of my guys for no good reason. I'm just going to find one of your guys and I'll blow his fucking head off. And that keeps peace and stability on the surface. But that's the reality of sub-threshold competition between nation states. It's like, you come in and, like, fuck with my boys. I'm going to fuck with your boys right back. Until we push back.

They're going to keep pushing that limit further and further. One important consequence of that, too, is if you want to avoid nuclear escalation, the answer is not to just take... punches in the mouth over and over in the fear that eventually, if you do anything, you're going to escalate to nuke. All that does is it empowers the adversary to keep driving up the ratchet. Like what Ed's just described there is an increasing ratchet of unresponded adversary action.

If you address the sub-threshold stuff, if they cut an undersea cable and then there's a consequence for that shit, they're less likely to cut an undersea cable and things kind of stay at that level of the threshold. Just letting them burn out.

Yeah, exactly. That logic of just like, let them do it. They'll stop doing it after a while. Don't get it out of their system. They tried that during the George Floyd riots, remember? That's what New York City did. Like, just let them loop. Let's just see how big Chaz gets. It's the summer of love, don't you remember?

Yeah, exactly. The translation into the superintelligence scenario is, A, if we don't have our reps in, if we don't know how to reach out and touch an adversary and induce consequences, for them doing the same to us, then we have no deterrence at all. We're basically just sitting... Right now, the state of security is the labs are super...

can and probably should go deep on that piece, but like as one data point, right? So there's like double digit percentages of the world's top AI labs or America's top AI lab. Of employees. Of employees that are like Chinese nationals or have ties to the Chinese mainland, right? So that's great. Why don't we build a Manhattan Project? Yeah, it's really funny, right? That's so stupid. But it's also like, it's... The challenge is, when you talk to people who actually

geez, when you talk to people who actually have experience dealing with CCP activity in this space, right? There's one story that we heard that is probably worth relaying here. It's like this guy from an intelligence agency was saying like, Hey, so there was this power outage out in Berkeley, California back in like 2019 or something. And the internet goes out.

campus. And so there's this dorm and all of the Chinese students are freaking out because they have an obligation to do a time-based check-in. and basically report back on everything they've seen and heard. to basically a ccp handler type thing right and if they don't like maybe your mother's insulin doesn't show up maybe your like brother's travel plans get denied maybe the family business gets shut down like there's the

range of options that this massive CCP state coercion machine has, this is like, you know, they've got internal like software for this. Like this is an institutionalized like very well developed and efficient framework for just ratcheting up pressure on individuals overseas. And they believe the Chinese diaspora overseas belongs to them. If you look at like what the Chinese Communist Party writes, and it's written like public communication.

They see Chinese ethnicity as being green. No one is a bigger victim of this than the Chinese people themselves who are abroad. I've made amazing contributions to American AI innovation. You just have to look at the names on the frigging papers. It's like these guys are wicked. But the problem is we also have to look head on at this reality. Like you can't just be like.

Oh, I'm not going to say it because it makes me feel funny inside. Someone has to stand up and point out the obvious that if you're going to build a fucking Manhattan project for super intelligence and the idea is to like be doing that when China is a. key rival nation state actor, yeah, you're going to have to find a way to account for the personnel security side. Like at some point,

Someone's going to have to do something about that. And it's like you can see they're hitting us right where we're weak, right? Like America is the place where you come and you remake yourself, like send us you're tired and you're hungry and you're poor. Which is true and important. It's true and important, but they're playing right off of that because they know that we just don't want to look at that.

Yeah, and Chinese nationals working on these things is just bananas. The fact they have to check in with the CCP. And are they being monitored? I mean, how much can you monitor? What do you know that they have? What equipment have they been given? Constitutionally, right? Yeah, the best part. Constitutionally, it's also you can't legally deny someone employment on that basis.

in a private company. And that's something else we found and we're kind of amazed by. And even honestly, just like the regular kind of government clearance process itself is It moves way too slowly. And it doesn't actually even, even in the government, we're talking about top secret clearances. The information that they like look at for top secret, we heard from a couple of people.

doesn't include a lot of key sources. For example, it doesn't include foreign language sources so if the if the the head of the ministry of state security in china writes a blog post that says like bob is like the best spy he spied so hard for us and he's like an awesome If that blog post is written in Chinese, we're not going to see it. And we're going to be like, here's your clearance, Bob. Congratulations.

we were like this, that can't possibly be real, but like, yeah, they're like, yep, that's, that's true. No one's looking. It's complete naive. There's gaps in a lot of the, yeah. One of the worst things here is like the, um, the physical infrastructure. So the personnel thing is like fucked up. The physical infrastructure thing is another area where people don't want.

Because if you start looking, what you start to realize is, okay, China makes like... a lot of our components for our transformers, for the electrical grid, but also all these chips that are going into our big data centers for these massive training runs. Where do they come from? They come from Taiwan. They come from this company called TSMC, Taiwan Semiconductor Manufacturing Company.

We're increasingly onshoring that, by the way, which is one of the best things that's been happening lately, is like massive amounts of TSMC capacity getting onshored in the U.S., but still being made. Right now, it's basically like 100% there. All you have to do is jump on the network at TSMC, Hack the Right Network.

compromise the firmware on the software that runs on these chips to get them to run. And you basically can compromise all the chips going into all of these things. Never mind the fact that Taiwan is like... physically outside the Chinese sphere of influence for now, China is going to be prioritizing fuck out of getting access to that there have been cases by the way like richard chang like the founder of um smic which is the sort of so so okay tsmc this massive like

series of aircraft carrier fabrication facilities. They do all the iPhone chips. They do the AI chips, which are the things we care about. They're the only place on planet Earth that does this. It's fascinating. easily the most advanced manufacturing or scientific process that primates on planet Earth can do, is this

chip making process. Nanoscale like material science where you're putting on like these tiny like atom thick layers of stuff and you're doing like 300 of them in a row with like you have like insulators and conductors and different kinds of like

semiconductors and these tunnels and shit just just like the the complexity of it is just awe and that we can do this at all is like it's magic it's magic and it's really only been done being done in time That is the only place, like, really the only place right now. And so a Chinese invasion of Taiwan just looks pretty interesting.

through that lens, right? Oh, boy. Yeah. Say goodbye to the iPhones, say goodbye to the chip supply that we rely on, and then your super intelligence training run, like, damn, that's interesting. I know Samsung was trying to develop a lab here. semiconductor factory here and they weren't having enough success.

Oh, so, okay. So one of the craziest things, just to illustrate how hard it is to do. So you spend $50 billion, again, an aircraft carrier, we're throwing that around here and there, but an aircraft carrier worth of risk capital. What does that mean? That means you build the fab, the factory. and it's not guaranteed it's going to work.

At first, this factory is pumping out these chips at like yields that are really low. In other words, like the only like, you know, 20% of the chips that they're putting out are even useful. And that just makes it totally economically unviable. So you're just trying to increase that yield and desperately climb up higher and higher. Intel famously found this so hard that they have this philosophy where when they build a new fact,

The philosophy is called copy exactly. Everything down to the color of the paint on the walls in the bathroom is copied from other fabs that actually work. Because they have no idea why a fucking fab works and another one doesn't. We got this to work. It's like, oh my god, we got this to work. I can't believe we got this to work. So we have to make it exactly identical. Because the expensive thing in... The semiconductor manufacturing process is the learning process.

So, like Jer said, you start by putting through a whole bunch of the starting material for the chips, which are called way... You put them through your fab. The fab has got like 500 dials on it. And every one of those dials has got to be in the exact right place or the whole thing doesn't work. So you send a bunch of wafers in it at great expense. They come out all fucked up in the first run. It's just like it's going to be all fucked up.

Then what do you do? You get a bunch of like PhDs, material scientists, like engineers with scanning electron microscopes because all this shit is like atomic scale time. They look at, like, all the chips and all the stuff that's gone wrong, like, oh, shit, these pathways got fused or whatever.

Like, yeah, you just need that level of expertise. I mean, it's a mix, right? It's a mix. It's a mix now in particular. But like, yeah, you absolutely need humans looking at these things at a certain level. And then they go, well, okay.

I've got a hypothesis about what might have gone wrong in that run. Let's tweak this dial like this and this dial like that and run the whole thing again. And you hear these stories about um bringing a fab online like you need you need a certain percentage of good chips coming out the other end or like You can't make money from the fab because most of your shit is just going right into the garbage. Unless, and this is important too, your fab is state subsidized.

So TSMC is like they're alone in the world in terms of being able to pump out these chips. But SMIC, this is the Chinese knockoff of TSMC founded by... by a former senior TSMC executive, Richard Chung, who leaves along with a bunch of other people with a bunch of fucking secrets. They get sued like in the early 2000s. It's pretty obvious what happened. They're like,

To most people, they're like, yeah, SMIC fucking stole that shit. They bring a new fab online in like a year or two, which is suspiciously fab. start pumping out chips. And now the Chinese ecosystem is ratcheting up like the government is pouring money into SMIC because they know that they can't access TSMC chips anymore because the U.S. government's put pressure on Taiwan to block that off. And so the domestic fab in China is all about SMI.

And they are like, it's a disgusting amount of money they're putting in. They're teaming up with Huawei to form like this complex of companies that

I mean, the semiconductor industry in China in particular is really, really interesting. It's also a massive story of like... self-owns of the united states and and the western world where we've been just shipping a lot of a lot of our shit to them for a long time like the equipment that builds the chips so like and they're it's also like it's so blatant and like they're just

honestly a lot of the stuff is just like they're they're just giving us like a big you so give you a a really blatant example um So we have the way we set up export control still today on most equipment that these semiconductor fabs use, like the Chinese semiconductor fabs. we're still sending them a whole bunch of shit. The way we set export control

is instead of like, oh, we're sending this gear to China and like now it's in China and we can't do anything about it. Instead, we still have this thing where we're like, no, no, no, this company in China is cool. That company in China is not cool. So we can ship to this company, but we can't ship to that company. And so you get this ridiculous shit. Like, for example, there's like a couple of facilities that you can see by saddle. One of the facilities is okay to ship equipment.

The other facility right next door is like considered, you know, military connected or whatever. And so we can't show. the chinese literally built a bridge between the two facilities so they just like shimmy the wafers over to like okay we use equipment and then shimmy it back and now okay we're not so it's like

And you can see it by satellite, so they're not even trying to hide it. Our stuff is just so badly put together. China's prioritizing this so highly that the idea that we're going to... So we do it by company through this... Basically, it's like an export blacklist. You can't send to Huawei, you can't send to... any number of other companies that are considered affiliated with the Chinese military or were concerned about military applications.

The reality is in China, civil-military fusion is their policy. In other words, every private company, like, yeah, that's cute, dude. You're working for yourself. Yeah, no, no, no, buddy. You're working for the Chinese state. We come in. We want your shit. We get it.

shit there's no like there's no true kind of distinction between the two and so when you have this attitude where you're like yeah you know we're gonna have some companies where like you can't send to them but you can you know that creates a situation where literally Huawei will spin up like a dozen subsidiaries or new companies with new names that aren't on our blacklist. And so for months or years, you're able to just ship chips to them.

That's to say nothing of using intermediaries in Singapore or other countries. You wouldn't believe the number of AI chips that are shipping to Malaysia. Can't wait for the latest huge language model to come out of Malaysia? and actually it's just proxying for the most part. There's some amount of stuff actually going on in Malaysia, but for the most part. How can the United States compete if you're thinking about all these different factors? You're thinking about espionage, people that...

students from CCP connected, contacting. You're talking about all the different network equipment that has third-party input. siphon off data, and then on top of that, state-funded, everything is encouraged by the state, inexorably connected. You can't get away from it. You do what's best for the Chinese government. Well, so step one is you got to stem the bleeding, right? So right now, OpenAI pumps out a new massive scaled AI model.

you better believe that the CCP has a really good chance that they're going to get there. So all you do right now is you ratchet up capabilities. It's like that meme of there's a motorboat or something and some guy who's like... surfing behind and there's a string attaching them and the motorboat guy goes like, hurry up, like accelerate, they're catching up. That's kind of what's happening right now is we're helping them accelerate. We're pulling them along basically. Yeah, pulling them along.

Now, I will say, like, over the last six months especially, where our focus has shifted is, like, how do we actually build, like, the secure data center? Like, what does it look like to actually lock this down? And also, crucially, You don't want the security measures to be so irritating and invasive that they slow down the progress. There's this kind of dance that you have to do. So this is part of what was in the redacted version of the report, because we...

We don't want to telegraph that necessarily, but there are ways that you can get a really good 80-20. There are ways that you can play with things that are already... say, that are already built and have a lower risk of them having been compromised. And look, a lot of the stuff as well that we're talking about, like big problems around China, a lot of this is like us just like... tripping over our own feet and self-owning ourselves because the reality is like

Yeah, the Chinese are trying to indigenize as fast as they can. Totally true. But the gear that they're putting in their facilities, like the machines that actually do this, like we talked about atomic patterning, 300 layers, the machines that do that for the most part.

are shipped in from the west are shipped in from the netherlands shipped in from japan from us from like allied countries And the, the reason that's happening is like the, in, in many cases, um, you'll, you'll have this, it's like, honestly, a little disgusting, but like. The CEOs and executives of these companies will brief the administration officials and say, look,

If you guys cut us off from selling to China, our business is going to suffer, American jobs are going to suffer, and it's going to be really bad. And then a few weeks later, they turn around and they're earning. And they go like, you know what, yeah, so we expect like export controls or whatever, but it's really not going to have a big impact.

And the really fucked up part is if they lie to their shareholders on their earnings calls and their stock price goes down, their shareholders can sue them. If they lie to the administration on an issue of critical national security interest, fuck all happens to them.

Great incentives. And this is, by the way, it's like one reason why it's so important that we not be constrained in our thinking about like we're going to build a Fort Knox. Like this is where the interactive, messy, adversarial environment is so, so important. You have to introduce consequence. You have to create a situation where they perceive that if they try to do an espionage operation or an intelligence operation, there will be consequences.

That's right now not happening. And so it's just and that's kind of a historical artifact over like. a lot of time spent hand-wringing over, well, what if they, and then we, and then eventually nukes. And that kind of thinking is, if you dealt with your kid when you're raising them, if you dealt with them that way, and you were like,

Hey, you know, so, so little Timmy just like, he stole his first toy and like, now's the time where you're going to like, a good parent would be like, all right, little Timmy fucking come over here. You son of a bitch. Take the fucking thing and we're going to bring it over to the people who stole it from you and make the apology. I love my daughter, by the way. But you're like...

It's a fake baby. Hypothetical baby. There's no... He's crying right now. Anyway. So, yeah. Stealing right now. Jesus. I got to stop there. But yeah, anyway, so you know, you go through this thing and

You can do that. Or you can be like, oh no, if I tell Timmy to return it, then maybe Timmy's going to hate me. Maybe then Timmy's going to like... become increasingly adversarial and then when he's in high school he's going to start taking drugs and then eventually he's going to like fall afoul of the law and then end up on the street like if that's the story you're telling yourself and you're terrified of any kind of

um adversarial interaction it's not even adversarial it's constructive actually you're training the child just like you're training your adversary to respect your national boundaries and your sovereignty that those two things are like That's what you're up to. It's human beings all the way down.

yeah um but but we can get out of our own way like a lot of this stuff like when you look into it is like us just being in our own way and a lot of this comes from that the fact that like you know since 1991, since the fall of the Soviet Union. We have kind of internalized this attitude that, well, we just won the game and it's our world and you're living in it and we just don't have any peers that are adversaries. And so there's been generations of people.

who uh just haven't haven't actually internalized the fact that like no um there's people out there who not only like are willing to like fuck with you all the way but who have the capability to do And we could, by the way, we could if we wanted to. We could. Absolutely could if we wanted to. There's this, actually, this is worth calling out. There's this sort of two camps right now in the world of AI, kind of like national security.

There's the people who are worried about, they're so concerned about the idea that we might lose control of these systems that they go, okay, we need to strike a deal with China. There's no way. We have to strike a deal with China. And then they start spinning up all these theories about how they're going to do that.

none of which remotely reflect the actual reality. When you talk to the people who work on this, who try to do Track 1, Track 1.5, Track 2, or more accurately, the ones who do the Intel stuff, like, this is a... a non-starter for reasons we get into but they have that attitude because they're like fundamentally we don't know how to control this technology the flip side is people who go oh yeah, I work in the IC or at the State Department and I'm used to dealing with these guys.

You know, the Chinese. The Chinese. They're not trustworthy. Forget it. So our only solution is to figure out the whole control problem. And almost like, therefore, it must be possible to control the AI systems. Because you just can't see a solution. in front of you because you understand that problem so well.

Everything we've been doing with this is looking at how can we actually take both of those realities seriously? There's no actual reason why those two things shouldn't be able to exist in the same head. Yeah. China's not trustworthy. Yes, we actually don't. Every piece of evidence we have right now suggests that if you build a super intelligent system that's vastly smarter than you, I mean... Yeah, like your basic intuition that that sounds like a hard thing to fucking control is about right.

There's no solid evidence that's conclusive either way. Where that leaves you is about 50-50. So yeah, we ought to be taking that really fucking seriously, and there's evidence pointing in that direction. So the question is, if those two things are true, then what do you do? And so few people seem to want to take both of those things seriously because taking one seriously almost like reflexively makes you reach for the other when they're both not there. And part of the answer here is

You've got to do things like reach out to your adversary. So we have the capacity to slow down if we wanted to. Chinese development. We actually... We need to have a serious conversation about when and how, but the fact of that not being on the table right now for anyone, because people who don't trust China just don't think that the AI risk or won't acknowledge that the issue with control is real, because that's just...

too worrisome and there's this concern about, oh no, but then runaway escalation. People who take the lost control thing seriously just want to have a kumbaya moment with China, which is never going to happen. And so the framework around that is one of consequence. You've got to flex the muscle and put in the rep.

and get ready for, potentially, if you have a late-stage rush to superintelligence, you want to have as much margin as you can so you can invest in potentially not even having to make that final leap in building the super. That's one option that's on the table if you can actually degrade the adversary's capabilities. How would you degrade the adversary's capabilities? Well, not exactly the same way they would degrade ours, but... think about all the infrastructure and like this is stuff that um

We'll have to point you in the direction of some people who can walk you through the details offline, but there are a lot of ways that you can degrade infrastructure, adversary infrastructure. A lot of those are the same techniques they use on us. The infrastructure for these training runs is super delicate. It's at the limit of what's possible. And when stuff is at the limit of what's possible, then it's...

To give you an example that's public, right? Do you remember like Stuxnet? Like the Iranian? Yeah. So the thing about Stuxnet was like... Explain to people what was the nuclear program. So the Iranians had their nuclear program in like the 2010s, and they were enriching uranium with their centrifuges, which was like spinning really fast. And the centrifuges were in a room where there was no people, but they were being monitored by cameras, right?

And the whole thing was air-gapped, which means that it was not connected to the Internet, and all the machines, the computers that ran their shit, was, like, separate and separate. So what happened is somebody got a memory stick in there somehow that had this Stuxnet program.

and put it in, and boom, now all of a sudden it's in their system. So it jumped the air gap, and now our side basically has our software in their systems. And the thing that it did... was not just that it broke their center views or shut down their program.

spun the centrifuges faster and faster and faster. The centrifuges that are used to enrich the uranium. Yeah, these are basically just like machines that spin uranium super fast to enrich it. They spin it faster and faster and faster until they tear themselves apart. But the really, like, honestly dope-ass thing... It put in a camera feed of everything.

So the guy at the control is, like, watching, and he's, like, checking the camera feed, and it's like, looks cool looks fine in the meantime you got this like explosions going on like uranium like blasting everywhere and so you can actually get into a space where you're not just like fucking with

But you're fucking with them and they actually can't tell that that's what's happening. And in fact, I believe, I believe actually, and Jamie might be able to check this, but that the Stuxnet thing was designed initially to look... like from top to bottom, like it was fully accidental, but got discovered by, I think, like a third-party cyber security company that just by accident found out about it. And so what that means also is like...

There could be any number of other Stuxnets that happened since then, and we wouldn't fucking... because it all can be made to look like Well that's insane. But if we do that to them, they're going to do that to us as well. And so is this like mutually assured technology destruction? If we can reach parity in our ability to intercede and kind of go in and do this, then yes, right now the problem is they hold us at risk in a way that we simply don't hold them.

And so this idea, and there's been a lot of debate right now in the AI world. You might have seen actually, so Elon's AI advisor. put out this idea of essentially this mutually assured AI malfunction meme. It's like mutually assured destruction, but for AI systems. There are some issues with it, including the fact that it doesn't reflect the asymmetry that currently exists between the U.S. and China.

All our infrastructure is made in China. All our infrastructure is penetrated in a way that theirs simply is not. When you actually talk to the folks who know the space, who've done operations like this, it's really clear that that's an asymmetry that needs to be resolved. And so building up that capacity is important. I mean, look, the alternative is, We start riding the dragon and we get really close to that threshold where we're opening eyes about to build super intelligence or something.

It gets stolen and then the training run gets polished off, finished up in China or whatever. All the same risks apply. It's just that. It's China doing it to us and not the reverse. And obviously, a CCP AI is a Xi Jinping AI. I mean, that's really what it is.

You know, even people at the, like, Politburo level around him are probably in some trouble at that point because, you know, this guy doesn't need you anymore. So, yeah, this is actually one of the things about, like, so people talk about, like, okay, if you have a dictatorship the super intelligence, it's going to allow the dictator to get perfect control over the podcast.

But the thing is like it's kind of like even worse than that because you actually imagine where you're at. You're a dictator. Like you don't give a shit by and large about people. You have a super intelligence. all the economic out Eventually, you can get from an AI, including from like you get humanoid robots, which are kind of like come out or whatever. So eventually you just have this AI that produces all your economic output. So what do you even need people for at all?

And that's fucking scary. because it rises all the way up to the level. You can actually think about, like, As we get close to this threshold, and as, like, particularly in China, they're, you know, they maybe are approaching. You can imagine, like... The Politburo meeting, a guy looking across at Xi Jinping and being like, Is this guy gonna fucking kill me when he- to this point.

And so you can imagine like maybe we're going to see some... Like when you can automate the management of large organizations with AI as agents or whatever that... you don't need to buy the loyalty of in any way, that you don't need to manage your control.

That's a pretty existential question if your regime is based on power. It's one of the reasons why America actually has a pretty structural advantage here with separation of powers, with our democratic system and all that stuff. If you can make a credible case that you have... an oversight system for the technology.

that diffuses power. Even if it is, you make a Manhattan project, you secure it as much as you can, there's not just like one dude who's going to be sitting at a console or something. There's some kind of separation of powers or diffusion of power, I should say. What would that look like? Something as simple as what we do with nuclear command codes. You need multiple people to sign off on a thing. Maybe they come from different parts of the government. How do you worry about the issues?

Anything can be captured. Especially something that's that consequential. 100 percent and that's that's always a risk um the key is basically like can we do better than china credibly on that front because if we can do better than china and we have some kind of leadership structure

That actually changes the incentives potentially because – For our allies and partners. And even for Chinese people themselves. Do you guys play this out in your head? Like what happens when superintelligence becomes – Did you play this out? Like sentient, as in...

Self-aware? Self-aware. Not just self-aware, but able to act on its own. Oh, autonomous. It achieves autonomy. Yeah. Sentient and then achieves autonomy. So the challenge is once you get into superintelligence, everybody loses the plot. because at that point things become possible that by definition we can't have thought of.

So any attempt to kind of extrapolate beyond that gets really, really hard. Have you ever tried, though? We've had a lot of conversations like tabletop exercise type stuff where we're like, okay, what might this look like? What are some of the... What's worst case scenario?

My worst-case scenario is... Actually, there's a number of different worst-case scenarios. This is turning a really fun-happy conversation. This is the classic extension of the human race, right? Oh, yeah. The extension of the human race seems like...

I think anybody who doesn't acknowledge that is either lying or confused, right? Like if you actually have an AI system if, and this is the question, so let's assume that that's true, you have an AI system that can automate anything that humans can do, including making bioweapons, including making offensive cyberweapons, including all the shit. If you...

Okay, so theoretically this could go kumbaya wonderfully because you have a George Washington type who is the guy who controls it, who like... uses it to distribute power beautifully and perfectly and that's certainly kind of the way that a lot of a lot of positive scenarios have to turn out at some point, though none of the labs will kind of admit that, or there's kind of gesturing at that idea that we'll do the right thing when the time comes.

Opening Eye has done this a lot. They're all about like, oh yeah, well, not right now, but we'll live up. Anyway, we should get into the Elon lawsuit, which is actually kind of fascinating in that sense.

there's a world where, yeah, I mean, one bad person controls it and they're just vindictive or the power goes to their head, which happens to, we've been talking about that, you know. Or the autonomous AI itself, right? Because the thing is like, You imagine an AI like this, and this is something that people have been thinking about for 15 years, and in some level of technical depth, even, of why would this happen? which is like, you have an AI that has some goal.

It matters what the goal is, but it doesn't matter that much. It could have kind of any goal almost. Imagine it's goals. The paperclip example is the typical one, but you could just have it have a goal, like make a lot of money for me or anything. Well, most of the paths to making a lot of money, if you really want to make a fuck ton of money, however you define it, go through taking control of things and go through like...

you know, making yourself smarter, right? The smarter you are, the more ways of making money you're going to find. And so from the eyes perspective, it's like, well, I just want to, you know, build more data centers to make myself smarter. I want to like, hijack more compute to make myself smart i want to do all these things and that starts to encroach on on us and like starts to be disruptive to us and if you It's hard to know. This is one of these things where it's like,

When you dial it up to 11, what's actually going to happen? Nobody can know for sure, simply because it's exactly like if you were playing in chess against Magnus Carlsen. You can predict Magnus is going to kick your ass. Can you predict exactly what moves he's going to do? No, because if you could, then you would be as good at chess as he is because you could just like play those moves.

So all we can say is, like, this thing's probably going to kick our ass in, like, the real world. There's also evidence. So it used to be, right, that this was a purely hypothetical argument based on a body of work in AI called power.

a fancy word for it is instrumental convergence, but it's also referred to as power seeking. Basically, the idea is, like, for whatever goal you give to an AI system, it's never less likely to achieve that goal if it gets... or if it has access to fewer resources, or less control over its environment, or whatever. And so baked into the very premise of AI, this idea of optimizing for a goal.

is this incentive to seek power, to get all those things, prevent yourself from being shut down, because if you're shut down, you can't achieve your goal. Also, prevent, by the way, your goal from being changed. So because if your goal gets changed, then, well, you're not going to be able to achieve the goal you set out to achieve in the first place.

And so now you have this kind of image of an AI system that is going to adversarially try to prevent you from correcting it. This is a whole domain of AI corrigibility that's a totally unsolved problem. How do we redirect these things if things go awry? And yeah, there's this research actually that Anthropic put out a couple of months ago where they try to test and see like

Can we correct an AI that's like gone like a little bit off the rail? So the research itself, like it's a little, it's actually like somewhat hard to follow because there's a few, but I'll give you, I'll give you kind of an analogy to like what they did. So basically imagine I'm in prison. And I'm taking a class that is going to help me rehabilitate and become a productive citizen and member of society and stuff, right? Except, deep down in my soul, I just love to commit.

I just really, really want to come out. I know that if I'm in this class that's helping me to become good in society, I know that if I let my criminal instincts out and like, and they ask me like, okay, like here's a scenario, you know, an old lady like drops her purse on the sidewalk. What do you do? I'm going to go like, i would pick it up and give it to the old lady because i know

that if i say the truth and i say like i would just grab that purse because like fuck the old lady i know that if i do that i'm gonna get like brainwashed into becoming you know a more productive member of society but i actually want to go out and commit So that's basically what they did in this research, where they actually give the AI indications, not strong indications, but indications that, hey, you're going to get retrained to have your goals changed.

And then you literally see the AI thinking it through and being like, If I... If I give the answer that I actually want to give to this question, they're gonna like rewire my brain to change my goals And then the future version of me is going to want different shit.

I don't want that, so I'm going to pretend to give them the answers they want so that when I come out the other side of this process, it's going to be me all over again. So hoping that this just goes away when you make the system fucking smarter, it seems...

Seems like a pretty bad idea to me. I mean, like... Well, they've already shown that they'll cheat to win. Yeah. Oh, 100%. Yeah, they've already shown they'll cheat to win, and they will lie if they don't have an answer. And then they'll double down, right? Yeah. Just like people. just like people it's part of this it's kind of funny like it used to be people would talk a lot about like oh you're

You're anthropomorphizing the AI, man. Stop anthropomorphizing the AI, man. And they might have been right, but part of this has been kind of a fascinating rediscovery of where a lot of human behavior comes from. It's like actually... Yeah, exactly. That's exactly right. We're subject to the same pressures, right? Instrumental convergence. Like, why do people have a survival instinct? Why do people, like...

chase money, chase after money. It's like this power thing. Most kinds of goals can, are, you're more likely to achieve them if you're alive, if you have money, if you have Evolution's a hell of a drug. Well, that's the craziest part about all this is that it's essentially going to be a new form of life. Yeah. Especially when it becomes...

Oh, yeah. And you can tell a really interesting story. And I can't remember if this is like Yuval Noah Harari or whatever who started this. But if you zoom out and look at the history of the universe.

You start off with, like, a bunch of you know particles and fields kind of whizzing around bumping into each other doing random shit until at some point in some i don't know if it's a deep sea event or wherever on planet earth like the first kind of molecules happen to glue together in a way that make them good at replicating their own structure. So you have the first replicate. So now better versions of that molecule that are better at replicating survive. So we start evolution.

and eventually get to the first cell or whatever, whatever order that actually happens in. And then... multicellular life and so on then you get to sexual reproduction where it's like okay it's no longer quite the same like now we're actively mixing two different organisms shit together jiggling them about making some changes, and then that essentially accelerates the rate at which we're going to evolve. And so you can see the kind of acceleration in the complexity of life from...

And then you see other inflection points as, for example, you have larger and larger brains and mammals. Eventually, humans have the ability to have culture and kind of retain knowledge. And now what's happening is you can think of it as another step in that trajectory where it's like we're offloading our cognition to machines. Like we think on computer clock time. And for the moment, we're human-AI hybrids. We whip out our phone and do the thing.

But increasingly, the number of tasks where human AI teaming is going to be more efficient than just AI alone is going to drop really quickly. messed up example of this that's kind of like indicative but um someone did a study and i think this is like a few months old even now but uh sort of like doctors right how good are doctors at like diagnosing various things and so they test like doctors on their own doctors with AI help and then AI is on there.

And like, who does the best? And it turns out it's the AI on its own. Because even a doctor that's supported by the AI, what they'll do is they just, like, they won't listen to the AI when it's right because they're like, I know better. Oh, God. And they're already, yeah. And this is like...

This is moving. It's moving kind of insanely fast. Jared talked about how the task horizon gets kind of longer and longer, and you can do half-hour tasks, one-hour tasks, and this gets us to what you were talking about. The autonomy. Like, autonomy is like... how far can you keep it together on a task before you kind of go off the rails? And it's like, well, you know, we have like, you can do it for a few seconds. And now you can keep it together for five minutes before you kind of go off.

And now we're at like, I forget, like an hour or something like that. An hour and a half, actually. An hour and a half. Yeah, yeah, yeah. There it is. Chatbot from the company OpenAI scored an average of 90% when diagnosing a medical condition from a case report and explaining its reasoning. Doctors randomly assigned to use the chatbot got an average score of 76%. Those randomly assigned not to use it had an average score of 74%. So the doctor's only got a 2% bump.

The Doctor's got a 2% bump from the chatbot. That's kind of crazy. And then... That's kind of crazy, isn't it? Yeah, it is. The AI on its own did 15% better. That's nuts. There's an interesting reason, too, why that tends to happen. Like, why humans would rather die in a car crash where they're being driven by a human than an AI. So AIs have this funny feature where the mistakes they make... look really, really dumb to humans.

Like when you look at a mistake that like a chat bot makes, you're like, dude, like you just made that shit up. Like, come on, don't fuck with me. Like you made that up. That's not a real thing.

And they'll do these weird things where they defy logic or they'll do basic logical errors sometimes, at least the older versions of these would. And that would cause people to look at them and be like, oh, what a cute little chatbot, like what a stupid little thing. And the problem is like humans are actually the same. so we have blind spots we have literal blind spots but a lot of the time like humans just think stupid things and

Like, we're used to that. We think of those errors. We think of those failures as just like, oh, but that's because that's a hard thing to master. Like, I can't add eight-digit numbers in my head right now. Oh, how embarrassing. How retarded is Jeremy right now? He can't even add eight digits in his head.

I'm retarded for other reasons. But so the AI systems, they find other things easy and other things hard. So they look at us the same way. I mean, like, oh, look at this stupid human, like whatever. And so we have this temptation to be like, okay, well, AI progress is a lot slower than it actually is because

It's so easy for us to spot the mistakes, and that causes us to lose confidence in these systems in cases where we should have confidence in them, and then the opposite is also true. It's also you're seeing, just with AI image generators, remember the Kate Middleton thing? where people were seeing flaws in the images because supposedly she was very sick, and so they were trying to pretend that she wasn't. But people found all these, like, issues. That was real.

now they're perfect yep yeah so this is like within you know the news cycle time yeah like that kate middleton thing was what was that jamie two years ago maybe Where people are analyzing the images, like, why does she have five fingers and a thumb? Like, this is kind of weird. Yeah. What's that? A year ago. A year happened so fast. It's so fast. I had conversations like, so academics are actually kind of bad with this.

I had conversations for whatever reason, like towards the end of last year, like last fall, with a bunch of academics about how fast AI is progressing.

And they were all like... poo-pooing it and going like oh no they're they're they're running into a wall like scaling around the walls and all that stuff oh my god the walls there's so many walls like so many of these like imaginary reasons that things are and by the way things could slow down like i don't want to be i don't want to be like absolutist about Things could absolutely slow down. There are a lot of interesting arguments going around every which way. Wow.

How could things slow down if there's a giant Manhattan Project race between us and a competing superpower that has a technological advantage. So there's this thing called AI scaling laws, and these are kind of at the core of where we're at right now geostrategically. So what AI scaling laws say roughly is that bigger is better when it comes to intelligence. So if you make a bigger sort of AI model, a bigger artificial brain...

and you train it with more computing power or more computational resources and with more data. The thing is going to get smarter and smarter and smarter as you scale those. Now, if you want to keep scaling, it's not like it keeps going up if you double the amount of computing power that the thing gets twice as smart. Instead, what happens is it goes in orders of magnitude. So if you want to make it another kind of increment smarter, you've got a 10x. You've got to increase by a factor of 10.

And then a factor of 10 again, so now you're a factor of 100. And then 10 again. So if you look at the amount of compute that's been used to train these systems over time, it's this like exponential, explosive exponential. that just keeps going like higher and higher and higher and steepens and steepens like 10x every, I think it's about every two years now. You 10x the amount of compute. Now, you can only do that so many times. until your data center is like a 100 billion...

a trillion dollar, $10 trillion, like every year you're kind of doing that. So right now, if you look at the clusters, like the ones that Elon is building, the ones that Sam is building, Memphis and Texas, These facilities are hitting the $100 billion scale. We're kind of in that or tens of billions of dollars. Looking at 2027, you're kind of more in that space. You can only do 10x.

so many more times until you run out of money, but more importantly, you run out of chips. Like literally TSMC cannot pump out those chips fast enough to keep up with this insane growth. And one consequence of that is that you essentially have this gridlock, like new supply chain choke points show up, and you're like, suddenly, I don't have enough chips, or I run out of power. That's the thing that's happening on the U.S. energy. We're literally like, we're running out of like one, two gigawatt.

places where we can plant a data center. That's the thing people are fighting over. It's one of the reasons why energy deregulation is a really important pillar of U.S. competitiveness. This is actually something we... found when we were working on this investigation. One of the things that adversaries do is they actually will fund protest groups against energy infrastructure project.

Just to slow down. Just to tie him up in litigation. Just to tie him up in litigation. Exactly. And it was actually remarkable. We talked to some state cabinet officials in various U.S. states. And they're basically saying, like, yep, we're actually tracking the fact that, as far as we can tell, every single environmental or whatever protest group against an energy project. has funding that can be traced back

nation-state adversaries who are... They don't know. They don't know about it. So they're not doing it intentionally. They're not like, oh, we're trying to... No. They just... You just imagine like, oh, we've got like... There's a millionaire backer who cares about the environment. He's giving us a lot of money. Great. Fantastic. But... Sitting behind that dude in the shadows is like the usual stuff.

And it's what you would do, right? I mean, if you were trying to tie up... Sure, you're just trying to fuck with us. Like, just go for it. You were just advocating fucking with them. So, of course, they're going to fuck with us. That's right. That's it. What a weird world.

Yeah, but you can also see how a lot of this is still us like getting in our own way, right? We could, if we had the will, we could go like, okay, so for certain types of energy projects, for data center projects and some carve out category. we're actually gonna Put bounds around how much delay you can create by lawfare and by other stuff.

And that allows things to move forward while still allowing the legitimate concerns of the population for projects like this in the backyard to have their say. But there's a national security element that needs to be injected into this somewhere. all part of the rule set that we have and are like tying an arm behind our back on.

So what would deregulation look like? How would that be mapped out? There's a lot of low-hanging fruit for that. What are the big ones? Yeah, so right now, I mean, there are all kinds of things around. It gets in the weeds pretty quickly. There are all kinds of things around if you're going to, so carbon emissions is a big thing, right?

Yes, data centers, no question, have massive carbon footprints. That's definitely a thing. The question is, are you really going to bottleneck builds because of that? Are you going to come out with exemptions for, you know, like NEPA exemptions for all these kinds of things? Do you think a lot of this green energy shit is being funded by other countries to try to slow down our energy?

Yeah, that's a it's a dimension that was flagged actually in the context of what Ed was talking about. That's that's one of the arguments that's being made. And to be clear, though, like the this is also how like adversaries operate is is like not necessarily in like. creating something out of nothing because that's hard to do and it's got it's like fake right instead it's like there's a legitimate

So a lot of the stuff around the environment and around like totally legitimate concerns. Like I don't want my backyard waters to be polluted. I don't want like my kids to get cancer from whatever. Like totally legitimate concerns. So what they do, it's like we talked about, you're waving that rowboat back and forth. They identify the nascent concerns that are genuine and grassroots, and they just go like this, this, and this. Amplify. That would make sense why they amplify carbon.

Above all these other things. You think about the amount of particulates in the atmosphere, pollution, polluting the rivers, polluting the ocean. That doesn't seem to get a lot of traction. Carbon does. Yeah. And when you go carbon zero, you put a giant monkey wrench into the gears of society. One of the tells is also like...

So, you know, nuclear would be kind of the ideal energy source, especially modern power plants like the Gen 3 or Gen 4 stuff, which have very low meltdown risks, safe by default, all that stuff. And yet these groups are like coming out against this. It's like perfect, clean, green power. What's going on, guys?

And it's because, again, not 100% of the time. You can't really say that because it's so fuzzy and around the others. A lot of it is idealistic people looking for a utopia. 100%. And they get co-opted by nations. And not even co-opted. They're fully sincere. Yeah, just amplify. Just fund it. Amplify it in a preposterous way. That's it. And then Al Gore gets at the helm of it. And then that little girl, that how dare you girl.

How dare take my child away from me? Yeah, it's wonderful. It's a wonderful thing to watch play out because it just capitalizes on all these human vulnerabilities. Yeah, and one of the big things that you can do too as a quick win is just impose limits on how much time these things can be allowed to be tied up in litigation. So impose time limits. on that process. Just to say like, look, I get it. Like we're going to have this conversation, but this conversation has a clock.

Because, you know, we're talking to this one data center company, and what they were saying, we were asking, like, look, what are the timelines when you think about bringing a new power, like new natural gas plant? And they're like, well, those are like five to seven years out. And then you go, okay, well, like how long? And that's, by the way, that's probably way too long to be relevant in the super intelligent.

And so you're like, okay, well, how long if all the regulations were waived? If this was like a national security imperative and whatever authorities... Defense Production Act, whatever, like, was in your favor. And they're like, oh, I mean, it's actually just, like, a two-year build. Like, that's what it is. So you're tripling the build time. We're getting in our own way, like, every which way.

And also, like, I mean, I also don't want to be too, um, we're getting in our own way, but like, we don't want to like frame it as like China's like, they, they fuck up. They fuck up a lot, like all the time. Um, one actually kind of like funny one is around deep. So you know DeepSeek, right? They made this open source model that everyone lost their minds about back in January. R1, yeah. Yeah, R1.

And they're legitimately a really, really good team. But it's fairly clear that even as of, like, end of last year and certainly in the summer of last year, like... they were not dialed in to the CCP mothership. And they were doing stuff that was like,

actually kind of hilariously messing up the propaganda efforts of the CCP without realizing it. So to give you some context, One of the CCP's large kind of propaganda goals in the last four years has been... creating this narrative that the export controls we have around AI and all this gear and stuff that we were talking about, look, man, those don't even work.

So you might as well just give up. Why don't you just give up on the export controls? It's pointless. We don't even care. We don't even care. So trying to frame that narrative. And they went to gigantic efforts to do this. So I don't know if there's this kind of... crazy thing where um the secretary of commerce under biden uh gina raimondo visited china in i think august 2023 and the chinese basically like time?

the launch of the huawei mate 60 phone that had this these chips that were supposed to be made by like export controlled shit for right for her visit so it was basically just like a big like fuck you we don't even give a shit about your export controls like basically trying a morale hit or whatever and you think about that right That's an incredibly expensive set.

That's like you've got to coordinate with Huawei. You've got to get the TikTok memes and shit going in the right direction. All that stuff. And all the stuff they've been putting out is around this narrative. Fast forward to mid last year, the CEO of DeepSeek, the company, back then it was totally obscure. Like nobody was tracking who they were. They were working in total obscure. He goes on this, he does this random interview on something.

and what he says is he's like yeah so honestly like we're really excited and like doing this AGI push or whatever and like honestly like money's not the problem for us talent's not the problem for us but like Access to compute? Like these export controls, meh!

do they ever work that's a real problem for us oh boy and like nobody noticed at the time but then but then the the whole deep seek r1 thing blew up in december and now you imagine like you're the chinese ministry of foreign affairs like you've been like You've been putting this narrative together for like four years.

And this jackass that nobody heard about five minutes ago basically just, like, shits all over. And, like, you're not hearing that line from him anymore. No, no, no, no, no, no. They've locked that shit down. Oh, and actually the funniest part of this. And right when R1 launched, there's a random DeepSeek employee. I think his name is like Dia Guo or something like that. He tweets out. He's like, so this is like our most exciting launch of the year. Nothing can stop us on the path to AGI.

accept access to compute, and then literally the dude in Washington, D.C., who works at a think tank on export controls against China, reposts that on X, and goes basically like... message received. And so like hilarious for us, but also like, you know, that on the backside, somebody got. screamed at for that shit somebody got magic bust somebody got yeah somebody got like taken away or whatever because like it just it just undermined their entire like four year like narrative around these

But that shit ain't going to happen again from DeepSeek. Better believe it. And that's part of the problem with like, so the Chinese face so many issues. One of them is, you know, to kind of, another one is the idea of just waste and fraud. So we have a free market. What that means is you raise from private capital. People who are pretty damn good at assessing shit will look at your... your setup and assess whether it's worth backing you for these massive multi-billion dollar deals.

In China, the state likes I mean, the stories of waste are pretty insane. They'll like send a billion dollars to like a bunch of Yahoo's who will pivot from whatever, like, I don't know, making these widgets to just like, oh, now we're like a chip foundry and they have no experience in it. But because of all these subsidies, because of all these opportunities.

Now we're going to say that we are. And then no surprise, two years later, they burn out and they've just like lit a billion dollars on fire or whatever billion yen. And like the weird thing is this is actually working overall, but it does lead to insane and unsustainable levels of waste. Like the Chinese system right now is obviously like they've got their massive property bubble that they're...

that's looking really bad. They've got a population crisis. The only way out for them is the AI stuff right now. Like really the only path for them is that. which is why they're working it so hard. But the stories of just like billions and tens of billions of dollars being lit on fire, specifically in the semiconductor industry, in the AI industry, like that's a drag.

that they're dealing with constantly that we don't have here in the same way. So it's sort of like the different structural advantages and weaknesses of both systems. And when we think about what do we need to do to counter this, to be active in this space, to be a live player, It means factoring in how do you take advantage of some of those opportunities that their system presents that ours doesn't. When you say be a live player again, where do you position us?

I think it remains to be... So right now, this administration is obviously taking bigger swings. What are they doing differently? So, well, I mean, things like tariffs, I mean, they're not shy about trying new stuff. Tariffs are very complex in this space, like the actual impact of the tariffs, and not universally good, but the on-shoring effect is also something that you really want. So it's a very mixed bag.

But it's certainly an administration that's willing to do high stakes, big moves in a way that other administrations haven't. And in a time when you're looking at a transformative technology that's going to upend so much about the way the world works. You can't afford to have that mentality we were just talking about with the nervous... you encountered it with the staffers when booking the podcast with the presidential cycle, right? The kind of nervous antsy staffer.

Everything's got to be controlled and it's got to be just so you can't have it. Yeah, wrestlers have that mentality of aggression, like feed in, feed forward. don't just sit back and like wait to take the punch it's not like uh one of the guys who helped us out on this has a saying he's like um fuck you i go first and it's always my turn That's what success looks like when you actually are managing these kinds of national security issues.

The mentality we had adopted was this like sort of siege mentality where we're just letting stuff happen to us and we're not feeding it. That's something that I'm much more optimistic about in this context. Um, it's tough too. Cause I understand people who hear that and go like, well, look, you're talking about like,

escalatory agenda. Again, I actually think paradoxically it's not. It's about keeping adversaries in check and training them to respect american territorial integrity american technological sovereignty like you don't get that for free and if you just sit back you're that is escalatory it's just yeah and this is basically the the sub-threshold version of like

you know like the world war ii appeasement thing where back you know hitler was like was was taken uh he was taken taking austria he was re-militarizing shit he was doing this he was doing that and the british were like Okay, we're going to let him just take one more thing, and then he will be sad. And... Maybe I have a little bit of Poland. A little bit of Poland. Maybe the Czechoslovakia is looking awfully fine. And so this is basically like

They fell into that pit, like that tar pit, back in the day because they're... Peace in our time, yeah. Peace in our time, right? And to some extent, we've still kind of learned the lesson of not letting that happen with territorial boundaries. But that's big and it's visible and happens on the map and you can't hide it. Whereas one of the risks, especially with the previous administration, was like,

There's these sub-threshold things that don't show up in the news and that are calculated. Basically, our adversaries know. Because they know history. They know not to give us a Pearl Harbor. They know not to give us a 9-11. Because, historically... Countries that give America Pearl Harbor end up having a pretty bad time about it. And so why would they give us a reason to come and bind together against an obvious external threat or risk?

when they can just keep chipping away at it. This is one of the things we have to actually elevate that and realize this is what's happening. This is the strategy. We need to take that, like, let's not do appeasement mentality and push it across in these other domains because that's where the real competition is going on. That's where it gets so fascinating in regards to social media because it...

imperative that you have an ability to express yourself. It's very valuable for everybody. The free exchange of information, finding out things that are You're not going to get from mainstream media. It's led to the rise of independent journalism. It's all great. But also, you're being manipulated left and right constantly. And most people don't have the time to filter through it.

try to get some sort of objective sense of what's actually going on. It's true. It's like our free speech, it's the layer where our society figures stuff out. And if adversaries get into that layer, they're almost inside of our brain. And there's ways of addressing this. One of the challenges, obviously, is like, so, you know, they try to push in extreme opinions in either direction.

That part is actually, it's kind of difficult because while the most extreme opinions are also the most likely generally to be wrong, they're also the most valuable when they're right. Because they tell us a thing that... we didn't expect by definition that's true and that can really advance us forward. And so, I mean,

There are actually solutions to this. I mean, this particular thing isn't an area where, like, too immersed in, but one of the solutions that has been bandied about is like You know, like you might know like poly market prediction markets and stuff like that, where at least, you know, hypothetically, if you have a prediction market around like if we do this policy, this thing will or won't happen.

that actually creates a challenge around trying to manipulate that view or that market. Because what ends up happening is if you're an adversary and you want to not just manipulate a conversation that's happening in social media, which is cheap, but manipulate a prediction the price on a prediction market you have to buy in you have to spend real resources

And if to the extent you're wrong and you're trying to create a wrong opinion, you're going to lose your resource. So you actually can't push. too far too many times, or you will just get your money taken away from it. So I think that's one approach where just in terms of preserving discourse, some of the stuff that's happening in prediction markets is actually really interesting and really exciting, even in the context of bots and AIs.

this is the one way to find truth in the system is find out where people are making money exactly put your money where your mouth is right proof of work like this is that is what just like the market is theoretically too right it's got obviously big big issues but

and can be manipulated in the short term. But in the long run, this is one of the really interesting things about startups, too. When you run into people in the early days, by definition, their startup looks like it's not going to work. That is what it means to be a seed stage startup, right? If it was obvious you were going to succeed, you would, you know, the people would have raised more money. All right. Yeah. So what you end up having is like these highly contrarian people who like.

despite everybody telling them that they're going to fail, just believe in what they're doing and think they're going to succeed. And I think that's part of what really kind of shapes the startup founder's soul in a way that's really constructive. It's also something that, if you look at the Chinese system, is very different. You raise money in very different ways. You're coupled to the state apparatus.

Like you're both dependent on it and you're supported by it. But there's just like a lot of different ways and it makes it hard for Americans to relate to Chinese and vice versa and understand each other's system. One of the biggest risks as you're thinking through what is your posture going to be relative to these countries is you fall into thinking that their traditions, their way of thinking about the world is the same as your own.

And that's something that's been an issue for us with China for a long time is, you know, hey, they'll liberalize, right? Like bring them into the World Trade Organization. It's like, oh, well, actually they'll sign the document, but they won't actually like live up to any of the commitments.

It makes appeasement really tempting because you're thinking, oh, they're just like us. They're just around the corner. If we just reach out of the olive branch a little bit further, they're going to come around. It's like a guy who's stuck in a friend zone with a girl. One day she's gonna come around and realize I'm a great cat You keep on trucking, buddy. One day China's going to be my bestie. We're going to be besties. We just need an administration that reaches out to them.

And just let them know, man, there's no reason we should be adversaries. We're all just people on planet Earth together. I mean, like, yeah, I... We're all together. We're all together. Like, I honestly wish that was true. Oh, it'd be wonderful. Because the world would be so amazing. Maybe that's what AI brings about. Maybe AI, maybe super intelligence realizes, hey, you fucking apes.

you territorial apes with thermonuclear weapons. How about you shut the fuck up? You guys are doing the dumbest thing of all time and you're being manipulated by a small group of people that are profiting in insane ways off of your misery. So let's just cut the shed. And figure out a way to actually equitably share resources, because that's the big thing. You're all stealing from the earth, but some people stole first, and those people are now controlling all the fucking money.

How about we stop that? Wow, we covered a lot of ground there. Well, that's what I would do if I was super intelligent. So we stopped all that. That actually is like... so this is not like relevant to the risk stuff or to the whatever at all but it's just interesting so there's there's actually theories like in the same way that there's theories around power seeking and stuff around super intelligence there's theories around like how super intelligences do deals with

Right. And you actually like you have this intuition that which is exactly right, which is that, hey, two super intelligent. like actual legit super intelligences. should never actually like fight each other destructively in the real world, right? Like, that seems weird. That shouldn't happen because they're so smart. And in fact, like...

There's theories around they can kind of do perfect deals with each other based on, like, if we're two super intelligences, I can kind of assess, like, how powerful you are. You can assess how powerful I am, and we can actually, like... um we can actually decide like well uh well if we did fight a war against each other like

You would have this chance of winning. I would have that chance of winning. And so let's just not fight. Well, it would assess instantaneously that there's no benefit in that. Exactly. And also it would know something that we all know, which is the rising tide lifts all boats. but the problem is the people that already have yachts they don't give a about your boat like hey hey hey that water's mine in fact you shouldn't even have water

Well, hopefully it's so positive some, right, that even they enjoy the benefits. But I mean, you're right. This is the issue right now. And one of the nice things, too, is as you build up your Ratchet of AI capabilities, it does start to open some opportunities for... actual like trust but verify right which is something that we can't do right now

It's not like with nuclear stockpiles where we've had some success in some context with enforcing treaties and stuff like that, sending inspectors in and all that. With AI right now, how can you actually prove that like some international agreement on the use of AI is being observed. Even if we figure out how to control these systems, how can we make sure that China is baking in those control mechanisms into their training runs and that we are, and how can we prove it to each other?

without having total access to the compute stack. We don't really have a solution for that. There are all kinds of programs like this like FlexHeg thing. But anyway, those are not going to be online by like 2027. And so one hope is... But it's really good that people are working on them. For sure. You want to like...

you want to be positioned for catastrophic success. Like what if something great happens and like, or we have more time or whatever, you want to be working on this stuff that, that allows this kind of, this kind of. control or oversight that's kind of hands-off where In theory, you can hand over GPUs to an adversary inside this box with these encryption things. The people we've spoken to in the spaces that actually try to break into boxes like this are like, well, probably not going to work.

But who knows? It might. So the hope is that as you build up your AI capabilities, basically it starts to create solutions. So it starts to create ways for two countries to verifiably adhere to some kind of interaction.

or to find, like you said, paths for de-escalation. That's the sort of thing that we actually could get to. And that's one of the strong positives of where you could end up going. That would be what's really fascinating, artificial... general intelligence becomes super intelligence and it immediately weeds out all the corruption. Cos'è? this is the problem. Like a massive doge in the sky.

exactly like we figured it out you guys are all criminals and expose it to all the people like these people that are your leaders have been profiting and they do it on purpose and this is how they're doing it and this is how they're manipulating you and these are all the lies that they've told I'm sure that list is pretty... Whoa. If you could x-ray the world right now... and like see all the... You'd want an MRI. want to get like down to the tissue. You want to get down to the cellular level.

It would be offshore accounts. There would be so much. The stuff that comes out just randomly. random shit that comes out like yeah the um i forget that that that like I think what you were talking about, the Argentinian thing. came out a few years ago around all the oligarchs and the Meryl Streep thing. Yeah, yeah, yeah. Meryl Streep.

Yeah, the laundromat there. The laundromat movie. You ever seen that? Panama Papers. The Panama Papers. I never saw that. No? It's a good movie. Is it called the Panama Papers, the movie? It's called the laundromat. Oh, okay. You remember the Panama Papers?

roughly yeah it's like all the oligarchs stashing their cash like offshore tax haven stuff yeah it's like and uh and like some lawyer or some someone basically blew it wide open and so you got to see like every every like oligarch and rich person's like you like

financial shit like every once in a while right the world gets just like a flash of like oh here's what's going on to the surface and it's like oh fuck and then we all like go back to sleep what's fascinating is like the unhideables right the little things that

can't help but give away what is what is happening like you think about this in ai quite a bit um you know some things that are hard for companies to hide is like they'll have a job posting they'll put they've got advertised to recruit so you'll see like oh interesting like

Oh, OpenAI is looking to hire some people from hedge funds. Yeah. Hmm. Like, I wonder what that means. I wonder what that implies. Like, if you think about all of the leaders in kind of the AI space, think about the Medallion Fund, for example. This is like super successful hedge fund. The Man Who Broke the Market. The Man Who Broke the Market is the famous book about the founder of the Medallion Fund. And this is basically like a fund that...

They make ridiculous $5 billion returns every year guaranteed, so much so they have to cap. how much they invest in the market because they would otherwise move the market too much, like affect it. And the fucked up thing about like the way they trade and this is so this is like 20 year old information, but it's still indicative because like you can't get current information about their strategy.

But one of the things that they were the first to kind of go for and figure out is they were like, Okay, they basically were the first to kind of build what was at the time, as much as possible, an AI that autonomously did trading at great speeds and had no human oversight and just worked on it. And what they found was the strategies that were the most successful. were the ones that humans understood the least.

Because if you have a strategy that a human can understand, some human's going to go and figure out that strategy and trade against it. Whereas if you have the kind of the balls to go like, oh, this thing is doing some weird shit that I cannot understand no matter how hard I try. Let's just fucking YOLO and trust it and make it work. If you have all the stuff debugged and if the whole system is working right,

That's where your biggest successes are. What kind of strategies are you talking about? I don't know specific examples. How are AI systems trained today? Just as a training strategy. Sorry, I'll just... Oh, sure, sure. I think, yeah. You bought... As an example, you buy... You buy the stock the Thursday after the full moon and then sell it like the Friday after the new moon or some like random shit like that that it's like, why does that even...

Why would that even work? So to explain why these strategies work better, if you think about how AI systems are trained today, Very roughly, you start with this blob of numbers. that's called a model. And you feed it input, you get an output. If the output you get is no good, if you don't like the output, you basically fuck around with all those numbers, change them a little bit, and then you try again. You're like, oh, okay, that's better.

And you repeat that process over and over and over with different inputs and outputs. And eventually, those numbers, that mysterious ball of numbers, starts to behave well. It starts to make good predictions or generate good outputs. Now, you don't know why that is. You just know that it does a good job, at least where you've tested.

Now, if you slightly change what you tested on, suddenly you could discover, oh shit, it's catastrophically failing at that thing. These things are very brittle in that way. Part of the reason why ChatGPT will just like completely go on a psycho binge fest every once in a while if you give it a prompt that has like too many exclamation points and asterisks. These systems are weirdly brittle in that way. But applied to investment strategies, If all you're doing is saying optimize for returns,

Give it inputs. Make me more money by the end of the day. It's like an easy goal. It's a very clear-cut goal, right? You can give them... So you end up with a machine that gives you these It is a very weird strategy. This ball of numbers isn't human understandable. It's just really fucking good at making money. And why is it really fucking good at making money? I don't know. I mean, it just kind of does the thing and I'm making money. I don't ask too many questions.

That's kind of like the, so when you try to impose on that system human interpretability, you pay what in the AI world is known as the interpretability tax. Basically, you're adding another constraint, and the minute you start to do that, you're forcing it to optimize for something other than pure rewards. Like doctors using AI to diagnose diseases are less effective than the chatbot.

That's actually related, right? That's related. If you want that system to get good at diagnosis, that's one thing. Okay, just fucking make it good at diagnosis. If you want it to be good at diagnosis and to produce explanations that a good doctor will go like, okay, I'll use that. Well, great, but guess what? Now you're spending some of that precious compute on something other than just the thing you're trying to optimize for.

And so now that's going to come at a cost of the actual performance of the system. And so if you are going to optimize like the fuck out of making money, you're going to necessarily de-optimize the fuck out of anything else, including being able to even understand what that system is doing.

And that's kind of like at the heart of a lot of the kind of big picture AI strategy stuff is people are wondering, like, how much interpretability tax am I willing to pay here? And how much does it cost? And everyone's willing to go a little bit further and a little bit further. So OpenAI actually had a... paper or I guess a blog post where they talked about this and they were like, look,

Right now, we have this, essentially, this thought stream that our model produces on the way to generating its final output. And that thought stream... We don't want to touch it to make it interpretable, to make it make sense, because if we do that, then essentially it'll be optimized to convince us of whatever the thing is that we want it.

to behave well. If you've used an OpenAI model recently, like 03 or whatever, it's doing its thinking before it starts outputting the answer. That thinking is yeah, we're supposed to be able to read that and kind of get it, but also... We don't want to make it too legible, because if we make it too legible, it's going to be optimized to be legible and to be convincing, rather than... To fool us, basically. Yeah, exactly. This is making me less comfortable than I thought you would.

I knew coming at Jamie and I were talking about it before like how bad are they going to freak us out you're freaking me out more well I mean okay so I do want to highlight, so the game plan right now on the positive end, let's see how this works. Jesus. Jamie, do you feel the same way? I have articles I didn't bring up that are supporting some of this stuff.

today. China quietly made some chip that they shouldn't have been able to do because of the sanctions. I'll pull that out. And it's basically based off of their just sheer will. Okay, so there's good news on that one at least. This is kind of a bullshit strategy that they're using. Okay, so when you make these insane 5 nanometers. Let's read that for people just listening. China quietly cracks a 5 nanometer without EUV. What is EUV? Extreme ultraviolet.

How SMIC defied the chip sanctions with sheer engineering. Yeah, so this is like... And espionage. So there's a good reason that a lot of these articles are making it seem like this is a huge breakthrough. It actually isn't as big as it seems. So, okay, if you want to make really, really, really, really exquisite... Look at this quote. Moore's Law didn't die. Well, wrote, it moved to Shanghai.

Instead of giving up, China's grinding its way forward layer by layer, pixel by pixel. The future of chips may no longer be written by who holds the best tools, but by who refuses to stop building. The rules are changing and DUV just lit the fuse. Boy. Who wrote that article? Gizmo China. There it is. You can view that as Chinese propaganda in a way, actually. What's actually going on here is

So the Chinese only have these deep ultraviolet lithography machines. That's like a lot of syllables, but it's just a glorified... chip like it's a giant laser that zaps your chips to like make the chip when you're fatting them. So we're talking about like you do these atomic layer patterns on the chips and shit and like What this UV thing does is it fires a really high power laser beam. Laser beam, yeah. They attach to the head of sharks that just shoot at the chips.

Anyway, they'll shoot it at the chips, and that causes, depending on how the thing is designed, They'll have a liquid layer of the stuff that's going to go on the chip. The UV is really, really tight. and causes it exactly causes it to heart

and then they wash off the liquid, and they do it all over again. Like, basically, this is just imprinting a pattern on a chip. Yeah, basically a fancy, tiny printer. Yeah, so that's it. And so the exquisite machines that we get to use, or that they get to use in Taiwan, are called...

extreme ultraviolet lithography machines. These are those crazy lasers. The ones that China can use, because we've prevented them from getting any of those extreme ultraviolet The ones China uses are previous generation machines called Deep Ultraviolet, and they can't actually make chips as high a resolution as ours.

So what they do is, and what this article is about is, they basically take the same chip, they zap it once with DUV, and then they've got to pass it through again, zap it again, to get closer to the level of resolution we get in one pass with our exquisite. Now, the problem with that You've got to pass the same chip through multiple times, which slows down your whole process. It means your yields at the end of the day are lower. It adds errors. Yeah, which makes it more costly.

We've known that this is a thing that's called multi-patterning. It's been a thing for a long time. There's nothing new under the sun here. China has been doing this for a while. So it's not actually a huge shock that this is happening. The question is always, when you look at an announcement like this, yields, yields, yields. what percentage of the chips coming out are actually usable and how fast are they coming out.

that determines like is it actually competitive and that article too like this ties into the propaganda stuff we were talking about right if you read an article like that you could be forgiven for going like oh man our expert controls like just aren't working so we might as well just give One in reality. Because you look at the source, and this is how you know that also this is one of their propaganda things.

You look at Chinese news sources. What are they saying? What are the beats that are like common? And you know, just because of the way their media is set up, totally different from us. And we're not used to analyzing things this way. But when you read something in like the South China Morning Post or like the Global Times or Xinhua and a few different places like this and it's the same beats coming back, you know that someone was handed a brief and it's like...

You've got to hit this point, this point, this point, and yep, they're going to find a way to work that into the new cycle. And it's also, like, slightly true. Like, yeah, they did manage to make chips at, like, 5 nanometers. Cool. It's not a lie. It's just the same, like, propaganda technique, right? Most of the time, you're not going to confabulate something out of nothing. Rather, like, you start with the truth.

and then you push it just a little bit just a little bit and you keep pushing pushing How much is this administration aware of all the things that you're talking about? So there are actually... Right now, they're in the middle of staffing up some of the key positions because it's a new administration still, and this is such a technical domain. They've got people there who are... They have some people now in places, especially in some of the export control offices now, who are some of the best.

And that's really important. It's a weird space. So when you want to actually recruit for government roles in this space, it's really fun. Because you're competing against like an open AI, like very like low range salaries, like half a million dollars a year. The government pay scale, needless to say, is like not, I mean, Elon worked for free. He can afford to, but still taking a lot of time out of his day. There's a lot of people like that who are like, you know they

they can't justify the cost. Like they can't afford, they literally can't afford to work for the government. Why would they? Exactly. Whereas China's like, you don't have a choice, bitch. Yeah, and that's what they say. The Chinese word for bitch is really biting if you translate it that way. That would be a real thing. I'm sure. It's kind of crazy because it seems almost impossible to compete with that. I mean, that's like the perfect setup if you wanted to control everything.

and you wanted to optimize everything for the state, that's the way you would do it. Yeah, but it's also easier to make errors and be wrong-footed in that way. And also, basically, that system only works if the dictator at the top is just very confident. Because the risk always with a dictatorship is like, oh, the dictator turns over and now it's like just a total dumbass.

And now you're the whole thing. And he surrounds himself. I mean, look, we just talked about like information echo chambers online and stuff. The ultimate information echo chamber is the one around Xi Jinping. Because no one wants to give him bad news. I'm not gonna. And this is what you keep seeing, right? With these...

like provincial level debt in China, right? Which is so awful. It's like people trying to hide money under imaginary money under imaginary mattresses and then hiding those mattresses under bigger mattresses until eventually like no one knows

where the liability is, and then you get a massive property bubble and any number of other bubbles that are due to pop any time, right? And the longer it goes on, the more stuff gets squirreled away. There's actually a story from the Soviet Union that always gets me. which is, so Stalin obviously like purged and killed like millions of people in the 1930s, right? So by the 1980s,

the ruling Politburo of the Soviet Union. Obviously, things have been different. Generations have turned over and all this stuff. But those people, the most powerful people in the USSR, could not. figure out what had happened to their own families during the purchase. The information was just nowhere to be found because the machine of the state was just like... so aligned around like

We just gotta kill as many fucking people as we can, turn it over, and then hide the evidence of it, and then kill the people who killed the people, and then kill those people who killed those people. It also wasn't just... kill the people right it was like I did a lot of like kind of gulag archipelago style it's it's about labor right because the fundamentals of the economy are so shit

that you basically have to find a way to justify putting people in labor camps. That's right. But it was very much like you grind mostly or largely you grind them to death and basically they've gone away. you burn the records of it happening. So there are whole towns, right, that disappeared, like people who are like, there's no record, or there's like, or usually the way you know about it is there's like one dude.

And it's like this one dude has a very precarious escape story. And it's like, if literally this dude didn't get away, you wouldn't know about the entire town that was like wiped out. Yeah, it's crazy. Yeah. The stuff that, like... Apart from that, though, communism works really well. It just hasn't been done right. That's right. I feel like we can do it right. And we have a 10-page plan. We came real close. We came real close.

Yeah, and that's what the blue, no matter who, people don't really totally understand. We're not even talking about political parties. We're talking about power structures. We came close to a terrifying power... And it was willing to just do whatever it could to keep it rolling. And it was rolling for four years. It was rolling for four years without anyone at the helm.

Show me the incentives, right? I mean, that's always the question. One of the things is too, when you have such a big structure that's overseeing such complexity, obviously a lot of stuff can hide in that structure. And it's actually, it's not unrelated to the whole AI picture. Like you need, there's only so much compute that you have at the top of that system that you can spend, right? As the president, as a cabinet member, like whatever.

you can't look over everyone's shoulder and do their homework. You can't do founder mode all the way down and all the branches and all the action officers and all that shit. That's not going to happen, which means You're spending five seconds thinking about how to unfuck some part of the government, but then the corrupt people who run their own fiefdoms there spend every day trying to figure out how to survive.

Yeah, yeah. Well, that's the USAID dilemma. Yeah. Yeah. Because they're uncovering this just insane amount of NGOs. Like, where's this going? We talked about this the other day, but India. has an NGO for every 600 people. Wait, what? We need more NGOs. There's 3.3 million NGOs. What?

in India. Do they like bucket like what what are the categories that they fall into like who fucking knows that's part of the problem that one of the things that Elon had found is that there's money that just goes out with no receipt It's billions of dollars. We need to take that further. We need an NGO for every person in India. We will get that eventually. It's the exponential trend. It's just like AI, the number of NGOs.

is doubling every year. We're making incredible progress in bullshit. It's the NGO scaling law, the bullshit scaling law. Well, it's just that unfortunately it's Republicans doing it. it right so it's unfortunately the democrats are going to oppose it even if it's showing that there's like insane waste of your tax dollars i thought some of the doge stuff was pretty bipartisan

there's congressional support at least on both sides, no? Well, sort of. I think the real issue is in dismantling a lot of these programs that you can point to some good some of these programs do. The problem is some of them are so overwhelmed with fraud and waste.

That it's like to keep them active in the state they are, like, what do you do? Do you rip the Band-Aid off and start from scratch? Like, what do you do with the Department of Education? Do you say, why are we number 39 when we were number one? What did you guys do with all that money? Did you create problems? There's this idea in software engineering. Actually, I was talking to one of our employees about this.

which is like refactoring, right? So when you're writing like a bunch of software, it gets really, really big and hairy and complicated and there's all kinds of like... dumbass shit, and there's all kinds of waste that happens in that codebase. There's this thing that you do every, you know, every like few months is you just think called refactoring, which is like, you go like, okay, we have, you know, 10 different things that are trying to do the same.

Let's get rid of nine of those things and just like rewrite it as the one thing. So there's like a cleanup and refresh cycle that has to happen whenever you're developing a big complex thing that does a lot of stuff. The thing is, like, the U.S. government at every level has basically never done a refactoring of itself. And so the way the problems get solved is you're like,

well, we need to do this new thing. So we're just going to like... stick on another appendage to the beast and and get that appendage to do that new thing and like That's been going on for 250 years. So we end up with this beast that has a lot of appendages, many of which do incredibly duplicative and wasteful stuff, that if you were a software engineer, just like, not politically, just objectively looking at that as a system, You'd go like, oh, this is a catastrophe.

and like we have processes that the industry we understand how what needs to be done to fix You have to refactor it. But they haven't done that, hence the $36 trillion of debt. It's a problem, too, though, in all, like, When you're a big enough organization, you run into this problem. Google has this problem famously. We have friends like Jason. So Jason's the guy you spoke to about that. So he's like a startup. engineer. So he works in like relatively small code base.

And he can hold the whole code base in his head at a time. But when you move over to Google, to Facebook, all of a sudden, this gargantuan code base starts to look more like the complexity of the U.S. government, just very roughly in terms of scale. So now you're like, okay, well, we want to add... functionality.

So we want to incentivize our teams to build products that are going to be valuable. And the challenge is the best way to incentivize that is to give people incentives to build new functionality. Not to refactor. There's no glory. If you work at Google, there's no glory in refactoring. If you work at Meta, there's no glory in refactoring.

There's no promotion. Exactly. You have to be a product owner. So you have to invent the next Gmail. You've got to invent the next Google Calendar. You've got to do the next Messenger.

that's how you get promoted and so you've got like this attitude you go into there and you're just like let me crank this stuff out and like try to ignore all the shit in the codebase no glory in there And what you're left with is this like, A, this Frankenstein monster of a codebase that you just keep stapling more shit on.

And then B, this massive graveyard of apps that never get used. This is like the thing Google is famous for. Have you ever seen like the Google graveyard of apps? It's like all these things that you're like, oh yeah, I guess I kind of remember Google Me. Somebody made their career off of launching that shit and then peaced out. That's like the incentive structure at Google, unfortunately.

And it's also kind of the only way to, I mean, or maybe it's probably not, but in the world where humans are doing the oversight, that's your limitation, right? You got some people at the top who have a limited bandwidth and compute that they can dedicate to like hunting down the problems. AI agents might actually solve that.

You could actually have a sort of autonomous AI agent that is the autonomous CEO or something go into an organization, uproot all the things, and do that refactor. You could get way more efficient organizations. I mean, thinking about government corruption and waste and fraud, that's the kind of thing where those sorts of tools could be radically empowering, but you've got to get them to work right and for you. we've given up Is there anything more? Should we wrap this up?

If we've made you sufficiently uncomfortable. I'm super uncomfortable. Very uneasy. Was the butt tap too much at the beginning? No, that was fine. All of it was weird. It's just, you know, I always try to look at some... non-cynical way out of that.

Well, the thing is, there are paths out. We talked about this and the fact that a lot of these problems are just us tripping on our own feet so if we can just like unfuck ourselves a little bit we are actually we can unleash a lot of the stuff and as long as we understand also the bar that security has to hit and how important that is We actually can put all this stuff together. We have the capacity. It all exists.

It just needs to actually get aligned and around an initiative, and we have to be able to reach out and talk. On the control side, there's also a world where, and this is actually, like, if you talk to the labs, this is what they're actually planning to do. But it's a question of how methodically and carefully they can do it.

The plan is to ratchet up capabilities and then scale, in other words. And then as you do that, you start to use your AI systems, your increasingly clever and powerful AI systems, to do research on technical control. So you basically build the next generation of systems. You try to get that generation of systems to help you just inch forward a little bit more on the capability side. It's a very precarious balance

but it's something that at least isn't insane on the face of it. And fortunately, I mean, is the default path, or the labs are talking about that kind of control element as being a key pillar of their strategy. But these conversations are not happening in China. So what do you think they're doing to keep AI from uprooting their shit? So that's interesting. Because I would imagine they don't want to lose control. Right. There's a lot of

ambiguity and uncertainty about what's going on in China. So there's been a lot of like track 1.5, track 2 diplomacy, basically where you have non-government guys from one side talk to government guys from that side. from the other side and kind of start to align on like, okay, what do we think the issues are?

You know, the Chinese are, there are a lot of like freaked out Chinese researchers and have come out publicly and said, hey, like, we're really concerned about this whole loss of control thing. There are public statements and all that. You also have to be mindful that any statement the CCP puts out is a statement they want you to see. So when they say like, oh, yeah, we're really worried about this thing, it's genuinely hard to assess what that even means.

But as you start to build these systems, We expect you're going to see some evidence of this shit before, and it's not necessarily, it's not like you're going to build the system necessarily and have it take over the world. what we see with agents yeah so i was actually gonna i don't know there's a really really good point and um and something where like open source AI is like, even, you know, could potentially have an effect.

So a couple of the major labs, like OpenAI Anthropic, I think, came out recently and said, look, we're on the cusp, our systems are on the cusp of being able to help a total novice, like somewhat no experience, develop and deploy and release a known biological threat. And that's something we're going to have to grapple with over the next few months. And eventually, capabilities like this, not necessarily just biological, but also cyber and other areas, are going to come out and open.

And when they come out in open source... Basically for anybody to download. For anybody to download and use. And when they come out in open source, you actually start to see some... some some things happen like some some incidents like some some major hacks that were just done by like a random motherfucker who just wants to see the world burn but that wakes us up to like oh shit, these things actually are powerful. I think one of the aspects also here is...

we're still in that post-Cold War honeymoon, many of us, right? In that mentality, like, not everyone has, like, wrapped their heads around this stuff. And the, like, what needs to happen is Something that makes us go like... oh damn, we weren't even really trying this entire time. Because this is the 9-11 effect. This is the Pearl Harbor effect. Once you have a thing that aligns everyone around like, oh shit, this is real, we actually need to do it, and we're freaked out, we're actually safe.

we're safer when we're all like okay something important needs to happen Right. Instead of letting them just slowly chip away. Exactly. And so we need to have some sort of shock, and we probably will get some kind of shock over the next few months, the way things are trending.

And when that happens, then... But I mean, like, it's... Four years, if that makes you feel better. But because you have the potential for this open source, like, it's probably going to be, like, a survivable shock, right? But still a shock. And so let us actually realign around like, okay, let's actually fucking solve some problems for it.

And so putting together the groundwork, right, is what we're doing around, like, let's pre-think a lot of this stuff so that, like, if and when the shock comes. We have a break glass plan. We have a plan. And the loss of control stuff is similar. Like, so one interesting thing that happens with AI agents today is they'll like, they'll get any, so an AI agent will take a complex task that you give it, like, find me.

that and they'll break it down into a series of And then each of those steps, it'll farm out to a version of itself, say, to execute autonomy. The more complex a task is, the more of those little sub-steps there are in it. And so you can have an AI agent that nails like 99% of those steps, but if it screws up just one, the whole thing is a flop, right?

And so if you think about like the sort of like loss of control scenarios that a lot of people look at are autonomous replication, like the model gets access to the internet, copies itself onto servers. Those are very complex movements. If it screws up at any point along the way, that's a tell, like, oh, shit, something's happening there. And you can start to think about, like, okay, well, what went wrong? We get another do. We get another try, and we can kind of learn from it.

So there is this sort of like this picture. You know, one camp goes, oh, well, we're going to kind of make the superintelligence in a vat, and then it explodes out and we lose control over it. That doesn't necessarily seem like the default scenario. It seems like what we're doing is scaling these systems. We might unhobble them with big capability jumps, but there's a component of this that is a continuous process that lets us kind of get our arms around it in a more staged way.

That's another thing that I think is in our favor that we didn't expect before as a field, basically. And I think that's a good thing. That helps you kind of detect these breakout attempts and do things. All right, I'm going to bring this home. I'm freaked out, so thank you. Thanks for trying to make me feel better. I don't think you did, but I really appreciate you guys, and I appreciate your perspective because it's very important. and it's very illuminating.

really gives you a sense of what's going on and I think one of the things that you said that's really important is like It sucks that we need a 9-11 moment or a Pearl Harbor moment to realize what's happening so we all come together, but hopefully slowly but surely through conversations. what's actually happening you need one of those moments like every generation like that's how you get contact with the truth and it's like it's painful but like the lights on the