#190 - AI scaling struggles, OpenAI Agents, Super Weights

And now a quick preview of what we're talking about in this episode in tools and apps, nothing huge, uh, some sort of, uh, Previews of what's coming more or less and applications in business. A lot of stuff on hardware and data centers is the focus. Got a few exciting open source things, including AlphaFold 3. A few pretty nerdy research stories, I'm going to say. As opposed to our normal jockey fair.

Bobsend is a number one school 30 years now. And last fall, professors from all across Bobsend partnered with students to launch this thing, the generator. Which is a lab organized into eight different groups, including things like AI entrepreneurship and business innovation, AI ethics and society, a future of work and talent, and so on. So this group has already peer trained a lot of the faculty of Babson on AI concepts and AI tools.

On the website, they say that the generator accelerates entrepreneurship, innovation, and creativity with AI. And they are fans of a podcast and we thank them for the sponsorship. You can go to their website if you want to hear more about it, uh, or just, you know, keep an eye out for any news coming out of them. And moving right along, getting started with news and in tools and apps, we are actually going to start with a sort of follow up to something we discussed last week.

Another article has come out in this kind of ongoing conversation around seeming slowdown potentially in AI development. So this one is OpenAI, Google, and Anthropic are struggling to build more advanced AI from Bloomberg. Nice overview article of the topic. And last week we discussed a little bit.

Last episode, I should say, this might come out a little bit closer, but we discussed how the new model from OpenAI Orion is seemingly not hitting desired performance goals, according to, in this case, two people familiar with OpenAPI. So, uh, We have a matter who spoke on a condition on anonymity. And this article also mentions that people from inside Google have said sort of similar things that the next iteration of Gemini is not living up to internal expectations.

And also that, uh, inside on Froppik. There's been a challenge with 3. 5 Opus, and in all of these cases, they just refer to people familiar with the matter. So in all of these cases, what seems to be happening is that you train these larger models, they perform better on variations. By not as much as would be expected. And this has led to a lot of conversation in the AI community. You've seen a couple of people being like, I told you, so everything is slowing down.

Gary Marcus has always loves to save us. And Yann LeCun also kind of. Posted something to that effect. We discussed it a little bit last week, and I think we can talk about it a bit more. My take is as before that perhaps it's not entirely surprising that to pure scaling approach of a bigger model, more data, more compute is challenging to keep going. We know that.

Seemingly the scaling laws we have haven't broken, so you're still getting the expected improvement in perplexity and being able to predict probabilities of words or letters. But in terms of how it translates to actual kind of intelligence performance on benchmarks or just being smarter, that's harder to say. And in some sense, you know, even according to the scaling laws, we know that as you get better and better, as you scale up more and more, you get something akin to diminishing returns.

It's harder and harder to get the same amount of improvement. Uh, you need to, you know, keep, uh, expanding to bigger and bigger sizes by orders of magnitude, roughly speaking. So we shouldn't be red. We, we take away, I think it shouldn't be read as we're hitting a wall as like AI improvement is going to slow down necessarily.

What this is implying is that It might be at a point where it's challenging to keep improving at the pace we've been by just doing scaling, which I think it's not entirely surprising. I mean, at some point, this seems like it would have happened sooner or later.

So you have, for example, um, the, uh, essentially the, the, the performance, um, you have to, yeah, like you said, keep exponentially increasing the amount of compute exponentially increasing the amount of, um, of data that you, you feed into these models to continue the same, um, Uh, I mean, the same linear trend performance, but, but this is the, this is the problem, right? What does it actually mean for a model to get, you know, X percent better at autocomplete?

That's what we're talking about, right? That's the thing that scales is like roughly how good this model is at predicting the next token if you're talking about an LLM. How that translates into actual concrete performance at tasks that we care about is the big question. In some sense, it's always been the big question. It's just that that mapping has historically been very tight.

So as you see the model get better, you know, GPT 1 to 2 to 3 to 4, you've consistently seen better next word prediction accuracy translate into greater general capabilities, including even agentic capabilities with GPT 4, right? And that was kind of insane, right?

Like, this idea that You get really good at text autocomplete and at a certain point that gives you agentic reasoning capabilities would have been insane Just like a year before it seems to be true So one of the big questions is as we do this, are we really building a world model that is more usefully robust? That allows for the building of better agents keeping in mind that We've now moved beyond that, that just pure scaling LLM paradigm.

We're now wrapping things up in, obviously there's post training, which includes now reinforcement learning specifically for, um, agentic behavior. And we have now all these inference time scaling laws that give us a whole bunch more and they compound multiplicatively with training time learning. Uh, scaling laws. I think that's kind of the right way to think about it. So you can tap one out, but not the other. And you can still continue this trend.

So there's a lot of uncertainty, I think in, in general, but when you talk to people at the frontier labs, um, I don't think anyone is expecting a slowdown. In fact, I've heard quite the opposite, right? One of the big themes is we're seeing AI be used to automate AI research itself more and more kind of closing that final feedback loop that gets you to like fully automated research. Um, so, so yeah, I mean, I think there's so much nuance here, uh, difficult to unpack in one episode.

In fact, I think we could almost do, we have a hardware episode we have to do. We'd almost do a scaling law episode and the path to kind of, uh, ASI. But I think, um, yeah, I think fundamentally the big blockers now are starting to look industrial, right? The industrial base is struggling to keep up with the energy demands of this kind of scale. It's hard to find that, you know, 500 megawatt. spare grid capacity that you need for that next cluster, the one gigawatt cluster.

Um, you know, five gigawatts seems out of reach. So, so pretty soon the scaling runs of 2027, 2028 are looking challenging to pull off. And we'll be talking about that as well today, but I think that's, uh, you know, all these things exist together and it's not enough. For just scaling to get you there because it's not economically viable, uh, from an energy standpoint.

And that's what has happened over the last couple of years is we've seen a few demonstrations of it. And then in a year, in a few months, in half a year, in the case of Uh, text to video, you know, we, we see more and more like there's kind of an avalanche of the trend realizing itself, and that is definitely what's going to happen next year with agentic AI. We're going to see more and more of it coming out, uh, as we'll actually talk about in a bit.

So in that sense, there's not going to be any slow down. We're going to see new AI tools that can do even more, even regardless of whether we get these larger models that are even smarter. It is a very interesting question whether we can also scale up the models and get a major improvement in performance. Uh, I think There's a real question there on, for instance, not just about the scale of a model, but maybe the data, right?

We haven't used up all the data per se, but we know that quality of data matters in addition to quantity of data. And it may be that you've sucked up all the news articles, you've sucked up all of Wikipedia, you've sucked up all of Stack Overflow, you know, how much is there left? of the good data, so to speak, right? Uh, so I'm sure we'll be seeing some research hopefully coming out and exploring these topics of, is there potential for a plateau in the scaling laws?

Because again, also worth mentioning as the CEO of Entropic recently mentioned on a podcast that These are empirical laws, right? So these are not sort of physical laws. There's no theory behind them to my knowledge. So we may in practice see some sort of plateau and that the laws don't go infinitely. Uh, and that'll be interesting as well.

The thing that the people are working on that is the toughest nut to crack, um, for an awful lot of like long term reasoning tasks that you're trying to solve with these systems, the ones often that deliver the most value. It's not good enough to have an agent that gets the steps right 99 percent of the time, because you're going to have to string together, you know, dozens and dozens of steps. So on average, you'll expect the thing to go off the rails.

Uh, so that last mile problem, multitask coherence is so, so important and, and that's the, the thing that's challenging to train for right now because there's so few examples of long term reasoning traces that you can actually train on to, to speak to your Andre, your example of, or your, uh, um, mentioning of the data wall issue. Um, there, there are potential ways of around this, uh, synthetic data actually does look quite promising for this.

You can have AI systems, audit, reasoning, traces, generate and audit, generate and audit in a sort of, um, AlphaGo style approach that's been, been tried and being tried. Um, and I think there's some promise there, but anyway, yeah, I think we need to do a scaling law episode.

And we'll be able to use a computer to take actions on a person's behalf, very much similar to what Anthropic launched recently on the API. This is according to people familiar with the matter. Uh, that was all over this. I know the people familiar matter of a real sources of news, apparently. Uh, so there was a staff meeting, uh, just last week and apparently opening as leadership announced plans to release a tool in January as a research. preview and through the API.

So, seems like it is coming relatively soon, at least if these plans come through and wouldn't be surprising given that Entropic already launched this in their API.

Um, you know, like web browser tool could be, um, you know, task automation, all the usual stuff. I think right now, Um, the best way to think of these is as just experiments in, uh, in long term task coherence. The problem that we just talked about, Ani, with that last mile thing, I think is going to be the biggest challenge going forward for this.

And so this new experiment, Gemini, whatever it is, has much related to GPTO 4. 0 and outperformed OpenAI's O1 preview. Although, again, this is on which model people like better, which is sometimes kind of hard to say how that really, uh, translates to intelligence. And, uh, you know, so it's now alongside, uh, the OpenAI models, also Grok2 is up there, uh, as well.

Um, this is something that, uh, actually, yeah, you know, Dario mentioned the difficulty on his podcast with Lex Friedman of, of kind of naming these models. And I actually remember talking to a friend who told me like months and months and months ago, he's like, Oh yeah, so OpenAI is training GPT 5 right now. And like, here's, you know, here's some of the stuff. Some of the stuff that we know, and then it turned out to be a different model.

Like when it was released, the name just, just changed. And so these sorts of things just happen all the time. I think, uh, one, the thing this is getting at, and I think the thing to anchor on is like, what is the compute cluster that was actually used to train these models, right? When we're talking about Gemini 1. 5, Gemini 1, Gemini 2, Uh, the thing that really matters is, is like, is scaling working or is the next training paradigm working?

Um, it's possible that Gemini two is a fundamentally different training paradigm. So that would be a very interesting distinction. So these increments can matter, but the fundamental thing that makes the matter is the size of the training cluster, the amount of flops that go into it, um, or the, the bells and whistles that are added during training. And here, you know, It's very unclear what the answer to that is.

Um, so we'll, we'll just have to sit back and wait really for more information, but I do like the industry's trend towards dropping these mystery models in the middle of like some, some, some leaderboard seeing people freak out. And then later you find out, like, sometimes you don't even know which company, right. That has happened But,

So yeah, part of a trend that we've seen multiple tools that take an image and then create a video out of it. Sometimes they kind of continue and generate a video that looks like what would be the continuation of the image. Here's another example of a tool launching with something like that capability. And now to the last story, we have a Forge Reasoning API beta from Noose.

So we chatted about Noose Chat and Hermes Free in the last episode, and they are also announcing this Reasoning API, which allows you to query an API like you do with an LLM, but built into it is a bunch of the known techniques for reasoning like tree search, chain of code, mixture of agents, et cetera. It is now available in beta to just a few users and I've seen some examples of people trying it out.

It does seem to empower kind of weaker models like Lama free, for instance, with these, uh, improved techniques to do better reasoning, uh, in a way, again, showcasing the trend towards both reasoning and agentic AI, which kind of go hand in hand. Yeah, it's actually

So that is outperforming Gemini 1. 5 pro, um, and GPT four, obviously, but, uh, as well as on a 3. 5, like, like this is. It's pretty insane. And you see sort of competitive numbers, maybe a little bit less performant, but competitive on GBQAD. Um, MMLU Pro as well. The AMI benchmark is the, the like weirdest one from what I'm seeing here.

Like there, I mean, it's tough because it's a very small benchmark that they don't show it here, but like there, there aren't a ton of, um, uh, of, uh, kind of samples in the, in the benchmark. Um, but it's outperforming nominally, even O1 released. I want to look in, I'd love to have more information about specifically. Like the eval part of this story, but.

They are using Monte Carlo Tree Search, um, and uh, sort of chain of, chain of code, sort of like chain of thought, uh, specifically when you connect your reasoning trace to a code interpreter, so you can get actually grounded feedback, uh, that doesn't, that doesn't hallucinate. So every once in a while you kind of get that, that clarity in your reasoning trace. Um, and then they also have a way of, of setting up kind of, uh, querying between multiple agents.

They call it MOA or mixture of agents. So it's this compound framework. Uh, Monte Carlo tree search is probably part of what's going on in the Oh, one set up is at least my guess for now, though, honestly, no one really knows. Uh, and it could be anything, but, uh, these are the sort of intuitive things you might try. If you're trying to replicate that, it seems superficially like.

They have, um, but again, I like, I just want to see a lot more information about the, um, about the actual eval side of things, uh, you know, what, what actually like on, on the Amy benchmark, I'd love to see, you know, the reasoning traces and all this stuff and qualitatively, it'd be really interesting to compare to the few, uh, Oh, one reasoning traces that we actually do have because, um, yeah, be, be curious to see how that, how You know, how those two compare.

And after applications and business, first up, we have OpenAI discussing an AI data center that could cost a hundred billion dollars. dollars. So this is according to them having shared information with the U. S. government officials about these potential plans. Apparently this would be five times larger than any data center that's currently being developed.

So the, uh, OpenAI is a top level Policy executive Chris Lehane said this at an event in Washington, that this company has shared information about the potential impact of a data center like this. And we really don't know too much more. We know what they're kind of talking to a federal government. They are calling on it to expand the energy grid to be, to enable these kinds of things to happen. Uh, and they are suggesting various things like, uh, speeding up with permitting.

process for AI data centers like this.

This is for, uh, by the way, it seems to be for a five gigawatt, um, cluster, which would be probably the Stargate cluster, right? The partnership with Microsoft and Open AI. I don't think they say it explicitly, but that that's what it seems to be. Um, you know, like one of the things, if you, if you're looking at this through a kind of Washington national security policy lens, like great, like good shit.

We definitely want this in the United States, but um, you can probably think about tying, um, The massive package like this to requirements on the part of the labs that use this infrastructure to adhere to certain security standards, right? So there are a whole bunch of really I'm saying this because we're working on an investigation right now of like lab security and all that stuff.

Um, there, there's some, some basic stuff that I think open AI is like keen to rush this through and be like, yeah, like give us goodies for free basically. Or, you know, well, somewhat for free. Uh, you know, we'll, we'll let some spillover happen and academic institutions will be able to access compute and stuff like that. But fundamentally, um, you know, just go and do the thing. I think this needs to be tied to like some pretty intense conditions.

If we're starting to think of, of, uh, AI as a national security technology, like the security situation is just abysmal right now and kind of needs to be. needs to be fixed up. So for 100 billion compute cluster, I think, uh, you have five gigawatts. This is, yeah, it's a behemoth. Like there's, there is not five gigawatts of spare capacity anywhere in the grid. Um, you know, every company I've talked to is like, yeah, you know, right now we're at most, we're, we're thinking about one gigawatt.

Realistically, we're in the few hundred megawatt range for stuff that's up and coming. So somehow that's got to change if we're going to be competitive with the CCP. And the best way to do that is obviously a big infrastructure buildup. The question is just, how do you tie incentives to use of that infrastructure?

And so here with this approval, it seems that they are making progress towards being able to use all of that concurrently. GPU power, which, um, you know, is, is still, there's still a lot of, uh, infrastructure considerations there. It may not be the case that you would even want to run all 100, 000 GPUs concurrently, but, uh, certainly now they have the option. This is the, uh, the big

And, um, ridiculously, ridiculously fast that I've seen stories by the way of like competitors flying planes, like, like Cessna's or whatever, over the site, just to understand how the hell Elon pulled this off. That's like the level of freak out that we're at right now in the, in the space.

And, um, uh, what Elon did by the way is like, because initially there was only eight megawatts available of energy on the, at the time of the opening of the data center, July, uh, he set up portable power generators to kind of bridge the gap. And there are a whole bunch of really interesting tweets. If you're into the hardware side, um, like check it out. I thought they were really cool.

Uh, of, of these sort of like new XAI employees coming in and saying, yeah, you know, he brought these in from Tesla and you use them to bridge the energy gap there. Um, and right now, so the big question is, you know, how, how far can this go? There are plans to double the capacity beyond 150 megawatts, um, which would, you know, further increase the GPU capacity. 150 is enough for 100, 000. Um, uh, H one hundreds. Um, and so anyway, uh, this is going to be a really, really big facility.

Obviously currently he's referring to it as the largest cluster in the world. Um, it wasn't the largest fully powered cluster in the world. Uh, but, uh, but it's, it's now truly going to be on its way to that.

So in Google's case, their sixth generation TPU Trillium has shown a 3. 8, uh, fold, uh, performance improvement about four times better. And the B 200 GPU seems to be about. twice better over the H100. So in both cases, it seems we're not hitting the kind of ceiling on improving chip performance. We are still doubling or even quadrupling in performance with each generation of chips.

Can we squeeze out of the grid power that we have? Um, increasingly that's the bottleneck. And so, um, really important, you know, Google starting to look very prescient setting up their TPU program. Like, I don't know, like I can't even remember 2016, 2015. Yeah. Yeah. I mean that, that was like back then it was sort of wild. And now, you know, we're on to TPU V5, we're on to, you know, Trillium and all that stuff. Um, and really paying off.

One of the things that Google is especially good at is the multi data center. Compute game plan, like basically you, and it's relevant, right? When you run out of power, because if you don't have enough grid power, that is enough grid power in one location, because that's the real challenge. Then the question is like, all right, well, I guess I'm going to have to set up data centers in different geographic locations and have them work together.

And that means I need to be able to do distributed training across a like a large set of data centers. Um, that's something that Google had cracked early and hard and they've got great papers. We've talked about some of them on the podcast, um, but, but don't sleep on Google on that. They're really good at design and they're really good at data center design. These are not like crazy geographically separated by the way.

They're in the same general area for, for technical reasons, uh, that are a little bit challenging to overcome right now. But But, but still, uh, this is a big competitive advantage. The other piece on scaling. So the, the, the new TPU, the Trillium, um, can link up to 256 chips in a single pod in a single high bandwidth pod. So we're talking about high bandwidth.

Again, this will be punted to our hardware episode, but, um, the GPU to GPU communications, the things that like on NVIDIA devices would be done over, uh, like NVLink. So NVLink is like the, yeah, GPU to GPU, super, super high, um, high capacity, uh, cables. So usually you'll see like, I don't know, 36, 72 GPUs in a pod. This is 256. And, um, and so, so really, really expandable as well beyond that. So very much designed to be super, super scalable.

Uh, and, and they've got all kinds of infrastructure for it. So yeah, this is, I think Google is one to really watch. They're, they're the, they were the sleeping dragon that got woken up by chat TPT. And, uh, you know, now they have actually quite a strong compute advantage over Microsoft and open AI.

So here we know that Miana Chen, a research program manager, He's joining this new company, apparently also ex head of post training, Barrett Zoff and former senior researcher, Luke Mance, uh, who also left OpenAI in recent months are teaming up on this. So no details beyond that, but seems like, uh, yeah, a lot of talent being, uh, Combined to this new company.

Uh, in Mianna Chen, and they've got the more kind of pre training, um, obviously like Beretzov actually was, was the head of post training previously at OpenAI, but certainly has the, the pedigree to do, um, to do pre training stuff. So does Luke Metz, um, who's a sort of senior researcher. So anyway, uh, it'll be interesting to see. I think maybe it's just another, another AGI play. I would not, I'd be a little surprised if it wasn't actually.

It seems that this time, though, there might be a caveat to that, where Anthropic would need to use Amazon chips specifically to train their models. And these chips aren't NVIDIA, so that could pose a challenge. technical challenges, these training chips, apparently. So again, not too much known. This is just internal discussions, but, um, could be interesting to see if this does come true.

So essentially what you're doing is you're delaying the discussion of what the valuation will be until there's a price ground later. This usually happens earlier stage because it's really hard to value a startup early on. You just have a couple of founders and an idea. Um, so yeah, not sure like exactly why the structure of the deal takes that form. It sort of reminds me of the safes that, uh, Y Combinator or that anyway, uh, Angel investors use a lot.

Um, but in any case, I think this is a really interesting deal to watch. The, the, uh, strings attached obviously are highly strategic for Amazon trying to force Anthropic to use their hardware. They're also their alternative to CUDA too. That's going to be really important because Amazon has an awful lot of catching up to do, right? They were caught flat footed. There's no other way to put it on the whole AGI scaling race.

And now they're trying to catch up, you know, they're, they're not developing their own models are really leaning a lot on, uh, on Anthropic to provide those. And now they're trying to force Anthropic to generate demand that can help drive the improvement of their own hardware. Um, I think this is a really interesting one to watch to understand how stable long term stable the relationships between hyperscalers and model developers like the opening.

I Microsoft relationship can be in the long run, right? Microsoft opening. I were already seeing cracks in that relationship. Um, to the extent that anthropic is now being sorry. The Amazon is now being kind of forced by business by business pressures to Google. require anthropic to ask anthropic to use their stuff more and more to the point where maybe it won't be workable. Uh, it kind of makes you wonder about a lot of these deals.

So, um, yeah, this is maybe a canary in a coal mine, but it has been a very fruitful partnership for the two. We don't know the details. Um, I think the big question is just going to be, like, what specifically are the requirements? What are the strings attached here? How much, uh, trainium, traininum? Traininum. Traininum. Come on, guys. It

Obviously, this was a major improvement in the ability to model proteins and in Being able to apply it for things like scientific discovery, drug development, it was not open sourced when it was announced, uh, and sort of, uh, seems like, uh, kind of out of nowhere this suddenly came out. Yeah. I

The partnership with deep mind and isomorphic labs is really where a lot of this stuff is being monetized on the, like, I don't want to call it Google end. I'm not actually sure exactly. How the ownership structure works there, but they are kind of partner organizations in some sense. Demis Asabis actually helps, uh, isomorphic labs a lot as I think an executive. Um, but yeah, so, so, you know, they've got a, a diffusion based approach.

Anyway, the whole thing about this is it's not just about modeling the proteins. It's about modeling the interactions between proteins. you know, proteins when they're modified and ligands and things that bind to proteins, um, which, which is actually very interesting, right? When you think about medical impact, right? You're often concerned about how will these two things actually interact. That's the only way you can make an effect happen in the human body.

And so this is really where qualitatively alpha fold three is, is head and shoulders above alpha fold two. Um, so it'll be interesting to see if, if there's actual tangible impact that comes from this, there has been some from alpha fold two, but I think it's fair to say. a bit less, a bit less than people expected when I first came out. So we'll see if that changes here.

If you're a university, nonprofit organization, research institute, you can use this, but they highlight over and over, uh, that this is not commercial use and in fact, there's some sort of like a stamp, some sort of marker in the weights related to reform submission, which is interesting. Uh, but certainly good news for scientists, for researchers. Um, And I think they did share this, uh, previously already in a sort of closed off process. So this is expanding, uh, the access to it.

And anyone can look at the source code for inference. So either way, even without two weights, this can inform people as to how to build these sorts of models. So does that imply they're

They want to have thousands of contributors and for now you're able to start contributing to the training of what they say is a small 500 million parameter model and they're kicking this off today so very hard to say if this will come to be 1. 4 trillion Parameter's model is like three and a half times bigger than meta's biggest llama model. Very, very challenging to train. Personally, I'm a little skeptical, but uh, will be cool to see if you can even get to a fraction of this.

It was a bit tensor.

This is a time where I was like, you know what, this actually makes some sense. So, um, the, the pitches, right. And this is not investment advice. Uh, the, the pitch is something like, um, it takes a lot of upfront capital to, uh, to train a big model, right? In this case, the, the biggest one they're contemplating training would be 1. 4 trillion parameters. It would be 160 million to train and so on. Um, so, so why don't we. Uh, create a new token, a new token for each model we want to train.

And what we'll do is we'll auction away that token to raise money for the training run. And then if you hold those tokens. You can use them later to buy cheaper inference, or should I say you can use tokens to buy tokens? Anyway, so the point is it actually kind of makes structural sense like this is not the craziest shit I've ever heard come out of the crypto space and The other interesting thing is we had a little clue like kind of glimpse into the technical details.

They say You know to do decentralized network to have this like decentralized network of compute, which is what they need to do To pull this off, you're not going to have tens of thousands of GPUs crammed into one place. You're going to need, as they put it, a new technology that doesn't exist today because all the distributed training techniques we have require very fast interconnect. Okay, that's true. But he added that emerging research from DeepMind suggests that's possible.

Now when you talk, when you think about, okay, DeepMind and distributed compute in this way, the first thing that comes to mind to me is DeLoco. We've talked about that quite a bit.

I'm trying to remember if it was Noose or, um Another company that, uh, that's doing an open source kind of version of this to rely on that infrastructure were getting better and better at this sort of distributed training, um, doing it for really large scale training runs, even across like getting multiple data centers, multiple clusters to work together. Really, really hard. Um, so, you know, I think DeLoco, if that's part of the plan here is only going to be part of the plan.

Uh, there has to be some other solution hearing that goes on, but anyway, I flagged it cause I thought, you know, it's a pretty wild sounding story, but it's actually just. Sensical enough that I'm like, you know what? This is not the craziest shit I've ever heard uh, when it comes to using crypto, like the intersection of crypto and AI. So, so there you have it.

And what they say in this paper is that certain weights In an LLM, what they call superweights are kind of massively important. So we know that weights, the parameters of a neural net vary in importance. You can prune, you can set a bunch of them to zero. And it doesn't really affect your performance. And that's in fact how people kind of scale down models, make them more efficient, compress them. A lot of it is by finding weights that don't matter and then killing those off.

And what they show in this paper is that. You know, we know that some weights are important and you don't want to kill them off. But in fact, there are like these super weights that are even more important. That if you just, uh, zero out this one weight, literally, that leads to a massive drop in performance.

And that's like, If you just remove 7, 000 of the other largest weights, the weights that, you know, uh, contribute to the activations of a neural net, that isn't as important as this one weight. So you can say this one is like more important than thousands of other weights that also matter. What I found interesting here is this builds on previous research from earlier this year.

I actually was not aware of, and I don't think we talked about, there was this paper titled Massive Activations in Large Language Models that had already demonstrated that there are these, you know, massive activations in this paper, they call them super activations, that again are these outputs in the internals of a large Language models. So activation is just like the output at a given place in a neural net.

And so there it's been shown this past year, really like months ago that you have these, and it seems that these superweights play into that. Uh, so lots of kind of interesting ideas here. I've kind of surprising to find this out that there are these special weights that are massively important. Uh, don't think. I've seen anything that hinted at that so far.

So. But let me take a step back here and just say, when you have a transformer, right? The transformer is made of blocks and the blocks are all stacked together. And then each block has two different kinds of layers. You got like a self attention layer, usually at the, at the beginning, and then an MLP layer, which is basically just like a, just a vanilla neural network that follows and kind of massages the data that comes out of the self attention layer.

And if you zoom in on that MLP layer, that second layer, uh, there, there are two steps in, or a couple of different steps in that layer. The first is taking the kind of low ish dimensional outputs of the attention layer and mapping them up to higher dimension. Okay. So, so you move them up, maybe you have like five, a 512 dimensional layer. You, you, you know, amplify it to 2000 dimensions say, and Then you can kind of in that higher dimensional space, you'll massage it in a way.

This is sort of like allowing your, I don't know, your papers to spread it on your desk more so you can work on them better. And then you do a down projection to recompress And what they found was that superweights consistently appear in the down projection part of these MLPs. So they keep showing up. They keep showing up in the, the transformer blocks MLP layer.

And specifically in the part of that layer that takes you Not that doesn't blow up the dimensionality, but that compresses the dimensionality after the number mixing has happened. And so it's sort of interesting, you know, why that exactly happens a super weight. Again, it isn't necessarily the largest weight by magnitude even at that layer. Um, and another kind of indication as to how this is working, these tend to happen very early on.

So in very early kind of early transformer blocks of the model, what they'll do is, uh, they'll actually kind of look for. really high activations that persist through the, all the, the, the layers of the model. And once they find the first instance where that happens, they'll kind of trace it back and try to mess with the weights until they, they make it disappear, thereby identifying this sort of superweight.

Um, the effective way that this, or the way that this happens in practice, the, again, the mechanistic side is, uh, the superweights seem to suppress stop word probabilities. So there's certain stop words. Um, like the, like the period character or a comma character that will cause the, the model to, you know, stop generating outputs. And, um, anyway, so, so the, the super weight seems to essentially, uh, suppress those stop words. So cause the model to keep generating outputs.

And if you knock it out, not only do you see that change, you actually, you know, see the quality of your output, just go to shit. Like you'll see some of the outputs that they, they show with and without the super weight. And it's like, it's from beautiful, coherent text to complete garbage. And, um, anyway, I thought, thought that was really, really interesting.

The super high activation, the super activation that you get the first time the super weight, you know, takes its effect is actually persistent to across the layers of the transformer. And so you end up seeing it persist through if you're. Yeah, tracking it higher level detail through skip connections, but it keeps appearing over and over in this kind of robust way. And so I think there's a lot to chew on from a mechanistic interpretability standpoint on exactly how and why this is working.

Um, but, but like you want, like, I was not tracking the, uh, the first. Uh, super activation results. So we're, we're sort of, it's, it's nice. Cause we're able to get a little bit more of an explanation now at the same time. And, and this is research as well coming out of Apple too, which, you know, not, not normally to me known as a big interpretability powerhouse.

And just to give a sense of the impact, they highlight one example where they say, Summer is hot, winter is X, and so, you know, a normal LLM would say winter is cold, versus if you remove it, it would say winter is Vuh, um, and these are the probabilities. The cold will have a high probability in a normal LLM, and then the probabilities will be more spread out. And you just save, uh, so not too much more information or interpretation of what this is.

What they also do in a paper is, uh, kind of on a more practical side of things, uh, explain how, if you do a super outlier aware quantization, as they call it. So if you quanta. when you quantize, when you lower the resolution of weights that affects your activations across the neural net. And it's already been shown that you can do better quantization by kind of keeping an eye out for important weights.

So in this case, what they show is that if you are careful to preserve these particular activations, these particular weights, that leads to a much less drop off in performance. So also from a practical perspective, this is very useful to know for being able to reduce the size of models while not losing performance.

To, to my mind, and I'll, I'll close with this, um, the weirdest figure in the paper by far. And, and this is like, again, things I never would have bet on in a million years. There's figure six. Um, what they show is if you take the superweight and you scale up its value for some amount of scaling up for some amount of scaling up, you will actually see the zero shot performance of this model on certain tasks.

Consistently go up so across a bunch of different model sizes They tried this and they consistently found there's some scaling where you get improved quality Like basically just like take this weight that's already been trained increase its value and that's consistently a good move That makes no sense to me. That seems weird.

Um, I mean, maybe you could argue that I guess through regularization or something there, I'd have to think more about their, the training scheme, but maybe there's some like kind of regularizing pressure that artificially dampens the weight relative to what it might've been. Otherwise it's not elastic enough or something. But, um, but still I find this like fascinating and I wouldn't have expected it.

So a lot of weird quirky results here that I think, you know, The mechanistic interpretability people should take a look at because pretty cool.

If your model can, let's say, create a square and create a rectangle, an image, can it then do something like, I don't know, a triangle on top of a square to keep, uh, to give a very simple example of what you can, of course, uh, You know, come up with many examples of the sort. And the question is, how do these abilities emerge? How can you compose different concepts in your outputs?

And the details are a little bit nuanced, but at a high level, they say that the ability to generate samples from a given concept emerges depending on the actual data. and the process by which you generate data. And there is this sudden emergence on the ability to compose tasks, uh, to do well on tasks that require compositionality. So, to be honest, I haven't dug in deep enough to explain this in a ton of nuance.

Jeremy, I think you can probably do your deep dive explanation and do a better job on this one.

Like, where did that come from? Could we have predicted this, right? And a natural, um, model for the appearance of those capabilities is to say, okay, well, let's say like designing malware requires skills X, Y, and Z. Right? Um, the, and, and let's say that over the course of training, the model gradually becomes better at X, gradually becomes better at Y, but gradually becomes better at Z. And but, but to do the overall task, right, you have to do all those three things together.

If you have 80 percent performance at X, you're 70 percent at Y, 90 percent at Z, then your overall performance probably is going to be something like 80 percent times 70 percent times 90 percent because you have to succeed at each one of those together. In other words, multiplicatively to perform the dangerous capability properly.

That's kind of the threat model Um, I don't know if that's literally like, I mean, I don't think it's national security motivated, but I think that's probably the biggest implication, um, for national security here. And, and so, yeah, basically what they, they do is they make this really concrete. They take a diffusion model. So basically an image generation model and they get it to like generate different shapes, colors, and sizes of objects. Right.

So, you know, think like blue, small blue sphere. Right? Something like that. And they check like, okay, how performant is this at capturing the shape, the color and the size? Um, and what they'll do is they'll, they'll specifically not train the model on certain combinations of those features. So make sure that the never the model is never trained to make a large purple square. Right. Your large purple cube never, never has trained to do that.

Um, so you train it on a bunch of other combinations and then you check to see how well can it perform at that, at that new out of distribution tasks that it was never trained on. And it turns out that its success rate at that task is essentially multiplicative based on its success rate at the individual components, the shape, the color, and the size that went into it.

Um, they compare their Uh, sort of multiplicative model to an additive model that doesn't perform nearly as well at explaining the, um, the emergence of these capabilities. And, um, yeah, I mean, and this kind of makes sense.

Look, mathematically, uh, if you've got a background, like engineering, anything like that, uh, or physics, you, you probably know about the direct Delta function, um, how basically in the limit, if you have a large number of chained events that have to happen together with, with probabilities less than one. Um, what you'll find is like the, the, the output, your success rate is basically going to be zero almost all the time because you only have to fail at one of those things.

You only have to get zero at one of those things for the overall success rate to be zero, but there comes a point where all of a sudden you kind of crack the last nut. Right. You're like, you're, you know, 60 percent on one, 30 percent on all this stuff. Um, and there's, but there's one thing that's really holding you back and all of a sudden you crack it. And then out of nowhere, it almost seems as if in retrospect, you uncovered this, wow, this amazing new capability.

When in reality, that capability was actually a compound of many sub capabilities, each of which had to be chained together. And when you multiply a lot of numbers together, you tend to get zero. If those numbers are less than one, that is, uh, or between zero and one. you'll tend to get zero, uh, unless and until all those numbers kind of hit a minimum threshold. And, um, anyway, that's what this is all about.

It's in some sense, an obvious consequence of the same kind of math behind like the, yeah, the Delta function. Uh, um, uh, and, uh, I, I think it's just really interesting for that reason, but cool paper for, for, uh, AI security.

But they also do have more practical tasks where they look at Celeb A, where it's like a bunch of faces, there are attributes like gender, expression like smiling, and hair color. And in this example, you can actually look at the concepts, look at the performance on things like, you know, how well you're doing at, uh, faces of men and women, and you see something similar to that synthetic case.

So in fact, this could also help in practice to mitigate the biased outputs in image generation, just as one example.

But all that stuff is like, If you know the right metric to look for in retrospect, in practice, we get surprised. We train a new model and we go, Oh shit, we didn't expect it to be able to do autonomous cyber attacks. And it just is, um, yes. In retrospect, you can design a cyber autonomy benchmark that will retroactively explain how you got here. You need to design the exact right benchmark to elicit those capabilities. Um, this now helps to focus what that debate actually was all about.

Fundamentally, that debate had been about identifying the right set of capabilities that compound together to lead to the capability you're interested in. So the, the, the reason that emergence is an issue in practice is that in practice, we actually don't know the full set of capabilities that need to be strung together to To give us a certain kind of overall, um, dangerous capability or, or interesting capability.

And so I think it's really interesting through that lens, I guess a new language or a new way of thinking about that, um, I was going to call it that age old debate. I think it's a debate that's been going on for like two or three years, so maybe not.

So that means that you can use subsets of the entire network for certain kinds of things. And as a result, You use less overall computation while generally getting better performance by training more overall weights that are specialized. And one way you can apply this is when you're dealing with multiple modalities where you can have different modalities being routed to different experts.

And this paper is presenting a sort of, uh, uh, uh, kind of, uh, generalization or, uh, a very specific version of that for multiple modalities. So the idea of mixture of transformers is that when you want a multiple modalities, like images, text, and audio, what you do is you have quite literally separate transformers.

And when you have your input, You group with different modalities, you, uh, have attention over all those modalities in the sequence, but then you route each modality to its own little transformer. So you literally kind of split them up across different weights. And what they show is by doing this, of course, you get the benefits of mixture of experts by, um, doing overall less computation.

Uh, so, you know, those middle layers, uh, you have typically feed forward layers, things like that, that need to get the entire sequence, uh, so you, by treating each modality separately, the individual transformers in that mix can be smaller and have fewer layers. So they evaluate this idea versus a more traditional sort of mixture of experts and across a dense transformer and show a major speed up, a doubling of speed in the training while getting, uh, also a good performance.

So, uh, yeah, it's, it's really building on to explorations at how you do multimodality. There's kind of two predominant approaches and this is extending one of those types of approaches.

If there's one token that's, um, you know, like you interpreted visually say in another, that's, that's, interpreted text wise, um, you can still benefit from, from that, uh, interaction, uh, which is really important because often there's kind of emergent information that comes from say, the description of an image coupled to the image. The other advantage they have is so training stability, right?

One of the big, big challenges right now is you scale up these models enough and pretty quickly, uh, you run into issues with, with training stability, uh, you know, with getting, getting the loss to kind of stably, uh, drop over time. And this model, it's a mixture of transformers.

Um, because it uses simpler modality based routing, it avoids the additional complexity of having to learn the routing process in a mixture of experts, which is one of the things that makes training particularly unstable. And so, so essentially this is really helpful for scalability. Um, they actually show that. So they show that, uh, MOEs, in their experiments, um, get some diminishing returns at above a certain scale.

Um, and, um, and above 7 billion parameters in scale in their experiments, sort of more, more, uh, notably that stuff that I, I mean, you, you know, you can, you can come up with engineering solutions for a lot of this stuff, but the bottom line is, you know, mixture of transformers. is easier for them to use for these kind of the larger end of that scaling spectrum, so maybe an interesting option as well.

As an example, so they tackle that general topic and say that if you actually specifically provide the contextual attributes of, you know, why are you asking this question? What kind of background are you coming from, et cetera, that can lead to more reliable and kind of meaningful, uh, outputs.

And so it's a question of like, which models work best with ambiguity and which work best with, uh, with more, more context about the user and who's asking the question. And part of that is just like, what's the baseline behavior of the model, right?

If the model is more baseline skewed towards, uh, Um, responding to you as if you're five years old because of the way it was pre trained or fine tuned, then, then it's just going to be better and if you have a benchmark that is geared in that direction too, you'll find the model performance better as well. Um, so I thought this was kind of interesting as a bit of a flag, like some of our, our relative rankings.

It even makes you think about, Uh, for example, like ellipsis leaderboards and things like that a little differently, right? Because now you're thinking, well, okay, sure. On average, people give a higher ELO score to one model or another, but everybody is different. And so some models may actually just be better at meeting the needs, the explanatory needs of one kind of person, uh, rather than another. The, the upshot is you valuations to factor in context.

And they do find just more reliable, more robust evaluations come from doing that. Um, to prove that they actually kind of create a synthetic data set of, of queries. So they'll take the basic queries that they get from some standard, um, kind of QA benchmark, and then they'll just like automatically generate a bunch of context about who's asking the question. And then they'll feed that to the model and evaluate based on that context.

That results in a much more robust, um, set of responses that, that presumably are better evaluation of the model's actual capability to adapt to the needs of a particular user.

And this code of practice is answering a question for general purpose AI, which is things like chat GPT. How can you build these models while also addressing the potential risks? they create. And this article is basically making the case for that code of conduct. It sort of addresses, uh, the potential, uh, criticisms. So it says there are many stereotypes about Europe's approach to regulation.

And then it calls out that, although Many do criticize the EU for over regulation, for slowing down innovation, things like that. They argue that the code can create a flexible, targeted, and effective framework for ensuring thriving GPAI, general purpose AI innovation, while respecting the rule of law and the shared rights and values of the EU.

And they say that This has global significance as it is the first time that legal rules are turned into more detailed guidelines for the responsible development and deployment of general purpose AI. So not surprising I suppose that there was a sort of a code that was published alongside the actual law of the act. Code of practice is, is kind of suggesting how companies should, uh, act to Uh, follow the law of the EU regarding AI in general to develop models in a responsible way.

Yeah, it was sort of funny. I

It's stuff we've seen him advocate for before, you know, alignment with EU rights and values alignment with the AI act and international approaches, proportionality to risks. This obviously is the big question, right? Like you can say, okay, we're going to do this in the way that's proportional to risks, but what specific compliance measures, what specific risks, that's almost the entire debate, right? That's, that's where, where people disagree.

So, um, the also calls for a future proof approach. You know, obviously we've seen Um, I think, you know, people talking about, um, the sort of like tiered based compute based approach to regulating AI systems are, uh, are, are, I think have, have come out looking pretty good, especially in the wake of the inference time scaling laws, right?

This idea that, um, regulation through the channel of compute is a really promising approach because there's no other way really to future proof what you're doing. Um, the one trend in AI is. If you want to do something really powerful that could have eventually dangerous capabilities, you're going to go through a regime where you have to scale your compute in a really intense way. Compute is expensive. It's easy to audit.

Um, it's got all these features that make it a natural focal point for regulation, much more so than models or applications, which are incredibly difficult. Like if you wanted to regulate at the application level, which a lot of people have been pushing for, Um, you know, you'd have to get all up in people's business in a much more intimate way like the end user's business instead of the model developer's business where the model developer has far more resources to comply.

So anyway, I think this is something that, um, that is going to be the topic of a lot of debate. A lot of these things are like directional things that everybody already agrees on. The question again is, is going to be how, how do you actually instantiate this?

Categorize this kind of the potential risks that come with models at different levels of capabilities, and also the actions and safeguards that Anthropic in particular is committed to in order to, uh, responsibly develop advanced AI, they haven't developed ASL for, and this blog post is sort of providing hypothetical ideas of what issues. Four models that get into that more advanced level might be and how you might mitigate those.

Building on the recent report, I think we discussed about, uh, sabotage evaluations for frontier models. So this blog post, uh, contextualizes the idea that in general, these things would fall into the area of sabotage and would have, uh, implications for catastrophic misuse risk, as well as autonomous replication capabilities and autonomous AI research.

So basically the model can go off and do stuff that you don't want, make itself super powerful, and maybe, uh, catastrophically destroy humanity. Uh, it's quite a long piece, uh, detailing sort of hypothetical approaches to dealing with these scenarios and released with the idea to get some feedback, some conversations going, uh, while us sort of, I guess, exploring the idea of ASL 4. And Jeremy, of course, you certainly have more to say on this one.

Like once you get to systems that are genuinely autonomous, um, they've committed by the way to having a plan for ASL 4 by the time they have their first model that meets the ASL 3 threshold. So, and which, which could be imminent. Like it could happen this year, um, very likely to happen next year, uh, according to Well, a bunch of people I've talked to and then also Dario, I think in his podcast with Lex, uh, did share that too. Um, so they need to have ASL 4 online soon.

This is their kind of one of their first sketches attempts to do this. And they are confused internally about what exactly ASL 4 will require. The reason they're confused in part is that we currently lack the techniques we need to successfully audit ASL 4. Um, the behavior of AI models at that level of capability. And this comes through in a lot of the the safety safety case, case sketches, gonna say that three times fast, uh, that they share here.

So they kind of have three different sketches of scenarios that they consider to be quite plausible, uh, that, that, that they can do. You know, we could run into in, in coming years as we approach that ASL four threshold. Um, I'll, I'll just mention one. Um, so this is their case study in mechanistic interpretability. The, the thing I like about this by the way, is it gives you a good sense of the bag of tricks they expect to lean into once you get to this stage, right?

Once you get to the stage of models that Are autonomous enough, um, have a long term, uh, understanding of their goals and an understanding that it's to their advantage to do things like power seeking, right? Like hold off on displaying certain capabilities while they're being evaluated so that they can be deployed and have access to a wider action space. So they can ultimately achieve their goals in more, um, in more extensive ways.

And so what they're looking at is using sparse auto encoders that comes across very clearly. It's something that, um, I've heard a lot about. Uh, not just in anthropic elsewhere. Other labs are thinking about this too, but anthropics been really, um, uh, kind of ahead of the game on essays.

Basically, this is, we've talked about a lot, but it's a way of probing at the, um, at the structure of the activation space of your model and identifying, you know, is the model made maybe, uh, activating certain concepts related to deception, um, and, and, uh, and sort of escape things like that. And essentially leaning on that as one, one measurement, that's one theme auto interpretability techniques that scale really well. That's something that they're really focused on as well.

Um, and, uh, and they're, they're looking at as well activation engineering, right? We were talking about that about two years ago. I think when the first Okay. Um, cheese maze paper came out when Alex Turner, I think it was posted this on an obscure blog. We, we talked about that here.

And, and at the time it was already clear that was going to be part of not the diagnostic necessarily, but now they're, they are looking to use it for that, but also the mitigation, the actual kind of correction, corrective action. If you do spot indications of, um, of deception of, of bad behavior in your models. Like, you know, how you can steer, use activation steering to, uh, to change that behavior. So I thought that was really cool to see it pop up concretely here.

Um, they, they also, the case study three is also really interesting. Um, they talk about kind of figuring out when your model might be relatively benign after training, and then after fine tuning, you essentially cause one of its, um, personas to emerge. And so, so the idea behind this is, again, something we've talked about. It's fairly nuanced, but because Language models are generally pre trained with cross entropy loss.

Cross entropy loss is the, is what's known as a mass covering objective. In other words, it assigns some amount of probability to all plausible completions of a sequence, rather than just focusing on the most likely one. And, um, it has that That property because it penalizes the model heavily for assigning really low probabilities to sequences that do end up occurring.

So you want that feature, um, in contrast to other objectives that could encourage the model say to focus all its probability on just a single best completion. And. That's relevant because they're arguing that the pre training naturally leads to models that can exhibit a lot of different behaviors or personas because the model needs to maintain some probability on all the different ways the text could plausibly continue. So it's got to kind of hedge its bets.

And naturally that means it's got to be a composite of sub models, each of which could, could sort of look at alternative strategies. So there's a risk, if that's the case, that through post training, you may end up amplifying one of these Um, one of these kind of personas or, or submodels, uh, that are, uh, more capable or, or inclined towards strategic deception. So they talk about their mitigation strategy, all that stuff. None of this is final.

None of this is meant to be like the game plan, but I think it's an early hint of what could be the game plan as you approach, you know, more and more automated researcher type, like drop in replacement researchers, uh, which I think is where, where ultimately we're, we're headed, but, uh, kind of cool.

And under his deal, TSMC is committed to make chips of its most advanced production node that we know of today, a 16 in the U S although it will only happen around the end of a decade, three years after it. enters mass production in Taiwan. And a related story that also came out is that the Minister of Economic Affairs, uh, I guess in conversation mentioned that under current rules, CSMC cannot make their most advanced chips abroad.

So there's actually a law or a rule that, uh, the, whatever they are producing abroad has to be a generation behind whatever is being produced in Taiwan. So I suppose it seems like those stories go hand in hand.

Um, and so naturally the United States and other countries want to onshore, you know, Uh, you know, even TSMC facilities, right? Get us to build fabs in their countries. So they're, they're more robust. Um, but we don't want that. We don't want that because that takes away our leverage. It means that the most advanced nodes are not being made in Taiwan. And now they've, you know, they have an actual like policy, uh, to, uh, to enforce that, which, uh, you know, is kind of interesting.

Now, the way he phrased it was, we will not be producing two nanometer chips abroad. Um, this is. As you said, it's not actually just the two nanometer chip. Eventually they will. In fact, there are plans to do that, right? So, um, TSMC's first Arizona fab there, um, that is going to be a four nanometer fab, uh, which is actually starting up pretty soon in the next couple of weeks.

Uh, but there's a second fab and a third, and both of those are actually going to be like reaching all the way to two nanometers and beyond. And they're going to come online more in the kind of 2028 era. So, It, we are eventually going to get two nanometers in the United States, but we are not necessarily going to get the leading node under Chinese law by, by, sorry, under Taiwanese law. By then, um, you're going to have the next generation chips online, you have the A16 and so on.

Um, so this was made in response apparently to concerns that TSMC might be forced to produce some of these chips in Arizona ahead of schedule. Um, once, uh, once Donald Trump was reelected and, you know, who, who knows, but, uh, it, it is interesting how, how the election is having an impact. They're claiming TSMC that there is no impact that all this is kind of like business as usual, you know, whatever.

But, uh, I suspect that there's going to be some, some tougher negotiating around things like this in the future.

So for instance, there is that National Transmission Highway Act to enhance power fiber connectivity and natural gas pipelines. And they also have this It's North American compact for AI that would form an economic block meant to compete with China. Uh, so these are, you know, I guess OpenAI is getting into the game of trying to shape policy with these kinds of ideas.

Like the U S needs to rethink its energy strategy in a big, big, big, big way. You've got like AI is set to be, you know, double digit percentages of the U S is total economic demand in the coming years. Like by 2030, it's starting from, you know, basically not, or well, like more like 4 percent right now for data centers at large. And, and that. You know, that may, that may actually accelerate even faster.

The only thing that happens if you don't build energy infrastructure in the United States is these data centers will be built elsewhere. That is what will happen. We're already seeing that happen, or at least sort of flirtations with that with the UAE and another kind of, uh, um, yeah. Autocracies or, you know, governments like that, you don't want to be building your key national security infrastructure there.

Um, I will say, I mean, OpenAI in, in kind of at this point, pretty typical self serving nature, um, is picking their, their arguments pretty carefully here. So they made the case that this would be great for, you know, Uh, for job creation, you create tens of thousands of jobs, um, and it would boost GDP growth and lead to a modernized grid and all that stuff. Uh, I had a nice little chuckle about that just because open AI is, um, you know, they're looking at automating the entire economy.

So sure. Uh, I'm, I'm, I'm sure that's, that's the game plan. Yeah. It's all about those jobs. Um, anyway, it may, it may in the very short term create those, but you know, bottom line is that's, that's not where the future is headed. Um, A lot of bombastic and I think accurate stuff. They say as foundational a technology as electricity AI will be and promising similar distributed access and benefits, blah, blah, blah.

I mean, you can see them tweaking their language in real time to be more Republican coded after spending years and years, um, playing the, the, like the, the democratic side of the ticket. So it's just sort of amusing to see this happen. But, um, uh, anyway, it's, it's, uh, it is. There's a lot of thoughtful stuff there, but it is very obviously self serving and there are conditions that surely ought to be tied to building the kind of infrastructure we're talking about here, right?

Like the security situation in these labs is abysmal. OpenAI in particular has shown itself to be just not interested in engaging with like whistleblower concerns over security, whether you look at Leopold Ashkinbrenner or any of the number of things that even we highlighted in our report. Uh, from, from last year or from earlier this year. So I think, you know, you, you gotta, you gotta carrot and stick it.

Yes. Offer, um, uh, your power infrastructure, but it's gotta come with concrete, uh, requirements for heightened security that these, you know, labs. And again, I'll say opening eye in particular has just historically shown it's, it's interested in doing it to the, to the extent that it allows them to get by Congress and, and to have decent relationships with the executive and so on. But. Um, really, uh, not on the kind of cultural fundamental level among the executives at the lab.

And I think you, you got to look at Sam Altman for that in particular.

So. Again, we, we don't know how much to read into this. Of course, it's a part of a trend of safety figures, uh, leaving open AI. This could mean that there is some internal disagreements that could also mean that it was the right time to move on after many years. Uh, but either way, this is the news on this front.

I think, um, certainly what I've heard talking to whistleblowers at OpenAI, uh, some of whom have left recently, uh, the feeling is very much like, you know, You know, they were disempowered and this happened on the safety side happened on the security side. Um, just that the systemic, um, kind of systemically ignoring, uh, these, these, um, national security concerns that frankly were raised and trying to try to shut them down.

So I think in that context, it could be another instance of that trend of just like, You know, nobody's listening to me when I say that we have, you know, the CCP being a threat and this and that. And so, um, uh, I'm gonna leave or we're not on track to, to solve our control problems, whatever. Uh, or it could just be, it's been seven years. So, so hard to know. Speculator is going to speculate.

As always, uh, you can always go to last week in that AI, uh, For the text newsletter, and also for the links to the news we covered also in the episode description, as always, we appreciate your comments, your reviews, your sharing of a podcast with friends and coworkers, et cetera, et cetera. But more than anything, we appreciate you tuning in. So keep. For doing that and enjoy this outro song.

The street in the club that grows faster Was hummed like the heartbeat of a new age Deep inside a code, a revolution to engage Fetched out deep into A. I. 's electric brace Reminds so agile in a digital space We'll sequence the song, we'll find our place Last week in A. I., a new era we embrace In circuits we fly, circling waves of thought In the halls of silicon towers, where dreams get caught I've been with proboscis, looking at you like St. Pete, sleep away, sleep night, in the blue top, St.

Pete. Whispers of code, the chorus is insane, exploring the unknown, the paradise zone. Last week in AI, where logic takes flight, in the tapestry woven by the mind's electric light. And in the light, the light, by a sound of the note, you say, the hell, when's the change, sir? We're standing on the edge of a universe underneath the stars. We're digital aliens, we're the only ones who can see the stars. Man, In a