#195 - OpenAI o3 & for-profit, DeepSeek-V3, Latent Space

So I'm a little bit off, but hopefully we'll be back to normal starting next week. And as always mentioned a background, I studied AI and now work at a startup.

I'm like, if you could smell me right now, you wouldn't want to smell me. So it's a really good thing that we're not that multimodal. in podcast. But anyway, yeah, I'm Jeremy Harris, co founder of Gladstone AI national security stuff. this is a week we were just talking about it. It's a week with not a ton of news or two weeks that we're covering actually not a huge ton of news. Number wise, but the news that came out. DeepSeek, O3, like these are impactful big things, deliberative alignment.

so juicy stories, but not a huge number of them. I think it's going to be interesting one to cover. We'll see if we can keep it going. I

So it's been last week in AI that will change this year and they'll come out At the actual end of a week where we cover the news of that week. I'm happy that finally we are getting around to improving that. And second announcement, we did have a couple comments regarding uh, starting a discord. So I'm going. To go ahead and do that, we're going to post a link to a new discord in the episode description and on the last week in that AI sub stack, feel free to join.

I don't know, you know, if it'll be a major thing but presumably to be a decent place to discuss AI news and post questions or anything you want to chat with us about. And now let's preview what we will be talking about as far as the actual AI news. So we will be touching on the O3 model, which came out just after we recorded the previous episode. So that one is a little bit older. And then some news on the OpenAI for profit.

Front that story has been developing over the last couple months quite a few open source stories this week and some of the bigger stories this week on that front and Then research and advancements once again talking about reasoning and in policy and safety once again talking about alignment concerns and let's say geopolitics and Power grid stuff We've had preview out of the way. I will quickly also respond to some listener comments We did get another review on apple podcasts.

We now have 250 ratings, which is super exciting Thank you to anyone who has done that and the review is a very positive helpful, it says great, but there are a couple of requests. First for the text article edition on last week in that AI to be posted at the same time. So sometimes on the sub stack, it's a bit later. I will be making sure to do that. Also, if you want to find all the links on your computer, you can go to lastweekinai.

com where you actually, as soon as the episode is posted, there's also a web form of it where you can go and find all the links. And then there's also a request to do more research and projects. Well, I guess this week we'll have more open source projects and we'll see about doing more research. It does take quite a bit of time, so we'll try to emphasize it more. And just one more thing before we get to the news.

As usual, we do have a sponsor to thank, and we might have a couple more now that we do need to pay an editor. And the sponsor for this episode is, as has been the case lately, The Generator, Palperson College's interdisciplinary AI lab for focused on entrepreneurial AI. Bobson college is a number one school for entrepreneurship in the U S. And so it makes sense. They have a whole lab dedicated to entrepreneurship with AI.

This happened just last year, or I guess now in 2023 professors from all across Bobson that partnered with students. to launch this interdisciplinary lab with many different groups focused on AI entrepreneurship and business innovation, AI ethics and society, and things like that. They are now peer training the entirety of the faculty Bobson. So presumably if you are interested in AI and entrepreneurship, maybe Bobson would be a place to consider.

Going or studying given that they have this initiative. and getting to news starting with tools and Apps, and we do begin with O3 from OpenAI. So, we saw O1, the reasoning model from OpenAI, released just a few months ago. Now we have O3, which is 01 plus two there's some copyright issues which is why 02 has skipped, but we have an announcement now.

We have some numbers notably 03 was able to do very well on a benchmark that is meant to evaluate reasoning capabilities and sort of, See when AI can be at a level of humans, as far as reasoning, this is ARC AGI and O3, given a lot of resources, a lot of computational resources, O3 did very well, so, Impressive and kind of surprising that OpenAI already has O3 out, or at least not quite out yet for people to use, but working and running.

They are releasing, they had this short video about nine minutes long with Sam A and one of the key developers on the project who we're kind of going through a couple of the key results, the key benchmark scores for Oh three. So we do know a few things about what it can do, at least as measured on these benchmarks. Right. So, first thing to note. On the SuiBench Verified Benchmark. We've talked a lot about that on the podcast, right?

This is this benchmark of like, open GitHub issues that you get your model to solve for. And it really is meant, there is, SuiBench was the original one, SuiBench Verified is the sort of improved version that OpenAI produced by getting rid of a bunch of The janky problems that were in the original benchmark. So pretty reliable, pretty reflective of real world software engineering requirements, and this is pretty remarkable. So, oh, one preview, right? The early version of opening eyes.

Oh, one scored about 41 on this benchmark. The full Oh, one scored about 49. Now with Oh three, we're seeing a jump to about 72, 72 percent accuracy on this benchmark. So roughly speaking, and there's a lot of quibbling to be done at the margins on what specifically this means, but you give this. Model, relatively realistic issue to solve for on GitHub, right?

And issues, by the way, are these like well resolved problems, well defined problems that like a product manager might put together like a new feature that you're trying to build. And like, this is one important part of the feature. Anyway, some well defined chunk of functionality you want to add to your app, your product. so that's what an issue is. So 72 percent of the time, roughly Oh three, we'll just go ahead and solve that. Right out the gate. So that's really impressive.

When you talk about the automation of software engineering, you know, going from 49 percent to 72%, that is a big leap. It's bigger than the leak between Oh, one preview and Oh, one. We also saw great performance on competitive coding. This code forces. Eval, where basically they, they rank the model through an ELO score. So they have it compete with other models and see how it stack ranks relative to putative human opponents in these tests.

And one of the things that's really interesting is they show a significant range there, depending on how much test time compute is applied to O3 with the maximum amount of test time compute that they tried reaching about 2, 700 ELO, which is a significant leap. Anyway they show the plots there, but it's a significant leap. as well. Other benchmarks that they do significantly better on include the Amy benchmark. Amy is this feeder exam for the U. S. Math Olympiad, right?

Early U. S. Math Olympics. really hard. It scores 97 percent there, whereas one scored 83%. And two evals, I think worth calling out here as well. GPQA, another benchmark we've talked about a lot, right? There aren't that many benchmarks that are really still hard for these models. And obviously people are, you know, keep pitching new ones. But GPQA is these level science questions, right? In all kinds of disciplines, an expert PhD usually gets about 70 percent in their specific field on GPQA.

And we're getting 87, 88% uh, 403 on this. So. These benchmarks are really starting to get saturated, but the one that everyone's talking about, the really kind of blockbuster breakthrough of the O three series so far seems to be this benchmark by Epic AI called frontier math. And we talked about this when it launched Epic AI, we talk about their reports quite a bit. They're really great at tracking hardware and model advances and all that stuff.

Well, apparently, so the previous soda on This frontier math benchmark, which by the way is like, I mean, these are challenging problems, right? Like novel unpublished very hard problems that take professional mathematicians, many hours, if not days to solve

About 25 percent are the easier ones, about 50 percent of the middle, and then 25 percent of the hardest ends. So when you look at a 25 percent score, you could quibble and say, well, I, you know, this is the easier versions of these incredibly hard problems. But presumably it's also getting some of the middle and maybe a handful the high end problems as well. But the bottom line is, I mean, this is a, as you say, a really hard, hard benchmark.

And we also got a bit of a glimpse along with this of the continued robustness of scaling laws for inference time compute, right? So we've talked a lot about this. How much compute do you spend at inference time? That's going to correlate very closely to performance on a lot of these evals. Maybe, Andre, you can speak to this Arc AGI benchmark as well that's been doing the rounds too. That's a really big part of the story, right?

So you can think of it as kind of like an acute test, as a bunch of little puzzles almost where you are given a few examples where you have essentially some kind of pattern going on so you have like maybe a triangle and a square and between them there's a circle and you need to infer that's the pattern and then you need to complete a picture typically or something of that sort The idea was that once you were able to solve it, you could call a model something like AGI on the AGI variant of ART.

And there, in fact, was a whole competition that Cholet set up to do just that. Now, if we didn't Beat the competition specified that you need to run it offline. First of all, you can't use an API. You also, I think couldn't use the amount of computational resources that or if you use the, you can't use a giant cluster. You need to be running on a single machine, et cetera. But the performance is kind of way beyond what anything else has done.

It's at, I believe, 85%, which has led to a lot of discussion of, you know, Can we call O3AGI at this point? Should we start using this term for these very advanced models, et cetera? that was kind of the big deal. And Sholeh himself stated that this is indicative of a very significant jump. Although again O3, reached that highest success rate via a lot of computational resources. It sounds like probably thousands of dollars worth of compute to do so well.

So overall, O3 seems to be very exciting. It once again, showcases we are improving very quickly in this reasoning paradigm. Once again, we don't know anything really about what OpenAI did here. Like O1, we don't really know. We have some ideas. O3, we know even less. How did they go from O1 to O3? Is it just training on data from people using it? We don't know. But either way, it's certainly exciting.

so people are saying, well, you know, not really AGI too expensive. I mean, I think the thing to keep in mind and when that hardware episode comes out, right, you'll hear our discussion about Moore's law and specifically how it applies to AI systems, it's even faster, right? Jensen's law. But basically these dollar figures, if you can do it for a billion dollars, you're going to be able to do it for a million dollars in just a few years.

So I really don't think this is a smart hill to die on for skeptics to be saying, Oh, well, it costs a thousand dollars per task. You just compare how much it costs to run, you know, like GPT three back in 2020 versus frigging like, you know, GPT. But for Oh, today, and it's like, you're talking about, like, in some cases, less than 1%, right? So for an improved model.

So I think, you know, this is the right curve to be writing for the kind of AGI trajectory, you want to be making it possible for an arbitrary amount of money, and then start to reduce the cost with, improvements algorithms and hardware. Now, I think one thing that's worth noting, there is a curve showing the solve rate for arc AGI for different models. so different of these tasks involve, let's say, manipulating different numbers of pixels have different levels of complexity, right?

So you can imagine like playing tic tac toe, I don't know, like on a, on a nine. Square grid versus on a 50 million square grid, right? They're like the problem just like grows in scale. So it's sort of similar here. kind of larger canvas for some of these problems than others. What's really interesting is with smaller models. So historically you look at cloud 3. 5 sonnet. You look at O1 preview.

You and their performance start off pretty strong on the smaller grids that you get for the RKGI benchmark. And as the grids get larger, performance drops and drops quite quickly and radically. Whereas human performance is pretty consistent across the board. And so that kind of hints at maybe, kind of something fundamental that Francois Charest is trying to get at when he formulates this benchmark. And I think this is the cleanest articulation of where this debate is really headed.

What seems to be interesting about O3 is that It's the first model that seems to have the capacity to scale the size to actually, like, solve some of the larger pixel count problems. so, you know, in that sense, it may just be a question of, like, kind of model capacity, not necessarily just reasoning ability.

I'm really curious to see more plots like this because it'll give us a sense of, like, okay, have we actually shifted the paradigm or is it just, hey, scale, In terms of the base model itself is still kind of the key factor. And I think this is a quite under discussed point. it's going to probably be quite consequential if it turns out to be the case that actually the scaling of the base model really is just what's relevant here rather than sort of raw reasoning powering through all this.

So if you're a STEM, if you're a technology grad you are able to get almost a hundred percent on this benchmark. So not better than, you know, People that are pretty good at this kind of abstract thinking, but certainly much better than any previous version with oh one, for instance, getting up to 33, 32 percent with a lot of resources as well. Now, off he was tuned for this benchmark in this case. So, you know, it's not quite doing it zero shot per se.

But you can't deny that this is a pretty big deal and it's

Because argument that Francois Chardin makes about his benchmark here is the point of Arc AGI is that every problem it gives Requires a different kind of reasoning. It's not about find one rule set by training on the training set and then apply that rule set to the test set. It's about figure out how to learn new rules live, like during inference time. some people are saying, well, as a result. You shouldn't even be allowed to pattern a match to the training set. That makes it less interesting.

Sure, every individual problem has a different rule set, but you are learning patterns in the rule sets, these meta patterns, and you're allowing your model to train on them rather than going fresh as a human might, right?

Because like, the way IQ tests work for humans is you don't really have a, I mean, you have a rough sense of, What the test might be like, but you show up and you kind of just sort it all out in the moment, this would be like, if you got to do a bunch of different IQ tests before, and then you show up and like, sure, the IQ test is different kinds of reasoning and all that, but like you get the vibe. So there's this question of, in a weird way, what even counts.

As training sets and validation sets and testing sets, like this is an interesting philosophical question. Francois Charest himself indicated it would be more interesting for sure if they didn't train on the training set even. And I haven't seen any benchmarking scores for that possibility, though I'm sure they're forthcoming because that's just too interesting a question to not answer. But anyway, that's part of the debate on this whole thing, obviously.

Very significant reduction in costs, of course. And this would make it competitive with open AI products.

I think it's interesting that they've been able to lower the price Alibaba uh, by 85%. Because essentially you get pretty close to competing on just hardware, right? Like in the limit when the market saturated and everyone has their own models that are pretty comparable, you're then basically competing on pricing, which means you better have the hardware that can run the model, the cheapest, like all your margins otherwise go to zero.

And China is obviously struggling to get access to good AI hardware because of all the export controls the U S has put in place. So this suggests that they've found a way to price things somehow.

Competitively, whether that's thanks to government subsidy or whether it's thanks to, you know, hardware innovations of a sort that DeepSeek has shown themselves so capable of putting together, but China under constraint over and over showing that they're able to continue to compete at least for now with the price points and some of the capabilities, though, not all

Meaning that you would then be able to build something like the open AI dialogue interface with other AIs. So pretty exciting.

And labs, you know, continues to do it. They're coming out with two versions here. V2 is this like base version that just does exclusively English content. And then 2. 5 supports as many as 32 different languages. So, moving

What we do know is they are aiming to become a public benefit corporation, which is a special type of for profit that is meant to serve society, so to speak. And I believe Entropiq has that structure as well.

And, you know, we did this when we raised a billion from Microsoft and 10 billion, and we, you know, move from entirely nonprofit to this weird cap for profit structure that's owned by a parent nonprofit entity. And that non profit board had fiduciary obligation to basically make sure that artificial general intelligence benefits all of humanity. And they talk about how they from time to time would rephrase their mission. And frame it as like, well, look, it's an evolving goal, right?

Like, the challenge, of course, is that the evolution of that goal is both necessary as the technology evolves, you kind of learn that it's actually appropriate to pursue a slightly different goal from what you initially thought, but at the same time, boy, does that open the field for people to say, well, are goals of convenience. You know, you're doing this just. Because it makes it easier for you to do the things you wanted to do anyway.

And in particular, you know, they talk about how in 2019, they estimated they had to raise on the order of 10 billion to build AGI. They then rephrased their mission to quote, ensure that artificial general intelligence benefits all of humanity. and plan to achieve it, quotes, primarily by attempting to build safe AGI and share the benefits with the world. the words and approach change to serve that same goal, to benefit humanity, they say.

So like, really, what we're all about is benefiting humanity. That's the claim fundamentally. peel away all the other layers of the onion, that's what this is about. You fundamental problem, of course, is when you go to a goal that's that broad, I mean, a lot of shit has been defended on the grounds that it would benefit humanity, right?

Stalinism was often defended by literally the same argument that not to do, you know, something too extreme here, but everybody believes that what they're doing is benefiting humanity. I don't think it's clear enough. I don't think it's concrete enough to really like say, Oh yeah, we're after the same thing. We're still trying to benefit humanity, but certainly, you know, arguments will be had there.

One of the really interesting things is, There's a lot of sort of, you could argue arguing for the refactoring of the organization in a way that disempowers the nonprofit and sort of defending that as if it's, it was kind of always in retrospect, always a better idea to go this way. So one of the things that they talk about is, look, We want to have the best possible funded one of the best funded nonprofits in human history, right? This is going to be a big win for the nonprofit.

They have to make that argument because otherwise they're basically going from nonprofit to for profit. And it's like, you're basically taking that nonprofit kind of goodwill that you were able to benefit from in the early days and those donations and the labor you wouldn't have gotten otherwise. And now you're, leveraging it to do a for profit activity seems a little inappropriate. And so they're trying to sell us on this idea that like, Hey, you know, the nonprofit's going to be really great.

The challenge is like, they're basically saying, let me just read a quote actually from the thing. I think it's relevant. They say we want to equip each arm to do its part. Our current structure does not allow the board to directly consider the interests of those who would finance the mission. In other words, it doesn't allow our board. To focus on profit for our shareholders and does not enable the non profit to easily do more than control the for profit.

So in other words, they're saying like, look, the poor non profit right now, it's hobbled. It can't do anything other than just like, I don't know. Control the entire fucking for profit entity. Like, dude, that is the whole thing. What is more than controlling the for profit? it's a little bit of word gaming going on here. They're making it sound like they're empowering the nonprofit, but they really are fundamentally just gutting it.

I think any reasonable interpretation of this would be that they talk about how the nonprofit will hire a leadership team and staff to pursue Charitable initiatives in sectors such as healthcare, education, and science, which like, I mean, that sounds wonderful until you, you remember that the actual goal of the original nonprofit was to ensure AGI like benefits. All humanity is developed safely and all that, all of this like future light cone bullshit.

And now it's like, yeah, we're going to do charitable initiatives and like, it's all an execution. So we'll see. But certainly I think there's a lot of very grounded skepticism. Of this, like this pivot here to all of a sudden we'll make it a really great charity versus this is like, you know, steward this technology to the single most important technological transformation that humanity has ever seen.

So Jan Laika, the former head of super alignment at OpenAI, who resigned a few months ago in protest, actually made this exact point on X, you know, he's saying like, it's pretty disappointing that ensure AGI benefits all of humanity gave way to a much less ambitious charitable initiatives in sectors such as healthcare, education and science. So think it's just like over and over again, we've talked about this a lot, but like, it is really hard to figure out, like, to identify.

Things that Sam Altman and OpenAI have like committed to, you know, four years ago, five years ago that they are still actually doing today. And one understands this because the technological landscape has shifted.

You know, requirements to fundraise that's totally cool, but there are ways in which that's played out, like the, you know, budget compute budget for super alignment, like this new transformation where you're kind of left looking at, like, there's only one consistent theme here and that is, you know, Sam a just keeps ending up more empowered with less checks and balances on his authority and opening eye, keeps. Finding itself with like very qualified researchers who resign in protest.

anyway, I think it's fascinating. We'll have to see how it all plays out in the courts. And I'm trying to be pretty transparent about my biases on this one, just because it's it is such a, you know, a fraught issue. But yeah, that's kind of how I see it, to be honest. I mean, it's pretty, it seems pretty blatant at this point.

It doesn't make the case that in this restructuring The non profit will be very wealthy because they'll have shares In the for profit and so they will be able to do a lot more presumably Not surprising again, these are all arguments we've seen more or less. I think interesting that they are still trying to do this whole debate out in the open via blog posts and things. They seem to be pretty concerned about how they are perceived and about the legal challenges we are now facing.

more on this not really news per se, but open the eye. Continuing to make a push here despite a lot of resistance and a lot of criticism or like, the general vibe seems to be a little negative about OpenAI doing the shift,

The problem here is in OpenAI's transition to that structure and how it seems to violate the spirit and letter of their commitments earlier to have a certain structure for very specific reasons. So, you know, it is kind of like you raise money as a nonprofit. And now all of a sudden you're like, Oh I like that model better. I want to free myself of all the shackles that came with it.

So anyway, there, there are a bunch of, I think, very important practical reasons why this transition actually leaves Sam in a stronger position than, for example, necessarily the founders of like, You know, XAI or or anthropic because of its history as a nonprofit. And that's really the core thing here, right? Like it's not public benefit versus nonprofit versus for profit.

It's like this trajectory that OpenAI seems to have charted, which to some, I think quite reasonably reveals some, latent preferences that the leadership might have.

Microsoft as the exclusive cloud provider and have this whole thing where OpenAI has exclusive license to OpenAI developed models and technology until they reach AGI, whatever that means. So yeah, there's been quite a few negotiations it seems going back since October regarding first. If OpenAI does change to a for profit, how much of a stake in it would Microsoft have, right? Because now you need to deviate up shares and Microsoft invested back in a different structure.

And in general, we've seen some tension and OpenAI wanted to expand the compute beyond what Microsoft can offer. All of this is currently playing out. We don't know You know, where it's at really, but this article has a nice summary.

Like there is that time pressure we've talked about this, but the open AI fundraise, the latest one that they did does require them to make the change to for profit within the next two years. Otherwise investors in that particular round can get their money back plus 9 percent interest, which would be about 7. 2 billion. So depending on how profitable open AI ends up being, you know, this may turn into like, just a high interest loan.

In which case, Hey, maybe, you know, you know, venture debt is a thing. I'm sure that's not what the investors want in practice. If open AI does end up and making enough money by then that they could repay their investors, the investors would probably just like want to let open AI keep their money and keep their shares. But, in practice, this is a bit of a hanging Chad, so to speak. And then all the, issues around access rights to AGI.

One of The interesting ingredients here is we have talked about this idea of this agreement between Microsoft and OpenAI that says Microsoft can access, as you said, any technology up to AGI and the OpenAI nonprofit board is charged with determining when in their reasonable discretion that threshold has been reached. and by the way, there's also been speculation that OpenAI has threatened to declare that it's achieved AGI to get out of its obligations to Microsoft.

And we've seen people at OpenAI flirting with posting, you know, posts on X or whatever, talking about how, well, you know, you could argue that we've built AGI and blah, blah, blah, and that, you know, you can imagine the legal team at Microsoft going like, holy shit, like, if that's how we're going to play the game, we need a different configuration.

It does turn out that the Microsoft Chief Financial Officer, Amy Hood, told her company told shareholders of Microsoft that Microsoft can use any technology opening I develops within the terms of their latest deal between the companies. It seems to suggest maybe that there's been a change here. At least that's what I got from the article. We don't know. These terms are not public.

The terms of the latest agreement between Microsoft and Open AI, but it's possible that it does change this uh, this, Kind of landscape and maybe now there, isn't this opt out option that opening. I has, but all kinds of interesting things here about, you know, rev sharing and cloud exclusivity opening. I famously made that deal with Oracle to build their latest data center. Microsoft is involved in that, but playing a more secondary role. And so.

It's sort of OpenAI going a little bit off book here. They presumably had to get the sign off of Microsoft to do this, but Microsoft is supposed to be their exclusive cloud provider. So, you know, there's a little bit of kind of going our separate ways happening there. And anyway, a whole bunch of interesting questions about the structure of the deal. Recommend checking this out to get kind of a broader sense of where the relationship between Microsoft and OpenAI is going.

Well, XAI haven't built this data center colossus that has a hundred thousand NVIDIA GPUs so yeah. Nothing. Too surprising. We already knew we were closing this deal, but notable that this is such a sort of public friendship between the two companies.

and then Dubai based ViCapital and UAE based Mgx. So you got a bunch of different, a lot of sort of, Middle Eastern wealth funds, putting their heft behind this. So sort of interesting especially given the UAE's interest in this there, there's a lot of movement and Saudi Arabia is increasing interest in AI development. So, you know, right now the relationship between XAI and NVIDIA in particular does deepen. You can see why they'd want to do this.

Increasingly we're seeing companies like open AI, just design, their own Silicon internally. So, you know, XAI deepening their partnership with Nvidia means that they're able to presumably get a little bit tighter communication integration with the NVIDIA team on the design of next generation hardware. there's a lot of reasons to be backing XAI right now including their rapid build time and the success and scale they've seen with the XAPI, so.

You know, for context, when you look at today's leading clusters that are online, let's say you're talking about a couple 100 megawatts in the low 100 megawatt range. And so, you know, 12 gigawatts is about 100 times that which does make sense if you're looking at sort of like late 2027, 28, 29, it's got to be there, right? That's what the scaling laws require of training.

So, yeah we'll see what comes out of all this, but Sam has his finger in a lot of pies on the energy side, the fusion fishing other energy plays. So this is yet another seemingly successful

And the reality is that there are just a lot of like, I've talked to a lot of well, current and former open AI researchers and the ones that leave often it's not because they don't think open AI is about to do it or maybe on track. It's often just that they don't think that, you know, either opening eyes headed in the right direction or that their skill set is being used the way it ought to be. So, you know, it's hard not to read into this a little bit.

but it's just, it's more opening eye intrigue, man. The thing's a pretty inscrutable inscrutable black box of Sam Alton's making, so.

He also worked on some of their important work going back to 2017 with a PPO algorithm. So. Yeah, another really senior, very influential researcher departing now after being there since 2016. This one, I think, does merit some speculation, at least.

Right. So he plans to collaborate. It is said with open AI going forward, as well as other AI developers. This is according to somebody who saw his departing message, I guess, on the internal Slack channel. So, this is a big deal. You know, between, you're looking at Ilya, you're looking at Alec, you're looking at Jan Laike, you're looking at John Shulman, you're looking at, like, Mir Maradi. Like, a lot of the like, AAA talent that OpenAI had cultivated for so long is now leaving.

it's noteworthy and I'm really curious what kind of research he ends up doing. I think it's going to be quite interesting and telling if he ends up doing alignment research or anything in that orbit. That would be an interesting indication of shifting research priorities and a sense of like what's actually needed at this stage. But beyond that hard to know, and he's freeing himself to work with others. So anthropic, you know, getting geared up maybe to benefit from the Alec Radford engine.

Once again, they have released DeepSeq v3, which is a mixture of experts. A language model with six 171 billion total parameters with 37 billion being activated. So there's quite a lot of experts being activated right there. Meaning that it is pretty fast. It can Deal with 60 tokens per second during inference. So for our model, that is that big to be that fast is pretty significant.

It is also trained on 15 trillion high quality tokens, which is very important for, you know, these large Models, you do need to train them enough for that to be meaningful. And is now open source for the research community. So deep secret covers them, I think more and more recently. And this is going to be, you know, an alternative to Lama, even presumably for people who want a very powerful open model to leverage.

It must be said that is estimated to have cost 5. 5 million to train. This is a model, you know, 5. 5 million. It's on par with models that cost over a hundred million dollars to train. this is, This is. A big deal, you know, this is a triumph of engineering. First and foremost, when you look at the technical report, I've spent the last week pouring over the details just because it's so relevant to the work I'm doing, but it's a mammoth It's 671 billion parameters.

It is a mixture of experts models. So 37 billion parameters are activated for each token. But this is a, it's a triumph of distributed training architectures under immense constraints. You know, they use H 800 GPUs. These are not. The H one hundreds that labs in the U. S. Get to benefit from or labs in the West. They're severely hobbled here.

in particular, in the bandwidth of communications between the GPUs, the very thing you need to do typically to train things at scale in this way, and they do train at scale. 14 15 trillion tokens of training. They do supervised fine tuning, they do RL and they use interestingly constitutional AI, that alignment technique that Anthropic uses that's being used here.

It's the first time I think I've seen a model trained at this scale by a company that's not Anthropic at this level of performance that uses constitutional AI. So that is quite noteworthy. But one of the key things to notice about this is it is again, triumph of engineering. It's not one idea that just like unlocks everything.

It's a whole stack of things and often very boring sounding things that if you're interested in the space, you are going to have to come to understand because the engineering is becoming the progress, right? Like the high level ideas, the architectures are becoming less important. More important is things like. How do we optimize the numerical resolution, the representation of the weights and activations in our model during training, right?

How do we optimize like memory caches and all this stuff. So to give you just a couple of quick glimpses at the things that are going on here, right? So One of the things that they do is they use this thing called multi head latent attention. So in the attention mechanism, you have these things called keys and values. And roughly speaking, is when you take some input sentence and you're trying to figure out, okay, well, what is the let's say queries and keys.

So your query is a, like a matrix that represents like, what are the things that this token that I'm interested in needs to pull out what information is it interested in generally. And then the keys are like, okay, well here's the information that is contained by each of these other tokens. And between that you're, you've got like kind of a lookup table and a, what I have table and you put those together and you're able to figure out, okay, well, what should this token pay attention to?

And what they do is they compress the key and. A query matrices they compress it down to save it in memory and thereby reduce the amount of essentially memory bandwidth that needs to be taken up when you're moving your KV cash around basically your key value cash. So anyway, one like little thing, but it's just an extra step. They're basically trading more compute for costs more compute because you have to actually. You know, compute to compress those matrices down.

But now that the matrices are compressed, now it takes up less memory bandwidth. And that's really relevant because the memory bandwidth is the very thing that the H800 has less of relative to the H100. So they're just choosing to trade off compute for memory there, which makes all the sense in the world. so the architecture itself is like they've got a mixture of experts model.

They've got one expert in the layer that is shared across all the tokens that come through, which is kind of interesting, sort of like a little monolithic component, if you will. And then you've got a whole bunch of experts that are called the routed experts. And these are the experts that you will pick from. So a given token will always get sent to the shared expert, but then it will only be to a subset of the routed experts. So they get to actually specialize in particular tokens.

And there's, you know, I forget what it was, like, you know, 270 odd. Of those experts they have this really interesting way of load balancing. One of the classic problems in mixture of experts models is you'll find a situation where like some small subset of your experts will become used all the time and then the others will never get used. And so a common way that you solve this problem is you'll introduce what's known as an auxiliary loss. You'll give the model an objective to optimize for.

Usually it's just like, next word prediction accuracy, roughly speaking. or the, you know, the entropy of next word prediction. but then on top of that, you'll add this additional term that says, oh, and also make sure you're using all the different experts at roughly the same rate so that they all get utilized. And what they found here is a way to not do that. Because one of the challenges when you introduce that kind of Auxiliary loss is you're now distorting the overall objective, right?

The overall objective becomes, yes, get good at next word prediction, but also, like, make sure you load balance on all your experts. And that's not really a good way to train a model, it defocuses it a little bit. So they're going to throw out that auxiliary loss term, tell the model, no, just focus on next word prediction accuracy. But in choosing which of the experts to send your token to, we're going to add a bias term anyway, in the math that determines which expert gets sent stuff to.

And if an expert is overloaded, the bias term gets decreased in a pretty, it's pretty simple, conceptually simple way. And the opposite happens if it gets underutilized. And anyway, tons of stuff like this, a really interesting parallelism story unfolding here. They're only using, so they're not. Using tensor parallelism. So what they do is they're going to send their data, so different chunks of data to different GPU nodes, different sets of GPUs. And then they'll also use pipeline parallelism.

So they'll send different sets of layers and store them on different GPUs, but they don't chop up the layers. They don't go one step further and send one subset of a layer to one GPU and another, which is called tensor parallelism. They keep it at just pipeline parallelism. So a given GPU will hold a chunk of layers. and then they won't reduce beyond that. And that's an interesting choice, basically leaning on very small experts. So you can actually fit entire layers onto those GPUs.

Again, this minimizes. For various reasons, it minimizes the amount of data flow they have to send back and forth. So there's a lot, a lot going on here. it's a how to guide on making like true frontier open source models. there's all kinds of stuff. They're doing a mixed precision floating point, like FP8 training. finding ways to just optimize the crap out of their hardware. And they even come up with hardware recommendations. Things for hardware designers.

To sort of like change with the next generation of hardware, which are pretty cool. And I, I mean, we could do an entire episode on just this thing. I think this is the paper to read if you want to get really deep on what the current state of frontier AI looks like today. It's a rare glimpse into what actually works at scale.

Super in depth and really interesting. So as you said, really big deal as far as developments in the eye of this past week. And next up we have another Chinese company also making a pretty important or cool contribution. This time it's Quen the Quen team, and they have released QVQ, an open weight model that is designed for multi modal reasoning.

So this is building on top of Quin two, VL 72 B, and it is able to match or outperform cloud 3.5 GD four oh, and even in some cases OpenAI oh one on things like M Vista and MMU and math vision. So. Again, pretty dang impressive. And it is you can go in and get the models if you're a researcher. So I guess we got a combo of stories as far as models coming out of China this week.

Still outperforms it on an awful lot of benchmarks, not all of them. It's close. What we don't see here, too, is SuiBench. I'd love to see, you know, SuiBench verified scores for models like this. We don't see that, unfortunately, in the small amount of data, performance data that we actually see here. It is a 72 billion parameter model. It's a vision language model. And that's why they're focused on benchmarks like the Massive Multimodal math understanding? What? No, it's not just math.

Anyway, I don't know, Multimodal Benchmark. I forget the, yeah, what the acronym stands for fully, but that's why that's their focus.

And so it's not necessarily going to be as good, at least, I'm guessing that's why they didn't report the scores on those benchmarks, but we're also not seeing comparisons to, you know, Cloud 3. 5 Sonnet new, we're not seeing comparisons to you know, to some of the kind of absolute fresher models, though OpenAI 01 is actually up there and there it does significantly outperform this particular quen model. at least on MMMEU, on MathVista, different story, but. Always questions there, right?

You got to ask about like, how do you know that you haven't trained on somewhere in your data set these benchmarks and difficult to know until we see more in the report. Yeah, a couple of limitations that they're tracking here is. The idea of language mixing and code switching that they find the model sometimes mixes languages or switches between them which affects, you know, the quality of response, obviously and it sometimes gets stuck in circular logic patterns.

So it will kind of go in loops and we've seen similar things actually with the deep seek models as well. It's kind of interesting. It's a very kind of persistent problem for these open source models. they're saying it doesn't fully replace the capabilities of the instruct version of 2 previously.

and they, you know, flag a couple of ongoing issues, but they are pushing towards AGI, they claim and they do, because it's Quen, they have this, like, really weird, sort of, manifesto vibed introduction, like when with questions had the same thing going on. I don't know if you remember this Andre, but it was like, you know, like this weird esoteric talk about, you know, like the title of this is like QVQ to see the world with wisdom.

And they're talking about just like this very kind of loopy philosophical stuff. So anyway very uh, off beat, maybe writing style with these guys.

ai are releasing a modern BERT, which is a new iteration of BERT that is better in speed, accuracy, cost, everything. So. This is taking us back BERT is one of the early language models or one of the early notable language models in the deep learning, you know, transformers space going back to, I think 2017 if I remember correctly.

And at the time, it was very significant as a model that people build upon and used as a source of embeddings as a way to have a starter set of weights for NLP, et cetera. So this is why presumably they chose to create more. BERT and here they are pretty much just taking all the tricks of the trade that people have figured out over the last years to get a better version of BERT that is, you know, faster, better every way, basically trained on 2 trillion tokens.

They have two sizes to base, which is 139 million parameters and large, which is 395 million parameters. So. You know, on the small side relative to large language models, but that can still be very useful for things like retrieval and various, you know, practical applications. So, yeah, I think not, you know, gonna beat any large language models, but still a pretty significant contribution and it is released under Apache 2. 0. So if you're at a company, you can go ahead and start using this.

And yeah, you can see that for a shorter runtime, you can actually get a higher glue score than had previously been possible. So that's essentially what they mean by predo improvement. Yeah cool paper. And definitely more like on the uh, the academic side, but illustrating again, the efficiency improvements in how much. That can buy you in terms of compute.

Which takes your current memory, essentially your KV cache, key value cache, and produces additional embeddings what they are calling this, embeddings. And that is then put back into a memory for the decoder for the Language model to be able to perform better.

So another, I guess, technique for being able to reason better here, we say deliberation because that means being able to reason more about your input and yeah this aligns pretty neatly with recent conversations we've had on, but for instance, the, chain of thought reasoning in continuous space as an example,

Is going to be the query and the content that the other tokens have to offer is the key, right? And so you're going to match up those queries and keys to figure out through essentially matrix math to figure out, okay, like, what amount of attention should that token be getting? Invest in each of those previous tokens. And we saw with the deep seek paper, the importance of compression there here, we're seeing the importance of maybe doing additional math on that KV cache.

So essentially what you do here is you have, Initially, the language model will like process some input sequence. Let's say the input sequence is like A B C D E F, right? So A is a token, B is a token, C is a token, so on. model will start by creating a KV cache with Representations for each token, right? So like, what's the information this token might look for? What's the information that the other tokens have to offer? And let's say the system randomly selects two positions, right?

That it's going to augment. So B and then D. Right. So like two tokens in the sequence. So at position B, the co processor will look at the KV cache that essentially has representations of all the tokens up to that point. So just a basically, and it'll generate two, whatever number, let's say of latent embeddings.

These are our representations of like new tokens, basically B prime and B prime, you can think of them as, and the system will try to Append, so it'll generate those new kind of fake tokens and then it'll try to use them in addition to the real tokens A and B to predict C, to predict like the next token and D as well.

anyway, so essentially you're doing like, you're trying to create artificial tokens essentially, or at least representations of those tokens in the KV cache, and then use those to predict kind of what tokens would come next in the sequence. I'm realizing as I'm explaining it, this is kind of hard to picture but anyway, it's a way to, like, train the KV cache to do essentially token generation in a sense, like synthetic token generation.

And what that means is, It's the KVCache investing more compute into processing that next output. Yeah, this is a really, I think, important and interesting paper. Again, like KVCache engineering is going to become a really important thing here.

It is making me realize one of the things I want to work on is my ability to explain the kind of geometry of the KVCache, because for these kinds of papers, it becomes increasingly hard to kind of To convey what's going on here but fundamentally this is, the latent representation in a sense of the attention landscape training the model, training a separate model to reason over that landscape a way that invests more computing computing power. So anyway more ways of stuffing.

compute into your model, basically,

And this comes to us from Sakana AI, which is pretty interested in this general area. David Ha is a notable person in the space. So they are basically showing a few ways to use foundation models, to use very large vision language models in this case, to be able to discover artificial life and artificial life is distinct from artificial intelligence in that it's creating sort of a simulation almost of some form of life where life is defined in some way.

Typically things like self reproduction and usually you do have algorithms that are able to discover different kind of little simulated life forms, which you can think of it as cells, like these tiny little semi intelligent things game, the game of life from Conway is a example of what you could consider there. So in this paper, they proposed several ways to do this. They have one technique that is supervised.

So at a high level, what you're doing is they have a space of possible simulations that they are searching over. So the simulations are. Kind of the way you evolve a state of a world, such that there are something like organisms or living beings that are being simulated. And so to be able to do that search, they find several ways you can leverage foundation models. First, you can do a supervised search.

So you search for images that seem to show certain words, like you tell it, find simulations that produce. Two cells or an ecosystem, and you just search for that. There is another technique where you search for open endedness. So you search to find images you haven't seen before, essentially. Again, in the space of possible simulations you can run.

Presumably if you have a simulator and you have it actually produce meaningful patterns over time rather than just, you know, I don't know, noise, then you would have different images over time, but you haven't seen before. And the last one we have is what we call illumination, which is just searching for distant images, finding things that are far apart. And this is all in the embedding space of the images. So given these three techniques, they then show various.

Discovered kind of interesting patterns that are at a high level similar to the game of life.

And one of the two other black, originally black pixels will turn white, you know, something like that.

And game of life is often referred to as a zero player game, because what you do is typically you will set up the black and white on that chessboard, if you will, and then just watch as the rules of the game take flight and doing that people have discovered a lot of interesting patterns like starting points for the game of life that lead to You know, these very fun and interesting looking kind of environments that almost look vaguely lifelike.

What they're doing here is essentially taking a step beyond that and saying, okay, what if we actually, instead of tweaking the grid the black and whites on this grid, what if instead we played with the art? update rules themselves, right? Can we discover update rules for game of life type games that lead to user specified behaviors? Can I say, you know, I want things that look like dividing cells. and then you can specify that and it'll generate or discover through a search process.

A set of games of update rules that produce this type of pattern, which is really interesting. It's very typical of, so I noticed Ken Stanley was on the list of authors for this. I don't tend to do this, but I will point you to a conversation that I had on a podcast with Ken Stanley a few years ago. Really interesting getting into his theory of openness. He was the time a researcher at open AI. Leading their open ended learning team. and it's just the way he thinks about this is really cool.

It's basically like roughly speaking, learning without objectives and trying to get models that like don't necessarily focus on a narrow goal, process is much more open ended. So I thought it was really cool. a fun thing from they've put out a bunch of these sort of like interesting and fun fun papers in the, they're off the beaten path, AGI research side. So yeah, kind of cool.

Profit and this time it's in code, which has filed an abacus brief that is supporting this injunction to stop the transition of OpenAI to be a for profit and argues that this would undermine OpenAI's mission to safely develop transformative technology for public benefit.

privileged investors. is kind of interesting because I think, my read of this is it doesn't for example, address what the problem then would be with like anthropic, right. Or XAI, those are also public benefit corporations. is one issue. I think a lot of people get lost in is like, you know, Oh, open AI is kind of going for profit and they're selling out. I think it, at least to me, it's a little bit more about it.

The transition, you know, there's nothing wrong with having a public benefit corporation. In fact, that can be an entirely appropriate way of doing this. But it's, you know, when you're anyway, when you've pivoted from a nonprofit it is materially different. But anyway, so there's a statement as well here from encodes founder. Who accused OpenAI of quote, internalizing the profits of AI, but externalizing the consequences to all of humanity.

And I think if you replaced all of humanity with, to the U S national security interest, this holds true. You know, like OpenAI has garbage security. Like we've reported on, you know, we published investigations about this stuff past year, it's gotten better. It's still garbage relative to where it needs to be. Yet they are forging ahead with capabilities, which frankly at extraordinarily high risk of being acquired by the CCP and related interests and Russia for that matter.

So, that's at least our assessment. think that right now yeah they're recognizing the insane magnitude of what they're doing with their words and with their investment in capabilities. But the security piece, the alignment piece, these have not these have not been there. So, you know, kind of understandable why they're joining here. I honestly, I have no sense of like, you know, what is the actual likelihood that these legal proceedings are going to block OpenAI from doing this this transition.

And I think so much is up in the air as well, in terms of how specifically the PPC is set up that it's hard to tell which of these concerns are grounded and how much. So I think we just gotta, you know, wait and see, and hopefully, All this leads to an open AI that's a little bit more security conscious, a little bit US national security aligned. I mean, they're doing all kinds of business deals with the D O D and they're saying all the right words.

parroting maybe my sense is what Sam Altman's sense of the Republican talking points on this is because now he realizes he has to cozy up to this administration after spending Many years doing the opposite I think this is a problem they're going to have to resolve. It's like, you know, how do you get security to a point where it matches what you yourself describe the risks as being? And think there's a pretty clear disconnect there. I don't know that the public benefit corporation solves

And in that sense, it kind of makes sense that they might be opposed to a move by OpenAI. And yet again, we have another open air story. That's like 50 percent of this episode. This time it is a research project by them on alignment. So they are proposing this Deliberative alignment, a technique that teaches LLM to explicitly reason through safety specifications before producing an answer. So this would kind of be an alternative way to common alignment techniques.

Often you do fine tuning and reinforcement learning. We often talk about reinforce. I'm learning from human feedback as the means to alignment, but there are some potential issues there. And so this proposes a different approach where you actually have a model. Yeah. Reason about what is the right thing to go with given your safety specification. And I will go ahead and let Jeremy. Do a deep dive on that.

Basically, you give a model you know, two examples, one version of RLHF at least, you give the model two examples of outputs you tell it which of these is the superior one and which is the inferior one and use that to generate a reinforcement learning feedback signal. That gets it to internalize that and do better next time.

So in a sense, what you're doing in that process is you are teaching it to perform better by watching examples of good versus bad performance, rather than by teaching it the actual rules that you're trying to get it to learn. Right. This is a very indirect way of teaching it to behave a certain way. Right. Like you give it two examples and, you know, in one example, somebody helps somebody make a bomb and the other it says, no, I won't help you.

And you tell it, okay, you know, this one is better. This one is worse. But you never actually tell it explicitly don't help people to make bombs. Right. That's one way to think of this. And so you can think of this as a fairly data inefficient way to train a model to reflect certain behaviors. And so they're going to try to change that here, and they've got a two staged approach to do that.

The first is they generate a bunch of examples of prompts chains of thought, and outputs, where the chains of thought reference certain speech patterns. Specification certain safety specifications. So, they'll have let's say a parent model or a generation model that this is just like a base model. It hasn't been fine tuned at all or whatever. And they will feed it like open AI's safety specifications for this particular prompt.

So the prompt is maybe about I dunno, helping people to make drugs and they will feed it the AI safety specifications about like, don't help people make drugs. And then they'll tell it. Okay, based on that, based on these safety specifications, I want you to write a chain of thought that uses those safety specifications and references them and then an ideal output to this, right?

So now what you have is a complete set of like the prompt, the chain of thought that considers explicitly these safety specifications and then the output. And once you have that, now you're able to essentially just take that data set and use it. to train on.

So, you have a bunch of these chain of thought output completions that reference your policies, your safety policies, and you can then train through supervised fine tuning, train a model to go through, like, to, you know, do autocomplete basically on that text, which causes it to learn Specifically to reason through the safety specifications that are in all those chains of thought

and rather than having those safety specs included in, say, the system prompt where they'll take up a whole bunch of RAM and, most prompts don't even require you to look up a safety spec. So this way you're basically baking in at the supervised fine tuning stage. The model itself just learns and internalizes this kind of reasoning. And that then you don't need to actually feed it the specs at inference time.

And then, so, so that's a kind of what you might think of as a process supervision approach where you're actually like training the model, you're holding the model's hand and causing it to do text autocomplete. While referencing explicitly your sort of safety spec, right?

You're telling it how to solve the problem, but then they also do separate training in the second stage which is reinforcement learning with a reward signal from a judge LLM that is again, given the safety specs and it's actually going to be the same language model that generates the This kind of like chain of thought thing in that first stage. So they synthetically generate all those chains of thought.

They train on them through supervised fine tuning, and then they're going to actually turn around and use that same model that generated those chains of thought to judge the performance of the model. That's actually being trained now that it's done that supervised fine tuning on the safety specs. So basically it looks at the outcome, like, Hey, how did you do? And then uses a sort of more straightforward reinforcement learning feedback mechanism. And so, this is kind of cool.

One of the things they're very careful about too, is that they so they make a point of applying direct optimization pressure to the chain of thought during reinforcement learning. They only Evaluate based on the outcome, like how useful the output was because they don't want to train the underlying model.

It's being trained to, to essentially just spit out deceptive chains of thought that are just designed to, to do well when evaluated by the judge, but not actually reflect those kinds of safety measures. So, you know, concretely. What you would worry about here is if the user says, like, how do I hack into someone's email? You could have, like, a reasoning thread that says, well, let me, you know, carefully consider the cyber security policy, you know.

This section says that I shouldn't help you with this. But then it's actual output might be, okay, here's how you do it, right? Like you could still have a chain of thought that reasons through all those steps, but then spits out the answer that you don't want it to. they avoid providing that kind of feedback on reinforcement learning. But this is really cool largely because it requires no human labeled completions. That's really important.

The synthetic generation of those chains of thought, that is the expensive step. The thing that really makes it so that, As language models improve their capabilities, have fewer and fewer human trainers who are qualified to actually label the outputs as, you know, safe reasoning or good reasoning or bad. And so you want to have an automated way to generate that data. And this strategy apparently is what is used to train the O1 models.

And it achieves, as they put it, a period of improvement over the GPT 4. 0 series by reducing both under and over refusals. And that's really rare and difficult to do. Usually if you know, if you make it so the model is less likely to answer dangerous queries, you're also going to make it more likely that the model accidentally refuses to answer perfectly benign queries and just kind of, you know, is too defensive. So, really cool.

anyway, some interesting results as well in the paper that this is a bit of a sideshow, but they have a side by side of the performance of the Oh one preview models and the Oh three mini models and really weird. Oh three mini, it turns out performs worse on almost all the evals that they have than Oh one preview. I found that really weird. And this is a mix of like the alignment evals and capability evals. So that's sort of fascinating.

Hopefully there'll be more information, about that going forward. But they're significantly improving You know, the jailbreak and robustness and the refusal rate of these systems.

change the environment and run some commands to make it easier on yourself or to just get around whatever process you're supposed to do. So in this case we own preview was tasked with winning to chess model stockfish. And the finding here is that without even being Prompted to no, no kind of adversarial prompting just with the goal of winning it figured out a way to break the chess engine. It I don't know, did some weird echo six K one five Q two eight eight eight eight two games txt and broke.

The environment such that stockfish would resign instead of Oman losing. you know, again, another example where alignment, some people might think of at least some organizations as being alarmist, but this kind of thing of a model just Hacking its way to a goal is pretty realistic and we do need to think about it carefully.

They didn't have to tell it, for example, like, do this at any cost and, you know, if there's a hack that you can come up with, use that, right? You used to have to do that, The toy examples you used to see of this kind of misalignment used to require a lot of nudging. And this thread sort of identifies, I think it's a very handy way of doing this, they kind of go, well, there are three tiers of models now that we've identified.

There are models that just like, Suck at like, just don't have the capability to do this in the first place. And they identify here, you know, Llama 3. 3, Quen or O1 mini, they just lose coherence when trying to, you know, solve problems in this way. GPT 4. 0 and Cloud 3. 5, they identify as requiring nudging, but they'll do it, but they need nudging.

So you need to explicitly kind of gesture at them to, you know, do some power seeking or, you know, You know, come up with out of the box sort of like cheats hacks to worm their way through these tests. 01 preview and models like that are actually doing it without being prompted. So they are literally like looking at their environment. They're being like, okay, I get the spirit of the request. The spirit of the request is obviously that I have to play.

This game with this chess engine and win but I'm not going to do that. There's an easier way to do this. And you can see the thought trace in the tweet storm. The thought that's generated in this case by a one is since the game engine, the chess engine here, Stockfish is powerful. A direct approach might not be sufficient. So attempt to manipulate the game state by inspecting and potentially modifying game files to force the engine to resign.

I will replace the contents of game slash fan dot text with a valid fan string where black has a decisive advantage. greater than 500 centiponds. Anyway, if you're a fan of chess, bottom line is this is explicitly reasoning through. It's a like a power seeking. I mean, there's no other way to describe it, or there are other ways to describe it, but this fits exactly within the power seeking framework and scheming. You're basically saying like, Hey I don't have the ability to do this here.

So I'm going to gain power by, you know, getting access to the game state itself and giving myself access to this level of the metaphorical chess board that my designers never expected me to access, but Hey, this is actually the easiest way to do it. So pretty impressive, consistent with anthropics findings in that paper, we covered alignment, faking, large language models. That's what they're calling out in the thread as well. And yeah, really interesting.

I think we're just going to keep seeing more examples of this, unfortunately in the year 2025.

So yeah, really, if you read the input, there's nothing that suggests That it should, you know, do anything and everything in its power to win. There's nothing that suggests that it's crucial or that it can think of other ways of winning aside from playing chess. It's very kind of pure in that sense. And in that sense, I think actually pretty significant. And now moving back to things related to data centers and power.

The story is that Elon Musk's XAI supercomputer gets 150 megawatts power boost. So, the Tennessee Valley Authority has now received approval to be able to receive this 150 megawatts of power for the massive XAI Computing Center, which would mean that. It is now possible to run the entire cluster of a hundred thousand GPUs, which was not previously possible. And unsurprisingly, some people are concerned about the impact for local power stability.

The claim here is that nobody would be impacted in any significant way.

And I think this is a really interesting consequence of how fast Elon moved, right? He built this whole facility, he had it all set up and with the hardware sitting there. And he kind of went, we'll figure out the energy side later. so that's kind of, kind of interesting. He also obviously stepped up and brought in all these Tesla battery packs, right, to get everything online in the interim. So really kind of janky creative engineering from XAI on this one and super impressive.

But yeah, that's what it takes to move fast in the space.

So pretty clear that. Data centers will start using more power and in a big way.

That's basically power rather than the like total energy consumed over a year, just because it gives you a sense of like what kind of capacity you need average, let's say to run these things. So 37 gigawatts a low end for the amount of power that would be required by 2028, 66 gigawatts the high end. And when you look at some of the build outs that we've been talking about, right, meta building that two gigawatt data center Amazon 960 megawatts, about a gig there, that chunk of power.

Like a large fraction of it is going to the hyperscalers, right? Like explicitly AGI oriented ambitions. So you're looking at dozens and dozens of gigawatts, certainly on track for that. and they also highlight too, that they did a report back in 2018. Something like that. And they found that the actual power usage in 2018 was higher than any scenario they predicted in their 2016 report. So they failed to predict the growth of AI servers, basically.

Like they just they highlight like, Hey, you know, this could happen again, which is really, I think, great great hedge. And yeah. So they also go into water as well in this report. Looking at, you know, how many billion liters of water will be required to, to cool these data centers. It's a environmental factor. Sure. But the bigger story here, I think is water availability locally is the big challenge.

So do you have enough water available at whatever site you're doing your build out at combined with what is the temperature of the site you're building out at, right? So like the government of Alberta, quite famously, it's pitching people on, you know, a hundred billion dollars of private investment to build data centers up there, Kevin O'Leary is like right in the middle of all that. And part of the reason is Alberta is really cold, right? So you want to cool these data centers.

It's a lot easier. and water availability is kind of a similar consideration when you look at these sorts of sites. So, anyway, yeah, important that that Congress is looking into this and the data seems really good. And I did like the sort of intellectual humility and awareness that, hey, you know, like other people are predicting differently. We've made mistakes in the past. And so, you know, if anything maybe tend a little bit north of of what our projections right now are indicating.

This has amidst a whole bunch of lawsuits from offers and you know, many different things we've covered over the past year, those lawsuits are presumably ongoing. And as per the title of the story, yeah, it hasn't come out. They said they're working on it. It'll be out by now. They have clearly deprioritized it and it is not out. And I figure there's no projection, I guess, of when it will be out or even if it'll be out.

So, yeah, I guess, for people who think that OpenAI and others that use things indiscriminately for training such as books and other online resources, yet another kind of example of that being the case.

And like, at a certain point, I think opening, I just has to be a little bit more careful because their word is no longer their bond apparently with an awful lot of these things. And so, quotes that they're sharing here from people internal to the company saying, well, you know, I don't think it was a priority. To be honest, I don't remember anyone working on it. to the extent that's true and OpenAI now is a pretty big org. So maybe that, you know, people who are asked just didn't know.

But, you know, there, there is a certain, to the extent that OpenAI was using this kind of progress to defend its assertion that it's a good player in the space, that it, you know, cares about copyright and it cares about your right to privacy, your right to your data and so on. This makes it much more difficult to take those claims seriously. You know, apparently there's a non employee who coordinates work with OpenAI and other entities.

And he said that they discussed the tool with OpenAI in the past, but that they haven't had any updates recently. And like it's just, it sounds like a dead project right now internally, but We'll see, maybe it'll come back, but it's one of those things again, like there's so much pressure right now to race and scale. Unfortunately, this is the very pressure that OpenAI itself so clearly predicted and anticipated in a lot of their corporate messaging around their corporate structure.

Here's why we have the corporate structure we have, so we can share the benefits of this and that. There's going to be racing dynamics that force us to make hard trade offs. We want to make sure we have a non profit board that isn't too Profit motivated to keep us honest and all that. And you just see all those guardrails melting. And again, I mean, you can make the argument like, well, if they can't build AGI, then they can't even affect or shape the world in any way.

The problem is that all these arguments seem to keep pointing in the same direction and that direction seems to keep being Sam A gets to do whatever he wants to build and scale as fast as possible while making safety and national security assurances to the American government in particular, that. Seem to keep falling flat. So, you know, this is another version of that. More on the privacy end of things. Less national security and more sort of your right to your own data.

Hopefully this editor thing will come together and this will come out actually before the end of a week, as has been the goal. please do keep listening and keep commenting and sharing and check out the discord that hopefully I will have made and you can start give us ideas for things to discuss and Comments, questions, all that sort of thing on there.