#197 - AI in Gmail+Docs, MiniMax-01, Titans, Transformer^2

My background is of having studied AI in grad school and now working at a gen AI startup. And not as usual. We do not have Jeremy. He's off doing some policy business maybe having to do with new administration. I don't know, but we do have some great co hosts from LatentSpace, the number one AI engineer Podcast and a community reaching over 2 million technical readers. They do a lot of interviews with very cool AI people.

So I think there'll be great co hosts and I'll just let them introduce themselves.

And I think you know, especially with the movement that was started since Chachi Beauty, like a lot of things that are now going into, into production and especially have the bar has been lowered a lot for, you know, Basically guys like me to be able to build AI products because we do interviews. That's, that's definitely a big part of our podcasts. And then we also do essays. We do conference coverage. So like we cover NeurIPS, ICMI, CLEAR and cover what the latest research is.

Alessio, did I, did I miss out anything?

So there's just some small updates. Primarily some, Pretty cool projects and open source and research stories similar to last week. And then some more updates on the actions that the Biden administration is going with before they close out. So yeah, a bit of a light news week and we'll probably be a bit short of unusual, but it'll give us some time to chat.

And then one more thing, we do want to acknowledge any listener comments and corrections seeing some more comments on YouTube, which is always nice. And we are having a bit of a chat on the new discord. So there is still the link in the description. You can go to AI discord, kind of different from latent space. And it's still, You know, still kind of forming, we'll see how it goes.

But the idea is I'll, I'll at least try to post some news stories and papers as we see them, rather than as we cover them on the podcast, and then we can chat about it there. So feel free to check it out. It's a, it's a fun spot and if nothing else, it's fun for me to see the sorts of people who listen to a podcast.

And before we get to the news, as usual, we do want to acknowledge our sponsor, which as has been the case for a little while now is Regenerator, Bobson College's interdisciplinary AI lab focused on entrepreneurial AI with Bobson being a number one school for entrepreneurship in the U. S. There was this initiative where professors from all across Bobson partnered with students to launch Regenerator.

And organize it into various groups like entrepreneurship and business innovation, AI ethics and society, the future of work and talent, and so on. They are peer training all of Bobson. So making sure that the faculty there is up to date on AI concepts and AI tools. And are, you know, I guess, guiding people in learning how to do entrepreneurship, innovation, and creativity with AI. So a very cool initiative.

And I guess Bobson as a college is going full in on AI And now onto the news, starting with tools and apps as usual. And to start with, we have kind of a duo of stories where neither is too huge, but I figured we can. Just pair them up. So first, Google is making Docs free, but raising the price of Workspace. So there are these Workspace apps you can pay for as a business like Gmail, Docs, and Sheets. And they've had the ability to pay for AI features 20 per user per month.

For the Gemini business plan. So Google is now making that free, but raising the base cost of subscription from 12 to 14, which kind of makes you wonder, I guess, if people were really into this Gemini business plan. And then just as that happened, Microsoft did something sort of similar. They have.

Co pilot for business, and they did some rebranding now it's free copilot for business is Microsoft 365 co pilot chat, and they are rebranding co pilot chat as Bing chat enterprise and sort of, emphasizing the agent angle here. And again, trying to encourage people to sign up for this 365 co pilot.

So yeah, real duo of stories there of Google and Microsoft, both having their kind of business offerings for people to pay and makes me, yeah, I do often wonder how many people find these things useful. The AI tools on Google kind of do email summaries. They do some automated meeting notes, some writing tools, nothing. So hard to say, I guess, how excited business people are about these subscriptions.

So what those companies are basically doing is like, Hey, look, you're already paying me 12 bucks. I'll just charge you 14 bucks. But then the way they're going to look at it internally, it's like, you know, instead of two bucks going to AI, it's like six bucks are going to AI and eight bucks are going to the previous product. So then the, the reporting or like the AI revenue and kind of like. ROI on the AI investments that these companies are doing kind of looks better.

So, a lot of it is financial engineering when it comes to this, but some of the, you know, maybe more midsize 10 to 30 billion market cap companies that I talk with, all of them started with their AI stuff for free. they're not trying to charge for AI separately because they don't really have a lot of leverage. In Microsoft's case, it's, you know, people are not moving out of, if you're a Microsoft shop and you're using teams, you're using SharePoint, you're using the stuff.

You're not going to move out, but it doesn't mean that you're going to pay for AI. But obviously Microsoft is under a lot of scrutiny to show that all this money they're pouring into AI and data centers is kind of getting ROIC. So you'll see, you'll see more. Of this, of this like AI bundling and pricing.

One thing, I was just kind of looking at their intro video, the co pilot chat intro video, and noticed, if you looked at the UI, it doesn't say what model they use. It doesn't say OpenAI anywhere. And this is a question of how much does Microsoft want to collaborate with OpenAI in the future? A little bit unknown, but you know, it's interesting that Microsoft used to have a very, very close relationship with OpenAI and now maybe less so.

they are seemingly kind of iterating on how to get people to do that with this Microsoft latest rebranding of the free version is co pilot chat. The paid version is just co pilot. They have the Various things that you pay $30 a month for. And then also they have these agents where you pay as you go, where they do research for you on the web and their graph of knowledge. So they're definitely expanding a product suite aggressively and try to get people to buy into it.

Okay. And next story, we have Google signing a deal with AP to deliver up-to-date news through Gemini. So AP is the associated. press. They do a lot of news delivery of breaking news. And now we are seeing yet another deal of a news organization. We've seen OpenAI already doing a deal with AP back in 2023. And now Google has done the same thing, presumably. You know, paying to integrate and give that breaking news. So we've talked a lot about, this is a trend with open AI over the past year.

And we are now starting to see, it seems other companies following in their footsteps.

But clearly, I mean, the like now this is possible because the AP has done it. Yeah. And clearly it's in the news organization's interest to strike a deal with every single LLM trainer on the planet to do this. Yeah, I mean, I really wonder how much they charge.

So Barron's is doing the same thing with the Wall Street Journal and some of these more financial services things where at the end of the day, they also need If Google, so to speak is going away and the social media algorithm that Elon has put in place downsize links all the time, you need to find some other way for people to come to your, to your product. So yeah, these deals can get quite expensive just based on.

What I've heard from, from people working in this order of magnitude, at the early stage, kind of like, you know, early companies are paying about, 750 K 1. 5 million for some of this data. I'm sure that, you know, OpenAI is maybe paying 50 million, 30 million, something like that, depending on the source and kind of like some of the more fine grain thing. Like, for example, can you also use for training, like things like that? There's kind of like a lot of parameters that go into it.

And perhaps that's also the case with others, and we'll, we'll get more stories like this.

And I think it's really interesting to think to view it from their perspective that they actually don't see that much difference between suing them and doing a deal with them.

It's, it's just a purely economic, like how, how can I get the most money out of out of my content play, which is like, just very interesting on the LLM lab side I think it's like kind of incriminating, like, you know, I'm not a legal strategist or anything, but like, you can say that AI training and training on data is fair use, therefore you don't have to pay for any of it, but like by paying and licensing, you kind of admit guilt so like, I don't think you can have it both ways.

But you know, obviously I think this will be played out in court.

Actually, it's very related that we have me straw signing a deal with AFP, the agent France, Press to offer up to date news in the chat. So the chat is there basically chat GPT or Gemini. It's a chat bot and very similar. It will now be able to give you access to stories as you chat to them.

So another example of that data war and that, almost search war going on as well, or I don't know if it's a war, but everyone is trying to get you to use their chat to see about stuff that's going on right now and Mistral trying to also compete in that chat bot space is, it's pretty curious because I feel like I don't know if a lot of

people are using let chat and I don't know if they'll be able to compete with chat GPT and Gemini and so on, which are already so dominant, but interesting to see them try. No comment there. And moving away from those stories to something a little different and dealing with open AI, we have a chat GPT having reminders and to dos. So we're adding this better feature called tasks in chat GPT, which is what it sounds like.

It can schedule future actions and reminders similar to stuff that Google assistant or Siri can do. It is available to paying people plus steam and pro subscribers. And you can have these one time or recurring tasks like daily weather reports or reminders. So again, kind of surprising to me from a product perspective to see them doing this kind of thing. To me, I always think of these chatbots as one of things as places you go. To do research or your work.

And here we're doing this thing that is very much like Siri and kind of positioning chat GPT is more of a personal assistant, which I don't think we've seen so much of in their product direction so far.

Do the same thing every time at a specific time. It's a, it's good. I don't know what the adoption is going to look like because it's kind of limited, right? You don't have a function calling, you don't have good interpreter. I think, we'll, we'll see what the numbers look like.

And this is a core loop, you know, of, what an agent needs to do and needs to be asynchronous. Lang chain happened to also release what they call ambient agents on Opening, I release tasks and you know, I think both of them are basically exploring spaces where you don't have to initiate the chat in order for the AI to do something. And I think like the general theme of ambient agents is something I can bet on or follow, explore throughout 2025.

I, I, like, I can see that chat GPT tasks is like on that path. It's just a very, very small step on that path.

And also to differentiate it because in my view, still there's not much of a distinction between chatGPT and Cloud and Gemini, you know, one chatbot is not too different from another. And so there's no stickiness there and whoever is cheaper and faster and better. is probably what you're going to go to. And I've personally been using cloud a lot more. So there is that product angle too, of how do you get people to stick around and commit to you.

And moving on to applications and business not too many exciting stories here, and we start with something pretty different from our usual kind of stories related to chatbots or LLMs. I think this one is pretty good to highlight and it is that Palmer Luckey's AI defense for company Endurall is building a 1 billion plant in Ohio. So Palmer Luckey, I guess, is being highlighted here as the pretty prominent CEO and founder of this company.

Previously, he did VR and started Oculus, what Meta is now doing. And after that started Enduro, which is a company building drones and really AI enabled drones for the military, for military applications, they have these Fury Roadrunner drones, Barracuda missiles with the idea of being there that these are AI enabled, more advanced and competing with maybe, I guess you could say a legacy Contractors and R and D shops for the military.

So clearly we're getting a lot of business considering they're building a 1 billion factory named Arsenal One. And I think really highlights that often we are not talking about AI and it's military implications. We haven't seen any sort of advanced AI so far be a major player in Wars or ongoing battles. But I think it's safe to say that it is going to happen at some point. And certainly people are investing and moving in that direction.

So, and it an important thing to keep in mind in this whole push for AI investment and that kind of rapid improvement of AI is that it is going to play into this aspect of technology as well.

Owning the infrastructure for a lot of these things matters and where the data centers allocated matters and how much, you know, what models running matters. So, yeah, it's not, it's, I think that's kind of the I think Anton from Chroma I'd done this like fake protests, basically like, They're like, it's illegal to not build autonomous warfare systems because you're putting human lives at risk instead, and so you should just send the robots to fight for you.

So I think you'll, you'll see more, more of this in the next few years, now that AI is kind of like in the, in the zeitgeist

So that's another angle here as well, where AI certainly is already massively being used for surveillance or in this case, I guess, detection of people that is, massively the case in China from what I've heard. And this company is now offering that to some degree in the U S as well. And one more thing I'll mention is I think we're Silicon Valley as a whole, my impression is, is warming up to the idea of working with the military and working with the defense sector.

We've seen Anthropic and Meta both announced that their products can be used for military applications by the defense system by the DOD. And in the past, Silicon Valley type software engineers are very resistant to that sort of thing. We've seen, you know, protests in Google when they're, I guess, AI division, cloud AI cloud division was working on some stuff for military. I think it was project Maven and that was met with a lot of outcry.

My general vibe check on Silicon Valley is that people are, Becoming more open to it if nothing else, because presumably it's very lucrative.

I think now there's a lot more maybe VC interests, which I think then makes people want to talk more about it because then they want to raise money to go do a company in that space. So there's more, more of that at play, but Yeah, I don't know. I mean, I'm becoming a U S citizen in five days. So my, my view, congrats,

We've seen them engage in content sharing and licensing deals, but this is the first time they are actually, I guess, helping a publisher expand. I would imagine that's to get closer ties to kind of become more closely aligned with these media companies. Another interesting development in this whole story of, I guess, what you call the data wars and not one I would have expected, you know, paying for media companies and their initiatives.

And on to another type of business story, one we see every once in a while, we have a new startup from a famous AI person focused on AGI. And this time it's Cholet, I would guess is how it's called. They have a new startup called Endia. And the story is that they are Going to shoot for AGI through, it seems like program synthesis. This is co founded with Mike Knoop, who they are already hiring. And Sholay, his background is having created Keras, which is a very popular I guess, platform.

Package that a lot of people use it for coding with, deep learning back like a decade ago. He worked at Google for a while. Kerris was bought out by Google or adopted by Google. And yeah, it's, it's part of a trend in a way where we've seen, obviously, people leaving open AI and doing AGI startups. And this is another example of that. And, I guess it's just the feeling now that you're close to AGI and you can kind of shoot for it, perhaps.

But my, can I actually play soccer together on Thursday nights? So I already messaged him about doing a podcast episode with us. So we'll, we'll learn more. Well, they'll, they'll talk more about publicly soon.

But in general people are aiming to do this in the next one, two years horizon and not super far in advance. And speaking of VC type things, we do have a couple of stories related to fundraising and evaluations. First, we have the company Synphasia, which is an AI video platform. They're focused on sort of AI generated, Videos of human avatars things that you can use for, I don't know, like marketing materials or maybe internal videos things like that. have raised 180 million.

Their valuation is now at 2. 1 billion. So not in the LLM space, not in the chatbot space, but we have seen Synthesia, I think Previously also covered them. And I do think that this is one example where AI video will be pretty successful with kind of this inclusion of synthesizing videos of people talking or, perhaps doing some things in commercials. Much cheaper than hiring models to do that for you, much easier to adapt to multiple languages to quickly iterate on messaging.

We've seen this done for politics, for example, in South Korea, I think we covered that story. So clearly based on this fundraising and the valuation of this company being at 2 billion, the, Presumed expectation of the company's bankrolling this is that many businesses are going to go for this.

And they're, you know, I think they, they do much more sort of broad things than just like helping people in the creator economy. Like this is much more like support or sales or anything like that. But I think it will keep growing. Like, Hey Jen is the other one that's kind of up and coming in this area. And I think we as creators, like literally like us in this room as podcasters should also think about using it.

Next cursor has kind of a few more features that set it apart, that are more advanced. I know I'm personally a user, a lot of people at my company are users. So it does seem like it is positioned to, perhaps lead in this very, very competitive space. There has been many startups, betting on doing, I guess, coding assistance, a lot of fundraising for that. I think you saw YC having like multiple startups in the space in a single batch. So very competitive.

And it seems right now that this company is a major competitor and potential one to come out on top.

Last year of people forking VS code and trying to compete the cursor with their really good execution. And they were basically first, like we, we actually had them on the podcast. I think we were the first podcast that they ever did and seeing them execute. And then also they, they came and spoke at my conference last year, like very sort of heads down determined. Has a, with a very strong point of view as to like how people should be coding.

Like they didn't promise, you know, super AGI agents or anything like that. They just said like, we will make a better idea for you with AI in it. And I think typically what people usually say is that, you know, challenges like this would force the incumbent VS code to do something. And VS code and GitHub have done something. They just. Are usually copying cursor. And I don't know if that's like the strategy.

I think composer, it's like just as good on the coding side and you're still within your environment and you know, it has a lot of advantages. So I'm curious to see where things go, you know, from here.

And then there's kind of like replit kind of, you know, it's kind of been a little left behind, at least in the Zeitgeist, you know, they're obviously making a lot of progress on the product, but it's a competitive space, the, it's hard to get developer mindshare and, When you only have the ID form factor, it's easy for people to switch around. You know, I can open the same repo and VS code and cursor at the same time. It's like the same, the same thing, you know?

So, yeah, I, I'm sure the, the worst will get even more intense this year.

My thesis has, has gone has died basically because now it's pretty clear that cursor is is rumored to be Anthropic speakers, customer now, something like that, like that, you know, they, they, they definitely ran into limits with Anthropic multiple startups claim this, so I don't know how true that is, or, you know. What the, what the exact positioning of this is. But yeah, I mean, I think, I think applications, you know, we're in sort of the applications layer of the podcast or whatever.

We they typically want to be multi model and then the the LLM labs, they all want to be sort of vertical, right? They, they want to build their chat GPT. They want to build their tasks and reminders and all that. I think it's one of these, it would just be a fight back and forth between applications and the foundation model labs.

So again, another example for a more domain specific company that is aiming to dominate a specific space, in this case, AI for legal applications, where it's obviously very important to not hallucinate, to give accurate information as you're doing, for instance, like research for a certain legal case. And Harvey was one of the early players in that space. And. Seemingly is already getting a lot of revenue. We saw them make some deals with major law firms covered that a while ago.

So another one that is seeming to probably stick around in my view.

And is this because the legal space is super hot or is this because the investors are very desperate to get into this thing? What's,

I'm not leaking anything, but but Harvey's kind of like broadening from like just legal to like all professional services. And same thing is happening with Habia, which started as more financial services focus to start. And now it's kind of like going towards all kinds of service businesses and knowledge work. So, they have a bunch of overlapping investors. So I know some of the investors are a little nervous about the collision path that they're on.

But yeah, I, I think theoretically the market is bigger. If you do all professional services work in practice, it seems hard to build a company that does serves every professional services market. So, yeah, I don't know, but I think since he was already an investor in cursor, so maybe they got to stick to it. and y'all always gotta like do the math also on the. How much dilution the founders are actually taking. So it's not always, it's not always just about the evaluation.

I'm just like there, there's supposed to be this thesis of like, you know, you raise once and you're done or, you know, you're, you're profitable or unicorn 10 percent startup. You know, that's, that's what gum loop is, is touting, but then like, you know, both Cursor and Harvey, they're kind of following the traditional path. I'm sure they, they haven't touched this in their previous round. They all say this and you know, money's just sitting in the bank.

And so what is it for sales and marketing? I don't know.

I'm sure Cursor is mostly spending it on compute.

Cause that's the pattern that's been established by GitHub Copilot early on. And if you're going to try and go above that, then To be very expensive and something like cursor where it's doing ambient completions for you. It's like, as you code it, just offering stuff for you all the time. It's also looking forward to see about suggestions, not where you are. It's able to do inference over multiple files, answer long questions.

Yeah. You've got to wonder if one 20 subscription per month is actually going to pay for what a typical programmer who uses cursor every day. Is querying and moving on to projects and open source. We've got a couple fun stories here, starting with Minimax oh one, which is all about scaling foundation models. So they have minimax text oh one and mini. Minimax VL, Vision Language 01. And the big story is that they're meant to handle lower longer context.

So they use lighting intention and mixture of experts to have a total of 456 billion parameters with 45 billion Active Webber token, pretty big for this class of models. And they're saying that these can handle context windows of up to 1 million tokens during training, and they can even take 4 million tokens during inference. So, this is one of the big stories since chat gpt came out. It used to be that the token windows were like this big.

4, 000 tokens, 8, 000 tokens as the default, wherever context window is, how long of an input you can handle, right? So can you handle like a book's worth of content? Can you handle multiple documents? So this is an example where by using a newer form of attention, Typically, that's one of the major tricks for being able to do this kind of long context. You are able to scale.

And so Minimax says that they offer context windows that are 20 to 30 times longer than other Open models, and these are publicly available. So another kind of class of open source model that is competitive with top of the line models like GPT 4. 0 and Cloud 3. 5 Sonnet.

QVQ, these are now major players in the space of models that are competitive with Lama often and other things that have been out there also deep seek stuff like that.

So mostly these are just kind of training models where you could kind of distill into smaller ones that you actually use. It's just interesting that they, they choose to launch their text models like this. Usually it's the other way. Like you remember Mistral launched their 8B. And then they went to eight by 22 B. Here it is. The Chinese are doing the opposite. They're just going big.

Lama or other models that basically lay out for you all of the nitty gritty of training models, of optimizing them, dealing with long context windows. So I guess another interesting aspect in addition to there being pretty competitive open source offerings for various LLM tasks, now also for vision language tasks, is there's no kind of secret Formula for having a good LLM as, as far as I know, it's just like, you need good data and you need to have the infrastructure.

And then the details of how you train and set up your model are not very much of a secret at all. And onto the next model, this one is called the MinMo and it is covering another aspect of a space that is less seen in open source. This is a multimodal large language model. That is more so focused on the voice integration and audio. So again, this is coming from China, Tongyi lab and Alibaba. This is an 8 billion parameter model, and it is focusing on voice comprehension and generation.

So this is dealing with speech recognition, speech to text and text to speech. Again, another example, I think we've mentioned recently how vision language is Area of it hasn't seen as many open source models and they're starting to come out. Now we have pretty impressive example of speech to text. It's apparently competitive with whisper large V3. It's getting 85 percent accuracy in language identification and 98 percent accuracy in dialect and other tasks. So. Pretty impressive.

And again, pretty interesting that you're seeing more and more models of the sword being developed. And in this example, they are saying that the code and models will come soon. So far, we just have the paper. And I get another example, I guess, of something you've seen where we've seen previews and papers come out. Before they are able to do a full launch here. We're not seeing a model or code yet, but we do see the numbers and the paper. So let's see if they follow through, I guess.

And onto the very last example here is a benchmark and that's another space where there are many, I guess, things happening all the time and many new benchmarks for different size of the AI. Inference game, I guess you could say in this example, it's halogen and the paper or the report has this fun thing with title fantastic LLM hallucinations and where to find them. That's referring to some Harry Potter movies. Anyway.

This one is focused on identifying on benchmarking LLMs for how many hallucinations they have, often how many times do they say things that are essentially false. This has almost 11, 000 prompts and automatic verifiers that look to see in the LLM outputs, how often they produce reliable. knowledge and facts. So it's interesting, I think they show that even top performing models can have hallucination rates as high as 86 percent in some domains.

My personal experience and kind of impression is that hallucination is Increasingly not a real problem. You know, when I do work on some sort of coding or some sort of, I don't know I guess some I just want to find out about some topic. I'm not super worried about hallucination and them just you know, importing. Packages that don't exist or doing some silly function call for something that isn't there.

But at least in this example, they are saying that there are still some pretty major hallucinations on things like coding. Text simplification, historical events, and so on.

And obviously it wouldn't know anything in, in that sort of like a beta API. So that was like very annoying. But also for large report generation, like the Gemini deep research of the world let's go back to from called bright wave that generates reports from a lot of data sources. And I run AI news, which also generates. from a lot of resources into, into a report that gets sent out every day. Yeah, every single day there's hallucinations.

It doesn't understand that, what sources are tied with what. So sometimes they'll confuse things and sometimes it will not fact check something that was like, obviously wrong. If you know, even like the little thing about like, let's say like Eric Schmidt, like running meta. I'm like, no, like that's, that's wrong, you know, but like the model doesn't know that. So like we've caught it. Like I get reports from my readers every single time this happens. And I'm like, yeah, sorry, man.

Like I'm trying, but the best I could do is just have another agent check the agent. But like,

In the research space on how to get beyond that, how to be able to update the models as you use them so that they have ongoing updating attention. One example of that is TTT, which we covered back in Like mid 2024. Well, now there's a new variant of that, that people are pretty excited about. It is called Titans. And the idea is we have a few ways to integrate it.

So at a high level, what we are proposing is a. Formulation of novelty loss of being able to see what you want to put into your long term memory based on how novel it is. They also propose a specific way to do forgetting. So they have this decay model where over time you can let go of information that's not relevant. And they also look into how to integrate memory in your architecture. So there's different ways you can have memory as what you're doing.

They have one Variant called memory in context, where you have multiple parts of your model. You have persistent memory, which is your typical LM that has knowledge that is there forever. You have your long term memory, neural memory, which is the memory that gets updated as you go. Context as architecture is appending into your Input. So you take some input sequence, you add some information from your persistent memory, and then you also add some knowledge.

from your long term memory into the input based on retrieving some of it with regards to the input. And then you do attention over all of that. And as part of your output, you update the neural memory with this novelty metric. Another example of what you can do is memory as gate, where you, Add memory into your attention, attention calculation. So you take your input and then as you calculate the typical sort of attention over the input, you, take into account the longterm memory.

And then another example they have is memory as a layer, which I forget is more traditionally what you do with things like recurrent models, where you pass the input through your memory and then you update the memory and then you And also have a kind of middle layers almost of the model, be the memory.

So anyway, they have this, test time memory where you actually update the models based on the novelty of the inputs and as you might expect, they show in comparison to various models of memory, like transformer plus plus, Mamba, Delta net, TTT. They outperform all of them. By a decent metric. So this is one of these problems that's unsolved for LLMs, right?

Where if you have an agent that's deployed out there, if you have an assistant that you're using, the fact that it can't remember things long term, things that you've seen, that you've chatted about is a real hindrance and a real, I think, reason that AGI is not possible if you're not kind of learning as you go. There's been more and more research on this, which is building on a lot of that. We have detailed comparisons to various things like Mamba and like TTT.

And people were pretty excited cause there's a lot of detail in this paper and the numbers of course, on various benchmarks look very solid.

You store some stuff and it gets added to the input here, actually updating the weights at test time. And that's much harder.

So. Again, as with any of these papers that look into alternative architectures, we've talked about Mamba a lot. And that addresses a similar problem where you can have ongoing memory over a lot of time. Not clear when we'll see some of these things actually have an impact on. Things that are out there in real world, but always exciting to see developments in the core architecture of how we build these large language models.

And what they do say is you're able to scale to context window sizes larger than 2 million tokens and maintain high accuracy and things like needle in the haystack scenarios. So. And a real surprise for me personally that we've been able to scale context so well without using some sort of memory like this.

And what we do is given your input, they have a few different ways to be able to decide on what set set of, what type of task it is, you could say, and then they. Take that as the right sort of set of weights to use. And then they put your input through the model a second time. And then that kind of gets you better. Performance.

So another example, in a way of we would say as adaptation over time, although in this case, I was a little disappointed in the general idea where they do need to pre train different specific variants of a model, what they call tasks, specific expert vectors. Which are then mixed in to the weights. So it's not quite as dynamic as something like memory here. But you still have just a few variants and you pick between them.

they do say that this outperforms things like Laura and is more efficient takes fewer parameters. And Laura is one of the very important techniques out there for taking Base model and customizing it to a certain task. So in that sense, this is pretty cool. And then there's some pretty nitty gritty details of doing SVD and so on. But, anyway, another example of research on adopting LLM students.

But one thing that I, I don't really know is like how this relates to representation fine tuning, because I mean, I just, I just searched the paper. It doesn't really mention REFT, but REFT was one of the best papers at NeurIPS last year. And like, it's kind of the same thing in terms of like people are understand that lower layers are good, but like tuning like representations over the entire model is better.

And transformer squared kind of does that, like it uses the inference to do the sort of representation discovery, which is kind of cool. But in all other respects, it feels like just representation fine tuning. And yeah, I guess people are exploring this, but at the same time, I am not so sure, like, if it'll get mass adoption yet, because we haven't seen Rift get mass adoption.

And so typically one way to get better images is just to do more denoising steps, more iterations of fine tuning your image towards the the Best, I guess, denoised variant of it. And that's how you train it. Typically is you take a real image, you add noise and you train the model to do these denoising steps. Well, this paper is looking at what else can you do beyond just doing more denoising steps to get better outputs. And what they find is.

One example is you can search in the space of what noise you are adding. So you don't just do more denoising, you also do different types of denoising and find the best output for your, diffusion model that way, which is, I suppose somewhat big deal, right? Inference time scaling is the hot thing now in LLNs and. Image generation is not seeing that much competition, I think. But if this could be one of the differentiators for ChajBT, for GemIIni, for the multiple kind of.

Places competing for text to image could be a significant part of a model that gets you that last kind of Delta in the quality of your output.

This is known as the interim final rule on AI diffusion and these are, doing some new, I guess, categorization. So there are. New now free groups, strong allies like Japan and South Korea, they are adversaries like China and Russia, and a third group that is most countries. And so each of these groups have varying levels of restrictions on AI chip purposes. So that's countries like Mexico protocol and Israel now face a cap of 50, 000 GPUs per country. Which is kind of surprising.

I mean, we've seen most of the restrictions on China, for example. Now it's on any other country. Nvidia has criticized these proposed rules. So these are not yet in action as unprecedented and misguided quote, because it does seem like a very. Big move to yet again, restrict the ability of Nvidia to sell to other countries in a very significant way.

Question mark on how some of these things will fair out. And the other thing is the restrictions are not geographic base, but about ownership base. So for example, if you are, you know, Volvo, which is like a car maker in Sweden, they're actually majority owned by a Chinese company. So they would also be limited to GPU access. There's kind of like a lot of. Implications of this thing. But also, you know, we're going to get a new administration four days. Right. Yeah. It's hard to

So they're involving the DOD and Department of To lease sites for these data centers. And yeah, another example, I think of, it seems like they're clearing out their priorities, signing a lot of executive orders, and this is addressing one of the popular limitations with how hard it is to get permits and so on for data centers. Again, we'll be curious to see what happens as the new administration comes in.

They are saying that we should streamline industry engagement with national security, establish export controls for these models. They also are advocating for publicly available information for model training. So in on the copyright question and so on. So, yeah, another example like regulation in the US is in an, in an open area, pretty fluxy.

So not too surprising that OpenAI is trying to influence wherever it goes and interesting if, if they released this blueprint to time it with the new administration coming in. And one last story in the section dealing more, I guess, with societal impacts, not policy and not quite safety, but I think sort of related, this is about a Pew Research Center survey that found that 26 percent of U. S. teens aged 13 to 17 have used chatGPT for schoolwork, which is Double what that was two years ago.

Although I don't know what two years ago was. Cause that's when the chat GPT came out, like a couple of months after that. So I guess maybe sort of like that. Anyway, over half of the teenagers also believe that it's acceptable to use chat GPT for researching new subjects. A lot of them, like more like 20%, also approved for essay writing. So I guess it's, it's interesting to see a glimpse.

It's often surprising to me and to people in the AI space, how little impact chat gbt and visa LLMs are seeing, how many people are not using them or not aware of them, even. To see that a quarter of us teens are using them. Clearly it has a massive impact on education already. And that's one of the areas where it is very disruptive. So that is really painting a picture on education and I guess, school work as being an area of that needs to evolve now that we have stuff like this.

So actually some statements he has made as part of this, I guess, the YouTube part of the story, the headline is he said, for example, YouTube, I think may end up hosting some stuff that people pirate for some period of time by. YouTube is trying to take that stuff down. This is again, dealing with LibGen that had pirated content. And he also said, and the vast majority of the stuff on YouTube, I would assume is kind of good and they have license to do.

So I don't know, I guess not really worth highlighting the YouTube. There's some other things that happened in the position. Zuckerberg claims he hadn't really heard of LibGen. He says, I get what you're trying to get me to give an opinion of LibGen, which I haven't really heard of, it's just that I don't have knowledge of that specific thing.

So, yet another thing he said, so would I want to have a policy against people using YouTube because some of the content may be No lots of statements by Zuckerberg in this deposition, nothing particularly surprising, but does point to this LibGen case actually giving us some updates and some interesting things where we haven't seen many of these legal battles have progress or have any problems. updates on what's going on in them. And that's it for this episode.

Thank you to our listeners for listening to yet another episode of Last Week in AI. As always, you can go to lastweekin. ai for the newsletter at lastweekinai. com for the notes for this podcast, all the links, if you want to open them on your Browser. Sean and Alessio for co hosting once again, Layton Space, very cool resource. Check it out. And yeah, again, thanks for co hosting.