HockeyStick #2 - LLMs in production - Chris Brousseau & Matt Sharp

It reverberated through a lot of levels of society, including, our own.

And, I think in many ways, technology and writing code might be the easiest use case for, this kind of models, right? Do you agree with that? I don't know if I completely agree with it, because, code is incredibly syntactically dependent, right? every developer who's worked with JavaScript or C++ and then moves to Python, they feel it, right? That's one of the biggest complaints is "I hate Python syntax".

"I hate that white space matters", it's a little bit more complex than just repeating whatever natural language happened, but you're absolutely right that is one of the best use cases so far. because, it's better structured than just spoken language, or is there any other reasons that make it so well suited for that particular application? programming languages are not real languages, right?

one of the things that makes it simultaneously very well and ill-suited for it is how much gets repeated, You use the exact same words. The exact same tokens to define every function that you make, but then the function's name can be whatever you want. And so using the exact same tokens is awesome. That provides landmarks for the probability as it's going through all of this.

But then that input to just say whatever you want and put it in camel case or snake case or whatever, tons of different formatting for functions. it makes it a little bit more difficult. Especially while you're trying to tokenize that, one of the big benefits with code is the amount of data we have around code. lots of people are writing code. they all have very similar ideas of what they're trying to do, of what they're trying to architect, of what they're trying to design.

and so we're not necessarily worrying about, hallucinations or fake news or, people disagreeing or other things like that. there's just a lot of data, that all agrees with each other and pushes in the same direction. It makes it good. there's obviously some negatives of just assuming, some of these LLMs writing code is going to do things well, but, I think Chris highlighted that already. it's actually really similar to how regular languages work.

If we have more python data, like Matt's saying, it's going to do better at python.

And that can create a little bit of a positive feedback loop with LLMs, where a lot of people want to get into python, and they're very good at it, but then when you look at emerging languages like mojo, for example It's really difficult to find that data and so LLMs are worse at it, similar to natural languages that have a lower number of speakers, a lower presence on the internet, So is the solution to use an LLM to generate a lot of Mojo and make it a significant percentage of GitHub?

that'd be fun, dude. I think there are some problems with synthetic data that can lead to stuff like model collapse. I don't know if we're going to see that in the code space, though. I think we could see that in natural language. So that might be a valid solution. Okay. the date is 13 February, the day before Valentine's Day 2024. I'm going to ask you for a wild prediction. Where do you see that going?

Should, all kinds of, or maybe any subset of programmers who, produce code as a job, should they start at least worrying? Is that something that's going to, decrease the pool of available jobs, no, I don't think it's really going to impact the amount of work.

I just think about my job, and even when I'm in very technical roles, and I'm spending 50% of my time on the keyboard, still, it feels like a majority of the work is still just communicating with stakeholders, understanding exactly what the problems are, technical writing, design docs, really understanding at a high level, what you want to build. To be fair, programmers have been automating the 'writing the code' portion forever, right? From the beginning.

yeah, with massive amounts of like scripts and configs that they use. And that's why they love Vim or Emacs still, right? It's because they have it configured just right. And they can move really quickly, because it provides a lot of that automation for them already, but this is just helping junior engineers already have all that configuration and set up really quickly, right?

It mostly will just make our jobs a little bit easier, it doesn't remove the need to really understand the engineering aspect, the architecture aspect, the design aspect that still is involved with coding. Oh, yeah. this is why we love comparing LLMs to a printing press. That Johannes Gutenberg. Because did that destroy the writing industry? All it did was it destroyed the monopoly that certain organizations had on publishing books.

Before you had to get a scribe and you had to pay the scribe and you had to have access to scribes You couldn't just walk up to a printing press and hit it and then boom you have a book. You have to have knowledge You have to have an idea. The printing press just gives you a lower barrier to entry Which is what we love, right? For coding, I think Matt is exactly right, that it's a lower barrier to entry for junior engineers to be able to produce significantly better work.

and in some ways it actually accelerates it, because when you copy and paste what an LLM gave you and it doesn't work, you have to go figure it out, right? With the junior engineers, it also helps speed up senior engineers, and staff engineers and principal engineers. it's good, and lowers the barrier for the entire industry, we like that.

Yeah. I've lately been spending lots of time writing chapter 10 of our book, and in chapter 10, we actually go through a project, where we help you build your own co pilot and we build the VS Code extension to get it in. if you want to be running your own LLM on your own computer with your own data, so that way, you can get your own things. we walk through all the steps to do that. And in some aspects, it's interesting cause sometimes. adding an extra feature, made the model work, right?

there's still just so much to learn about it. ultimately, it comes down to your data, right? how good is your coding data? is really how well the co pilot works, right? SQL is one of the most repetitive of all of the programming languages. but true skill with SQL does not involve being good at SQL. It involves knowing the data, right?

It's knowing which tables to query, how to merge them, how window functions, all of that stuff, knowing exactly what you need to be looking at is the true skill in SQL. And we're hopefully getting to a point where we can help the model know the data, right? We can give it some sort of context for the data that it's going to be looking at, so that it can generate good SQL that's a really good point. I've actually had, lots of mentees who are trying to learn SQL for the first time.

that's a perfect example, understanding how to write the code is only half the problem.

Understanding how to integrate it is really the bigger problem. What's the most terrible use case, that people are currently trying to use LLMs for? What does LLM in general, or LLMs, what do they suck at the most? I'm going to say they, they suck at, sequence prediction, which sounds so off.

Because that's what they're made for, but one of the things that I'm seeing people do, is try and automate entire workflows with LLMs, and they're trying to get the LLM to just do the whole workflow and they suck at that what they need all of this stuff to help it.

They need tools, they need rag, they need specific fine tuning landmarks and they need few shot prompting, they need all sorts of stuff to make it work, and then it's still up in the air about whether or not it will do the right task in the right order. Yeah, I was thinking, I don't know how much I'm seeing this.

But, three months, six months ago, I was hearing a hundred horror stories about, essentially CEOs being like, "we need LLMs" and like their magic, they can do anything, And so it didn't matter what the problem was, "oh, we need to, do outlier detection using LLMs". No, use stats for that. yeah, outlier detection is really a statistical problem. It's really a data and math problem. LLMs are good at natural language.

And so when we can solve a problem using words and communication, that's when LLMs can get in. But problems like, outlier detection or weather prediction or these other things, we have, algorithm. stock market prediction, Super Bowl prediction, All these things, we have better ways to make predictions. And it's called math, right? Fourier transforms, other machine learning algorithms, other things like that.

LLMs are not good at doing those things, cause we don't talk about them in natural language. we've invented other languages like math just to describe them And that's why they're not good. we can make tools, you can build functions for an LLM to use to do Fourier transitions and whatever else, right? But getting the LLM to know that it needs to do that is really difficult.

Probably just as difficult to, as explaining what the Fourier transition is to an LLM within your training data to get it to be able to replicate it. This is one thing that makes it almost miraculous when stuff does work, and that's that feeling that we're chasing right now, and that's the replicability that we're trying to help people get to in a book.

how do you actually do it, and how do you make sure that your scope is small enough, that it will work repeatedly and you can build a product off of it, that's difficult. I'm a big fan of chess. And, since ChatGPT came out, lots of people have been making memes, or just

it just magically creates a knight out of nowhere, and magically, will take its own pieces as it moves its pieces around, it's always pretty funny. And even though it's cheating the entire way, it almost always loses, right? Cause It doesn't have an understanding of chess, like it doesn't have that model underneath it. sure we can talk about it in language, but not really, right? So we, we still have better ways to play chess, alpha zero, et cetera.

Stockfish, like there are engines out there that play chess really well. And we don't need to make LLMs good at chess, but that's a very good example of one of the things it's not good at. I've seen someone on Twitter who said "I'm gonna give LLM $1000 or whatever initial amount, and I'm gonna ask it how to best invest it. I didn't follow where it went. But I think a lot of people had the same idea. this is some kind of genius system.

I'm just gonna be its flesh and bones agent in the real world. and hope for the best. So I think that kind of goes back to your chess thing. So excuse me for that, but I have to ask you the AGI, Artificial General Intelligence. Any chance for that happening anytime soon? What's your prediction? not with our current systems. No, I don't think AGI is ever going to come out of quadratic equations, like not a single chance.

maybe if there are better dropping sub-quadratic replacements, stuff like hyena, I've tested that out. I think it's really cool. But, the fact that attention, the query key value attention, ultimately generates complex numbers. I think that is a little too much for AGI at the moment. So you're not one of those people who secretly hope that OpenAI has something they're gonna release soon. I don't think they have it, right? I'll be hopeful, sure. If it comes out, that's great.

Yeah, I'm of the same mind as Chris. I hope they keep pursuing it. we've gotten major breakthroughs from what they pursued. It's very possible AGI will happen in my lifetime, I'm still pretty young We keep on making advances really quickly, but are we relatively close to it? Probably not. No Oh, the thing about progress though is that it's very rarely linear, It tends to have a very weird curve. So that's why all the predictions are so funny, but hey, I had to ask you anyway.

No, I think it's a great question. Okay, let's delve a little bit into, a portion of your book, It's basically describing the two options that you have today. you can either go and pay some money to OpenAI, maybe Google, or somebody else, or you can build, So you've got buy versus build. Could you talk to me a little bit about how someone would decide about this as of february 13, 2024. What's the things to consider, and what's the weights that you would put in, and biases?

the basic consideration is just your use case, right? If you just want to test something out, you're a student and you don't have a lot of budget, and you want something up and running so that you have LLM experience, I would say just, shell out for that, ChatGPT+ or buy Anthropic or Google Bard has a fantastic API, or I guess Gemini now just do it. it's not that big of a thing.

If your product that you're trying to ship is inconsequential and you don't need it to be right every time, you just want to sprinkle the AI pixie dust on it, just buy it.

If your use case goes deeper than that, though, if you want to be able to build your own, if you need to make sure that it says the right things all the time, if you need it to behave a little bit more deterministically, There have been probably a thousand case studies in the last year of people building products on top of ChatGPT and then OpenAI rolling out an update that changes how chat GPT behaves, and they don't have any way to measure all of the different

ways that it will change it, right? There are 176 billion parameters in GPT-3 alone, they don't know it's going to break your program down the line. they're just going to update it for what they consider to be better. And those programs break constantly. that doesn't mean you can't fix them. It's just a much bigger problem of maintenance, than I think a lot of people are expecting going into it. So If you want to have to maintain it less, build your own.

Yeah, I think the other aspect is like you want that control, right? there's lots of examples of companies who, essentially built a small shell around ChatGPT that did something unique. And then, months down the line, now ChatGPT just does that out of the gate, right? their value proposition just completely disappeared. And that's because they didn't have control over the model. They didn't have, control over, what it did it's just interesting, right?

Because I say these things and things have changed over time. But when ChatGT first came out, it was free, it was a demo, and they were specifically doing it to collect data. And that's what they did, they used collected data to improve their models. And that's what they continued to do for a while, right? Oh no, they're back. They it's terms and service, right? If you want them to save your chat, so that you can return to it and ask more questions, they get to train off of your data.

So if you want to put anything private or sensitive in there, like it's over, you've just leaked it.

they're back and forth about what data they're collecting, what data they're not collecting, and if you're with an enterprise customer, like maybe you can make certain rules and things like that, and oftentimes they won't, it's a minefield, for how people are using it, and so it's just something important to take into consideration, if your LLM model is doing something magical, that's really core to your business, that is really driving customers. You want to control that.

You want to make sure that the model is working exactly as intended. You're not getting updates randomly, that break your application. You're also controlling the data flow, you're making sure that you're not accidentally training your competitor's model, and other things like that. And there's just lots of aspects where it's just important to make sure that you own it. And, no, that's not necessarily everyone's concern, right?

if you're a student or you're just doing some side project or anything, there's lots of APIs out there that are very cheap that can get you up and running, there are literally hundreds of hugging face spaces that are free APIs. With, have LLMs running behind them and you can just hit them whenever you want, right? unless you're queuing behind a thousand other people. yeah, exactly.

I liked the example you gave in the book, I think people at Latitude, the Dungeons & Dragons people would agree with a lot of what you're saying now, but can you tell the story of what happened with them? Latitude, is a local company, that was here in Utah. it was put together by, two guys from BYU. GPT-2 came out several years ago. They're like, "Oh, this is mind-boggling.

Let's build a game off of it!" And what they came up with was like a dungeon crawler, a text based game it was really neat, because it would just generate, an infinite amount of opportunities. And so it created this 'choose your own adventure'. It got relatively big in the space, and lots of people enjoyed playing it. things were going really good, and then OpenAI GPT-3 came out, they offered it to them, hey, we can, we have this new model, it's a lot better, why don't you try it?

they played around with it, and "oh yeah, this is, it's much more descriptive, it's much more interesting, it's really great", There was a lot of excitement around it, however, it turned out that the model itself, had a propensity to, generate smut, and it got really concerning people would write like, "I'm an eight year old girl", and then the model would complete it saying "....and I'm wearing a skimpy outfit", And oh, whoa, like the player didn't want that, but like the model generated it.

there became this big feud between OpenAI and Latitude about creating filters. "hey, we don't want your players doing that. We don't like that". And, Latitude's "okay, we'll create some filters" and things like that. And it devolved really quickly. Latitude being a very startup, not necessarily knowing everything they were doing, they built a very shaky filtering system, and then OpenAI was "that's not good enough".

So then they started banning players, and so eventually we got to this territory where players - paying customers would be playing a game, the model would randomly generate, something that the filtering system didn't like, and then they would get banned. Cause it's like the game just did itself. It was a very complicated time, and there was lots of back and forth between Latitude, who's a small company, and OpenAI.

There's lots of ' he said they said' going on, but ultimately, it's just this position where Latitude They had this game that was completely dependent on OpenAI's model to generate good output, and it really caused a lot of drama between the players and Latitude and, OpenAI in the background and that is a critical example of LLM was very critical to their business, If they owned it, then they could have controlled it, they could have made sure that from the model aspect,

they could have trained the model to make sure it didn't do any of those things. And then they would never need to play the little blame game, right? Nobody likes to play that game. That's whose fault is it, that the model is generating bad stuff. Is it the player who's prompting it? Is it Latitude who has some systems for tokenizing and preparing player output before it goes to OpenAI? Is it OpenAI because their model is generating that?

Is it Latitude for post processing the content from OpenAI before they serve it to the player. I don't even know if it really matters who's to blame. it's just a sucky game to play.

and that's like the ultimate example of why you might want to consider build versus buy is if you buy from any provider, we're picking on OpenAI here, because they're a big player, but you buy from Anthropic, you buy from the guys down the street, the startup that just barely came up and they're offering for half the price of whatever, Buy from anybody, and you will eventually have to play that blame game.

we had another example in there of some lawyers who generated, cases that didn't exist they asked ChatGPT about cases and it came up with a perfect response. a little too perfect. It hallucinated stuff that didn't exist. and, is it ChatGPT's fault? Is it OpenAI's fault for, allowing their model to make stuff up and behave dishonestly? Or is it the lawyer's fault for not checking it? who cares? the problem is that it's not locked down. It's qnon deterministic.

Yeah, in a way, as I was reading the chapter on that, it makes me think of using a machine to maybe do some farm, work. Let's say that you're plowing a field and you're using a horse versus a machine, right? A machine might break, but in a predictable way. And if you've got a mechanic around, they'll come and fix it. A horse can get scared, or it has a bad day, or it can be moody. And it can come up with something new. So you always have to be careful with that.

is that an accurate feeling of someone who's working with this LLMs day-to-day? You work with some kind of animal? One of the most annoying things is even if you set the seed of it, so the random generator is going to be the same every single time, you can still give it the same prompt and get something different out. The truly awesome thing about LLMs is the number of non-linear activations that are going through the model, right?

It's creating incredible, non-linear jumps throughout that dimensional space that the embeddings are in. you just can't really predict it. It is a little bit like an animal. the fact that like we can prompt engineer at all. it's a little bit telling of where we are, right? Cause like prompt engineering, you can change the spaces, the white space inside of your prompt and it can end up giving you a completely different result.

we're still in a very interesting area, where we're trying to create better ways to communicate with the LLM and get predictable outputs. But, the fact that we can do that at all is. This is a bit of a miracle, right? you can't do that with a human. a human isn't going to be tricked into saying something different. humans are tricked all the time, but not necessarily in the same way that we do with LLMs. it's a very interesting world we are in, and a lot of people are having

that horse versus machine experience. let's talk about the cost a little bit. you mentioned that it's super cheap to pay some big company to use their thing. let's focus for a minute on the cost of actually building your own LLM. if I wanted to build one of this foundational models, Let's say that I take one of those 75TB corpora from the internet and I'm feeling particularly GPU poor that day. How much money do I need to have in my little piggy bank to get something useful?

That's difficult, man. because you're either paying for a GPU, right? Or a suite of GPUs in order to parallelize it so that you can ingest that over a short period of time. Or technically with a lot of this stuff, you can load it onto a [Geforce] 3090, I've done this personally, you can train in FP16, you can train up to, about, 13 billion parameters pretty effectively, and pretty cheaply, on a 3090.

You have to be a little bit smart about your data loading, you have to make sure you're streaming stuff you have to pay for the data storage anyway, it's incredibly slow, you have to do gradient checkpointing, you have to, do like gradient accumulation steps, which slow down the training even more, I trained a little bit bigger than that, it was about a 20 billion parameter model on my 3090, but what I don't, generally talk about is it took a year of just running to do that.

it was horrendous and that all culminated in a company giving me a cease and desist, so I couldn't even release it, so you're either paying. A lot of money, hundreds of thousands of dollars in order to get something quick. Especially with 75TB of text or more, grab your own data, get more data, and you're paying to store and to process all of that. And that costs tons of money.

Or you are not paying the money, but it takes a really long time and makes all of your shareholders really frustrated because you're ruining go to market. You're taking too long. You're not going to be the first in the space, It's a huge trade off as with many things, you can trade time or money, and training an LLM is very similar. I think they estimated, huge models that we see, like ChatGPT things. You're probably paying somewhere like what was it like a half million?

I think they say, and that's just for the training, we're not even talking about all the experts you have to pay and buy in order data curation, man. on the very far end on the expensive side. it gets really expensive really quickly to train these models, just because. buying enough GPUs in order to parallelize this to do it within, reasonable time and just the sheer volume of data you have to run through to train all the parameters.

It gets really expensive, but on the other end there's lots of good open source models that have done that main pre-training already. And so you can grab one of those, you can train it with something like Laura, which you, only need a handful of samples and maybe like 10 minutes if that, and you can train it on a very, simple GPU and you have something fine tuned for what you need, and you can get under $200 is very reasonable. $150, $20.

It's very possible to train, these models with certain methods to get what you need. So does it mean that in a kind of natural, almost biological like evolution we're going to end up with few primary models that a lot of the different models branch off of, instead of, reinventing the wheel? That's where we're at currently. I hope that it doesn't stay that way, because I really enjoy seeing new people create new models for new use cases and all this stuff.

so I hope it doesn't stay that way, but I do see a lot of value in creating industry standards, at least around how you are actually writing the binary files, how are the weights actually being stored? What do the different layers look like? I, think that standardizing what the model looks like so that you can load it as flexibly as possible is awesome.

I would like to see more open source models, which is funny considering there are thousands of open source fine tuned versions and hundreds of open source foundational models on the Hugging Face Hub right now. I want more, right? I'm greedy, man.

To me, it sounds like basically every week there is another one that's better at something and if you look at the Hugging Face LLM leadership board, it's changing by the hour, literally and it looks like a gold rush in many ways but I like this gold rush much better than the crypto one, couple of years ago Yeah, man, there's a lot higher chance that you'll come out of this gold rush with a great product than with the crypto one.

yeah, there's a lot there, and just to summarize that into one sentence, you can probably fine tune even a gigantic model for around $200 to $500. And you can go lower than that. Even if you are smart about how you're doing it, versus training from scratch, which either is going to take an inordinate amount of time or will cost thousands and thousands of dollars. So I'm willing to bet money that a lot of our listeners are going to pause this now and start Googling furiously.

How do I fine tune a model? Where would you point them as a good starting point? any particular paper, any particular, company, anything that's, a good place to start with that a bit selfishly, I would say you should buy our book. We talk about probably the main ways to train in chapter 5 of our book, I was going to say that, but, I was going to say it last, right? Cause we do go over it.

The book is primarily about production environments, but you can't really put a model in production if you don't know how to work with it. So we have stuff on fine tuning. We have stuff on perimeter, efficient, fine tuning on low rank adaptation, the whole deal. YouTube is actually probably one of your best resources right now, because it has amazing content creators that show you how to do it in whatever format you're comfortable in.

So if you're a C+ developer, there are YouTube videos on how to fine tune a model and create a Laura using llama CPP, right? It's not even all that difficult. You just have to convert a model into a GGUF format and Boom, you're there. You can do it on a CPU. it'll take a long time, but you can do it in whatever quantization you want and everything.

YouTube will meet you where you're at if you want to learn something a little bit more industry-standard so that you could potentially, get employment in this area, PyTorch has an amazing documentation, fantastic tutorials and they're one of the best at really making it feel like you're playing with, let's say "big boy Legos" You're like building the model using their little Lego pieces pretty cool If you need something Bit more high level than that.

Hugging face, I think is the industry standard for, working in between a whole bunch of different frameworks, whether that's PyTorch or TensorFlow or, whatever other framework you're working with Onyx. HuggingFace has abstracted away a lot of the difficulty of setting up models for fine tuning cause in PyTorch you have to build out the exact model architecture just to load the weights and then fine tune it. HuggingFace already has the class built for you.

I would point to those if you need more explanation, like Coursera is a fantastic place. Deep learning AI on Coursera and on their own sites felt like that's Andrew Ng's education stuff. That's where I got my start with machine learning was Andrew Ng's machine learning course on Coursera. It was Awesome. Fantastic. Jeremy Howard is also amazing in that area of creating content for people starting out and learning from beginner to advanced level. He's a fast AI.

I, yeah, I strongly recommend all of those and your book. yeah, we ingested a lot of those in order to write the book, our book is a very nice high-level overview of the key things you want to be looking at and like different methodologies from training from scratch to basic fine tuning to. model distillation to, Laura and Path and things like that. we definitely give a high level overview, we give code samples and show you that.

But, ultimately if you really wanted to get into it, yeah, there are other resources out there. I know Manning has another book coming out, specifically around all about training LLMs. there are definitely other places you can go, but. If you're looking for the quick, summarized version of all of these things, our book is actually a really good resource for it.

One other thing that I like about your book is, the part where you build up the different, breakthrough moments, throughout the world of mathematics, that ultimately led to 'attention is all you need', and

what is it, seven years later now? the gold rush that we're observing. but just before we jump into that, there is a little bit of vocabulary and that one needs to have in order to basically talk or even read a lot of this papers, could you. Talk us through briefly that vocabulary. I'm talking about phonetics, syntax, semantics, pragmatics, morphology, that until I read your book actually made me think mostly of blood tests and semiotics.

Could you give us like the MVP version of what you need to know about these things to be able to read papers? Oh, absolutely. Matt has been learning a lot of this too, he might be better at it than me. I will throw other jargon into it. writing this book with Chris over the last year has been, mind-opening for me.

until you can Understand these words like you were saying it's really hard to dive into the deep end but we go over in our book just because we do find it so valuable, It really helped me understand very quickly. "Oh, this is what my LLMs are good at. This is what LLMs are not", and that was one of the first things we started with but the first one semantics, that is just like the structure of words, how things go, whether or not it sounds correct. that is what LLMs are really good at.

They're really good at making sure like the semantics of words align really well. but after that, you got pragmatics, which is what LLMs have no idea about. That is all the information around. That isn't said, right? So when you say I'm going to find the eggs the Easter Bunny left, right? you have to understand what, Easter is, what the Easter Bunny is, why a bunny has eggs. there's a lot of context around it that you have to understand, and that's all pragmatics.

it's information that isn't said. And that's what LLMs generally lack. Actually, I'm gonna, I'm gonna jump in here real quick. Miko, did you like the Velkanot example that I gave in there? Yeah, I thought it was Yeah. Was that pretty good? I just wanted to ask because I remember experiencing that in Slovakia.

Like I lived there for years and that was a hugely beneficial portion to me to help figure out that 'no, tons of people have tons of ways of looking at things', and LLMs don't know about it. you would have to explain every bit of it to them in order to get them to understand the same things as you.

Anyway, sorry, Matt. I find like those two words in general, semantics and pragmatics, understanding those is going to get you significantly farther and just understanding how LLMs work, what they're doing. there's obviously a lot of other words that we talk about, like morphology and stuff. And I'll hand it off to Chris to talk about what he wants to add to there. I would agree with Matt.

Just understanding semantics and pragmatics would get you probably 60% of the way there, and you could read new papers that come out and immediately see like where are they amazing? Where are they failing? I end up using The relationship between those two, just the literal encoded meaning of your words. if I say, "I'm married to my ex-wife", there's immediately, boom, semantic problem there. How can I be married to my ex-wife? The words don't agree with each other.

Versus, exactly as Matt was saying, if we talk about Easter, if we talk about traditions, if we talk about rituals that people have, just like the stuff that you say, if you ask someone in Slovakia, they're going to respond to you. That's normal. it's a question, they respond. LLMs don't have that, and you have to have them ingest tons and tons of data in order to even get as far as giving a response. the other ones that we can think about, syntax, I would say that syntax is largely solved.

At this point, syntax is your structure around the words, like what order do the words go in for them to be correct? Is it 'I go to the store' or is it 'I to the store go' or all of that stuff. That's syntax. It's the structure that holds your sentences, your utterances together. Morphology is delving into something that I consider to be very important in LLMs. I'm not going to say the most important, cause I think that's still semantics. There's a lot of work there.

but morphology would be how words are built. what are the fundamental units of meaning the morphemes do those even exist that sort of stuff. and we don't have to delve really deep into that. That's largely solved by tokenization, but we can see. with newer models that come out that really matters. You have much smaller models that have more novel tokenization, more novel morphology that end up outperforming larger models on tasks that they didn't even train on all that much.

if we can put it all together really quick. The model solves syntax. Embeddings try to solve semantics, but semantics is difficult, and so they're not perfect. Pragmatics is stuff like RAG, your Retrieval Augmented Generation, and having repeated sequences within your training data, it gives it landmarks, it's context around the syntax and semantics.

Morphology is your tokenization, which, if I would Give that an example, your tokenization provides your model with stuff that it sees, it changes from text into what does the model actually see. And, your embedding strategy is moot if you don't have it. Just your morphology gives your model glasses, if you want to call it that. And then phonetics is the one that we haven't even talked about.

Phonetics is the reason why we are doing a podcast and we're talking instead of just texting each other or emailing each other. Can you imagine trying to ingest a podcast that's just emails? It's horrendous. And it's because there's so much richness and depth in meaning in the language that is just lost when you strip it of its phonetic, I'm going to call it a medium.

And that can lead people to think that it has to do with sound, that's the most common modality for people, but sign language has phonetics, they have particular places where they, make signs. They have particular ways that they do them to inflect and express more emotion. Their phonetics exists even outside of the verbal modality. that's important because that's where I see the most improvements coming to LLMs in the future is being able to process.

phonetic information without having to convert it into text or process phonetic information and compare it against the text. that can be incredibly helpful for your model's understanding. those are the five features of language that we break things down into in the book. And they're largely agreed upon. There are some other linguistic features that are incredibly important, stuff like dialogue, that we haven't even covered. beyond that. Yeah, we can talk about semiotics too.

That's, Charles Sanders Peirce, smart dude from the 1800s just created, a lot of structure and organizations we dive into that very lightly in the book. I don't think that you need a grounding in semiotics in order to improve your ability to interact with LLMs. But it is helpful for organizing all of these other concepts. how do we create a mental map for how stuff needs to be processed within a machine learning pipeline?

How do we make sure that we're not mixing things up and inadvertently destroying our model's ability to see things, right? If we put embeddings before tokenization, it breaks your process. it's helpful for organizing things and it's also helpful for understanding how conversation happens and how I say something and it moves through

your mind to create an interpretation. that's by far like the most theoretical out there concept that we get into in the whole book. And together you came up with this language definition as being, as a concept, "an abstraction of feelings and thoughts that occur to us in our heads". And I'll be honest, I initially thought it sucked. because it's a little bit, it's a little bit wishy washy. I wanted something a bit more concrete.

But then, as I looked up all the other definitions in different contexts, I was like, Okay, I can clearly not come up with anything better than that. So I think I'm ready to yield now and say that this is actually capturing it pretty well. Putting abstraction in it, sounds also vaguely techie, so that helps.

How did you come up with that definition? I didn't. I would love to take credit for that. No, that definition has been around for a long time within the linguistic community, and one of the best examples of why it really works is babies, right? Babies have no idea how to express their thoughts, but somehow they get it across.

when a baby is happy, we can tell when a baby is crying, we can infer that it needs something, babies are able to communicate without language, meaning that language is something that we created to shorten the conversation. The reason I called it an abstraction is we have abstract ideas. You probably come up to a situation where you're feeling something, and you don't know the words to really express it.

I think that's a pretty universal human adult thing that has happened at least once in your life. That's happened to me a bunch of times, and it really illustrates that "Oh man, the language that we use is actually describing "what's in here", it isn't "what isn't here". it's a hard concept. Once you get there though, it really helps with LLMs, because you realize that the language that we're using is a crutch. And that's all that the LLMs have in the first place.

And so this is another thing that goes towards the miraculous nature of them working at all. Is they're dealing with an abstraction of an abstraction at least. In order to communicate with us. So let's say that I buy that. my first question, would be going back to your baby example, isn't what the baby's doing some form of a language? what's the line I'd like it to what is and what isn't? what's the line between, a language and communication? I like that.

That's a question that a lot of people I bet have and It'll probably go in the appendix. We'll probably talk about this in an appendix for curious readers so the line between just straight up communication and a language is the ability to talk. there, there are a lot, but one of my favorite ones is the ability to talk about something that is not physically present. bees have communication. gibbons have communication. Babies have communication.

Babies, though, are unable to express any ideas about stuff that is not physically present, you can't talk to a baby about theoretical physics. I mean you can, but what are you gonna get back? You can talk to a baby about my Star Wars posters, right? I can point at them because they're right there, but if I'm in a different room, baby's not gonna be able to talk to me about them And that's the difference, It's one of them.

That's the one that I'd like to highlight though is that the fact that we can speak about things that are not physically right here with us, that we can point at, that's the distinction between communication and language, because babies are communicating.

But once they get to that point, it really deepens the interaction that you're able to have with them. So now, equipped, with all that knowledge, I'm gonna try to prompt engineer you and give you this prompt. I'm a five year old baby, that has language now, and who's very curious about understanding how we got from bag of words, counting frequencies all the way to LLMs and ChatGPT and people worrying about the Terminator actually coming into life.

Could you walk me through the high level ideas that were important, build up to what we're seeing today. The bag of words is really easy to think about, especially if you keep your tokenization incredibly easy. Sorry, this is, I'm already out of five year old territory. You just count words. If I take that sentence, "you ; just ; count ; words". Each of those has a count of one. If I add another sentence, "I like Star Wars". All of those still have a count of just one word.

And then if I add another, "do you like Star Wars?" You and star and wars all go up to two. That's it. That's a bag of words model. why is it important? what can it do? I think that bag of words is The first model that we really have to explain being data-driven. It's just keeping track of things. if you look at a bag of words model for your workouts, it's just how often do you do certain things? how often are you doing a bicep workout versus doing a pectoral workout?

How often are you doing which thing? it's just being data driven. It's the first step, right? You're not looking at any features. You're really caring about how these things interact with each other. You're just keeping track So I guess with that information from your example, I can guess whether you, are skipping leg days, and I can see what's important to you.

Or, if I'm counting, words in U. S. presidents speeches, I can say, like you described in your book, whether it's a wartime or a peacetime president, and what they really try to get across. this is something that you can use for anything you count in soccer which players make goals how often that is a bag of words model. You're not tracking words. It's a bag of goals or it's a bag of, whatever else. So what's the next step from there?

bag of words was really monumental just because it's so simple, but it's so powerful because know words you use when you're describing sports is very different from the words you use describing politics And so just picking up on certain words and their counts helps us understand the overall subject of what it is. But it really lacked, any sort of structure, because the order of words also matter, right?

So the cat in the hat versus the cat's hat, they both have the word 'cat', they both have 'hat', but mean different things because of the order of the words, and so that kind of led to, n-gram models. instead of just simple words, we would also take n-grams, which are, n number of words in a certain order, and we would start cataloging those. And so, more than just words, we're getting n-grams.

And that is improving our understanding of the language because now we have embedded some syntax in it. We understand some ordering of words and that's able to improve our categorization. however, from there though, we're not really able to make any predictions

of what next words about to come up or anything like that, when it comes to bag of words or n-grams they're really more for categorization. And so that kind of led to Bayesian techniques and so not to really go deeply into Bayesian statistics, but Yeah. I'm sorry. Sorry to all Bayesian fanboys. We're going to go about as deep into this as we did to pragmatics.

it's just you know, based off of the priors of the words that came before we can then predict the next word to come up and so if every single time after in text we saw 'I am a man' then it's going to predict that the next word is man instead of other words that easily could have come up like woman or girl or boy or cook or professional athlete.

certain things that could come up that are gonna be a lot rarer Like I am an astronaut like a lot less people have been astronauts in order to say that it's gonna have a very low probability of being the next word predicted but it gives us this opportunity to look at what is the next word predicted. from there, we move on to what's called Markov chains we're swinging back towards the n-gram model But it gives us a bit of prediction next.

I actually really love Markov chains because they provide very fast predictive text like Markov chains is essentially what's been fueling like the predictive text like for Google search and things like that has been the technology that's really been leading that charge for a really long time. and it's just a very basic way that we're using Ngrams now to make predictions of the future. You can think about it there, that is obviously I'm reducing it.

that's not exactly how it works, but it's a bag of n-grams where you take a state, at each point in a sequence, and look at all the times that Previewings have occurred in that sequence, and then from that you can model probability about what comes next. Instead of just looking at each n-gram by itself, you give it state. and it's a bag of n-grams. It's really fun. It's a probabilistic bag of n-grams. That's how the chains work.

One of my favorite parts, and I like that you kept track of this quote here, that Markov models represent the first comprehensive attempt to actually model language, which is funny, because Markov was not trying to model language initially, he was just trying to win an argument. And He eventually used it to, he looked at distributions in particular Russian authors. He looked at distributions in, Russian government official speeches.

he knew what he had and he believed in it, and I love that, what a great piece of history anyway. continuous bag of words. Is where we, start essentially taking the logic of a Markov chain where, "oh, if we keep track of where things appear and how often they appear there, then it helps us, be able to model for what could appear next", right?

And this is the first moment where we're really coming full circle all together and going right back to bag of words and just adding context for position and adding context. from the context of the bag of words, the literal counting of things, we're able to create embeddings, right? I don't know if a lot of people are aware, but bag of words is how Word2vec came to be.

Word2vec was huge in, I think, 2015, 2016, and it stayed huge, Gensim is still one of the most downloaded natural language processing libraries in Python for Word2vec and for GloVe. Continuous bag of words, just adding that one little thing. adds all this context so that we can create embeds. We can create vectors that we can compare between words. this all comes from the logic of I forgot that dude's name. Tell me the company that a word keeps, and I'll tell you what that word means.

just what's around the word. influences its meaning, which goes directly against a lot of previous linguists' thought that, syntax and semantics are absolutely not related at all. That's one of the big things from Chomsky, the colorless green ideas sleep furiously, nonsense, there's some semblance to it. There's some sense to it and taking advantage of that with continuous bag of words, we can create. like I said, these vectors that we can then compare, and that's really interesting.

that is what fuels LLMs now, is this exact same continuous bag of words modeling technique. It's been built upon a little bit, but that bag of words is still fundamental to how embeddings are created. Bag of words and positionality and, like we can get into, the rope scaling, all of these rotational, plugins that you can use to get longer sequences embedded correctly, or at least better. that's one of the hard things when we're talking about language modeling is what is good and what is better.

a lot of people like to appeal to, this is how humans do it. I don't know if humans are incredibly efficient when we do it, but. Like it's fine. then we get into the 1960s, the very first perceptrons, Before we go there, can we spend a little longer on what the embeddings actually are? You mentioned words to Vec, you mentioned the words vectors and embedding, but for somebody, listening to us, from the start, that's probably not clear what that is. can we delve a little bit? Yeah, absolutely.

So embeddings are the vectors that come out of models like continuous bag of words. when you look at a modern machine learning pipeline, there are multiple models that you go through and we just attract all of it and call it model, just one model. When you look at GPT-3, ChatGPT, it has a model that they call it, a byte pair encoding model to do its tokenization. And then it has a model to do embeddings. that model is fundamentally a continuous bag of words.

It's built on top of it a little bit with, like I said, keeping track. Not just how many times a word occurs, but how many times a word occurs in particular positions. and then on top of that, it keeps track of the, flip. It's either an odd or an even position within a sentence and it assigns it cosine or sine based on whether it's an odd or an even position.

in order to try to insert some of that meaning back into it, that was taken out from the tokenization, cause tokenization is just assign each token a number in a dictionary, and you have a way to get all words into that dictionary, and then come back out of that dictionary. So it takes all of the meaning out of it. It's just one number. The embeddings attempt to put some of the meaning back into it using positionality, using continuous language modeling techniques.

embeddings really simply, they're not perfect, they're just an approximation of that meaning, and because we are able to put it into a vectorized space, we're able to take these words, put them in a vectorized space. We can start doing things that start to make sense and start to make us feel like we're headed in the right direction. the classic example is, when we first discovered embeddings, we took the embedding of 'king', we subtracted 'man' from it.

We then added the embedding of 'woman' and we got the closest. Embedding to that was 'queen' to that, we start to get this vectorized space that starts to make sense. We start to, these words start to have connection to each other and they start to make semantic sense to us as humans. however, embeddings are still an approximation, right?

So if you were to do that with kind of every combination, it's interesting, what do you get when you start, taking words, That don't necessarily make any sense, like adding or subtracting them together. what do you get a good quintessential example of that is you take the vector for 'king', you subtract the vector for 'wolf', and you add the vector for 'prince', and you get the vector for 'village'. Or at least pretty close to it.

That doesn't make any sense, there's still lots of, okay, these are starting to add meaning, not always, but sometimes, like it's an approximation and embeddings ultimately.

it's something we're constantly trying to learn and improve If your listeners are wondering how to keep up in space, like embeddings are probably the number one thing to keep track of OpenAI recently released, logic for being able to change the size of embeddings, to me, like being pretty deep into this, it feels groundbreaking.

Because normally you have to structure these vectors so that they're all the same size and each point within that vector represents meaning negative or positive and it's very structured and not malleable and so the idea that you could take you all of your embedding space and change the size of it at your whim Is just amazing. that's one of the things that I see as a huge groundbreaking piece of technology that OpenAI is continuing to lead in.

yeah, and if you're ever in doubt for oh man, is this paper important? If it's about embeddings and doing really cool things with embeddings, probably. I think the one question for anybody to like picture that, so what's the dimension of all these vectors? Is that the entire vocabulary? Are there different techniques? yeah, currently the, number one, dimensionality that is an unspoken industry standard is 768. that's a number that pretty much every NLP practitioner knows.

like the reason OpenAI's embeddings initially were like really cool and they thought they were super dense is they were, what, 536, or 1536, which is 768 doubled, right? You're gonna see multiples of 768 all over the place here. And that's not because that number is super significant, that's just the first embedding space that we found that tended to work better than the others. So that's the more art than science part of this for It's the brute force testing.

Yeah, before going through and testing, 767, 766, 765 and landed on that one and it worked, that's the best one that we've found so far. Even the doubled embeddings from open AI offer a marginal improvement

in that understanding space. I think we can move on to the multilayer perceptrons. Okay. a perceptron is essentially just a linear transformation of data. If you look at it from a statistical standpoint, if you have three things about something, You can just add those things together and you get a description of that thing, right? Just summing them and, that's like abstracting it a little bit much, especially if machine learning practitioners are listening to that.

Like we can do linear trans transformations. that's like the easiest way to think about it for me is you perform one. action on a group of features and you get something out of it. That's not by itself. really helpful.

once you get into having multiple layers of the, this is the MLP, the multi layer perceptron, once you get into multiple layers where you are adding these transformations together, and in between those layers you have non linear activation functions so that you can, create, you can create nonlinear relationships between sets of linear transformations. You can get into really cool spaces.

And one of the first things that any machine learning practitioner learns, at least in a lot of the cases that I've talked to is that just adding more layers does not make it better. In fact, the cool part is finding the minimum number of layers that you need in order to model the relationship between two points. that's a little bit abstract, I think the quintessential example is like detecting which type of iris flower.

It is from an image, the, we don't necessarily know how many features there are, but we can vectorize the entire picture of an iris flower. And then we can discover that the, I think minimum number of layers is like five in order to go through and actually get really good accuracy on detecting which iris flower it is. yeah, multi layered perceptrons are The feed forward networks.

Those are the basis of everything that comes after it whether it's recurrent or even Transformers have feed forward networks inside them and that's the basis of it right there. How do you choose the sizes and is it all just trial and error as well for the number of layers, the sizes of the hidden layers? Are there Not any rules that always work?

Yeah, so going through a feed forward network and this comes from trial and error, it comes from a lot of people trying different stuff, but generally you have your Initial dimensionality could be something like 768, right? Your initial hidden layer. that's a good number for it. That's an embedding dimension that we're familiar with, but then we want the next hidden layer to be double that. And then we want to go smaller and smaller until we hit our final output classification layer.

So we want to have a big jump and then small. What to think about that theoretically is you want to model the number of features that you are looking for, and then you want to just model double that is just a good way of saying all the features that we might not know about that we might not even be keeping track of. Let's see if the model can figure them out mathematically. And then we want to narrow it down. Narrow it down.

Narrow it down until we get to our actual classification, which in language modeling is what is the next word, right? Got it. So double it and then boil it down to the size that you're actually looking for across a bunch of layers and hope for the best. Okay. and that's why when OpenAI doubled the embedding layers, it was a marginal improvement, but it's predictable because that's normal. People do that.

Are there any particular, well known kind of configurations of this neural networks that just work for a bunch of problems that, something that you keep seeing over and over, or is it more custom for every problem you just follow the heuristics that you just described?

as far as model architecture, no, it's basically the heuristics that I described, and then people will experiment and tune them and find that, oh man, statistically, If this layer of the model is bigger, then it works better, but it follows that general structure. I think, one of the papers that I would point to for this is a bit, MFIT, where it was, it's basically a methodology for fine tuning.

But it experiments with gradual unfreezing of layers where when you're training, you will start with only the very last classification layer and everything else is exactly the same. And you only train that one. And then you unfreeze, unfreeze, and test each layer as you're training. And that tends to help things like even now that is abstracted within the hugging face trainer class. And that's abstracted within pretty much every. model.fit methodology because it works.

Awesome. What's next in our journey? probably just the fact that multilayer perceptrons struggle with sequences, right? even if you try to embed things and try and keep some of that positional encoding within your embeddings, they struggle to model. Multiple things where the order of them matters, right? which language, which the order matters sometimes, right?

Sometimes it's normal to say gibberish and knowing when is, which is extremely difficult and to solve that, I don't know if we need to necessarily go into recurrent neural networks, but we definitely need to talk about LSTMs, the long term short memories, which are recurrent neural networks to, start with, but they added some really important things, which, for example, when I'm talking, you are.

Kind of consciously predicting what I might be saying, you can hear what I'm saying and you're trying to figure it out as it goes on to understand it. we call that active listening. that's what happens. long term short memories, model that a little bit in that they take the sequences and they allow the model to try to predict both going forwards and backwards. instead of just doing the one way. So that bidirectionality it's computationally expensive.

It takes a lot longer, which is why I think these are not used as much anymore, but it's really novel and it did help a lot in predicting sequences. it was phenomenal for language modeling. beyond that, they like solving the attention.

Within LSTMs, like when attention came out, adding attention to whatever you were doing was phenomenal where it added an extra layer of non linearity when it was going through and trying to search for what word might come next, it not only had all the modeling that we've already talked about, it also had the ability to search now and search for not that exact thing, but something similar. And, that just exploded in popularity because it works, it was phenomenal.

However, the difficulty with long term short memories is they're computationally expensive, they're slow, it's a lot of math that you have to do in order to get through every single layer of it, let alone trying to predict and stream those predictions in a sequence, you're going at one token per 30 seconds. And that's difficult for having models that are the same size as transformers, for example.

so yeah, it was a lot of really cool stuff that helped us solve basically how to get to the next step. It was just computationally expensive and slow. basically, not very practical in use, but important. talking about practicality, I think it's great that it's accurate, right? I think accuracy is incredibly practical. I don't think that from a customer experience that's practical, right?

Customers don't like waiting a long time for the right answer because they might be able to find the right answer in that amount of time anyway. and then from there, do we jump to the attention? at this point, we've gone through the history of, the field modeling language, building up and we finally reached attention, right? And attention is, the backbone of transformers, which is what LLMs are built off of. And, attention just adds a non linearity.

And it was just a breakthrough and how we're able to connect the words, so attention really quickly is just, creating these dictionaries, key values of, every word to every other word in the token space. and then it's able to query it. for each other word, we're able to build. importance of the other words that are important to it.

And it's in a quadratic space, so it's much more than a linear space, but it's a reasonable amount of time, to compute these kind of dictionaries, the key values, and then query them and understand the importance of other words It's the backbone of what all these, different models are doing.

and even as Chris mentioned, like we could inject attention into these previous, RNNs, LSTMs, et cetera, but, it was the backbone of building the transformer model, which, came out, in the catchy paper, "attention is all you need". where essentially all they use, a meme, right? That we've seen a whole bunch of other papers afterwards. They're like, "no, this is all you need".

or no, this is all you need, or no, you don't need, but the reason it's a meme is because they took out everything that was, supposedly novel about the long term short memory, the LSTM. They used only attention and feedforward networks Could you give us an example of what that would look like on a very stripped down thing? What does that dictionary look like? for visualization and decode. no, just for the attention itself, right? You mentioned a key value from basically every combination.

You have to pre compute every combination within the vocabulary. You can take a sentence that you're feeding in to the attention algorithm, the cat in the hat, since I used that earlier. and so essentially you would have a dictionary where the is comparing to every other word, cat in the hat, and it's coming up with assimilating metrics of the importance of all the other words.

And then you would do that for cat, it's going to do it for the in the hat, and in the cat, the hat, and it's going to come up with A dictionary, essentially, of key value pairs for all the other words, helping you understand, the importance of the other words that are in there.

and then the query algorithm, that runs, that essentially helps us understand being able to predict the next word that's coming afterwards based off of how important the, all of those kind of dictionaries are, and adding them. And so all of, this happens to happen in quadratic time. one of the nice novel things about this is that the query And key vectors, your query vector is the word that you're looking at in the utterance and your key vector is the key in the dictionary.

those two vectors are not one hot encoded. The way that a lot of we haven't even mentioned this. But that's a vector that is 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, that's how a lot of these things had been represented previously, coming off of the bag of words, The idea that, hey, we can model these things. We can create vectors that are just did this word appear. Or did it not? And where did it appear?

That was a positionality and, attention is all you need, you can immediately see a problem with one hot encoding in the it's very sparse, especially as you're getting into 768 dimensions, right? You have just one 1 and a whole bunch of zeros and those zeros don't really matter. And so one of the breakthroughs here was using dense vectors for queries and keys in order to get values that are also dense. I think one of my favorite visualizations of it, it's from Jesse Vig.

It's called BertViz on GitHub. I've used this in production environments in order to show that hey, Our model is not understanding this because look at the attention, all of it is factoring in, all of the queries are related to the key of the wrong word. If you look at words with semantic ambiguity, I think the quintessential one is "time flies like an arrow". Where flies is also another word that could mean multiple small little bugs buzzing around. How do we know that it's not that word?

It's because of the position in the sentence that we know that it is a verb. and it's referring to time and it's referring to arrow. And we can see that predictably within attention, because that word is determined to be important. That query is determined to be important as it relates to the keys of time and arrow within query key value attention. That's what that dictionary looks like. That's why it's useful.

And, I guess the representation of the importance, how do we actually come up with that I think it's dot product. we're comparing the vectors between the query and the key. dot product attention is, I'm pretty, that's not where it started, but I think that's where we're at right now. That's like the industry standard that everybody uses. It's just, multiplying the vectors together.

Essentially you take the dot product of the two vectors, and that's where we get the comparison and the relative importance values. it's not magic, it's math. kind of the same thing from time to time? Okay. And then with that, we've got the GPT, the generative pre trained transformer model. What's so groundbreaking about that? as opposed to the original transformer, they only use a decoder.

So the original transformer had attention based encoders, which changed your embeddings into essentially another embedding that was then taken by your decoder and used to predict the next word. So it had two networks linked together in the middle in order to produce Your next word and the reason this is important is it goes back to that original idea that we talked about a language as an abstraction, right?

The authors of attention is all you need looked at that abstraction and we're like, Hey, can we model that? And that's what an encoder is. When you look at models like BERT, it's taking your input and putting it into a new abstract space with lots of nonlinear trans transformations and it's taking your input and putting it into a new abstract space with lots of nonlinear trans transformations and it's taking your Incredibly useful.

And so the GPT models were groundbreaking, because they were like, we don't need that. we just need the decoder and we're just going to use syntax basically. And the thought process there is that syntax is related to semantics deeper than linguists are able to really conceptualize in an easy to understand way. We know that it's true, And we know that it's predictive with especially looking at how good GPT-3, GPT-4 are.

And even looking at the open source stuff, LLAMA is a decoder only network and it rocks, right? I have a suspicion that we're going to hit a point later where, Google is going to blow everybody out of the water with another T5, like another, version of that puts the encoder back in. I don't know how we're going to get to that point, though, because the decoder only models work so well.

And they're faster, they're less computationally expensive, because you're taking, probably, a third of the model and just throwing it away. So you mentioned Llama, and I think that might be a good segway from what essentially is, about a third of your book. so for everybody else who wants to go and jump into more details and see actual Python implementations of a lot of what we just covered,

the book is called Production LLMs. It's available on manning.com, and I'm pretty sure you're going to love it. So going back to Llama, let's do a little hall of fame, rundown of the kind of landmark important models from the last few years. Where should we start? I would probably start with the original transformer, like they deserve credit. A lot of the, Vaswani and all, a lot of the people who wrote that paper have gone on to found or co found companies that are now competing in this space.

Whether that's Anthropic or Character. ai, those are the people that created that Transformer and they're still building on it. I think that's the first one that I'd say for the Hall of Fame. what would you say, Matt? think part of this question is what is the first LLM versus what is, the first, Hall of Fame model and yeah, like Transformers, Bert, like Bert, is incredibly powerful, I think, because it's so small, it's not in the LLM space, it's often overlooked.

And I think many companies are still looking at these massive LLM models for problems they could solve with a simple BERT model. But because they're only getting into this space now, they think immediately, hey, we have to use an LLM, right? And they didn't care in 2017. And And over what was there. and I go back, I said it before, I love Markov chains, like they're amazing and they're really powerful for what they do really well.

And even then, a lot of people could just use Markov chains for a lot of the problems that they're trying to solve with LLMs, but, LLMs. They do give that flexibility, just their massive levels of computation. I think if I was to point, to a model that I thought was just really powerful, it. It would be Bloom, actually. Bloom was essentially the first, LLM massive, large model that was built. And it was built, completely transparently. it was a research, project.

funded, a large part by, the French government. And just, it was built completely transparently and completely in the open space. and even though the bloom model today, isn't seen as, a very competitive model, but like a lot of the open source learnings, a lot of what we have nowadays is because of what those researchers figured out while they were working in bloom. we got amazing, libraries out of it from like deep speed and other things like that.

it really boosted the open source community, which has been one of the major driving factors of LLMs today, and probably a large part of why we could even write our book, cause the open source community wasn't. At where it is today, like there wouldn't be much we could really tell people other than oh, You got to go work for Google or Microsoft or how would We, know any of it, right?

Yeah. we know, about it largely because, we've been involved in the open source and we, built off of what those scientists at Bloom did. Big science. So that's 2022, right? That's a couple of years now. Yeah. and then we had llama that became important, and llama2 Yeah, even more important.

Yeah, and it's largely just because, I don't remember the username of who did it, but whoever put that PR on the original llama GitHub that had the torrent link to leak the weights, that's the hockey stick moment for LLMs, right? That's what made them available to everybody.

That's what enabled Stanford to create alpaca and show that, oh man, you can make the model better with like only 50 K responses like you don't need tons and tons of data in order to fine tune and get very good results and improve in every metric. yeah, that everything since then has just been building off of that exact same momentum of whoever leaked that first llama and Meta has benefited greatly from it too.

they now have a very open, I wouldn't say completely, but a very open attitude towards the space because they recognize how, advantageous it is to have other people building on top of their model and be considered an industry standard. Yeah they've really leaned into it recently, right? And like how big was their stock jump? right? all of the underlying architecture, right?

Like these open source programmers or even just like the video programmers, like they're able to go in and because they know everything about Lama, they're able to optimize, cuda kernels and everything. And so Lama has gotten faster and more proficient, Lama CPP, we're able to run it with, just on a CPU, there's lots of benefits that because they, gave us the architecture, it was leaked, but now, they've, leaned into it. They essentially they've given it to us.

And so Yeah, we just need them to release the data that they used to train on it And it's completely open, right? but even the data, they've told us a lot about what the data is, right? we don't have the exact data, but we know essentially red pajama, what those data sites were built off of, what they were. And so we're able to. replicate it really closely in the open source community.

Llama, I don't know, if we have a really good list of Hall of Famers because it's difficult to see what's going to stick around partially because it's so difficult to evaluate these models as opposed to BERT right? large BERT had 300 million parameters. You can run stuff to see how well those parameters are, like you can hyper tune them. you can run evaluations to see how each one is performing and still go relatively fast.

When we're getting into the 7 billion parameter range and the 13 billion parameter range and the 70 billion parameter range, it's much more difficult and computationally expensive to evaluate on that level. And we don't even have the ability to describe what all the parameters are doing. and so our evaluation metrics are difficult to gauge. You look at MMLU, you look at a lot of the benchmarks that people are running, and they're useful.

But ultimately at this stage, we still have to go download those models and test them against our own use cases to see if they perform better. And that's incredibly time consuming. like we could talk about a lot of the models that have come out, like Capybara, we can talk about New Zermes, we can talk about WizardCoder, and they're all great. I don't know which ones are going to be the hall of fame.

The next industry standard though, there's definitely some other models that we love and we talk about in our book, like Falcon, which came out of the TII and Abu Dabi, right? Like amazing model. It's, Micu. the latest Falcon is one of the largest open source models and it's come, under the Apache 2 license. So it's completely open source. the very first model that's fully open source. there's definitely amazing, progress being made and lots of different models to be paying attention to.

But yeah, One of the biggest ones to pay attention to. right now, I think is Olmo, not because it's competitive and performant, but because like Falcon, it is 100% open source. You can see the data they trained on. You can replicate exactly their experiments. that's going to be one of the biggest drivers in this field where, you look at a lot of the, innovation that's happening and it's happening over on files that people are passing around on torrents.

It's happening on like random users on Reddit are coming up with NTK aware scaling and rope scaling after that. And they're coming up with more stuff because.

They have time, and they want to help and a lot of these people are experts and they're just anonymous and that's Incredibly important for the space because we're finding that people who deal with these models and use them 24/7 Have skills that the researchers don't necessarily have and that's difficult to admit being on the research part of it But it's true. so that's the one coming from Allen Institute for AI, right? The one it has, yeah, I think they're also open source in the

actual training code as well. the whole they are the whole thing. That's pretty awesome. So with that caveat out of the way, hedging your predictions, we don't know what's going to happen tomorrow. Do you see any one company kind of getting ahead of the others? The GPT-4 is still holding up well against a lot of these models, which makes me think personally that they have a few. Tweaks and hacks they haven't shared, which helps with their multi billion valuation.

Do you see anybody like running away from the crowds or is it too late now? The cat's out of the bag and the progress is going to come from the mass of people. I don't know I know that, I was texting with a couple of people the other day talking about GPT-4 and, how it is still relevant, even, people talk about the performance decrease, but it's still relevant, and every week, every model is, that's coming out getting compared against GPT-4.

And they're finding that most models are more performant in GPT than GPT-4 on certain things, right? It's comparing the Rain Man to an average human where, and asking like what tasks they're good at, right? If you, if it's going to McDonald's and ordering your own food, Rain Man is not great. And you just got to find the model that's better. a good example for that with GPT-4 is math. if you need a model to perform calculations for you. That's not it.

you have Alpha Wolf, you have, Goat, you have, even just Vanilla Llama 2 is better at math than GPT-4, even though they weren't explicitly training on it. And I think that they currently have that first-to-market advantage more than anything. That's not to say that it's bad. That's not to reduce the work that OpenAI has done because it is phenomenal. But that's what's keeping them really afloat is the first market and the ease of use.

One other question I was holding, as you were speaking with, you mentioned mixed role and, What is it called? Mixed of, mix of experts. what's Yeah, mixtral. Yeah, it's routing. it's being smart and saying, hey, we don't need a dense feed forward network for every single thing. Let's have a whole bunch of sparse networks and just based on the input route it and tell it which expert is actually going to be the best. It results in much larger models that are smaller on disc and faster to run.

Is that more similar to how the human brain works? Because it's obviously not fully connected. It's got different regions and stuff like that. I would love to appeal to that. authority. that didn't rock. I don't know though, because like you look at MRIs and you can see, Oh man, this portion is lighting up when you're experiencing that emotion or seeing that input. But who we don't really have a really great mapping of every person's brain.

I think the connection between a neural net and actual neurons has been lost a long time ago, right? how does the human brain work and how does it really compare to modern day models? Like it's hard to really make that argument, we're still learning about how we learn. And as we do, and as neuroscience filled advances, like ultimately leads to advances in the AI space and vice versa.

there's definitely connections there. but yeah, as far as your question goes, I think it's anybody's guess. I think this is a perfect note to end. A little bit of suspense. we're going to have to get you back at some point when you've finished your book and talk a little bit more about the actual technical problems and challenges. We haven't really touched upon any of that yet, but today I certainly learned a lot from you and I hope a lot of our listeners will as well.

It was an absolute pleasure to meet you both. Thank you so much and see you next time.