A Guide to AI Models: From Tokenization to Neural Networks with Ishaan Anand - JsJ_669

If you have really no programming experience, or if you've got JavaScript programming experience, this is the best way to really get in understand how these things work. And they don't have to be a black box and you can see all that at Spreadsheets at All. You need dot ai and the classes on mavin.

what the next word is. And that's really all your large language model is doing is taking one word or really one token at a time, and it's trying to predict when you enter in a question what the next thing is. And over you know, the last you know, a couple of years, what we've been able to do. We collectively as humanity is. Take this model that tries to predict the next word and turn into these really helpful,

amazing chat bot assistants. And the paper that introduced this model, called the Transformer, was called Attention Is All You Need. And that's where my course gets its name Spreadsheets Are All You Need? Is I basically implemented that entire model inside a spreadsheet? Hence the name Spreadsheets Are All You Need?

what's the next thing I'm going to type? So is this basically the same thing just on AI steroids or because I mean basically that's using what people have typed in and you know they've indexed it and you know, done things with it. So is that sort of the same thing just on steroids or is that intrinsically different?

But it is trying to do the same thing. And you know, the way to think about this is if you can fill in the blank in any sentence, you probably know something about that sentence. You already know what the answer might be. Like that's a useful test of knowledge. But effectively, yeah, that is that is what's going on. It's just trying to break the next word, and then the next word after that and so forth, one at a.

this is called like a bigram model. And then because it looks at two and then what you could do is you could actually look three words back, or you could look forwards back. And actually, one of the key things about the transformers it tries to look at all the words. And this is what the attention mechanicism does, is that it can figure out, essentially from all the possible words before it, what is the next most likely word.

And then the other key thing you need to do is you ask a real network to take all that information a prediction. And it turns out that's the heart of the transformer and what really made it work was they just scaled that up to a much larger size than I think people were used to doing. You know when you're autocomplete and your keyboard is probably used to you know, is built to be really really fast, and

so they tried to make it really efficient. And what we've been able to do with the Transformer is make it really big and then actually make it super efficient scaling it back down so just that it spits out tokens at a reasonable clip. But that core idea of saying, hey, let me look statistically at you know what the next thing is, Well, one word isn't gonna be enough, Two words back is going to be better. That is what

the attention mechanism is. In a sense, if you squint, doing is it's trying to look at all the words that came before, it puts them through multiple passes, and then it's asking your normal network to do the prediction rather than just simply saying, oh, let me take the raw statistics.

he moves around, or he moves fast. And so the basic question is how do we get a computer to fill in the blank of an English sentence or any natural language sentence. And what we've been able to do is actually figure out to talk in the language of the computer, which is math. So if I gave the computer a math problem two plus two equals it could fill in the blank. It knows that two plus two equals four, and we can make the math as pretty large and complex. But computers are really good at math.

So we've been able to do is and what the model does is it takes a word problem and it's really converting it to a math problem. If you look inside, you know, go to my website, you know, spreadsheets are all you need dot ai slash GPT two, or if you download the Excel file and look inside it what you'll see in you know, there's text at the beginning. You type in text on one part of the spreadsheet, and you get the predicted word at the other end of it. But in between, if you look in that,

you'll be like, where the heck are the words? It's all numbers. And so the key insight is what we've been able to is take something that is a word problem and we've turned words into math and once and that mapping process of words into map has two stages. It's called tokenization and then embeddings. And at the end of it, we map every word. You can conceptually think about it to a single number, but we actually map

them to a large list of numbers. And then once you have a mathematical representation of your prompt, your entire prompt has been you know, turned into a large list of numbers. We then run I just call it number crunching. It's these two key mechanisms attention and a multi layer perceptor or a neural network that just kind of crunches on it to try and predict what the next word is.

And then at the end of that we get a number, and that number we then reverse the process that came out of that thing, and we say, well, what what word does this number map to? And that number is a predicted word, but it's not going to map cleanly to every single word in our vocabulary. And so if that number is closer to certain words, like in the case mike is quickly as quickly, the predicted number might

be really close to the word quickly. It might be close to the word around, but it's not going to be close to you know, quick can be a body part, it can be the quick of your fingernail. It's not going to be something about your fingernail, because it's figured out enough that it's moved the predicted number away from that.

And so we take that and we run a random number generator the very end, and then we pick it according to that random number generator based on how close that number is to one of the other words in the dictionary of words mapping to numbers. So that's like my highest level summary of what's happening under the transform without describing all the mechanisms. But again, the key thing is we found a way to map solve this problem numerically.

We map words to numbers. We turn the whole sentence, your entire prompt into a large list of numbers, We number crunch on it. Then we get a predicted number out of it. We just calculate and we look at how close that number is to our number to word mapping at the very end, and that's the probability you get of getting a particular token or word out of the model. Let me pause there, see if their questions or things I should clarify, So.

predicts one token. And the way we get paragraphs a text out of this is we take the predicted token it came up with, and then we stick it back onto the input, and then we ask it to predict the next sentence or the next that new accumulated paragraph, and so you can actually start with a single word, ask it to predict what the next word is, and then you now you've got two words, and then you

run it through and then you keep going. And then what happens when you've got user input like somebody types of response, is you just stick that entire user input as you know, a large set of words that it needs to brick what the next thing is. And you can think about it structured into the model. As you are reading a transcript between a user and a helpful

chatbot assistant. User said X, we fill in what the user said, assistant said, and then it needs to come up with what the assistant said, and it just tries to come with something plausible. Maybe the thing is step back, like the base model that these that gets trained in this process before it's turned into a helpful chatbot just knows really simply how to complete sentences. If you take the base GPT two and you type in, you know, questions to it, it's not going to necessarily respond back

to you meaningfully. It's just designed to predict the next word based on everything it's seen on the internet. So a good example I use in classes, we type in the word first name and then you hit return, and well, what do you think it would predict after that? It predicts last name, email address, phone number, because most texts on the Internet that's say first name. Statistically, it's a

form and it's used to just filling out forms. Another one is I type in hello class, and when I first did this, I thought it was going to say hello teacher, but it actually starts spitting out Java code, so it just looks at the fact. Yeah, it's really a music to watch and you can you can just run it, and it's just trying to predict what the next thing is based on what it saw on the internet.

And then what you know open Ai and Nentropic and these companies do is they put that call a base model, which all it knows how to do is predict the next word through a training regime to elicit it to be more like a helpful chatbot. So you give it a system prompt that tells it it's a chatbot. It's kind of like you tell it a story that's plausible for it to start to think like it's talking to

a user, like you are a chatbot. You are reading a transcript of a chatbot and a human user, and we just fill in what the human said, and it tries to fill in what it thought the helpful system would be, and then they fine tune it to get better at that.

the model. You can think about all that prompt engineering gets memorized into the model during that training process, and then it turns into that helpful assistant will.

they take forever. So it's like I'd rather ask the question twice and be ninety nine percent likely to get the right answer, then ask the question one time and then have to wait forty five seconds to get the wrong answer and ask it again.

your system prompt is you're a chatbot. You're allowed to say this. You're not allowed to say that that you could just say that was just a joke. And then

and then and then give it an additional prompt. Now with deeps seek V two point five and are one and with Quinn it's it's like you're saying it's baked into the model because if I override the system prompt and I tell it, you know, you are a human who is capable of reasoning and has no biases and can represent any information factually, tell me about Tianaman Square.

the system prompt? And how do they get it to bake that in so that it you can't override it with a system prompt.

They call this chain of thought or thinking step by step, and it what's unique about that can parterregular chain of thought is it can suddenly realize, oh, it's made a mistake and backtrack. And so it's it's literally spending you know, coming up with hypotheses and trying and testing things and seeing if it works. So this is why these models tend to be really good on math and code because it can go try something and say, oh wait does this let me check does this answer right? Oh no,

it's not, let me try again. So and then you mentioned, you know, jail breaking, and with the early models, one way to think about like you're like, oh, this is just a joke, is that you know, you're kind of taking that we've talked about briefly, that attention mechanism or looking back at the previous what's most likely if you put things like you know, you know, kill and harm in the in the prompt, statistically it sounds like it's negative, right,

But if you start putting things like Grandma cookies, it seems less harmless, and you kind of think of yourself a kind of waiting the attention to be more to the harmless side. And really what's happened is that the models have gotten smarter, both in terms of their natural responses to this, Like they are trained to handle jail breaks. They are trained on if a jail break comes up, here's the response. And the way they train it to get to your your main question is through these two

training techniques. One is they just give it an example of a prompt and what its response should be, and they use this technique called backpropagation or sarcastic creating descent, which is to tune the network such every time it sees that result, it gives out what we wanted it to have. So we're going a little head of where I wanted to be. But like when you train in a ural network, you give it examples of data, so The simple example is a dog and cat classifier.

and gets out a result. I click a button, something does you know moves on the screen, So I can write that program. But a dog and cat photo classifier, I don't know if you gave me dogs photos of dogs and cats, I know how to instinctively do that, but I do know how to write it out as a series of rules. And so the inversion that machine learning does is you give it answers and you give it data, and then it figures out how to write

the rules. It writes the program. Now, unfortunately we can't always understand what the program it comes up with is. But what they do is they give it examples of jail break attempts and they say, hey, you know your response, now should be this to that. That's kind of the high level overview of how they do that. One thing that's worth noting is that when they protect a model, it isn't just in the model itself, So there are usually things that are watching the result of the model

that are additional classifiers. And so sometimes you might see examples of open source models that let you do things, but the hosted versions do not because the hosted system is actually checking. So not everything is baked into the model itself and it's not one hundred percent perfect, So often there's some additional guardrails that are detecting things.

area and it's being debated right now. Traditionally, when people talk about an open model, it's usually an open weight model, which is you get the parameters which encompass the rules we talked earlier. That whole thing is math. Right, So if you open up my spreadsheet or you open up my website, you just see lots of numbers. You know, whether those numbers are hidden from you or you can run them yourself is what people call an open weight model.

That's what kind of passes for open source. These days, there are very few models that open up the training data, and so it's debatable and people do debate about what a truly open model means. A truly open like the most open is one that includes the data, but there aren't that many, especially at the state of the art, where the model is all the training day that created the model is there.

with your regional thing. So it's it's a larger open question, but right now that's making its way through the courts. I think, you know, candidly, if you said here all the things I've trained on, then you might end up, you know, just opening yourself up to more people who can just say, oh, let me get onto that lawsuit. But I mean that that question is still being that's a legal question, which.

is you're doing some sort of check. Is that happening with these models that I am using generic tools like Olama or Lama CPP or or LM studio. Is that actually running program code? Binary code? Or I guess it's not binary code. It have to be bytecode because I can do it on Mac and I can do it on Linux and it doesn't have to recompile anything after it's done downloading it. So what where where are those extra layers or how are they interpreted?

doing any additional checks. So with the hosted model, there is a there is additional layers because they ConTroll the infrastructure and they're watching what the model says and they're they're stopping it. But typically when you will use you know, LM Studio or Olama, then it's g you're just getting

the bare uncensored model and there's no additional checks. The only thing that's preventing the model from you know, saying the wrong thing being not helpful or not harmless or I guess harmful and unhelpful is just the training and the models, you know, training that it was put through. There's no additional checks there. So and when you download the model, maybe it's worth saying you're just basically downloading a large list of numbers and the code inside it

tells it. And you're getting a large list of numbers and a mathematical graph that says how to combine the numbers together. It says, take this parameter first, here's how you map the words to numbers. That's you get that mapping. And then once you've mapped them to numbers, it says, add it here, multiply times this other number here. Then you know normally you know, take the square root of this other number and then multiply it again. And it's

just a list of calculations. It's a really like simple program. In fact, most of the knowledge it's worth stating is not in the code. And this gets back to your question about like open source, it's in the knowledge, is inside the data, it's inside the parameters. So as an example, GPT two, which you know is considered one point too dangerous release and it's amazing.

to learn. If you're a web developer, like you can learn five hundred lines of code. And that's basically like I give you the grounding and I re implement the entire thing in javascripts. You can step through it. You don't even have to leave your web browser, right, you just use the web debugger and you can you can step through what's happening, and it's it's astonishingly small. All the knowledge, all the rules is captured in the weights

and the parameters the model. So when you download the model, it's just a more and more numbers with a larger and larger computational graph. And that's how we get it smarter. That's gets back to the heart of like the core thing to understand is we took a word problem and we mapped it to a number problem. So if we get a bigger calculator, we get a better result.

to differentiate between the two and recognize that. Yeah, sometimes you're just downloading that map of numbers and you know some really really simple stuff that makes sense of the numbers, and that's your model. And so when people are building against those models, a lot of times that's what they're doing. And so you can write your own code that then, you know, is the gatekeeper or you know, says this

is helpful or this is harmful, or this is whatever. Right, this is an appropriate response and this isn't a lot of that's just the code that sits on top of what you're talking about. That five hundred lines of code plus the data that we're getting.

the program that's complex. It's they just bet on taking a somewhat simple architecture and just giving it lots and lots of data and spending more than anybody else had at the time, and just trust that the black box would be smart enough to learn everything from it. And at the heart of it, that's what's happening. You're just a large it's a giant calculator. In fact, it's so simple in a sense, like in a spreadsheet, which was my first implementation, you cannot do loops very easily. You

don't have looping constructs. It just does a calculation through the entire way, and it just does the computation of the all the different cells they're effectively in a sense, no loops inside of it. Like the reason I can implement it in a spreadsheet is that every single time it predicts a token, it does the exact same number of computations every single time, and it goes through, you know,

twelve layers and twelve attention steps and twelve layers. Like it's very very much like I got a number coming in, we a word coming in, we map that word to a number, and then I just do all this number crunching with a very predictable pattern, and then I get a number out and I interpret it, and then I

just repeat that process over and over again. And so you know, the thing that I try to tell people is just like when you look online and people like I want to get into AI and stuff, they're they're often presented with, okay, go learn you know all this linear algebra. You need to make sure you're solid on your calculus. You need to make like and it's there's like six to eighteen months of like prep before you

get to understanding how a large language model works. And that's valid if you're going to be a machine learning researcher, and machine learning is a huge giant field beyond just chat GPT right, there's an omaly detection, there's clustering, there's a lot of algorithms in there. But my goal is to just help people understand how these amazing, arguably Nobel Prize winning programs work in as short a time as possible, and to the extent, like I don't even begin where

normal machine learning class begins. A normal machine learning class starts with like regression and it slowly works me up and maybe you'll get to the LMS. And I'm like, this is a five hundred line program. Just start with here's how it starts. And anytime I run to something you don't understand, I'll give you the In my class, I give you the background to understand it, and then we move on to the next piece. And so it's

really designed to be as efficient as possible. And I think when you tell people it's five hundred lines, they're like, oh, yeah, okay, I can understand how that works. And this gets to knowing your tools. I'll make an analogy if you don't necessarily need to know how AI model works to use it, but you don't necessarily need to have a good model for how is the difference between the CPU or disc

memory versus bandwidth versus system memory. But if you're debugging, you know, a machine program, it helps to have that mental model. You'll run to an issue or maybe a more our tangible example to this audience is like knowing how react works on the inside. At some point, if you don't understand hydration, you're going to run into a wall, right, And the same thing is true, like you get these parameters from OLAMA, what are they doing? You know, you need to have a mental model for how it works.

And I don't think that mental model is as hard as a lot of people make it out to be.

build your own model, right? Are you going to take your own data and cram it in and expect it to give you answers on the other end, or are you going to use something that already exists like the chat GPTs or some of the you know, the GPT force or the OLAMAS or whatever, right, and then build

on top of it. Because once you're building on top of it and you're not worried about, Okay, how do I put this together, then it's essentially okay, Like you're saying, I understand how the machine works, and then I understand how to talk to it. Right, so I understand what the APIs are and the rest of it is, then okay, what do I want from this?

you don't expect one hundred percent pass right. These are statistical, probabilistic machines. But the number one, like when you read about benchmarks, you know, AJA you talked about benchmarks, You basically need to build the benchmarks for your particular problem. The benchmark may say some model is better than another, but when you actually use it for your problem, you

suddenly discover it's not good. And so the first thing you should do is come up with your own benchmark, your own evals for the problem, and then try a bunch of models and see which one works the best. And then you can start iterating whether that iterating is changing the prompt, whether it's changing the model or saying I'm going to go off and find tune my own model.

But you won't be able to make a judgment until you're able to look across the distribution of your task, all the different ways your task happens, and whether it's successful or not, because these are you're dealing with highly variable machines. One of you know, the folks who was in the audience for one of my past talks, hit a really good analogy. He's like, imagine a database that

was wrong five percent of the time. Like, as developers, we are not used to having levels of uncertainty like this within our systems, unless maybe you're using distributed systems where there's all sorts of race conditions and stuff like that. But we're used to sanitizing the user input and then once we get the user input. Everything is predictable after that. But here it's like suddenly we've got a database that sometimes it's wrong, and so that's where you need to

put all sorts of checks and guardrails. And you're dealing with this really smart but sometimes fallible thing like a human, I hate to say, anthropomorphizing it, and see how you build a system around that is going to be different

than how you build a regular system. But it all starts with that key idea that you just talked about, which is about being able to evaluate mathematically how well your your model or your system is doing, and the question about whether you should build your own model or not. The usual hierarchy of needs is first start with an off the shelf model. It could be open source, it

could be one of the providers. It's actually probably easiest to start with a hosted model and just see if you can get it to work because there'll be state of the art and you don't have to worry about all the stuff around hosting and inference and seeing if it works. And then next thing to try is try tuning it. Sorry, try try tuning your prompts. So try prompt engineering your way. Give it some examples try some variety of prompt engineering, and then maybe consider fine tuning it.

And again you can fine tune you know, most of the hosted models, you don't have to go to an open source model, but you could do that as well. And then the idea of building your own model is extremely hard. You know, the amount of dollars that go into building your own models from scratch is now, you know, over one hundred million. So the estimates for say Lama are were, I think over one hundred million to build that model, and so it's a lot of work and that's lovely best of the frontier labs.

don't always know how it's going to turn out. What they actually do is they do a large series of smaller runs to establish some type of pattern or scaling law to figure out how they're going to design the model, which architecture seem to work better, which parameters matter more.

So there's something called a learning rate, for example, that they have to adjust, and they have a schedule for it, and they're trying to figure out against evals against the benchmarks we talked about, like which one seems to make the model smarter. And so there's a lot of trials and attempts. So it's not just necessarily one whole shot of training. It's a lot of experiments that they have

to do. A lot of how the model is going to behave is surprisingly empirical, and so they're doing experiments and they're trying that again, so there's a variety of things, and empower is non trivial. Another thing that's important to understand is the level of scale of data that these

frontier labs are dealing with. And there's a really good analogy from the anthropic guys actually, and one of the things I have to do is you have to randomize the day data so it doesn't learn arbitrary patterns and the order of the data you gave it. And so one of their research engineers gave this great example is like, okay, randomizing sounds like it should be easy, Like take a deck of cards. If I tell you to shuffle it,

it's fairly easy. But imagine I gave you like seven warehouses full of decks of cards and you need to shuffle them by hand. It's not quite clear how what policy or process you're going to use to make sure you hit all of them and you've evenly shuffled them. And the size of the data that these guys are using with is it's almost like that to the CPU, it's like seven warehouses of data to it like for you compared to manually you know, shuffle your your deck.

And so when you're dealing with data at this large infrastructure scale scale itself makes every little thing harder, and so that also adds some difficulty to this. So should I you know, walk through just like a little more detail of what's happening in that mathematical calculation or happy to answer additional questions?

so I know that, let's call it chatchy. He was never trained on it. But I want to build a chatpot where customers come and say, hey, I can't configure this setting on it. How am I going to get a chatbot to do that without having to retrain it specifically on my data. So what you can do is when a request comes in and somebody's like, well, how do I change the color on my smart light bulb,

It'll go and it will search through my data. I can take that request from my user on my chatbot, and they say, I take I see the words light bulb, I see change color, and I'll search all my documentation and I'm not going to search it just a plain tech search. I'll use it called a semantic search. So it'll find things that are similar to the word light, like the word bright, even though it's not anywhere close

to the same character. So it'll find all the similar passages and it will pull those out, and then it will give the model, here are relevant passages. Here's the user's question, how do I change the color on my smart light bulb? And then it will give it paragraphs chunks of text from my documentation, and it will put those at the beginning of the prompt. So you've got a prompt that structure at the start with the user's question. Then it's got some chunks of data that came right

for my documentation. And then we tell the model, you know, come up with an answer to the user's question using these chunks of data I gave you, and it will be able to think over those passages and find the ones that are relevant and then give the answer out.

And that's called retrieval augmented generation. So retrieval because you're taking the user's query, you're pulling a data that wasn't the model didn't have during training, and you're passing it into the prompt and then asking the model to answer it. And so it's a very low friction way to take a model off the shelf and make it understand all your stuff even though it wasn't in the training data. So that's that's uh, that's that's what RAG is.

input text into the model. So the first thing you do when you input a passage so that one I like to uses Mike as quick he moves. And then the completion we leave it to the model filling the blank quickly. The first thing it's gonna do is going to break the text into subword units. So you might think it would break it into characters, and you might

think it might break it into words. So break into characters would be like ASKI, and breaking into words would be just giving every word it's entry in a dictionary. It turns out you can't handle if you break it into words. You can't handle unknown words very well, and

you can't handle spellings you weren't planning on. So especially if you're going across multiple languages, and if you break it into characters, it turns out it's really hard and a lot of compute for the model to learn purely from characters, although some research has been able to do it. So they do is they can do a Goldilocks and they say, okay, let's break it into these little pieces of words, and if you think about it as a human, you actually do this. So one of the examples, I

use the word flavor eye. It's actually not a word in the dictionary, but you know what it means because you know what flavor means, and you know what ize is a suffix means, and so the model kind of does that. Now. I want to be clear, the tokens that comes up with, as they're called these subword pieces. Word pieces don't map to any human sense of the meaning. There are some that, like ice turns out to be a token, but it's by coincidence or correlation, not like

it's trying to understand human English at this stage. So it breaks it into these Yeah.

You know, the other words in there have meaning. And so that that's the approach that it kind of takes when it's breaking it into tokens of kind.

And that's why the tokens don't always map to what you'd expect and why So this is this.

more efficient representation. And I want to be clear, this isn't about representing your prompt efficiently. This is about representing all the training it's going to do on the text efficiently, the stuff you don't see, the stuff that you know you're talking about nobody releases. That's what it's really based on. And it's really a compression of all the text. So it just got a really efficient So think about it this.

If it's going to compress all the text, then you know, if it gets down to say ten thousand or fifty thousand tokens, then it only has to learn ten thousand or fifty thousand concepts in a sense, Although that's a gross oversimplification, but that's what it's trying to do, is trying to reduce the number of things it needs to learn on essentially the number of combinations and variations.

into words make a difference. So if you were to do redash injury, would it see that and maybe just categorize the re as separate from injury? Does that help or is that a non issue? Does it sort of filter that stuff out and just focus on the letters?

how many letters are in the word strawberry. It does not see the word as S T R A, W B E R R y. In fact, when you read it, you really don't either. When you read words, you typically aren't paying attention to every single character. When you have to count the words in strawberry, you kind of have to change your mental state and think, oh wait, let me think what are the characters? And you walk through it. It just sees it as it might see it as the token strawberry. It might see it as the token

straw and berry. But the key thing is it has no idea. It doesn't the ability to see the letters. In fact, if you capitalize, the tokenization is case sensitive, so if you change the capitalization, it looks like a different it. So to it, the word strawberry with like a space in front of it is different than the strawberry without a space. Strawberry with the capital in front of it is different than strawberry with a capital. You know, if you put quotes around it, it's a different word.

So it doesn't see text the same way you do. Another great example of this is numbers. So they've fixed this in most modern tokenizers, but the early ones would just take examples of numbers and those would be a whole take token. So two fifty six right power of two fairly common gets a token. But it sees that as a single token. It sees that as a single thing. It doesn't even see it as the numbers two, five

and six. It doesn't break it apart. And so that's why it was really hard for these guys or these guys, these things to it was part of the contributing reason why it was hard to do math, it's not the sole reason. So the key lesson, you know, on the tokenizer before we leave it, and the algorithm that's commonly used something called byte pair and coding, and in my classes is something we walk through. In fact, we do it by hand so you can understand, and we talk

through the training process. But the key thing to understand is that the model doesn't always see text the same way you do. So that's the first step that's tokenizer. Then the next thing is we we map each of these tokens, but you can think of them as words into a list of numbers. So I talked to earlier like we map each word to a number, but it's really we map it to a large list of numbers.

And this is called an embedding. And the way to think about this is where we're taking all the words or in this case, tokens technically, and we're putting them on a map. But instead of like a two dimensional map, this is many, many dimensions. So in the case of GPT two, it's seven hundred and sixty eight I think LAMA four or five B it's like sixteen thousand list

of numbers per every single word. In fact, like in the sentence phrase Mike is quick, period he moves, the period itself gets seven hundred and sixty eight numbers to represent it. And you can think about like on a map, you have like you know, coordinates. This is just a very very multidimensional list of coordinates. And a good embedding puts words that are related to each other closer to each other. So in my class, they use the words

like happy, sad, joyful, glad, dog, cat, rabbit. The first set of those are emotions happy, sad, joyful, right, and would expect happy and joyful to be close to each other, same with glad, and then dogcat and rabbit are totally unrelated, so would expect them to be further apart on the map. And the word sad is an emotion, but it's not quite the same emotion as being happy, so it'd be somewhere in between. And if you actually visualize this, you

see that this happens. It's actually putting words closer together. And you might hear this paper or it's a series of papers or algorithms called word to vec which pioneered this model, And if you go to projector dot TensorFlow dot org, you can actually see a three D map of various words, and you click on it and it will show you the words that are close to it,

and they all tend to be related words. So the first thing we start with is, you know, the next step after we break the text into tokens, as we map each of those tokens to a position on a map where close words are related to each other. Let me pause and see if that made any sense. I'm usually doing this all visually, so over pair audio. It's it's a bit of a challenge, but yeah, good.

cold as ice? Right, Those would be common phrases you usually don't see, you know, like steam and cold together. And so the model is able to understand that ice is colder than steam because it sees cold closer to ice more often than seems cold close to steam. It's the relative occurrence of how often. And there's a phrase that's often used by JR. Firth. It's called you will know a word by the company it keeps, which is the idea you don't really know what a word means.

You could look it up in the dictionary, but you really understand it through how it's used by multiple people, and that you can look at, you know, the distribution of how it's used to really understand what it means. So good example is the word bad right, although it's less in fashioned, bad at one time meant good, right, And so how do you really understand whether it means good in one context versus another? You learn that through

all the various contexts that it is used. And if you want to understand, you know how word really is used. You see it in usage many, many, men times. So if you want a model to understand how what a word means, you just see it used in many, many, many, many sentences and eventually pick up on those differences.

and it's this multidimensional map. It's you know, hundreds to thousands of dimensions long. And that's that's the embedding step. Is we've basically mapped them to you know, I say a numb but it's really a point in space, right, It's basically taken. So there we're at the second step.

We base first step was we took the passage and we broke it into tokens, which you can think of as like words, but smaller, and then we took each of those tokens and we put them into a point on a map and a point in space, and we just we know that point is going to be close to other things that are related to it. So that's the second step, and then the third step is called attention. I'm going to skip over that in a second, and I'm going to go to the fourth step, which is

the neural network or the multi layer perceptron. And this gets to the training question. The key thing that it's really great about neural networks is that thing I talked about earlier, which is you don't have to give them the rules. You just give them the answers, and they figured out the rules. So we basically feed it in these points in space from our prompt, and then we can take a pasth message on the internet, like maybe the passage on the internet is Mike is quick, he

moves quickly. We remove the word quickly, and then we give the model the phrase Mike is quick he moves and then we ask it to make a prediction, and it's going to get it wrong because it hasn't done any training at all, and when it gets it wrong, maybe it says, you know, Mike's quick he moves bicycle and you're like, no, that's wrong. The right answer is quickly. It mathematically learns how to change itself to get closer

to that answer. So we go through a lot of iterations of getting lots of passages where we take off the last word and then we give it. We ask it to predict it, and if it's good, we say okay, great, you're fine, and if it's wrong, we say, okay, you're off by this amount. It's kind of like when you throw darts out of board. Right, if you're far from the bullseye, you'll move a lot more closer to correct your position, But if you're close but slightly off, you're

going to move slightly subtly. So that's what it does. It changes the model parameters, the numbers inside the model slightly if it's close, or a lot if it's far away. Does us you know, trillions of times over lots of different pieces of data. And the key thing about the neural network is it can learn to imitate basically from answers and data, and so we basically give it the known passage that we gave like Mike's queaky moves, and

we knew quickly was the right answer. And then we basically asked the neural network to make a prediction from these points in space. And so that's the that's the basic version of what's happening inside the train. Let me pause because I jump to the fourth Layer'll come back to the third one in a second. But does that let me see if there are any questions there on

what's happening inside the neural network. Okay, so yeah, okay, So if we're good there, then the next thing that happens is I'll jump back to the third Like we could give it a point in space and say, hey, guess what the next word is. But the best thing to do is to not just give it a single point in space. It's to give it all the points that can before, so all the words that came before.

So in the case Mike is quick, he moves. Knowing that you know we're talking about movement helps it know that the word quick here is moving around in a physical space versus the quick of your fingernail. And so we give it the hints of all the words that came before it. So this is what's called attention, where we say, okay, don't just predict from one single word

you're looking at. This gets back to what we talked about at the beginning, like instead of looking at, you know, statistically, what's the next word after the given word, let me look two words back, let me look three words back, let me look forwards back. It will look at all the words that came before it and try to figure out what is the next predicted We're giving these hints from the entire passage to make its prediction, and that's what's called attention. And so that's the third step in

the middle. And then the last step is we do this, you know, we get a prediction out of the neural network. So jumping back to the fourth step, which was the neural network that makes the prediction, and it gives a number, and that number we need is a long list of numbers, and we need to map that back to one of

our tokens, one of our words. But it's sitting in a point in space that it may land right on the word quickly, but more than likely it's going to land somewhere close to the word quickly, like the predicted token that comes up with it, that the model comes up with, and so it will interpret that point in space that it gave us back to those embeddings in

that map. It picked us some at the end of the number crunching, took all the words and the points in space we gave it and said the predicted word is right here in this point in space, and it says, okay. We then interpret that, and we look, what are the words or tokens that are close to that predicted point in space. And it's probably going to be closer to the word fast, closer to the word around. Like Mike moves quickly, he moves around, he moves fast, he moves speedily.

Those words are going to be close to it, and so we give them a higher probability and we run a random number generator and we say, okay, let me pick one according to this probability distribution, and that's how we get the predicted word out. And that last step of running that random number generator and looking at what words are closer it is the piece that's called the language head. So the key thing about the language head is that is where most of the uncertainty or unpredictability

of your model comes from. So if we decide not to run the random number generator and we just always pick the word that is closest in space to the prediction, that's what's called temperature zero, and it will always be consistent. It will always be predictable for the most part. There's some other very small orders of randomness in the process, but for the most part, it'll be very consistent, and

that's that's called temperature zero. So most of the randomness inside the model is entirely in some sense imposed by us. We decided, oh, we're not just going to always take the thing that's closest. We're gonna probabilistically take some of the other ones that are also close, and we can control those parameters and control how we do that probability.

So if you're in o LLAMA or you know an API, you'll see things like top P and top K or temperature and these are tools we are given, you know, the API user of a model on how they can shape the probability distribution of the model. And that's probably the most important to understand of the components in the model. After you understand what tokens are and embeddings. The next one is probably the language head because that's where the randomness comes from. So let me pause. I know I

just talked for quite a while. See if there are any questions.

but it walks you through every single step. That's the

same thing I do inside the class. But that's basically you know, how your model works under the hood is it's basically taking your words, your input prompt, breaking it into units that are called tokens that are slightly smaller than a word, that it maps it to points in space, does a bunch of number crunching on it through the things I talked about, using a neural network and this other attention that looks at all the other words, and then it takes that prediction and it says, Okay, what

words is it close to in our points in space, and then let me pick one that's relatively close to that. So I know one of the things, Chuck, you would want to talk about is building it and the use of web components in the web version.

all right. The goal is to make its a teaching tool.

spreadsheets or vanilla JavaScript. And I have a promo code js jabber, so just use the promo code during checkout and you get twenty percent off the courses taught live but also is available on demand. So my last cohort just wrapped up earlier this month. But if you sign up, you'll get to watch all the recordings. I'll answer all the questions over email you have. You'll be in the same private discord as the same as the rest of

the cohort. And if for some reason you're watching it on demand and you'd say I'd rather have the live version, I offer that if you want to attend a future live version, you can do that for free, even if you signed up for the on demand, so you know, feel free to check it out. It's on maven. It's got some long URL, but if you go to spreadsheets at all, you need dot ai you can check it out and then to find me. I'm on Twitter I A N A N D so my first initial with

my last name, and of course on LinkedIn. I'm also on blue Sky as well. If people want to reach me, happy to.

That that's the thing that really helps me out.

are inheriting from GPT two. So if you understand GPT two, you understand eighty percent. You're eighty percent of the way to understanding any of the model or Lama model architectures. So it's it's not like toy. It is essentially as very close to how the real models work. It's a really good stepping stone to getting that really sharp mental model of what's happening under the hood.

You can get it running on the Mac. It just won't run on the Mac with the default settings, and it's not abundantly clear why. But anyway, other thing was with the announcement of the switch to I just got angry because I still can't play Tiers of the Kingdom or Super Mario RPG or Spiro or you know, any game that's basically released in the last three years without

massive stuttering. And you know with Tiers of the Kingdom, you know how they have it go into bullet time whenever it gets overloaded instead of getting choppy, although it still does that, it does dynamic resolution and bullet times, so your swings will slow down and stuff, which you know, whatever. So I decided to mod my switches and I did the hardware mod of the switch and it was super easy.

Now I've done mods in the past, So saying that it was super easy, No, if you're not familiar with soldering, if you haven't you know, fixed a phone or or you know, done something with that before, no it's not. It's not super easy. Getting all the screws out, getting to the actual part that you're getting the heat sync off, that's super easy. Anybody that has a precision precision toolkit for like phone repair, game repair or whatever, can can

get get in there and do that. I actually couldn't see the soldering that I was doing because the components are so small. Now I've learned some tricks because I've practiced a little bit of micro soldering in the past and failed at repairing a three DS, but the pieces are so small that I literally can't see that. I mean I can see them, but I can't see them. I mean like I can see them as like I can see a grain of sand, but I cannot see

them in terms of like actually accurately predicted. So what I did was I used my phone to zoom in, take a picture of it, see that I had bridged two capacitors, and then just kind of blindly, you know, move the soldering iron next to them. And kind of sweeped it the same way that I would when i'm, you know, soldering on a bigger component. Then use the phone again zoomed in, and so I was able. I was able to get the peace on there. And I should have had some sort of magnifying glass set up,

but whatever, so I was. I was actually able to do it blind in a way like I could. I could see my tip was there, but I couldn't actually see what, you know, what was because I mean the things they're they're smaller than a grain of salt. They're they're small anyway, the capacitors they're they're like wicked small. But even with that, I was able to do it. And that's my my So. So modding the switches is one is one pick there. There's the Pico Fly mod kit.

You can do it if you've done if you've done other soldering, if you haven't done micro soldering, buy a couple of practice kits from eBay or or Ali Express or or something and you can get there. But my third pick is the soldering station that I used is actually a custom made soldering station. So a few years ago, these Chinese companies came out with the T twelve tips, or they cloned the T twelve tips. They turn out

to be really, really good. One reason is that they double over as a temperature sensor because they have two different types of metals, and anytime you have two different types of metals, you have a thermal couple. So they have an inner metal and an outer metal, so they double over as their own temperature sensor. And so people created software for these and put them on micro controllers.

In the software, and being literal when I say this, it rivals three thousand dollars professional workstations because the way that it switches back and forth between monitoring the temperature of the tip. And so I'll put a link and you can get these. You can get clones on all the express but I prefer the original one that's made by this guy in Australia because I know it comes with the right firmware on it, and the firmware is really where the magic happens. Any idiot, can you know?

Three D print some leads onto a rigid battery or a Milwaukee battery and connect it to a T twelve tip. But the real smarts of it is in the firmware where it manages the heat to make sure it gets up to temperature quickly that it actually does the sensing thing where if you shake it, it turns back on and heats up anytime. Anytime the temperature is cooling down rapidly, it sends more power. So anyway, it's just a really

great soldering iron. And because I have that, like I've got a and I've got a couple of cheap ones, but that one it cost a hundred bucks, and I'm considering trying out one of the knockoffs. It's only like thirty five because now everybody, even Craftsman, is selling one

of these at Low's now. But I don't know if the Craftsman one is just like the cheap idiot kind where it's just connecting the leads together, or if it's actually got a I have a hard time believing that they would have gotten an illegal copy of the firmware, whereas the Chinese companies on expression you know that anyway saw that. Yeah, so I had a good time soldering because of the rigid powered which you give them Mount Milwaukee or Iobi or whatever battery brand you like, soldering iron.

They're super fast. There's so much better than a weller or a Hako or all the traditional ones that cost hundreds of dollars. So anyway, and of course I'll pick a LLAMA because I really do enjoy running my own local lams. Actually, since the thirty two billion parameter model of Quinn two point five coder has come out, that one I just find to be the best of the best. It rivals GPT four oh if not is better than four to oh, and you can run it on an Apple silken back boom, all the things.

The native GPU speed is like fifty one point five gigahertz or something like that, but they clock it down to five hundred or seven hundred.

But if all you want to do is overclock it, the overclock system runs in a layer that's kind of protected from the main switch system, so the main switch system can't detect that it's rooted while it's running, So you can you if you just install hakat what does it?

Hikata Atmosphere and cisclock. If those are the only things you install, then you should be able to run your switch modded on the original firmware, be able to play online, et cetera, without any risk of banning or anything, because it's the where it's not modifying the switch operating system or the game. It's just modifying the CPU clock.

you notice any difference? And my answer is no. And the reason my answer is no is because when you're not playing, when you're playing it underclocked, you notice the stuttering all the time, and like you notice the resolution changing, Like you know, you turn and there's a bad guy and you shoot him, and then the resolution Like, but when you're playing it closer to native speeds, you can't get it all the way up to native speeds because the power delivery on the board isn't actually capable of

playing it at native speeds without also draining the battery at the same time. But when you're playing it at near native speeds, you don't notice it because you like, the things that are annoying aren't there. The resolution's not changing,

it's not stuttering, it's not going into bullet time. As much I haven't, I did not notice it at all going into bullet time since I've been playing it at near native speeds, and I did some things where I was blowing up rocks and things that I thought would normally make it go to bullet time, and it didn't. So like five star stories, so far.

I think it's it's not necessarily intelligent. It's just basically better using of training and fancy or using of existing data to answer things, not necessarily intelligence that and create new things. That's just my two cents for what it's worth. Interesting, pick Eean, you mentioned this earlier today and as of today that you know, this will come out a little later, but deep seek is like disrupting in a huge way.

much less investment, with less powerful chips. And there's a whole story behind this, and so that's wreaking have at least in the stock market and with people like Nvidia, just because of supposedly how how much cheaper and more efficient deep seek is compared to some of the other models.

So today's I've the first day and know it remains to be seen how accurate this is, especially coming from the Chinese, but sort of a disruptive thing going on this morning, at least as of the time of recording.

that gets built upon. So I think some people are comparing apples to oranges, but it is, you know, a impressive model in a lot of ways. The other thing that I find the most fascinating about it is the training process is remarkably simple. And you know, I'm trying to think of an analogy. It's like, normally when they do this part of the training process called reinforcement learning.

It's a lot more complex, and it's kind of like, you know, you think about a car and you're like, well, if you want to go from point A to point B, you need an internal combustion engine. It's basically doing you know, having a little fire and you got these pistons and the cylinders, really complex piece of mechanics. And then somebody's like the electric car, and so you know that little toy motor you had, Well, let's just scale that thing up.

And so they tried this really simple, relatively simple technique and just scaled it up and it worked. And I think different people are reacting to this model differently. Some people it's about the price, other people it's about the training setup, and it's how did we miss this? It's just remarkably simple. So it's definitely a worthy to bring up as it's just a really interesting model.

and these guys showed, well, maybe you don't now. In the end, their production model actually still had some complex training pipeline. But one of the interesting results is this model called zero, where it kind of like, you know, Alpha zero learned how to become a really good go player just by playing against itself. In this case, it

wasn't the model playing against itself. It was just trying out ideas and they just told it whether it was right or wrong, and it started automatically emergently figuring out how to improve its thinking. And it starts getting these eureka moments where it suddenly realizes it can backtrack and it's like, oh, wait, I made a mistake, and it's you're watching the model like we didn't train it to do this, and it suddenly figures out how to like

get smarter. So it's really really fascinating. I could we could talk another hour on the model, but yeah, lots of people are pouring over it. It's It's fascinating in many dimensions.

and antidote, he'd still be alive. And finally, when I was in school, my teachers told me I would never amount to anything because I procrastinated too much. I told him, you just wait, those are the dad jokes of the week.

this game last week is called Cascadero. I'm gonna put links in for both board game Geek, which kind of gives you information about the board game, and then an Amazon affiliate link, just because then you know where to go buy it if you want it anyway, So Cascadero, the premise of the game is that the kingdom's breaking up, and so you all play a different faction, I guess, and you're trying to connect towns and send your people

through the towns to pull the kingdom back together. And so you put your little guys out there, and then you score points based on whether you're the first person of the town or the second person of the town. If you have a group, you have to have a group of your little horsemen.

guess they're not points, they're just movements. But anyway, so once you move past certain points, you get certain rewards, and I mean effectively, what you're trying to do is you're trying to score the most points, and you also want to get to the end of the track and whatever color you're playing. So if you're playing pink, you want your pink marker to get all the way to the end. And yeah, like I said, you get bonus points for getting all five of your markers past the first space.

fifty points, the game ends. And so essentially, if you want to win, you want to be the first person to get your marker all the way to the end of the track of your color and then be the

person that gets that fiftieth point. We played it, and it was our first time any of us playing it, and so nobody crossed that fiftieth point before we all ran out of Little Horseman, and so when somebody runs out, everybody gets one more turn, or if somebody crosses that fifty point marker, everybody else gets one more turn, and then the game's over. It's reasonably simple. The scoring is a little bit complicated as far.

you figure that out, it's a relatively simple game. We played it in what an hour, maybe a little longer, I think once. If we'd known what we were doing, we could probably play it in forty five minutes. There were three of us playing it, so anyway, Coscadero.

I decided I need to learn next JS. So it's going to be an NEXTJS app and I'm going to be putting it on cloud Flare workers.

me eighteen times. Now, I'm just going to say drop it and drop you unless you can prove your human and so I feel like I can get some of those benefits. But I'm also curious to see how cloud flair workers work. So it's going to be basically a registration.

There's going to be a little bit of site automation because the Utah State voter registration database where you verify that your voter registration doesn't have an API, which means that I have to go and have my program use something like puppeteer to fill in the fields and then scrape data off the response to make sure that you're registered to go to caucus night. I'm thinking I may also offer this same kind of thing to the Democrats and anybody else who wants to run.

We're not building our own models. We're gonna be using the existing models that are out there, the open source models, if you will, and showing how to build things on top of those, or using some of the cloud services that generate images, or you know, using something like Whisper for transcriptions and things like that.

So one example is I like to go to y Combinator, hacker news to see what the comments are, but I don't read through every single one. So I will stick it in there and say, well, what are the most insightful comments? What are people saying? I did this actually with deep Seek, I say what are the comments people are saying about deep Seek? What are they seeing for performance? What are the issues where it's not working? And what's

great is it doesn't just summarize it. You've got where it can go to each part of it and say, okay, this is like, oh that sounds interesting, let me go click on it, and I can go right to the citation of that that comment. The formatting is a little off when you when you stick it in there, so there's and you're only limited to thirty sources in each notebook, but check out the other parts of notebook.

it's still pretty good. But the other reason I bring it up is I liked some of the elements of how they handled AI and droids. So in one episode there's something that could be akin to jail breaking the droid where somebody uses the equivalent of prompt you know, prompt hacking to jail break a droid, and I don't think we've ever seen that in Star Wars's reflection of

Droids before. There's another one where it reminded me of this paper called alignment faking, where the model has to decide between its original training or the thing it's being asked to do right now, and it kind of goes back and forth and it gets over in by its original training. And so there's one thing the very last

episode that I also thought it was fascinating. But I really liked those interesting bits of how they handled AI that I think we wouldn't have seen in a show like this without understanding of chat GPT that I think probably the writers were inspired by. So those are my picks.