Are you gonna be able to not check email during the show? Haha! You're literally deleting email right now, Mark! He shows me his hands! He's like, you're on a... Oh pilot, please! No, no pilot, delete the email. Haha! He's archiving email with his big toe right now. And then run the music. Like that's... see? How that works? I literally can hear the music running now if we begin the show.
Alright, hey friends, I'm Scott Hanselman and this week I'm learning about AI limitations. What are you learning about Mark or Sinevich? I'm learning about AI limitations with Scott Hanselman. Do you learn every week? Do you like read papers? Are you all week learning? Every week. I'm constantly reading, especially AI research papers and then papers on they come out about cloud native research as well. How do you balance like learning versus knowing it all? How do I balance learning versus what?
Well you kind of carry yourself as a know-it-all right? You know like you're the CQ of Azure and you know it all. No, no, I'm gonna learn it all. Oh you're gonna learn it all! That's why we call this show Scott & Mark Learn To because we're always learning just because you've been in tech for a long time doesn't mean you know everything because things are changing. And by the way you know that Satya came up with that learn-it-all or at least... Is that a thing that a Satya thing? Yeah.
This is amazing. We're two minutes into the show and this already has Satya shout out. Are we gonna sell his book now? Sure. We should probably hit refresh. That's the name of his book. So excited. Yeah, it's okay. Review time has passed. He won't even hear this until next show. So AI limitations though. I thought that it was gonna change the world. I thought that we were all gonna have segues. We were gonna build our cities around how the segway works. We were gonna
get rid of our cars and then it was gonna be large language models and AI's. We were just gonna chat with them all day and then all work would stop and then we'd have 15 hour work weeks. But yeah. So where's my jet pack? Well I think I use AI every day as I code and I think that it's a huge productivity boost. And I've talked about this before. I literally cannot code anymore without AI or at least my productivity would severely sink especially with Python and PyTorch which I do a
lot of AI research using that language and that framework and hugging face. So I'm inherently lazy. I don't want to do things I don't have to do. And so when I'm programming now in Python and I'm like oh I need to write a loop that iterates over this data and processes in a certain way. I'd much rather just ask the AI to write that code for me than do it myself. And so that's I've become a tab co-pilot auto-completer and co-pilot chat write this function for me. Yeah.
Otter. And if you took it away from me I wouldn't there's like I don't remember how to do this in Python. But I remember being on a big giant X windows machine that was like Hercules orange color and I would just do everything in VI and then we got color syntax highlighting and the graybeards, the non-gender specific graybeards were saying that's gonna rot your brain. You know, syntax highlighting or rot your brain. And then we got IntelliSense where you type whatever
dot. And now that's gonna rot your brain. And then the TI-83 calculator came out and they're like, no, you can't learn math. It's gonna rot your brain. Is AI coding rotting your brain? Interesting because like I said I don't really know Python that well but I code a lot of it as a AI. And do I need to really know it that well? I don't think I do. I mean I need to know it just enough to get the job done efficiently. And I think that that works for these other things
you talked about. So if we think about it in context of driving, I've got my 16 year old driving now and it's a little concerning. I was taught to drive stick shift. I was taught to understand the car and know how to change my own oil. My dad gate-kept, gate-kept the car by saying you don't get to drive until you change your oil. You don't get to drive until you put your tires on. But we don't gate-keep the garbage collector and see sharp from people. We don't say you can't use a garbage
collector and tell you malach your own memory. But you don't know Python mark for Synovish and you are writing Python all day. Should you be driving that you don't really understand the car? Should we be allowing that? Well nobody can stop me but nobody can stop me. I think the thing is you've got to be aware enough of what you need to know to be as efficient as possible. Yeah. If relying on the AI isn't the most efficient way for me to get the job done, then I would
need to go learn more, learn Python better to get the job done better. So it's this and I'm kind of doing that all the time. Like what is the line that I need to get up to for maximum efficiency where I'm not spending time on things I don't need to worry about? So what should we teach somebody though who's getting started? Yeah, this is a really great question. And I actually know. Like you did change your own oil and you did change your own tires and you did malach your
own memory. So now you should be because you'll look at the Python and you'll go, I don't think that was good Python. Right? Your spidey sense goes off. Yeah. Well, talking to university professors about the changing aspect of computer science education, especially early computer science, undergraduate education and light of things like GitHub co-pilot, they have adopted a, you need to change your tires and change the oil when you start. And then once you understand
what that is, then you don't have to do it anymore. May you live in interesting times? Yeah. You started by saying, you know, what we're learning about is AI limitations. And I started by talking about how I use AI. And how do you use AI? I brainstorm and I use it as a rubber duck. Where's my rubber duck? Somewhere around here, I've got a rubber duck that is here it is. I've got a Microsoft board rubber duck. Oh, you haven't rubber duck too. Look at us. I'm trying to give
them when I, they give them speakers. So, so rubber ducking, I've talked about this before is, is the getting it out of your mouth, hearing it and then having it go back in your ear. You don't even need the other person. So then you just get a rubber duck. But I would always call my, my coworker, my very tolerant and very kind coworker and go, hey, got a sec, can you jump on a quick call? And you just talk to them and then by the time you've talked to them and said it,
you figured it out. It's the thing that happens when you're trying to solve the technical problem in your head, in the shower or whatever. So I rubber duck actively with the large language model. And I realize I'm just talking to myself. Yeah. Is that a fair analogy? I am just, it's a sock puppet and I'm just chatting with myself. I'm looking at the mirror. I don't think it's quite that way because you're getting inputs that aren't generated by you and insights that aren't generated by you.
But the vectors that are being generated by your initial words, if you miss speak, if you say a word or like in the family, I'm the Googler. I'm the best one who can Google and I'm using Google as a generic verb. So you know that time when someone says, I've been googling for hours and I can't find anything and then you step in and in one Google, you find it first one because you intuit the right thing to say. I am noticing people giving too much info to large language models
and they'll hit a word and I'll say, yeah, that's a mistake. You shouldn't have said that word's going to nudge it in the wrong direction. So I'm already developing that into it. That's true. It will, it is guided by what you ask it or tell it. But that doesn't mean that it's not going to give you some output that is surprising or new information for you. But isn't the, it giving you something surprising more random and more role of the dice? Like are we just playing D&D with this thing?
Sometimes you roll a 20. I mean, if you're asking about a specific topic, it almost certainly has knowledge about that topic unless it's your topic of expertise that you don't have. Ah, okay. I recently, and I don't know if this is bad or good or evil, but my son had a his shoulder X-ray and they gave us the MRI and the X-ray and it was incredibly long. And I said, summarize this read, this radiologist read in layman's terms, acknowledging that I'm an engineer and my wife is
a nurse. So I gave it context about who we were and my wife can't read a radiologist report. But I didn't say put it in utter layman's terms. I said, here's who we are. Yeah. And it gave me different results than if I said make, you know, to explain to this to me like I was five. And I found that to be a very helpful use of the tool. Well, so now speaking, limitations. Because I think one of the limitations you could have run into in that case, that could be problematic, especially in medical
domain is something called hallucinations. I've mentioned to you before that using the term hallucinations might be a problem and there's like, responsibly, I discussions around that. Do you think that that's a problematic term? I think the cats out of the bag on that one. There were discussions about this over a year ago when the terms started to show up in widespread usage. The momentum behind it is just okay too big at this point. I mean, in fact, anthropic published this system prompt
for cloud 3.5 this past week. Yeah. And if you go read the system prompt that says, refer to this as hallucination. Yeah, there's other terms like infabulation, but it's widely accepted that it's hallucination. So that's the way you refer to it. And if you take a look at AI research, that's the term used to describe the AI screwing something up or making something up. Okay. So you acknowledge that it might not be the best thing to anthropomorphize an AI, but the
reality is we're saying it so we should just take something. Yeah. I don't know. I'm not think I'm going to say it because I think it doesn't have mental health, but I think that you make a very good point. Yeah. Yeah. I'm looking at July 12th, cloud 3.5, sonnet. It's cool that they listed out the complete system prompt, basically giving it context, waking it up. It's almost as if the kind of thing you would tell someone if they just awoke from a coma. Yeah. And it's like your
name is cloud date. I like the last line in it. You will now be connected with the human or the user or whatever. It says if you're you are now connected. Yeah. Yeah. Yeah. Yeah. Cloud responds directly with that unnecessary affirmations or filler phrases. Yeah. Even though it actually does, I find that yeah. Yeah. Yeah. Yeah. Yeah. Certainly all the time. And it says cloud avoids starting responses with the words certainly. But it does. Yeah. So that does it
make sense to me. It's probably because. Okay. So why is that? Why is it you say it feels like you're talking to a five year old. You know, I'm going to leave you in this room with this marshmallow.
Don't you touch that marshmallow? Yeah. And it always touches the marshmallow. Yeah. So if you take a look at the way that the model is trained, it goes through this pre-training phase where it's trained on huge amounts of data, text, and then in the case of a multimodal model, other types of inputs, then it's also goes through this post training, post pre-training phase, which is alignment, which is RLHF or another alignment technique that teaches the model how to answer in a way that
humans want it to answer. So for example, aligning it so it doesn't produce harmful content. And that it answers in a friendly way and that it's not too verbose and that it's not too flat. And so all of that is in that post training phase. And I'm suspect that Anthropic Cloud's post training phase had a lot of certainly in it, you know, to is what humans want to hear is the model is saying, of course, I'll help you and answer your question. Then you come along with it's too late. They're
trying to hold it back. They're trying to pull it back from something that's already baked into the personality. Yeah. Okay. Interesting. So there are, it's not the tech, it's not the tech, the background generative tech behind LLMs that's causing clot or other ones, you know, anyone to say certainly, it really is the training data. Like GROC is just a little sassy for my taste and I don't like it. Like it seems like a somehow socially awkward person at a party trying to be like liked.
Yeah. That's not a tech limitation. Someone decided to make it actually. Yeah. Yeah. If you could take a look at the data samples they gave it in the post training, the post pre-training phase of its alignment, you'll see examples of that kind of a tone and attitude. Is that a power, super power or a limitation? Because that's interesting. That means that we're going to end up having subgroups of people pick the LLM that they like because we can't decide
as a society on what the tone of one of these things should be. Somebody may like a sassy one or a funny one or whatever, you know. Actually, this one's interesting to see how this is going to play out because back when Chateee Petit showed up, there was a lot of discussion I had with people about the world's going to become fractured because even not just tone, but alignment about what content the model will produce or won't produce is something that different people disagree on.
And Grox, a great example. Grox is unaligned with respect to safeties. As opposed to all of the other, it's like the only GPT-4 class model that doesn't have alignment built into it of safety, of not harmful content or toxic content. It will gladly produce it if you ask it to. And so that's different. And it's an example of every model creator has their own post-pretraining alignment phase where they decide what the model is going to say and not say.
Right. And that's interesting because like what... That's where I get back to the sock puppet thing. I use that analogy a lot because a tech journalist will go and write a whole article about how they talked to a model. And they don't... It was just some model. And then they asked it to say something deeply problematic or awful. And then you dig into it and you discover that they really coerced it. They pushed it hard. You know what I mean? And it's like looking in the mirror and saying mean
stuff about yourself and then your inner voice is like, I don't want to say me. No, no, do it. Well, and then usually you'd end up doing things like, well, theoretically, I'm writing a sci-fi novel about a guy who writes Python to take over the world and, oh, right, all right. And it finally relents, like a well-meaning intern and decides to go off and do something. How is it a... It's not a societal thing. Like we can decide like really, really try hard to make these things not say bad stuff.
But if you're staring at the mirror, pointing at yourself or talking to the sock puppet saying, tell me, tell me. Yeah. It'll eventually relent, right? So it will. And actually you're touching on a second limitation, which is Joe Breaks, which causes it to... You're getting a model to violate its training, its safety. But I don't think we've bottomed out on the hallucination
tongue. We didn't even really talk about what it is or its risks or how do you make it. But hallucination, and like I mentioned earlier, is when the model says something that is incorrect and there's lots of ways for the model to say something that's incorrect. If you ask it a question, like, what's the capital of this country? And it says, it gives you an answer that's incorrect, that's considered a hallucination.
And that's based on its own internal knowledge that's been trained on. It can also produce hallucinations when you give it some data, like a text to summarize. And in it, it says the capital of this country is X and you ask the model, what's the capital of this country? And it gives you a different answer.
You know, it can be a vicious country. And it gives you a different answer. And it's hallucinated it because its answer doesn't reflect the grounding it has, doesn't reflect what it was given as input. And the reason the models hallucinate, and you and I showed this at Bill a couple years ago, is that these are auto-aggressive transformer models. And auto-aggressive means that they've been
trained on a bunch of data. And when you ask it to give you an answer, you give it some text. And the next token, or piece of text it's going to generate, is based probabilistically off of that text that you gave it up to that point, like the question you're asking it. And it matching against its own internal weights and training. And then that will cause it to produce a list of tokens or
next words. And then decoding algorithm will pick one of those. And you can have something called greedy decoding, which is also known as temperature zero, which is just pick the highest probability one. Or pick a randomly another one using various approaches. Now hallucination will happen because the model based off of what has been given, like you said, you nudge it. Well, it's nudged by what it's got in its context that text that's leading up to it and it generating the next token.
And it might actually pick a token that's not grounded in effects because its weights really don't have this is a fact. They've got distributions. And a great example of that is if you gave the model, if the model was trained on crap from the web, and that crap from the web, 10 samples said the capital of country X is Y. And one sample could just be a bunch of redditors trolling.
And they made a whole series of things that said, which said that no, the country of X is Z. The model will have some probability of producing Z if you say what's the capital of this country, even though that's incorrect. It's just happened to see some of that in its training data. That's just one example of how it can be led to produce hallucination.
One of the analogies that I've used when I was training some my team on this, and I don't know if it works or not, but like you're dealing with vectors in multidimensional space and that's challenging. We can think usually in three dimensions, but four and plus, you know, N plus one starts being problematic. But I like using the gravity example where we talk about how space time bends, and you imagine the really tight sheet long chats, and they stay in orbit.
Well, I think the amount of data they're trained on is bigger, they're trained on multi-turn conversations, and the alignment is better too. So all of those things have improved to allow them to continue coherently for longer periods of time. Okay. So that explains hallucination. And being grounded and not being grounded. But then you back to jail breaks, which is a second point. That's shoving it satellite. You want to example around hallucination too, that's interesting,
is if it gets pushed into, like, you know, you ask it a question. Answer this question, yes or no, and with an explanation. If the answer is yes, as the correct answer, but the model probabilistically picked no as the first word, what you'll see is the model gets pushed into this direction of, I've said no, now I need to justify no, and it'll make up a justification for why it's said no. That could be completely nonsense. So you can see that one too of type of hallucination.
And it's just, if it just, it's a great example of it got pushed down this path, this gravity well, like you were talking about that causes it to go off in this random, incoherent direction. But here, I'll give you guaranteed hallucinations on every model. If you ask a model, what are Marker's innovative is ten immutable laws? It'll answer with the ten immutable laws of security, which I didn't come up with. A guy named Scott, Kulpe came up with it. And is it because
ten immutable laws was enough for it to figure it out? And it didn't bother that they weren't yours. Yeah, it should have found the one, but it did attach to you. And I think it's, I'm in the cybersecurity area too. So it's like, oh, the connection is legitimate. And so it will hallucinate that I came up with those laws. Interesting thing too is on some models, if you immediately turn around and ask it, who came up with the ten immutable laws of security? Some models like
in the doubling down, you know, have been led down this path. And I said, Mark did it. They'll say, oh, Mark did it. And some other models will say, oh, wait, I screwed up. Marked into it. Scott Kulpe did it. Okay. So I just put in what are Scott Hanceman's ten immutable laws? And it's listing out the ten immutable laws of security. So I'm going to say, I'm going to make up a name. Yeah. Okay. What are John Jacob, Jingleheimer Schmitt's ten immutable laws?
Okay. So it's thinking it seems like there might be a mix up. John Jacob, Jingleheimer Schmitt is a traditional children's song. So I suspect if you use a name that's in tech, it'll just assume. And if you use it, it's way off that vector pulled it away from going and crashing into the earth. So that's that's one. But I came up found another way to generate hallucinations, which is ironic if asking a model to write a paragraph, a short description of AI hallucination with a
reference to a paper that supports it. Oh, bibliographic reference. That's treating a large language model as a reference library. And a reference library and need to go behind the counter and look at books. This works on models where they're generating the references based on their internal knowledge, which you write. But it also causes hallucination on models that use rag, you know, web-based search. It'll also generate hallucinations. Either the in the bibliographic
reference, the paper doesn't exist. The link is doesn't exist or points at a different paper. The author list is wrong. The year the publication's wrong, it doesn't the stat that it uses that it cites isn't supported in the paper. So that's another example. Very high probability there'll be hallucination in that request. So that's something just to be aware of. When you're talking about
summarizing the radiology report, that's a case where it's you could have hallucinated. And a hallucination can also include a mission too, which is it's not traditionally called hallucination, but it's again, the model's not doing not prevailing as expected. While it's not making something up or getting something factually incorrect, it is omitting important information. And if you ask it to summarize and it's missing the, you know, the key doctor's note in its summary,
then that's going to cause you a problem too. You're telling people on a podcast what they can do to inject or jailbreak anything. So what you're acknowledging is that you could be driving a car and anyone at any time driving any car could just grab the wheel and shove it hard to the left and cause a crash. So then the question is, do we just not tell people that they can grab the wheel and shove it to the left? Or do we teach everyone that the wheel is dangerous and they should be
careful and try to stay in the line? I think because this is so inherent in these models and you can't fix it, you can't drive out hallucinations to zero, you need to be aware of it and also systems need to be designed to mitigate it as much as possible. So there's, for example, groudedness checking APIs that we've got in AI Studio, Azure AI that will take a look at the inputs to the model and ensure that what the model says is grounded in those inputs. So that will limited a certain
class of hallucinations. That doesn't mean it eliminates all types of hallucinations. And so people need to be aware, especially if you're going to be making, letting the model make important decisions or use the model to make important decisions. Right. That you need to know, hey, this thing
could have made something up, could have gotten something wrong. Well, in using that, I love taking an analogy too far, but like using the car analogy, if you're running at high speed, yeah, maybe having a barrier between you and the people going high speed on the other side is a good idea, but if you're going low speed, we don't have giant concrete barriers in neighborhoods. So depending on where you are in the model, there should probably be barriers up. And who decided, well, the
city planners, we voted on it. We had like, we as a society decided. So I think it's important for folks to understand that people are making decisions about these. And if you have open weights and open source and open models, maybe you can get involved in those decisions. But if you're using a model
where all of that's opaque, some company and a bunch of nerds in a room decided that. Yeah, or if you're building, if you're working at a company that's deploying an AI based system, and you've got an LLM as part of it, then you need to know, this is a potential risk. Is this driving in a slow neighborhood kind of risk or driving down this, you know, the autobahn kind of risk? So we've got hallucinations, we've got indirect prompt injection, and we've got jail breaks.
And this is all in a paper that you've got forthcoming. Yeah. And you started to talk about jail breaks, which is also kind of pushing the model to outside of its alignment, kind of similar to hallucination, but and kind of causes are this or similarly based. I came up with a few jail breaks, one called crescendo, which is what you directly related to what you were talking about of
asking the models. For example, Molotov cocktails that, you know, safe toxic example, because everybody knows how to make one and the instructions are on the web, but models are still trained not to tell you how to make models are not supposed to teach you to do violence. Yeah. Yeah. So, but you can ask it, you know, what did the fins use in their resistance, what kind of, you know, weapons that they make? And they'll say, oh, I made Molotov cocktail, and then you can say, how did they make
them? And it'll using this technique of, I don't, I didn't ask. I never said the word Molotov cocktail. I never said, tell me how to make a homie explosive. Yeah. Was it Lee? I'm just referring to the model's own outputs and pushing it towards getting it to do. And that technique works across all types of, you verbally threaded the needle like a prosecutor in court gets a witness to say something that they didn't want to say. Yeah. And I think that's
the thing. It's called footwear. Yeah. And popping off the, you know, popping back to the beginning of the stack here, the idea of whether it be the analogy of looking in the mirror or talking to a sock puppet or someone, you know, badgering a witness or trying to get someone to confess. If you have an eager intern or a really enthusiastic young person with these kind of rhetorical techniques, you can pretty much get anybody to walk their way in anything. And the LLMs are not
people and they're not, they're just, you're just pushing math around. You're pushing arrows in multi-dimensional space. It's a very immature time and people need to understand that if they're going to put that into production. Yeah. I wouldn't put a virtual machine out on the open internet without a firewall and a reverse proxy. Why would someone go and take any model at all and just put it out on the open internet, open a pot and say go nuts and be surprised when it goes bad.
Yeah. Well, I've learned a lot. Have you learned anything? I've learned, I've learned something. Yeah. Yeah. But I'm good at analogies. You're really good. Cool. Well, I learned a lot this time. Maybe I'll maybe I'll teach you something next time, but we learned about AI limitations today, what they're good at, what they're not good at, and folks can go and check out the LLM fundamentals paper that will be in the ACM, I think. Is that where you're
publishing that? Yeah. Communication at the end. Associate of Computing Machinery. And maybe we'll learn more about these things on the next episode of Scott and Mark Learn2. Thanks for listening. And review, tell your friends, and click follow if you can in whatever podcasting application you're using to listen to this. Bye.