HockeyStick #8 - Generative AI in Action

Amit is a principal group technical program manager at Microsoft, where he leads the engineering team that builds the next generation of AI products and services on the Azure AI platform. He has over 25 years of experience in technology and product development, including the artificial intelligence and cloud platforms fields. And yes, you will learn what his mom thinks about ChatGPT. Welcome to this episode and thank you for flying hockey stick. let's start right away.

why did you write the book?

When did you decide to go into AI? obviously you've been in it for a while. It wasn't as hot as it is right now back then. Can you tell us a little bit about your story and how you ended up doing it? What you're doing?

And one of the technologies among a few was AI, a few years ago. In my role of looking at emerging tech, is how I got into AI. Of course, Gen AI or these underlying architecture principles that power these things today didn't exist. But I was quite fascinated. it was still a side job in the sense it was one of a few areas of emerging technologies to go dig and deep into. And then as that started getting more traction, I was the one eyed king in the kingdom of blind.

Because, I knew more than the others, didn't know, doesn't mean I know most. And then I was stuck with that. And then grew into that and got fascinated.

it could be brand new things that we're building with them or for them, or it could be improving existing features. So once we sign those contracts, those land on my plate to go deliver from a platform team perspective. So I'm responsible for a lot of the custom engineering on the platform, which is this. That's my first bucket of responsibilities. My second bucket of responsibilities is whatever we do custom in the first, make sure it's in the platform.

Because if you keep being custom, then there's no platform left. So the way I want you and the listeners who will get to this to think about it is, these large deals that we sign are the catalyst for us to go do things in the platform that we already are thinking, maybe it's not prioritized enough. so they are a forcing function to go improve the platform at the end of the day. and that helps not just that one specific customer, but all the rest of them as well.

And then my third unofficial one is anything and everything related to, Azure OpenAI coming from our, CEO and what we call our SLT, which is CEO and his, direct reports, in the context of customers where, it's a top of mind for many and. For many folks, their understanding is varied, which somewhat ties back to the genesis of the book.

so when Satya meets other, CEOs and they have a question or they're not happy about something or they need guidance, those get sent over and say, here's the team is going to go help you. And so then I go in and from an engineering point of view, support, see what they need or what they want. So those are, that's my day job, right? So custom engineering. And then supporting, Azure OpenAI related things, from our leadership team.

Obviously in our team, we do work from an engineering perspective closely with OpenAI and they're a massive partner to us. So I think in some cases, maybe we are a little more empathetic, I would say, because it's a little more closer to home. And, it's one big virtual team is loosely speaking, how to think about it.

I think the difference really is my mom didn't know about it, nor did she care. now she does. so I think, where we're coming from in some ways it's not new. And it's just become more in the limelight and people are becoming more aware, but We've been at it for a while, and both from a research perspective, products perspective, it's just more in the limelight now.

So one of the questions that I keep asking everybody is. Their reason to think why GenAI is such a massive deal, right? Why is it such a big deal and why again, your mom, why does she know about it now? And she didn't before, I don't think she knew about BERT. I suspect, but she does know about ChatGPT and there's a good chance she's using ChatGPT, which is, next level. And, what do you think was so special recently?

What's the like hockey stick moment from your perspective of what's changed that it became, a household name?

And the breadth of what a language understanding, it can do versus as we call it now, the traditional AI, which is very odd, by the way, in the first place, but, in the old AI, the pre gen AI, which is not old again, it's very much valid, of course, today. was very task specific, where you go deep in a certain task. So if you're in a company, in an enterprise doing a certain thing, using that, you understand it, you get its value, you know why it's powerful.

But you can't have a generic, free ranging, wider set of, conversations and thoughts, around it. So if you take a previous chatbot, for example, which is not powered by GenAI, and you say, and if Miko goes and says, hey, I'm hungry, It won't know what to do, I'm sorry. Whereas these things understand, they adapt, so I think the simplicity from a using perspective is the power. And that's why the likes of my mom and others in the world is talking about it, right?

Because it's not technical mumbo jumbo that a handful of people understand and you geek out in the corner. I can just use it.

So I think if you look at a consumer sense, like my mom, that is an iPhone sort of comparison moment.

And it's basically plotting, the human performance benchmark, which goes from minus 100, meaning that it's pretty bad and goes all the way to zero where it's comparable, I think, or maybe equivalent to human.

And, For everybody who's listening to that as a podcast and not seeing this on video, it's showing different, machine learning, AI, trends, it's got handwriting, recognition, speech, recognition, image recognition, and then it's got the reading comprehension and language understanding and what's mind blowing to me. And I suspect this is why you chose this particular graph is that. We've got the handwriting and the speech recognition that kind of goes slowly, looks linearly.

there was a little bit of progress and then somewhere in mid 2010s, it just goes out of control and, it goes all the way up, to very good results. And then 2016, I think on the graph starts the reading comprehension. It's basically, An arrow going straight up, same for language understanding. This is within two years. It goes from nothing literally to basically comparable to human performance. why did it happen then?

What needed to happen this is not even a hockey stick. This is just like the right angle here. How do you explain that?

But I think crucially, maybe equally, maybe more crucially is availability of data at the scale you need. And then also compute most specific GPUs to, train and crunch through these. I think that it's that perfect storm of those three things coming together. if one of them didn't happen as much, it would be still slower. And that's why you see the linear progression in the others versus I don't know, is that a rocket thing?

And then, cloud capabilities in the context of GPUs and compute, improved. I guess there's a fourth one, which is inherent, but A lot of system engineering things started coming online, right? How do you run these? Because it's not like running them on one GPU, for example. You need clusters of machines. So there's a fair amount of systems engineering, in the sense of reliability, resilience, and so on, under the covers that, that has to make it all happen. Otherwise it won't run.

Lots of Physics and computer science. I keep saying that to my team, for example. so I think that's maybe a fourth dimension, which most people don't talk about, but, I think those are the things that perhaps enabled a bunch of this to go where we are right now.

And, the figure is making a distinction between the ones that are basically open source, publicly available, and the ones that are closed source, and it's only. graphing, up to, GPT-4 and LLAMA-2. So there's, way more of that. I think at some point I saw that, hugging face had a hundred thousand models uploaded to it. And I suspect after Lama three, it's probably doubled since, it gives you a little bit of a perspective. It's not just ChatGPT and it's certainly not just OpenAI.

And it, it shows you how much variety there is. And, frankly, I've been looking at this things for a while now. And I still, there's probably like half of this graph that I haven't actually even heard of, let alone, tried. I keep using this word Cambrian explosion, but it really does feel like that. They're just crawling out of every rock and hole, which is amazing. This is, such an exciting time to be alive, is that it's the right way of putting that.

why did you choose that figure, for your book?

So in that sense, I felt I would be doing a disservice if I didn't make folks, at least aware, because you just know what you know. So I think that was my one aspect. I think the second aspect was also. Showing lineage because a lot of these models are complex, as base models of training. they're super expensive, both in the sense of data gathering, cleaning it up, actual training costs, and so on and so forth, which many don't really have the appetite.

or have the ability resources-wise to do that. So what I also wanted to show was, at the end of the day, it's still only a handful of base models that are further trained or fine-tuned and derived from. so it's a lineage aspect also I wanted to, because that gets lost in the noise as well. and again, the framing of the book is mostly in enterprises, so if you're in an enterprise setting, you just need to know the roots of the model you're using and the lineage it has.

So you can make an informed decision if that's the right thing or not the right.

And they were surprised that it kept getting better. Sounds like this one was, trained on a much smaller corpus of text. how do you explain, why it's so good?

Relative to, bigger models is because, a good chunk of the data is highly curated. There's two aspects to it, which we also publish. So there's this other paper, this textbooks is all you need if you or your readers have seen it. So basically a good portion of the corpus is high quality textbooks as input into the model to train on.

And then the second aspect of, data is not common crawl sucking stuff off the web, but again, highly curated, web data, or a very small subset of the web data, combined with the, Textbooks. that is also an interesting research thing where it's going now to say is like for these smaller models. How much higher quality data sets does carry a lot of weight. and that's really a lot of what you're seeing.

So there is other models helping when you say how do you create it, at least in the context of this, it is synthetic data generated using, GPT-4, but then the humans are involved to make sure that, it is curated. Again, it's not an army of people, but it's. machinery evaluations and so on, machinery running to

One shouldn't think of it as, 'hey, I can go use an LLM, spit it out, and then use that to go train my own model', in the breadth, in the broad sense of it. but you'll also hear of more recent papers coming, and more recent news where, in general, this is not Phi-3, but in general, we have reached the points where we are sucking in all of the available Internet that one's reachable or allowed to reach.

And to train the models more and more, we are also then complementing it with the synthetic data, which, other AI is, generating, So I think you have to go put it back in which aspects of your existing models not doing great on, evaluating those, and then using that as a basis to strengthen that dimension, rather than just a more horizontal generic, if that makes sense.

And I think what I appreciated, I actually haven't seen the shortcut, the abbreviation SLMs for small language models until I opened your Oh, okay. which is, an indication of just how much focus we put on LLMs, the large language models.

And, I think that, to me, at least, I don't know if it's just like the part of me that loves running things on Raspberry Pis and gets excited about the possibility of actually running a decent enough model that I can speak to that actually runs on my phone or something like that. so 3 billion parameters, does that mean roughly with 4 bit quantization that we can run it on effectively any phone at this stage? Like it's going to need maybe a couple of gigs

So yes, these are small language models and we do believe how LLMs after ChatGPT became a lot of hype. Some is good, some is not so good. SLMs will be the next set, in the context of the hype. But as I go to remind many of the folks I talk to, it's a small language model in relation to a large language model at the end of the day, I think 2.8 or 3.8 or whatever parameter we have on the mini one, because this is Phi mini, Phi-3 mini. It's also a family of Phi models.

This is the smallest of the ones that should be coming out and the paper touches on the others, at the end of the day, three billion parameter or whatever the exact number isn't small just from a computer science perspective, it is still a pretty big, complex thing. Yes. Compared to hundreds of billions of parameters, it is small, but it is not small. I think I have to go remind people that.

In relation, or relative to an LLM, yes, it's small, but by itself, it is still pretty complex and beefy in the sense of compute requirements and GPU requirements and, what it needs. It doesn't mean you'll go off and deploy a bunch of these on your Raspberry Pi with inference and, milliseconds and whatnot.

YouTube review of that and I think they basically destroyed it a little bit by showing just how long you have to wait because it's effectively just uploading it to a cloud somewhere and then downloading the response and it's just not there. And, with Phi-3 and like the smaller models, all of a sudden everybody's thinking the same thing. Can we make it native? I think Apple announced some things about how they're going to work on making sure that the hardware in the newer iPhones.

to run this stuff at reasonable speed. And this feels like this would be, another hockey stick moment for this things. There's small language models where Siri doesn't suck Hey Google. "Okay Google" works. And Alexa actually listens to me, that kind of stuff. do you think it was that one of the motivations of the smaller model?

It takes time to get results back and there is no ways to cut those corners at the end of the day. So where you're seeing latency or things are slower, it's because of that. from our perspective, there's also another dimension to run this at cloud at Azure level globally across the hundreds of data centers and what have you. that's not simple or cheap, So if we can reduce our costs to run this at scale, we can make sure the service is cheaper for our customers as well.

I think this is also where, we as humans are awesome and we forget things. Because many of these models are exposed as an API. we as, at least developers for sure, have the expectation that it's an API call, so I'm going to get my response back in, milliseconds and what have you. because that's what we have been used to. The difference is, yes, it's an API call, but the machinery that's running behind, including the models itself, is super complex. and when things are slow, we get unhappy.

So I think that also needs to be a mentorship. So if you package it all of this up, that's a big motivation of, in some cases, a small language model would make more sense. But I also want to outline this. It doesn't have the same power as a large language model. I see a lot of comparisons to the bigger models and all, which is good. It's early days, but at the end of the day, not an apples and apples comparison.

For example, A lot of people, including me, have been guilty of just using GPT 4 as a knowledge database, more and more people are instead of googling or binging or whatever you do, you just ask the thing. So you're using it as a big, fancy database.

So if I put that in the sense of the world knowledge, again, it's not factually correct whether it doesn't have the world knowledge, it only has the publicly accessible knowledge as of its cutoff training, but ignoring that point, the small language models will not have that because they've not been trained on that volume of data. So I think the other dimension is whilst the compute profile, what we've been talking is one, you have to think of the SLMs in the right use case.

What am I trying to do? If I'm trying to do understand an entity in a workflow, I can use a small language model. I don't need the power of these large language models necessarily. equally if there's different languages that one has to use, not English, for example, a small language model may not be as powerful or as good as a large language model. So the way we should think about it is they shouldn't be competing. They're complementing each other.

in what you're trying to solve at the right step, use the right model because the beauty again is they're an API call. So it's not that if you're developing an application, you're stuck with one thing for the whole duration. You can choose at the right step for the right thing, for the right power. So I use.

Often with my teams and others, the analogy, like if a GPT or pick your model is like a Ferrari, if you're going to racing, you need a Ferrari, but if you are, if an SLM is like a Honda, and by the way, I don't get pick your brand, and you're stuck in morning rush hour traffic, and Honda is better than you pick the right thing for the right purpose. Is this really what I'm getting into? I would show them the compute profile.

when you were saying all those things, I was wondering whether there is a certain kind of minimal level where it will be, a certain number of tokens per second that will feel to most humans as real time is really what we're talking about here and beyond that point, probably doesn't matter. if you can't read it faster than it's being produced and you're not going to have that, that feeling of slowness. And there are some interesting.

things like Groq, the one with a Q at the end, I think there's suing Elon Musk over there, that has some dedicated hardware, and I saw some demo, I was doing something ridiculous, like 800 tokens a second on LLAMA 3. So was it 70B or something? Is it not just like the matter of waiting like 5-10 years for the dedicated hardware to get cheap and plentiful enough. And it won't be so much of an issue?

I think I have to also remind people when you are doing comparisons or the expectations you have to think of in the sense of the load, the traffic, how much, now, we as a cloud provider, that's a lot of our headache and a lot of customers saying like, why do you think I'm paying you? but I also go back, yes, and laws of Physics don't change as well. but overall, I think, Nvidia announced a whole bunch of new stuff. at their conference quite recently, network speeds are improving or have been.

So if you step back for a second, I think, just history of computing has been that hardware scales up and improves and helps the software. I think the one thing that I'm not, personally speaking, I can't predict the future. The one thing is, This is one of those back to your hockey stick points. The scale is almost at a global level. okay, it's not every human on the planet using ChatGPT or LLMs in some fashion, but quite a big percentage of people are.

In some manner, on a daily basis, for some people, it is eight hours a day. My mom may be once every other, once a week or whatever it is she does.

but the breadth of humans using it is much more broader than it ever has been. So with that context, even as other underlying system and hardware improves, I think the perception of, is it actually getting improving, maybe slower than perhaps in the past where it was still more niche, if that makes sense.

it's a whole thing, but, once she gets some kind of interface that's built into her phone, or whatever, where she can just talk to it, that kind of clears a lot of barriers and, a lot of people are picturing this feature where your phone is slowly turning to effectively listening device from Star Trek and, It's just doing what you want it to do. And maybe integrates with all the apps. I ordered that Rabbit R1. I'm still waiting.

I don't know where the delivery is supposed to be, but, that's one of the visions of the future is right there. You just talk to it and the model does things on your behalf, goes to this dodgy apps and clicks things, and you don't have to worry about that. And you don't have a learning curve. And I think that's a vision of the future that excites a lot of people. And, I suspect we might see something like that in the near future because I don't see any roadblocks for

And I can give you two, real examples. for example, I'm originally from India. And in India, as much as the country's made progress, there's still a, decent percentage of the population who is not very literate. Either they haven't finished school, or they dropped out early, or they've actually not gone to school. Now, it may be a small percentage at a country level, but if it's a country with 1.4 billion, a small percentage in absolute numbers is still a big number. a chunk of humanity.

And in that, we're seeing, for many people who are not comfortable reading or writing, some of the cheaper devices they have, it's not an iPhone or an Android phone, but they have a speech. So they print, there's a big mic.

In the middle of the phone, they can press that and talk to it and actually in natural language, in their language, they're asking questions and talking to it and that is, as it happens in some of these cases, it's out of some of our speech AI, which is understanding it and then responding back. So it's lowering the barrier and opening up this to a broader segment, which in the past was not possible.

so that's one example, because they don't need to know the language to go type it in or what have you. They can just talk to it normally how they would talk to it.

And then the second one was actually more of a ChatGPT example, which I think, Microsoft also published where it's, plugging in different languages for, again, in rural areas in India as farmers, like India is not, For those who don't know, it's not like the U. S. and others, where you have big farms with, hundreds and thousands of hectares or acres. They're usually small farms. Usually it's, the family which owns it.

And they don't really have the muscle at an individual level, to go understand pricing and markets and what's happening as they want to go sell there. green or whatever they're growing. So in that sense, we talked about democratizing was how they're using ChatGPT to actually, getting basically real time market information.

So they're empowered to go make a better decision, which until now was impossible because you need a computer, you need a modem, or you need to be online and those are the barriers. And it's like, they don't know how to use it, or in the language that they understand.

So these are actually happening today, like in production, so to speak, live, and the way we want to think about it as is democratizing AI, which is when I go back to how you started asking me the question, when we started talking, if I go back to my mom's or the example you used with your mom of the barrier of a new app or a new interface, if we free them up or make it easier in many ways, those are the democratizing, elements that is happening is not only about your, Okay.

how literate you are or not, but it's the, it's easing barriers basically. So of course it doesn't do everything. It doesn't mean all barriers are gone, but we see a lot of real examples, day to day life things that people are using it, which is absolutely fascinating.

And there's, like a million startups doing things around that at the moment, it also obviously has the dark side, right? Where people are worried that what does it mean, Can you go and sway an election now by just calling everybody in the US and telling them something that they want to hear and, personalize the message.

It is a brave new world, a weird world that we're entering here, Where some things that, you could always technically go and call everybody in the U. S., but it would take a while. now with those things, maybe you can do it convincingly in a shorter period of time, And maybe not that expensively. does that scare you?

How do you know what's true or not? So I think there's of course dimensions into it where we as humans have to recalibrate ourselves on, do I trust it or not? For example, Robocalling has been around for decades. The fact that I can cheaply call everyone is not the problem. now it is, it may sound like Amit or Miko's calling, which in the past You know it's not Amit or Miko calling. I think that's the really, things to think about and worry about.

The way I reposition it as well, from a Microsoft perspective, and also I have a whole chapter in the book on that, is, the, there is new emerging threats from a security. So if you think of a traditional security aspect of your application or developer, DevStack, The way we're saying is look, there's additional new security threats you have to go think about.

And it's easy to get wrapped up in all the negative, but if you step back and say, look, as there were paradigm shifts, as you went from client server two tier, and I'm going to show my age now, applications to distributed applications and then to web applications, There's a lot of goodness, but then it also opened up the exposure to, a different threat vector. The surface area was different. In some cases it was broader, in other cases it was actually contracted.

And in that sense, this is no different. There is new emerging threats you have to think about and be cognizant of, and then also understand what is the risk of that. And sure, a threat could happen, but how often will it happen? And how do I mitigate that? Uganda will solve 100 percent everything. But you have to then hone it back down into what's your use case, how you're thinking about it, and so on.

so instead of, either ignoring it, which is not good, or putting your head in the sand like it's all doom, neither of those dimensions are going to be helpful. So I think part of it is understanding that, yes, there is a new set of threats that are emerging. Be aware of those. How do you solve for those? How do you manage those? And then In the context of a use case, in the context of how you're using it.

we got a technology that made things possible now, that we're relying on them just not being practical from, time and cost perspective. And now we have to deal with that. And the genie's out of the bottle, as they say, I think. and the cat's out of the bag.

my one PSA is please use a password manager. 10 years, 15 years ago, if you were chatting, the concept of a password manager would be so alien. And here now, I'm sure as you do, I do tech support for my family. Unpaid, of course. my everything is go use the password manager. Here's how you set it up. And why shouldn't you reuse passwords? And let the thing do the heavy lifting for you. But you save it, right? With your master password and whatnot.

I think it's, yeah, it's the same analogy in that sense. it goes back to your thread vectors change society. Things are changing and, part of it is adapting. Some is good. Some is not good.

There is obviously the intro that covers everything that you would expect in terms of what is generative AI, the architecture, high level, what it means, references, overview of LLMs, transformer, smaller language models, that kind of stuff. And then it turns into basically a guide to show you what's possible with it, show you how you can go and call some API and get magic text being generated.

It shows you how to generate pictures, shows you how to generate other things like music, video, I think briefly code, all that kind of stuff. So I'm picturing this really as a kind of guide that you get yourself when you want to get into this without wasting any time on things that are not necessary for your journey, I will get you from zero to one on that. is that accurate description? Am I doing a good, marketing pitch here?

A lot of these large enterprises, but this is not about just large enterprises, it's about if you're a company, you have a set of products you want to improve or make new, how do I use this GenAI and ChatGPT and LLMs and everyone's heard about it. And they don't know where to start or how to start. So that's really what I was trying to do, right? There's broadly speaking three parts to the book.

The first part is introductions and because you just know what you know, I can't just go dig into things without giving you some context and basis on what's possible, what's not possible. and that's the first part you're touching on. What I stay away from is it's not a science research book. I link to papers where there are people who generally are curious or they want to go deeper.

So we leave those crumb trails in a way saying, if you want to dig more in your own time kind of a thing, here's the things you can go read up and then that'll expose you to more dimensions, right? So it's not a science book, techie book in that sense, because At least in an enterprise setting, most developers and CTOs and CIOs or CEOs, they want to see like, how is it going to solve my business problem? How do I do it?

Some are interested in the science and the depth, but most just want to know at a high level, how it works deep enough, but not in the guts at least on the AI side of the science. so we leave the breadcrumbs and the trails pointing to papers where people can go deeper should they want to, but If you're a developer and you can use a set of APIs and SDKs, that is really for you and the way we say is because these, at least these LLMs are exposed as an API.

You really don't need to know any of the AI sort of mumbo jumbo, any developer can pick it up easily. So that's certainly why I was trying to position it. Part one is getting you a sense of the world from a technical perspective, but not go super deep. And then part two and part three is where we start going deeper on, okay, how do I use this in my production, solving my business problem, what I'm trying to do.

This is used for scenarios where the data is complex and diverse, requiring realism, suitable for high quality images, data augmentation, style transfer. So it's prescriptive in a way, I would say, you give people, what they need to, get to get cracking with it.

speaking of which, let's talk a little bit about the images because you do cover a few interesting things like the VAE, the GANs, diffusion, vision transformers, to give people a sneak peek of what they're going to expect. Can you talk about why they're interesting and why they might be, something that you should be paying attention to. What are the breakthroughs?

and image understanding, image editing, the power of it on one hand, wherever you go on whichever social thingy with stable diffusion came out, lot of creativity on the image generation was there, but in a social setting, the thing really is, how do you expand that in a corporate, application setting and what can you do? It's one is like fun and wonderful in a personal social setting. But then how do I transfer that and then which area do I use in a work setting?

Not even have to be work, each of these techniques have their own power. I think most people don't really care or maybe nor should they care, but in some cases where it would matter It's good to know what is the underlying tech, so I know what to ignore versus not to ignore. Because, again, the hype wraps up a lot of this. If you come back to it, it's more of helping people ground themselves a little, because at the end of the day, the tech is still the tech, right?

What it is meant to do and how it is meant to do doesn't fundamentally move. if you're trying to solve one set of images, one kind of things, like diffusion models would be great for, that set of categories. And now there's multiple diffusion models. You can go pick which one you want. versus a transformer model. So again, we don't go, I have a few diagrams and images to, outline at a high level how these are, because there's papers on each topic, you can go read like hundreds of them.

but the intention is just to know, look, there's different buckets and categories. Each has its own strengths. And if what you're trying to solve for, just make sure you connect those dots. I guess the other analogy is if you're writing a book, word is easier than notepad kind of a thing, right?

And the way we also tell many of our customers and I tell is look, you have to start thinking about prompts as your IP in many ways, and I'm not talking about simple prompts. Like I, in the book, I use simple prompts to make the point. So 'tell me a story about a panda' is not really IP in the context of a prompt. and then prompts are also closely tied to how a model understands it.

So again, outside of simple prompts, where you are When you're using this concept of RAG, for example, as you start using a specific model or a family of models, which are closely related, you start picking up nuances on how the models interpreting things and working with things and so on. And then you're tweaking your prompts along with that, right? So it's cohesive together. and that intuition as you learn is also part of your IP and how you want to think about prompt engineering.

That also means. There is no universal prompts. Again, outside of the simple ones, I'm not talking about the simple, straightforward prompts. So you should not, or one should not just say, if I'm, let's say, using GPT 3, 4, whichever, the same prompts, which are complex ones, I can pick up and expect to work on, let's say, LLAMA or something else. They will work, but do they work at the same level and the same evaluation, the same criteria?

Probably not, because they are quite tied into how the model behaves. This is very loose, right? It's not a scientific thing, prompt engineering is quite crucial. It is how we talk, even though you are calling an API, but how you're talking to the model is through those. so I think it's worth spending time to understand how, what these are, how they work. There's a lot of hype around prompts as well. I would say don't believe all of it.

The one final point I want to make on it is prompts is also one of the new threat vectors. So I touch on it in a later chapter, I touch a little bit on prompt injection in the chapter you've seen. but we go a little more deeper in one of the later chapters. But prompt injection, as an example, is one of the new threat vectors. It's not the only one. so again, understanding that as well, but prompts, in today's world gets quite crucial.

At the end of the day, it's how we, in quotes, talk to the model.

Saying it one time doesn't help, you have to keep repeating. The way I want, generally speaking, folks to think is, your model's like a toddler, you have to keep repeating, keep thinking about it, right? and as silly as it may sound, it's like basic stuff, like for example, one of the side effects is what's called hallucinations where, non grounded. So you will get responses back, which are made up and it's not factual. That may be okay in one dimension.

If you are writing a creative story, it may not be okay in another dimension where in a business setting, you're answering things based on some policy or information or what have you. so in the prompt it figures like simple things like do not make up any information, only answer from this, you would think that would be obvious, so your intuition of a cat is not very far off

Basically, it is also a little clunky right now, but what it does is, as the name suggests, the model that you're using just knows what it knows, what it's been trained on, which is public data. That's one. And then as with these things, there's a training cutoff, right? At some point, you say, okay, I'm done collecting data.

I need to go off for a few weeks or a few months or whatever it is and go train this thing and then spit out a model and go through a bunch more other alignment and this and that, and then eventually have a model available. So Online, when you go and see, a lot of people using RAG to get fresh information, which is post training data, that is absolutely valid use case. For many others, the other thing is my proprietary information.

So especially in a company setting, your proprietary internal information, corporate knowledge, the model doesn't know because it's never seen it. In fact, if it does know that, then fundamentally there's a different problem. Because it shouldn't know that. for your business workflow, you often need to bring in your internal proprietary knowledge, whether it's a CRM or a database or an ERP, or you're solving a ticket or what have you, depending on the use case.

The only way the knowledge, you can bring in the knowledge is through This technique of RAG, so retrieve, augment, generate. Retrieve means I'm retrieving the information, which could be from my corporate enterprise systems, or, Google or Bing to get more fresh information. I'm augmenting it in my prompt, which goes back to prompt engineering. And then based on that, I'm saying please generate, or whatever I'm trying to do.

Generation could be a summary, or entity extraction, or depending on, whatever I'm trying to do. But that's what RAG is doing. It also gets clunky, by the way, because, it is the first generation. I do expect things to get improving in that.

you talk about complexities of RAG, but if I have to get proprietary in-house information, if I have to get more fresher information, the only ways I can do that is through RAG, without retraining a whole model, which, in theory, is an option practically for, I guess 99% of people is not an option. I don't know if that was for a five year old, but

but before I let you go, I wanted to, ask you for a few predictions.

from where you stand, where, are we going to see the next evolutions and breakthrough,

But more and more multimodality, because our real world is that as well, right? So I see one and that happening. I do see SLMs to accelerate more, as we touched on. Again, they're not better, they're different. there's times you need one, and there's times you need the other, and there's times you need both. But, I do see more and more on that front because for many use cases, I need simple things. I don't need all the other power. so I do see that accelerating a lot.

And then I also see a third dimension is the, underlying, systems engineering things improving to be it cost effective from how much hardware and GPUs I need to run it to, latency around it, things like memory profile and so on and so forth. so I do see those sort of three and I guess I want to sneak in a fourth one, which is also all of the responsible AI aspects, which is one of the later chapters, we touched on the likes of prompt engineering.

I know I talked a little bit on hallucinations, but the new harmful things one can do. That's also a cat and mouse thing. I do see more research breakthroughs

So on one hand a lot of people get happy Hey, if I have a longer token, my context window is longer. It means I can stuff in more things I can ask you more things or I can generate more things On one hand, that's good. People get happy about it. What then I come and have to remind them like each token one length Increase is a quadratic increase in compute So it's four times, extra costly in the sense of computing profile. So just having a longer token, context window isn't necessarily good.

So there is research going on now to say, how can we do that? How can we, derivatives of the transformer architecture, which is, how can we increase? the token windows without having a quadratic increase on the compute profile. and that ties back to the attention mechanics of how the transformer architecture works. the way I would say it's the first one which has, reached the scale. And then now there's other research damage that's happening to make those profiles better.

will there be another big two? Which is, better than this, I'm sure, in the sense of humanity, absolutely.

I think what was the paper from Google talking about basically a method of achieving infinite, attention span, or was that some other research that you're alluding to?

There's other things, for example, like when we touched on RAG, I said, it's a clunky way of doing things. we didn't go deep in it, but it's a clunky way. So there's other things happening, like graph, can I do graph with RAGs, and so on and so forth. it's not only in one dimension, I was using one of these as an example, but across multitudes of dimensions. There is active research going on to improve those.

And as you start, as that starts formulating, cause look, research is one, getting something as a product that is deployable and running. That you can consistently, is a whole separate sort of scale and of its own separate complexities. but across these multiple dimensions as they come together, it'll just suddenly get improved. To your point, this is like version one, and it's a mad race across the board from academia to, commercial and whatnot. So it'll just be improving is how I see it.

So even if you have an open source model in the sort of near term, there's only a handful of companies who have the technical know how have the sort of the muscle, have the compute profile to be able to do that. so more and more until again, unless, as there's more fundamental breakthroughs to open that up more. a lot of the open source, it's back to that, the ant analogy used for, with the, one of the diagrams we have from the paper in the book.

the part I was trying to show there also is the roots are very few models that are derived from that. So I think once there's a lot happening, it's at the end of the day, there'll be just a handful of people who are publishing those and exposing those, that others are deriving from. So until that happens, as in fundamental breakthroughs at a cost point of view, where it becomes cheaper.

It all doesn't need hundreds of thousands of whatever it is, GPUs plus I don't know how many billions of tokens of data, to train them.

But the fact, if you, the other way If you just look in the last one year, 12 months, which is nothing in the sense of humanity and technology, just see how much progress and how much improvement the models have made across both dimensions, whether they're open source or closed source or what have you. It is fascinating. and, the fact that there's literally new models every day is a good thing, but also not a good thing. So it has to stabilize to some extent. At some point it will.

but I think the open source community is absolutely critical. On the flip side, a lot of research breakthroughs also are coming from the research labs where, there's, at the end of the day, deeper pockets and muscle sports in the sense of financial and compute and data as well. It's a fascinating world we are in, which is your, one of your opening statements, because at least for a geek and somebody in the industry, these are far and few moments that one gets. So it's absolutely fascinating.